AM bug
You probably heard about the famous AM bug on the SQ80. You may also have heard that this is an OS glitch. No, it isn't. (Repeat after me: No, it is not an OS glitch.)

The best thing is: You don't need to believe me, just point your disassembler of choice at address $ED24 ff. (i.e. offset $6d24 in OSHI of SQ80 OS1.8) and have a look yourself how waveform addressing is handled. See? Told you so.

It indeed is a bug in the DOC that for whatever reason overwrites the modulated oscillator's bank-select signal with the modulator's, leading to accessing the wrong sample ROM.

Long story short: keep your hands off bank select if you want to use AM ...

You can have either AM or Sync, right?
Following the data sheet, both should be possible even though it is controlled by the same bit:

• If set for an even-numbered oscillator, that will lead to AM of the following odd-numbered oscillator.
• If set for an odd-numbered oscillator, it will sync to the previous even-numbered oscillator.

DOC memory expansion
Already with the SQ80, Ensoniq decided that the DOC's native 128kB (well, more like 4*32k as 32k is an oscillator's maximum waveform size) addressing capabilities were not enough. As only 8 audio channels were required, they just reassigned one channel-address bit for waveform addressing. While in theory this approach could be extended, you'll obviously sacrifice audio channels for waveform memory size which is not exactly what you want in a hybrid synth.

Luckily, there are other methods, too, which however weren't explored by Ensoniq: they require some more effort as you now need to sync to the internal workings of the DOC. It is then left to your choices whether you want to attach an extra memory (that'll e.g. hold 32*8 or 32*16 waveform addresses) or go with a more baseline, yet electrically more complicated approach.

• Each full cycle, the DOC will perform two refresh cycles and an arbitrary number of oscillator cycles as defined in the oscillator-enable register. Bus-wise, refresh cycle and oscillator cycle are identical, so the analog voice demultiplexer needs to be kept from distributing spurious audio data. This is controlled by the channel strobe signal: During ordinary oscillator cycles, this strobe appears during E=1; during refresh cycles it doesn't.
  
  That way, we have an easy way of synchronisation: E=1 && CSTRB=1 identifies refresh cycles and hence the start of sound generation. We now use this as a reset signal for some counter of choice that is clocked with E=0 in order to output the currently serviced oscillator number.
  
  
• For the RAM-based solution, a 5-bit counter is required. During E=0, these 5 bits are used for addressing, during E=1, the CPU-provided address bits are used in order to allow writing to these locations (and hence setting the extended page address).
  
  The RAM-based solution is wasteful on the RAM itself, as essentially only 32 memory locations are used. It has the benefit of making full DOC use, though, i.e. 32 oscillators or 16 pairs. Back in the days, this would've been an easy hack featuring common 74189 or the more obscure 78218/318. These days, you'll hardly even get small SRAMs like 6116 or 6264 anymore. You'd hence do this in a small FPGA nowadays (which could easily hold an extended DOC design, but I digress ...). This solution could also be used with an ESQ1/SQ80 without having to rewrite the entire system software.
  
  
• We can do w/o a full-blown RAM also, but this requires a twist as we now pair four oscillators: oscillator 0 and 1 will not participate in sound generation, but just provide their wave page address (easily available on A15:8 with 256-byte waves). These will get stored in an 8-bit latch each to be output with the cycles of oscillator 2 (receiving WPA[0]) and oscillator 3 (receiving WPA[1]). Correspondingly, only a 2-bit counter is required in order to generate the according enable signals.
  
  This solution is wasteful on the DOC, as only 50% of the oscillators are available for sound generation, so you'll end up with a 16-osc configuration but the grunge (aka playback rate) of a 32-osc one. This solution could be nicely fit into a small CPLD (even small by 1990s standards) or a handfull of standard TTL chips. And you could even use the lower 8 address bits to access a saw-down wave and use that one for the above-mentioned poor man's (digital) resonance ...

True AM
On the DOC, AM is an all-or-nothing setting. Wouldn't it be nice to have it scalable?

• Well, for this we just don't use the DOC's AM bit. The modulating oscillator reads its sample value, scales it with its volume setting, and outputs the value -- together with an "illegal" audio channel address that is not used for routing but declares "use this for AM". The output signal is fed into an S&H stage that will drive a two-quadrant or even four-quadrant multiplier, with the other input coming from the modulated oscillator. This wastes one audio channel and hence might collide with a waveform expansion hack, but otherwise comes without real drawbacks. (We could even use the BS bit for this as it is buggy anyway.)
  
  
• We could also go for an all-digital approach by scaling the modulating oscillator's data reading together with an additional scaling value (more external parameter memory, yay!), sample this into an intermediate register to then be added to an inversely scaled modulated oscillator's data readout. As timing is tight and EPROMs are slow, this however might not work. Alternatively, the wave ROM would need to be increased 256-fold with the scaling factor applied as another bunch of wave addresses, which is also not really practical.
  

Can we do something about dynamics?
This is a harder one. Unlike the E-II that used 8-bit mu-law sample encoding (expanded to a 12-bit value range via a specialized D/A converter), the DOC uses 8-bit unsigned linear. Where both machines only know 256 different sample values, the DOC sounds much flatter in comparison. So what can we do here?

• We could opt for an external D/A converter. Instead of feeding the waveform readout from sample memory to the DOC, we could feed it directly into one of those ancient expanding DACs. If we feed the DOC a 0xff instead, its own output value would directly respond to the DCA setting which we then could use for VCA control (or DAC reference value generation). However, arbitrary sample size (i.e. not page-sized but 0x00-stopped) will be slightly tricky as here either 0xff (in case of a valid value) or 0x00 (in case of sample end) need to be forwarded to the DOC. Never the less doable, although those mu-law DACs are pretty much unavailable these days. Could be applied to the ESQ1/SQ80 w/o software change and make it shine, though. Hmmm ...
  
  
• As the DOC's internal D/A conversion is a multiplying DAC approach where volume scales sample (or vice versa), this could be used to apply sample expansion. For this, we would use an oscillator pair: both oscillators play the same waveform at exactly the same settings. With AM activated (and the first oscillator muted), we will now play back the square of the sampled value and, resultingly, a sample range from 1/256 to 256 in exponential scaling. However, while this indeed gives 16-bit dynamics, we have pretty much no control over it as here we'd just play the squared wave. And, of course, we need to keep those two oscillators in sync.
  
  Instead, we go for a single-osc approach: the oscillator's read-out is processed according to mu-law rules where only the lower four bits are fed to the DOC and get converted. The topmost four bits are turned into an 8-bit exponential number that serves two purposes: 1) generating an analog scaling factor that is applied to the DOC's output and 2) generating an offset voltage that is added to the scaled output.
  
  Timing is a little bit more relaxed on this one, but scaling/adding should be set up with the channel strobe to avoid audible effects.

That sounds like some hacking. Is there no other way?
Of course we could go the Amiga way and mimic Delitracker's 14-bit play routine, i.e. skipping the DOC's DMA ability completely, essentially just using it as a 32/16-channel multiplying DAC. That way, the CPU could access all sample memory it wants and preprocess the data in any form needed, to then feed it to the DOC.

But of course, there's more than one catch ...

• The DOC processing window: unfortunately, the DOC will insist on performing memory accesses, so CPU access may only take once per E=1 cycle. The external fast CPU will hence have to synchronize its writes to these cycles. (This is a general problem of all non-6502/680x CPUs that don't know about E/Q bus timing.)
  
  
• The DOC datasheet states that the wave data register is only CPU-readable, not writeable. (While I don't believe that, I haven't checked so otherwise, so let's assume the datasheet tells the truth.) Hence, sample writes must even be synchronized to the DOC's internal state so that a) the volume setting is applied before the next sample is read (but after the current sample is played) and b) sample needs to take place exactly at the designated oscillator's time-slot.