About mass storage interface/adapter/add-on design concepts - for Atari ST

[Petari]

Will be mostly for Atari ST case, but will be little generic, so may stay for other micros too.

I write this because of postings in style that only original Atari design is really good or compatible. Concrete, many Atari user think that only ACSI port is good for mass storage, and that other solution are: problematic because of compatibility. Slower. Causing bus overload, errors  etc ...  Must say right at start that all it is pretty nonsense and just show lack of knowledge.

Mass storage: first need to clarify it's purpose, how it works by micros,

Purpose is reliable and fast saving of data to some media(s), which keep data even without power. And of course simple and reliable, fast data retrieving into computer and use.

Data organisation:  data is organised into files of variable length and folfers. So, we save and load files. Directories are just to make all it easier to use.

Physical storage:  we have some standards now, established decades ago. Data is stored in 512 byte long sectors, in as many as is needed by file length. It stays for hard disks, Flash cards. By CD/DVD sector length is dfferent, but it changes not the concept. Minimal data what goes to be stored amount is always 1 sector. It may be not used all if data self is shorter,

After this academic, boring lead we can go on concrete storage solutions in Atari case :

What hard disk, hard disk adapter (IF, port) needs to do is right what is described above: to transfer sectors to media and from media to micro.

The choices of media/drive type:  IDE hard disks - 3.5, 2.5 inches and CF cards. Buying new IDE drive is insane - what to do with 500 GB and Atari ? So, CF cards or some little older PATA drive. 4 GB should be good for all needs, and over 8 GB is megalomania, I think.
SCSI hard disks:  new ones are extremely expansive and too big capacity. So, some used - may find in second hand shops for not much $ .
SD cards - with Satandisks. Likely the cheapest solution if we look only price of media.

In any case I think that today CF or SD card is far best choice:  capacity is just OK. Small, low power consumption - in case of IDE IF even no need for external PS. And data transfer with some PC/MAC is very handy.

Because Atari ST and it's ports to outside :  ACSI and cartridge port (parallel port is also usable for data transfer, but it is too slow in compare with first 2, not to mention serial or MIDI - both very slow) were designed 25 years ago it is normal that they are not usable directly with modern mass storage devices. Actually, they went obsolete pretty fast, especially ACSI port, so Atari 'removed' it from Falcon. Cartridge port is special case - in fact, by ST it is used mostly for diagnostic purposes (service cartridge) and diverse add-ons, which were mostly something other than ROM expansions. The reason is clear: it is connected directly to CPU data bus, so allows fast transfer to ST RAM.
It is very bad to think about cartridge port as only ROM port, just because it is intended for it by designers.  We should look at by used digital logic, lines available there. Then can see that it may fit many other purposes.

Doing concrete IF/adapter:  the goal is to make it practical, not expensive, fast and if possible to work with established SW (drivers). Other things to care: attachable to some existing port from outside, need for extra power etc. Which aspects will prevail depends on priorities of designer and potential users.
Some will go on solution which works with existing drivers in first place. Some will prefer speed. Some will go on with cheapest possible solution. Etc.

Is it possible to have all best features at once ? Hardly - it newer works in real life.

What is mandatory by mass storage IF ? Work with standard devices, medias - mostly because of price.

Some interesting facts related with compatibility:  some people thinks that HW is main reason for incompatibilities with some Atari ST(E) SW, mostly games.  Truth is that it stays only for Falcon - and main reason is different CPU, then different bus, video etc.
STE is very compatible with ST. If something works not on STE, the reason is 99% TOS version. By running game under TOS 1.04 on STE it is solveable in big %. Examples: Defender of the Crown.

So far I saw only one game which worked not on STE because of HW problem:  Grand Monster Slam. But it is because of bug in code, which overruns palette address space, and on normal ST it harms not, but on STE it writes trash in scroll registers, what screws display. Of course, in later cracks it is fixed.

Other STE case:  game Stardust works only on STE and Falcon. But not on STE with internal IDE interface ! It is again game's fault:  when it detects IDE IF it considers that it is Falcon, and loads Falcon executable, what fails on STE !  So, again not IDE incompatibility but stoopid code !


Now back to IF design self:  concrete example, fresh case:  ACSI-Compact Flash IF - just done design (circuit) :

It is based on fact that Compact Flash cards support 8-bit transfer - so good for 8-bit ACSI port. But ACSI port command set and parameters are pretty different than IDE ones. So, to achieve work with current popular drivers as Hddriver or ICD + having autoboot from TOS we need pretty complex solution, which can translate all commands and parameters served. In any case it needs some microcontroller with program. It has bigger price and may be not so fast as simple logic circuits.  Other way is to forget current drivers and going on solution what will need new, custom driver, but adapter self can be much simpler.
I went without much dubt on second - partially because I wanted to do it fast. Other reason is that I can make hard disk driver, and already have it - what means that must correct only HW accessing parts  + some related ones.
First idea was to use PIO mode of CF, and with simple logic activating DRQ line. As DMA transfer end detection I planned to use IRQ or DASP lines, as they should (de)activate right after sector transfer(s).
Unfortunately it worked not - mentioned lines (de)activate not enough fast, and I had more DRQ requests than needed. It screwed up whole thing. But DMA transfer self was error free and very fast. Then I thought about using 512 stage synchro counter (must be sync. to react fast at end). But such counters are very hard to find. 74LS491 is doscontinued for example.
So, PIO way failed.  Next option is to use DMA mode of CF cards. But there is a problem:  CFA specs say that DMA should return Command Aborted error if try DMA transfer in 8-bit mode !  
Despite it, I went on with DMA + 8-bit. Partially because all it started as test of DMA chip top speed. If it would fail, at least I could see what is real DMA top speed - with 16-bit transfer, where half of data would be lost ...
So, made corrections, and after some time it worked well. With Sandisk Ultra II 2GB card. With 1GB Kingston only 256 byte blocks were transferred - so it worked in 16-bit mode.
In any case, both cards did not follow CFA rule:  Sandisk done DMA in 8-bit mode. Kingston did not aborted command, but worked in 16-bit mode even if was set to 8-bit one prior.
I say that Sandisk did good choice - why no 8-bit DMA ? Will it harm someone ?  

Then made simple test program for measuring transfer speed - but I already knew that it will be very high, after seeing DMA timings:


Later I made driver, did reliability tests ...  
But as we know appetite coming while eating, so I decided to improve it by allowing work with cards not supporting 8-bit DMA + allow cowork with another ACSI adapter.
Solved first with single-byte transfers of DMA port in ST, while CF is in PIO mode, where 8-bit transfer is mandatory. The speed is little lower than could be, because of some DMA access restrictions.
Cowork is made by adding ENABLE bit to master latch. So, adapter will respond only if enable bit is set to high. Of course, I choosed bit 7 for it. Because usual SCSI commands all have bit 7 low. With it, sending SCSI/ACSI command for targets 0-3 will always reset ENABLE bit, so CF adapter will be inactive when other ACSI target is accessed and active. And later tests proved that it works - with Mega STE internal adapter flawless. With UltraSatan - very good. With AdSCSI Plus ST DMA transfer failed, but bytewise worked. All in all - it is usable for some copiing between drives/cards.

After all, what we get with this unusual adapter ? Unusual because is somekind of mixture between SCSI and IDE, that uses irregular CF mode for high speed ...
I say:  cheap and simple device, good for DIY. Fastest ever made for ACSI port (at least I never heard about faster). Support for 2 CF cards at once - as Master & Slave.  Works with ST, STE, Mega ST, Mega STE (Stacy, ST Book ?) and likely TT ? .
SW (compatibility) side: needs custom drivers - solved. Autoboot works not without TOS mod.
Compatibility with Atari SW:  it is practically same as with regular ACSI solutions (Megafile, ICD etc...) - because Atari DMA chip works in same way, only setting it is little different.
But mine drivers offer better gaming compatibility than others - as result of game adapting over years.
It means low RAM usage, independence from Timer C, easy solving of tricks like Hole, moving driver in High RAM and similar.

Now go to draw schematic