Atari ST family and mass storage

[Petari]

What I write here is based on my about 30 years (since 1992) experience with hard disks, mass storage with Atari ST family. Making partitioner/formatter, hard disk driver SW for. Which is acclaimed, recommended by many, and for sure is very good for easy data transfers between old Atari STs and modern computers.

Details are here:  http://atari.8bitchip.info/ASTfamMS.html

Here I make shorter version of it, trying to explain with simple language, but some computer terms are inevitable.

Work with hard disks is solved in 2 layers (that stays for most of  OS (operating system) by diverse computers) :
Filesystem - deals with files, directories  .
Hard disk driver - it is what accessing HW (hard disk/Flash card adapter) - and it gets requests from OS - like autoboot, sector transfers .
This concept allows work with wide range of adapters, and only driver part is what needs to be written for new type of adapter.
In TOS 1.xx there is only ACSI autoboot support. At 2.06 IDE autoboot is added , as IDE drives started to take over market - lower prices for drives and adapters too.
And to add: there is no hard disk driver in regular TOS versions. Only short code for autoboot - what will load hard disk driver in RAM. The reasons for it are: limited space in TOS ROM. Flexibility, being ready for some new protocols which may appear. And that's happened with ICD extension of ACSI protocol.

The limitations: most of limitations is in filesystem. Which is FAT16, and partially DOS FAT16 compatible. And because that it needs byte-swapped (low Endian) parameters in boot sectors, FAT (file allocation table). Sadly, compatibility goes only for partitions up to 32 MB size, above it it is not compatible, and the reason is 16-bit sector addressing used in TOS filesystem (16 bit addressing means exactly 32 MB with regular sector size of 512 bytes).  So, above 32 MB TOS uses so called large logical sectors like 1 KB, 2 KB ... 8 KB (TOS 1.00-3.06, 4.02) or even 16 KB (TOS 4.04) . And that's not good. All what I saw in last 10 years indicate that the culprit is DRI's Alcyon C compiler - was not good with unsigned integers.

Partition  size and count limitations:
TOS 1.00-1.02 :  max size is 256 MB.  Max 14 partitions, C-P .
TOS 1.04-3.06, 4.02:  max size is 512 MB.  Max 14 partitions, C-P .
TOS 4.04: max size is 1024 MB.  Max 14 partitions, C-P .
Max count stays for all partitions on all attached storage - so if there are for instance 11 partitions on first one, only 3 from second can be handled.

Compatibility with modern computers, OS:
As is said above,  TOS FAT16 is compatible with widely used DOS FAT16 only up to 32 MB size partitions.  But there is a way to make TOS/DOS compatible partitions - that needs special sized reserved sectors, FAT, etc.  Only way to create them is special partitioner program for that, and it exists only for Atari - so don't think about doing it on PC (well, except with emulator) . And of course, special driver for it is needed. Then data transfer with some modern computer is really peace of cake. But as usual, there might be some problems - in this case it is Windows long file names (LFN) - such files will confuse TOS, which is not ready for something created around 1995, so must avoid them.  I use Total Commander in Windows, where can set it to not use LFN . Unfortunately newer Win versions tend to write automatically some system DIRs and files on every attached Flash card and partitions on it.
Even if delete it, will write again in short time.   So, since  2014 I added LFN filtering in my Atari hard disk driver SW - that's best defence.
And it is included in improved TOS versions, of course.

Disk space usage efficiency :
Used space on disks/Flash cards is always bigger than sum of file lengths. The reason is that there is minimal space what one file can use on disk - and it is called cluster. FAT is where is data (records) for all clusters in 1 partition.  If it is 0 it means that it is free. If is $FFFF it means last cluster of file.  Some other value there means number of next cluster of file - that is normally next cluster, but if it is not free when file is written to disk next cluster of file will be first free one in partition. This allows that file  can be written to partition even when no so long free segment as file length . And then file is fragmented, what means slower work.  But let's stay at space usage:  so, if 1 cluster is for instance 16 KB, and that's case with partitions in range 257-512 MB (TOS 1.04 and higher)  it means that some very short file will take 16 KB on disk. Or if file is 16 KB +1 byte, it will take 2 clusters, so 32 KB.
So, space usage efficiency is worse with many short files and longer clusters.
FAT16 actually means 16 bit cluster records,  so should have up to some 65530 clusters. But it is not case with TOS, it can have max half of that, so 32765 records - and that's not good for space usage eff.   It is even worse with TOS 1.00 and 1.02 - they can have max 16382 clusters - and that's the reason for smaller max partition sizes too.
What can user do if space is tight, and there are many short files to save/copy ? Using more smaller partitions instead one big.
Well, with today large capacity and cheap Flash cards, this is not so big problem. More chances are that problem will be how to utilize some larger capacity card.  Because 14x512 MB is 7 GB - and that means that no sense to get some larger than 8 GB capacity card for Atari ST(E) with regular TOS.
But soon may happen that will be not able to find such cards in shops.  Solution can be my improved TOS  iTOS, on what I worked in last 4 years.  It has fully DOS compatible FAT16 filesystem, while is compatible with TOS type partitions too.
Limits are:  max 30 partitions of max 1 GB size - ergo can utilize 32 GB Flash card well (and they are never full 32 GB capacity, usually about 10% less - stays for other size cards too).  Disk space usage is more efficient, because max cluster count is 65530. And all it with even less RAM allocated by driver.