CRPG's are, in many ways, just very elaborate database programs. They read, write, manage, and display a wide variety of data to the user, in the form of graphic maps, text, and information. About 60-70% of the code is devoted to user interface and data management. To store the large amounts of data, disk files must be used.
The important element in the design here is to identify what data must be resident at all times, and what can be stored and retrieved as needed. Very common data, such as text messages from the UI, should be kept resident. Object names, map data, monster data, even graphics are not needed all the time.
The TI disk file system is built around a 256-byte sector. (Modern PC's have 512-byte sectors.) Depending on the format of the disk, you can have data on one or both sides, 9 or 18 sectors in a disk track, and 40-80 tracks on a disk. The classic TI disk controller was double-sided 40-track, so the theoretical maximum disk size was 180k. (40 tracks * 9 sectors * 2 sides = 720 sectors * 256 bytes = 180 kilobytes.)
The most common 'upgrade' for disks was a disk controller capable of double-density, which increased the number of sectors in the track. The typical increase was to 18, which gave the TI a 360k disk, equivalent to the early IBM-PC in capacity. RAMDisks emulate a disk, offering even more capacity, and hard drives are typically used to create a series of volumes, all acting like normal floppy disks to the TI.
One constant throughout all this, though, is the sector size of 256 bytes. This can't be changed without completely altering the TI disk file system as it stands. It IS possible to create your own custom formats, if you care to. Many Apple II programs did this, mostly for copy-protection purposes. I want the disk files to match the original TI file system, though, and I don't intend to encrypt the program.
A final consideration is the total number of files. The TI file system can support, at most, 127 files on a disk. This limitation applies even to virtual disk systems, like RAMdisks. So, if a player wanted to have all the files for the game in one place, it cannot exceed this count.
This means that caution is needed not to exceed this limit, for the game as a whole. At present, my file count is around 60. So I should be fine; I just need to avoid too many memory-image files. Since each memory-image is a separate file, it's easy to go overboard.
The TI file system offers a variety of soft types for file management, which can be very useful in defining how the data file is structured. In this case, most of our data file information is fixed, so we don't need to worry about identifying it, we can simply store file specifics.
One 'magic' file type on the TI is known as the memory-image. As the name indicates, a memory-image file is a perfect copy of a block of memory stored in a disk file. All you need specify is the number of bytes of data and an address in VDP RAM, and it will save that block of RAM to disk, or load it from disk into VDP RAM. BASIC and Extended BASIC programs use memory images for programs under 12k in size.
For our purposes, memory-images are incredibly useful. Data for a saved game can be placed in a large data array in VDP RAM and saved to disk. Graphic character sets can be loaded directly, leading to nearly instant character changes.
For a lot of our other data, though, we need only load one or two records, so in this case, our best choice for file types is internal and relative. Internal files are stored in binary form and ignore display character issues; this is important as even in-game text and dialogue will be encoded for space and encryption. Relative files can be accessed in any order, which is vital for speed.
One small issue with relative files is they must be fixed in size; that is, all records are of identical length. This is where the importance of sector size comes into play. If your record sizes are not based on a binary two system (2,4,8,16, etc.), then the file system will fill up the sector as much as it can, and then move on to the next one. Consider if you have a file size of 129 bytes. In this case, it only stores one record per sector, which means your file is taking up twice the disk space! Even lesser cases, such as size 48 records, still leave 16 bytes per sector unutilized.
Interestingly, the maximum size a record can be in the TI file system is 255 bytes. One byte short of the sector? This actually isn't as bad it sounds; the file system actually loads the entire sector into the VDP anyway. If you wanted to, you can retrieve any missing sector data from the file buffer in high-VDP memory. Why TI decided to make size 0 mean the default of 80, and not 256, is a curiosity. It also means your best maximum record size is 128 bytes; you will fully utilize the sector in this case.
So what do you do with data that doesn't fit exact sizes? Such as, for example, dialogue? Variable files seem like the best solution, but then the file must also be sequential. Which means if you wanted record #69, the file system has to traverse 68 records before finding the one you want. For games, delays are bad. In this case, the best thing to do is categorize your data into multiple files, each with a different fixed size. By using an encoding algorithm to further compress the text, you should have sufficient room for a good amount of dialogue without sacrificing flexibility. You'll have some waste, of course, but if you have enough size variety with files, you can cut this down to a minimum.
The most complicated problem with multiple disk files is how to manage them. Different users will have different needs.
A user with a hard drive system will want to place all the files in a single location, and never be prompted to swap. He will also have a potentially long path name, to include some form of directory structure.
On the opposite end of the spectrum, a user with only the two necessary drives will have to know when to swap disks, and what disks to put in. No matter the number or type of drives the user has, all this setup data must be stored in the game engine somewhere.
The peripheral access block (PAB) is the primary method of defining a file in the TI system. It consists of 10+ bytes of data, which must be placed somewhere in VDP memory. The variable amount is due to the filename. The original TI file system allowed up to 10 characters in a filename, but you could also put disk names into the path, so that it would search the file system for a drive of that name. (Although in practice this doesn't work very well.)
My plan is to not define a full PAB for every file. It would take too much space in VDP memory, and is a waste. Instead, disk file information will be stored in a fixed-size block, allocating 16 bytes per file. This is sufficient space to define characteristics, and also can be easily loaded as a memory-image from the root source drive, so that the engine knows where to go for everything else.
This does require a bit of extra processing; a PAB must be created anew for every disk access. But compared to other CPU-intensive operations, this is pretty simple. The only caution that must be taken is to minimize VDP access, the bottleneck in the process. A single read of 16 bytes (PAB constructor data) and 40 bytes (Core path name for a particular disk), and a subsequent 50+ byte write should happen well within 1/60 of a second.
The TI can, of course, can have multiple files open at a time, as many as you wish to allocate space for. The default value is three, and space is allocated at the start for that many in the top of VDP RAM. However, I don't intend on having multiple files open if I can avoid it.
The first reason is that multiple file reads/writes can make the logistics very complicated. The primary purpose in doing such a thing would be if you were consistently accessing a file in real-time, and you wanted to have it open indefinitely, but still open and access other files while doing so. This could cause all sorts of headaches, though, not the least of which is that a sudden crash or an accidental 'quit' could leave a file wide open.
For one thing, if a file must prompt for a disk swap, it can complicate matters a lot. The only files you know for certain you can access freely would be on the play disk. The only real potential value I see is that in the event that code must be actually kept on disk and loaded as needed, then you could use the multi-file system architecture to accomplish this. However, if all you're doing is loading up data for a single-time use or view, there's no need.
I recently purchased a CF drive system on eBay, from Jaime Malilong. This is a very clever little device that simulates the TI disk system (and optionally the 32k expansion) through the use of a CF memory card.
I was a little apprehensive when I first bought it... raw circuit boards can have that effect. But to my surprise and delight, it worked perfectly! I was able to transfer disk images directly from MESS onto the CF card, and they loaded without issue on my vintage TI. No more bulky fire hose cable, no more noisy fan, no more worries about floppy disks going bad! The virtual disks it creates are 1600 sectors in size, or 400k. Access time seems very good as well, equivalent to the original TI controller, and likely a little faster. (Anyone do any bench tests?)
I highly recommend this product to anyone looking to write and test software between emulation and original hardware. It's also great if you just always wanted a disk system for your TI console, but couldn't find or afford a Peripheral Expansion box. Please note that his ad says a power adapter is optional. However, he told me that it's now required, as not all TI consoles can supply the requisite power.
Also a word of warning, the CF drive system is NOT compatible with a PE box. So if you're looking to port disk software from the TI to the PC, this doesn't work as well, unless you have some form of persistent cartridge memory you could store the data in to transfer to the CF system.
One common element of CRPG's is the big world map. This is a massive sprawling map upon which you can walk from one end to the other. In some games, the map was literally a world map; reaching the edge would wrap around to the other side.
So, when I was in initial design, naturally I wanted one.
The first question is, how is the map stored on disk? In the case of Ultima IV, it's stored as a 64k data file, with every byte representing a tile on the map. Of course, you never load the whole map, you only load a portion of it at a time. Ultima IV uses a single kilobyte buffer, or 32x32 tiles, for all its in-game maps. Earlier Ultima's allocated more space; the maps in II and III were 64x64 tiles, around 4 kilobytes. This isn't surprising, since the engine in IV is more complicated than its predecessors and takes up more code space.
An interesting 'bug' on Ultima IV and V was that the map loader was not very picky about filenames. In fact, it may not even have accessed the disk by filename at all; it just loaded raw sector data. Players discovered that if they placed a different disk in the drive when prompted to re-insert the world disk that it would not check the disk name, and would load any data it found. This could be lucrative, as the treasure chest tile number matched the number for a null byte in their sector format.
On the TI, I had the problem that with level-3 routines, the whole sector is not available. It was early in the design and I wasn't fully convalescent with the TI file system, so I decided to make the map 240 tiles wide, and make it slightly longer in rows to compensate for the loss. I wrote a 'blank' map creator, which simply generated a massive internal/fixed 240 file full of 'grass' tiles. I also wrote myself a map editor and viewer program so I could start playing around on my new world map.
So, what happened? Well, when I wrote a "world walker" program to test out the concept, it worked, but the game play was terrible!
The problem is that the sector-to-record ratio is 1 to 1, the worst-case scenario for loading records. The TI file system will load the entire sector where a record is located into VDP RAM, and will refer to this buffer as long as possible. (Which means it's still a good idea to load relative records in a sequential order if you can, as it will do less reading from the disk.)
For my internal map, I had allocated enough space for a 48x48 size map, or about 2.25k. When the map moved, I was pushing the "window" on the map about 16 tiles one way or another when you hit the 'boundary'. As a result, on every map load, there were 48 sector accesses.
The first problem was speed. Even in emulation (which is all 16-bit 0-wait memory) and with a slightly faster emulated floppy drive, doing a map load took 3-6 seconds. Which, when you're trying to move about, seems VERY long. When I tried it on my original hardware, using my CF drive system, it actually performed all right, but since the CF drive is pretty quick on access, this isn't an accurate measurement of standard performance.
The second problem was that my threshold lines were static. As you approached a map border, the next section would load, which had overlaps to prevent the player from seeing over the edge of the map. However, if you tried to back-track, or move about in a zigzag, you could force continuous reloads. I could compensate for this by loading smaller overlaps, which would push the threshold lines away from you, but then I would need MORE reloads on average to move in a straight line, which coupled with the speed problem above was undesirable.
As I analyzed the problem, I realized that Ultima IV and V suffered from the same issue. Both run very badly on the 8-bit platforms, such as the Atari, Apple II and Commodore 64 systems. And their behavior mimics what happened on the TI, a long access time to load the new map section. You don't notice these kinds of problems in modern emulation as much, and certainly not in PC versions, since they're running off the hard drive.
So how to fix it? Well, from a technical standpoint... you can't. The only real option is to get a faster drive system of some kind, such as a RAMDisk or hard drive. This is certainly possible, but my original goal was the game WOULD be playable on a TI. I expect a certain degree of drag from older authentic hardware, but this was simply too much.
I could write a 'disk sector reader' routine, instead of using the standard level-3 file access commands. This doesn't solve the problem, though, of 48 sectors being read; at best I buy myself a little optimization in the reading process.
The true source of the problem is how the data is stored on disk. Long horizontal lines that consume a whole sector, when you only need a small part, is very inefficient. A solution is to split the map up into smaller pieces, in separate files, which could then be stored more efficiently, with multiple rows in one disk sector. (A 64x64 map, for example, can be stored in 16 sectors, a third of the sector accesses of my above routine.)
The problem with map splitting is the edges. What happens when a player approaches the edge? Does he just see a black wash that when crossed over, shows the next map? That would certainly be easier to do, but it breaks the perception of a continuous map completely.
A related solution would be to have "buffer" strips along each map edge, just big enough for the player never to see beyond, so that as they approach the 'edge', the map reloads. A nice idea, but an enormous waste of disk space for a purely visual effect (A 6-tile distance of viewing on a 64x64 size map is 1392 bytes of waste.) Any programmer worth his salt would be revolted with such a solution.
So, my solution, in the end, was simple: lose the big world map.
One element that lead to this decision was the realization of how I was designing it. I was having fun, using my graphics to create neat and interesting places, when I began to realize that I was repeating myself, and creating a lot of artificial barriers, such as mountains, over and over, to break up the rather stark landscape.
I had already realized before that size isn't everything. I had originally planned to have two huge 64k maps, each self-contained, with traveling from one half to the other at the mid-game point. As I considered it, though, I realized that too much space was bad. Would I have enough for large variety of towns and dungeons to visit in both places? And dialogue? Probably not. So I cut it back to a single world map. I wasn't planning on a true 'world' map, with looping and 65-70% of the area being ocean tiles. So I had plenty of space for what I wanted.
However, my mind really couldn't encompass that kind of space. I was thinking at the small detail level, how each area would appear to the player, not an overall view. In addition, I was still finding myself running short of ideas for terrain. And when the speed issues with the big map came up, it made it that much easier to abandon.
So, instead of a big map, we have a series of small maps, anywhere from 16x16 to 64x64 in size. I call this technique 'zoning', as it is what a lot of the larger contemporary online games do in order to break up large spaces into smaller, more individual areas.
The idea works surprisingly well. Since each map is self-contained, only one load is needed. A few seconds is fine when it only happens when you first enter the map. It also means that at the engine level, there is no discernable difference between world, town, and dungeon maps except the graphics sets used, and some accessibility issues. Very nice indeed!
There is also the added advantage of different projection schemes. Why have a square map? With 4096 bytes of space, it can take any shape I want. You could have a 32x128 size map, for example, for a long thin area. And with an engine tweak, you can have 'slanting' maps, so they run diagonal. This works great for coastlines that are intended to slope, and saves you wasting space on empty ocean areas.
The one flaw, of course, is the edges. You can project a single character in "off-map" areas easily enough; the obvious choice is an ocean tile for the illusion of "endless" seas... but that doesn't work so well inland.
The best solution here is to just make certain the player can never quite see beyond the boundaries. This is tricky, as some overlapping will likely have to happen, but if the player enters a city from the south gate, and exits from the north, could he really tell he's entered a new map? Some waste is likely, but not nearly as much as with the buffer option.
As you can see, there's a lot to consider with the design, and how the engine will interact with the data. In next month's article, I'll talk about a very interesting topic, the line-of-sight algorithm.