Left Field Productions, Inc. is a game developer in Westlake Village, California, which concentrates on the console market, specifically the Nintendo N64 and Gameboy systems. Their web page is at http://www.left.com/. I joined Left Field in September of 1998 to do Gameboy programming. Since that time, Linux has been used on the company-wide network offering various services, all running on a lowly Pentium 90 with 16MB of RAM and a single 6.4GB hard drive. The machine is known by all as “the Linux box”.

The company uses 10BaseT for all networking. The Linux box also provides the main gateway to the Internet. It is running the named name server, and the Apache web server for company internal web pages. The Linux box has dual mechanisms for connecting to the Internet: one through ISDN, the other through a conventional 56K modem. ISDN is the preferred method, but during occasional outages, it is necessary to downgrade to the modem.

The Linux box provides public Samba services, making storage area available for file backups and exchange among the artists and programmers. The fetchpop program was used to pull mail off our ISP and forward it to individual accounts on the Linux box. When some problems with fetchpop arose, mostly due to the lack of a timeout feature during mail retrieval, I replaced it with fetchmail, which we are still using. Periodically, employees' preferred desktop mail client connects to the Linux box to retrieve their mail and send any outgoing mail off to the Internet. Sendmail also provides outgoing mail service.

The Linux box has enjoyed up times of more than 60 days. The few times it has been rebooted were due to planned outages for power utility service, and upgrades such as adding a new hard drive. As far as I know, there has never been a system crash. I think it is sometimes rebooted unnecessarily by some of the other Linux-literate employees, as it can be more convenient than specifically restarting a single task after modifying a configuration file. Employees with lots of DOS/Windows experience have a kind of “when in doubt, reboot” philosophy.

When I first started at Left Field, we used a free assembler/linker combination for generating the code for the Gameboy ROM images, and we used commercial painting and graphic arts programs for art generation. The tools for converting and compressing graphics files were all developed in-house. We used Win32-based machines for everything. On my Pentium II 400 machine, a complete rebuild, where all source files are assembled and then linked, would take from 30 seconds to 2 minutes, depending on the project. We were working on a basketball game called Kobe Bryant 3 on 3 and a Disney title called Beauty and the Beast, A Boardgame Adventure.

Once the ROM image was generated, we could download it to the Nintendo debugger system for testing, or more frequently, we would run it on a Gameboy emulator. The emulator was very usable, but it had quite a personality and was temperamental at best. I found if I exited the emulator and reloaded it a few times, it would invariably crash the system. Also, it was unusable for testing the sound aspects of the games—the quality of its sound emulation was painful to hear. The emulator was actually a DOS program working with an extender, and knew nothing about the Windows 95 windowing environment. I think legacy is the most likely cause for its instability.

After finishing up these first two projects, I found myself with some free time before the next project began in earnest. I had long been thinking of writing our own emulator because of my dissatisfaction with the emulator we were using—the frequent crashes and inability to get the source in order to repair it ourselves was quite frustrating. So, within a couple of months of starting at Left Field, I began some minor work writing an emulator for the Gameboy CPU. All my programming was done in C. I installed the public-domain DJGPP compiler and used it under DOS. In fact, I was able to significantly speed up build times by using the MAKE utility from DJGPP, replacing the Watcom MAKE we had been using.

Unfortunately, the demands of the project caused me to stop work on the emulator before I could even test it. Then, about a year later, I had some free time again and was able to get back to the code. This time, I chose to move away from Win32/DOS and switch to gcc under Linux—a move that made the programming infinitely more enjoyable. Surprisingly, there were only a few mistakes in the emulation code, and in a short time, I had the CPU “working”. It appeared to be behaving correctly. The next step was to emulate the video hardware of the Gameboy. For display output, I chose to use the SDL library, which is a multi-platform gaming library. One of the supported platforms is Win32, so the benefit there was that any code I wrote could be used by other employees who were still running Windows. I was using SDL under an X Window System environment. After some solid work, I had the video emulation working quite well, and it was a joy to see ROM images actually work.

Finally, I had to add the sound-emulation code. This proved to be the easiest task, and after a short time, the emulator was producing quite accurate and acceptable sound, again using the SDL library. With a simple recompile using a cross compiler, an .EXE executable could be built. The emulator worked under Win32 as well. There were a few quirks related to sound under Windows 95 that had to be worked out. Windows proved incapable of servicing the audio interrupts at the 64Hz rate I had been using without problems under Linux. I had to compromise and lower the rate to 32Hz so Windows could keep up. I never determined whether the problem was in SDL's Win32 code or in Windows itself.

The assembler/linker we had been using offered a version for Linux, but I wasn't happy with it. The Linux source wasn't as up to date as the DOS version—the two versions were based off different source trees, and it was clear the DOS version had priority. My options were to use the older Linux version or port the DOS code to Linux. I chose instead to abandon the assembler and write my own. Using core code that originated from my own ACC C-like compiler, I managed to create an assembler with a syntax similar enough to the assembler we had been using. I took the opportunity to make changes in syntax when it was convenient. I knew how the assembler was going to be used, and some features weren't important, so I never implemented them.

In the end, my own assembler reported lines-per-minute assembly rates over 30 times faster than the old assembler. With the small source files, assembling each was practically instantaneous. The next part I needed was the linker. Again, I began with the ACC code and modified it to suit. Linking was also much faster than before.

Now, to test the assembler/linker combination, I took our game source trees and made the necessary syntax modifications in the source files. I used the Beauty and the Beast code, and spent about four hours going through all the files to get something without linker errors. Naturally, the resulting ROM image didn't work, but after a day of hunting for bugs in all parts of the system, I got a ROM image that actually came up on the emulator looking like the real game—very encouraging.

In a project like this, debugging problems can be tricky. When no single part has really been tested, a bug can be anywhere; thus, I found myself frequently hunting in the wrong places for bugs. Sometimes, I was surprised to find the assembler actually did something right, and the emulator was to blame—and vice versa.

In testing the ROM images with the emulator, it became apparent I needed some debugging functions built into the emulator. Even before beginning work on the assembler, I had added disassembly capability to the emulator. Doing so had been very helpful in finding bugs in the emulator. After I had the assembler working, I added some nice features like symbolic debugging, break points, expression evaluation, memory viewing and instruction execution history. For text display and entry, I added a scrollback buffer, name completion and tcsh-style line editing.

Bugs became more and more rare, and were easier and easier to find. It was clear the new system was completely viable for developing, and an in-house suite of tools offered very strong advantages that we would never get by using outside software. For example, any desired feature could be added easily, since the source was ours. For portability, I had written everything in standard C. One thing I kept in mind was that at any moment I might have to retreat from Linux and switch back to DOS, and I wanted the tools to work there as well. The assembler and linker compiled perfectly with DJGPP.

I was pleased to note build times were reduced to almost nothing. A complete rebuild that had taken 30 seconds before took three seconds now, on the same machine. It must be noted that those times reflect two different operating systems as well as two different assembler/linker pairs, so an actual breakdown of how the speedup was occurring can't be made. I never bothered to do a detailed analysis; I was happy just to be using the new system.

On the other hand, my emulator, written in pure C, placed significantly more demands on the CPU than the DOS emulator we had been using. The author probably had hand-coded x86 code sprinkled liberally throughout. I also assumed this was the source of most of the crashes caused by that emulator.

To complete the Linux Gameboy development environment, I had to port the various tools used for converting graphics files and dealing with data files in general. Some I had created myself, and these ported almost without modification because I had used DJGPP as the C compiler. The only change required was related to the DOS custom of having CR/LF (carriage return/line feed) as the end of a line, rather than the UNIX-style LF only. DJGPP header files define the flag O_BINARY which must be used when opening a file, to specify that CR/LF should not be converted. Under Linux, O_BINARY is not defined, so compiler errors would result. The solution was to place the three lines

#ifndef O_BINARY
#define O_BINARY 0
#endif

early in the source files, so the source would work without change under both DJGPP and gcc/Linux.

The main graphics manipulation tools had been written by someone else at the company, and I had to make more extensive changes to get them to compile under gcc/Linux. There was the O_BINARY problem as before, but in addition, some of the programs did wild-card expansion in the program itself. The DOS shell doesn't do wild-card expansion, so DOS programs must provide that service for themselves. Unfortunately, the functions for performing this were nonstandard C and so wouldn't carry over. In the end, I hacked out those sections of the code and relied on standard UNIX shell wild-card expansion to do the job. Also, the header file “windows.h” was included frequently, and that dependence had to be removed as well as the structures and system calls in the code that required it.

Another problem which came up is the DOS/Win32 standard practice of file names being case insensitive. Under UNIX, case is significant, so errors popped up in source files where a file such as “Elmer.h” was being requested, when the file in the directory was actually elmer.h (or even worse, ELMER.H). Also throughout the code, programs would create an output file with an extension and the extension would be in upper case, so I'd generate files with names such as “cpaused.CHR”. To me, this was jarring and unattractive, so I modified all the files to produce lower case extensions. This required another round of changes when the wrong file name was being referred to, and error messages were appearing.

In the end, I managed to mirror on Linux every tool we had under DOS/Win32. Developing code under Linux was vastly more enjoyable than under DOS/Win32, mostly because it was faster and more stable. There is also something very rewarding about using your own tools in any work. In total, from start to finish, the time to bring everything up on Linux, as well as writing the new emulator, took about three to four weeks.

Shortly after the next project began taking form, one of the artists asked if he could do builds himself, as he wanted to tinker with animation frames and see how they looked in actual use. Rather than go to the trouble of getting all the tools set up on his machine, as a quick solution I copied the source tree to a directory path that my machine had been making public with Samba. The directory /d on my machine appeared as //dave/d on the network. I set up a simple script to check for the existence of the ROM image file. If the file wasn't there, it would do a rebuild. So, to get a new ROM image, the artist would copy over the modified data files, then delete the ROM image and wait a moment for the new build to appear magically. With Linux's excellent disk caching, I was usually unaware of when this happened.

One final problem caused me some worry. I was currently the only person actually using Linux as a development platform. What would happen if one of the other programmers wanted to contribute to the project? I didn't want to force my Linux preference onto anyone else, so I had to be able to deal with this. Already, all our tools had equivalent Linux/DOS versions, so that wasn't a problem. The problem would be from multiple programmers merging code changes to the same source tree. Under Win32, we had been using Microsoft Visual Source Safe for that. Although I suspected there must be a Linux client for talking to Source Safe, I never looked. Instead, I studied how to use CVS (Concurrent Versions System), the traditional source-code revision-control system used in a vast array of open-source projects. After experimenting with CVS and learning enough about its quirks, I found it performed at least as well as Source Safe. It conveniently handles the end-of-line problem by storing the source files in the repository in UNIX format and adding/removing the CR as needed when dealing with a Win32/DOS client. I'm using CVS on the command line, and I prefer that to mouse clicks; CVS seems faster at updating the modified files.

Even though I'm the only programmer on the current project, I am checking my code changes into the network CVS server, just to get into the habit of doing so, as well as to provide an additional level of backup and modification history. As you might have guessed, the CVS server is running on the Linux box.

Related URLs