This is the final article in a three-part series (see Resources) on reducing boot times for in-car computers. The impetus for the project has been my company, CarBot, where we're trying to get computer hardware to behave as it should in a car. In the previous two articles, I discussed a range of solutions for solving the problem of slow boots. In the months since those articles were written, we've thoroughly explored the possibility of using a BIOS replacement on the VIA Epia-M boards in an attempt to reduce boot times to the theoretical minimum. During this time, we've also discovered some hacks that can be used to minimize the effect of boot times on the user experience.

In our earlier articles, we compared the unnecessarily slow boot process to that of a car radio. The car radio boot times have climbed from nearly instant in the 1980s to several seconds today, but they still are rapid enough to be hardly noticeable. If you watch a modern radio right after you start the engine, you may see it do a small power-on self test, flash all the lights on the unit--much as dashboards in many cars do--and then power on the amplifier, producing sound within a second or two.

We decided to use the radio as our benchmark of rapid usability and appliance-like behavior. We attempted to minimize the following two variables: time from computer power-on to video and time from computer power-on to audio.

Our testing rationale was if we can minimize the time before the arrival of audio and video in the boot process, we later can optimize our software so that whatever the user was seeing and hearing last can be restored. This is what a radio does, this is what a television does, this is what a computer should do.

It takes users a couple of seconds to orient themselves to the task at hand, such as configuring a car computer. If the screen looked like it did the last time they shut it off, it at least would give the illusion of being on and would be a little less annoying. In the book The Humane Interface, Jef Raskin, father of the Macintosh computer, asserts that there are no real technical reasons why a computer should take any boot time at all.

Reducing boot times can be done by brute force or through finesse. The solutions we have found or been told about can be broken down into three basic categories:

Long ago, I saw a comment on the Linux BIOS page that EPIA-M support existed but was unstable. That fact had not changed since July 2003. I e-mailed the maintainer of that branch, and he said "The fastest time that I have is 5s with some optimization of [the] init script."

Wow, five seconds. We wanted to get that.

Initially, I tried to find a commercial vendor to replace the BIOS. General Software was kind enough to send me a sample of its BIOS replacement for the EPIA-M, but the company did not yet have a similar version for the EPIA M2, another motherboard we use in our products. Also, the entry price was surprisingly consistent among the several vendors I asked--around $10,000 dollars. Recouping the effort of making a custom BIOS requires this sort of pricing, I suppose. Part of the problem was these BIOS implementations did all sorts of great stuff, and what we were in search of was a BIOS that did almost nothing.

So we set out to see how much time we could recapture by throwing away the BIOS and booting Linux ourselves. The idea was to throw away all the useless BIOS functionality, show video at the boot loader (that is, GRUB splash screens) and start playing audio as early as we possibly could in the boot sequence.

Our Linux engineer, James, was sent off with a pair of Epia-M boards, a bootable 256MB Flash drive, the first two boot articles and a pointer to the LinuxBIOS site. I initially thought that we could get enough Linux into the BIOS chip itself to do something useful, such as squeak and Flash the monitor. That would have satisfied our light and sound objective, but James discovered that we did not have enough space without going to the Flash drive. So the new goal was to reduce the BIOS boot as much as possible and then launch a stripped ZipSlack distribution on the compact Flash IDE drive.

The main benefit of LinuxBIOS for our application also was the main drawback. The primary development focuses on making clusters of supercomputers that need to be able to reboot quickly. These clusters were designed for network booting, however; we wanted to boot off a disk or a Flash device.

The EPIA-M boards, although quite popular in the hardware hacking scene are not popular as clustered supercomputing nodes. The EPIA is not the fastest motherboard, running between 600MHz and 1GHz with a comparatively feeble CPU.

As the LinuxBIOS project has progressed, some of the code to do things other than bootstrap supercomputer nodes has fallen out of sync with the documentation. Downloading and reading most of the LinuxBIOS mailing list archives helped to reconcile the discrepancies. For instance, for a while it seemed as though the Etherboot modules would be adaptable to booting off of other devices, such as the compact flash/IDE adapter combo we were using. Eventually, James switched to using the FILO bootloader.

The hardware aspects of flashing a BIOS were the most adventurous. The first few compiles of a new BIOS image naturally produce a machine unable to boot. The essential tools for BIOS programming work is two of the same motherboard and the correct chip tweezers.

Restoring a fried BIOS seems wrong when you're doing it. When you fry your BIOS in computer A, you boot computer B with the fixed BIOS and then carefully extract its chip with the computer still running. Then, you put the other BIOS chip in its place and burn it with a backup of the original BIOS. Then, you can put the chip back in the other computer and continue on your way.

Getting a usable BIOS was no small feat. Etherboot would fail in many charming ways, for any reason, such as not finding the Ethernet card, not finding the kernel on the disk and so on. After a bad flash, either there would be nothing at reboot or LinuxBIOS would fail to load the BIOS itself or FILO. After a dozen or so failed attempts, a bootable BIOS finally was flashed, and ZipSlack was booted successfully off the CF card.

The following several sections, written by James, detail what was discovered along the way.

The steps go in this order: power-up loads FreeBIOS, FreeBIOS loads a bootloader (FILO), the bootloader loads the kernel and so on. First, then, you have to build the bootloader. The FreeBIOS people like to use Etherboot, but Etherboot currently is broken as far as booting from disk goes. So, FILO is the thing to use. FILO is like GRUB, but it has no need for the BIOS.

Unpack FILO, switch to the FILO directory and run make. This creates a default config file. Edit config, then run make again to create filo.elf, which is what we use for the LinuxBIOS payload. Here's my FILO config.

Next, we need to add a video BIOS. Actually, the video BIOS on the Epia-M is pretty useless (more on this later), but it serves to let X know that we have built-in video, so in it goes.

Because the video BIOS is copyrighted by VIA and cannot be distributed independently, you have to grab it from your original BIOS. Do so in one of two ways. Either boot Linux and enter:

dd if=/dev/mem of=vgabios.bin skip=1536 count=128

or extract it from the BIOS distribution image, if you can find the offset. Cat the two together, like so:

cat vgabios.bin filo.elf > vfilo.elf

Therefore, vfilo.elf is our payload--a bootloader plus a video BIOS.

The traditional LinuxBIOS now is called FreeBIOS, as it can load things other than Linux. A new rewrite, called FreeBIOS2, doesn't appear to be ready yet. I therefore used FreeBIOS, which is available only from CVS. To get it, type:

cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/freebios login
cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/freebios co freebios

FreeBIOS builds and configures in a separate directory, so doing a CVS update doesn't hose your existing config files. So, in the same directory as where you have /linuxbios, make a build directory by issuing mkdir build. Then, create an epia-config file; here's our epia-config file.

Notice all the full paths in that file? LinuxBIOS likes full paths. To build the BIOS, do this:

python /home/jamesh/linuxbios/freebios/util/config/NLBConfig.py
epia-config/home/jamesh/linuxbios/freebios/

where /home/jamesh/linuxbios/freebios is where I put FreeBIOS. Then run make, which should produce a file called romimage. This is what you burn to your BIOS chip.

You also are going to need some utilities from the FreeBIOS folks. cd into freebios/util/mkelfImage and issue

./configure && make && make install

to get mkelfImage, which should end up in /usr/local/sbin. Then, cd into ../flash_and_burn and do make to get flash_rom. This is what you will flash your ROM with.

On a side note, some other tools are available to produce ELF kernels--don't use them. Most of them are broken, including the one from the Etherboot project that is mentioned in the HOWTOs.

The Epia patches include one called epia-fbdev, which leads to this encouragement from the LinuxBIOS mailing list:

El Jueves, 13 de Mayo de 2004 18:46, John Laur escribi?:
> If you have VGA correctly working with LinuxBIOS (Ie you see the
> linuxbios logo on startup), then you have VGA working as best it does
> with LinuxBIOS.
>
> Then you can worry about the kernel. VESA Framebuffer does not work with
> EPIA-M as the VGA bios doesn't initialize the video card up to where
> vesafb can take over. You have to use viafb (part of the -epia patchset)
> to get video on the console. Make sure you have it compiled into the
> kernel and not as a module. Make sure vesafb is NOT compiled in either,
> otherwise you'll need some commandline trickery to make it work. Also,
> viafb doesn't work with bootsplash in silent mode on kernel 2.4 if you
> are using that. I have not tested with kernel 2.6.
>
> John

However, all the flavors of epia-fbdev I tried hung the machine or produced scrambled text.

Anyway, one should build a kernel in the normal way. I unpacked the kernel, configured it and issue

make dep && make bzImage
&& make modules

Then, to install the modules somewhere so I could copy them to the Flash, I entered:

INSTALL_MOD_PATH=/home2/jamesh/linux-fast/ make modules_install

That command created /home2/jamesh/linux-fast/lib/modules/2.4.26-fast/. Next, make an elf image out of the kernel. This lets you put the command-line arguments in the kernel file itself and allows the bootloader to determine how big the kernel is:

/usr/local/sbin/mkelfImage --kernel=linux-2.4.26-fast/arch/i386/boot/bzImage 
    --output=bzImage-2.4.26-fast.elf --command-line="root=/dev/hda4 console=tty0 console=ttyS0,115200n8 rw"

Note that I set root=/dev/hda4. I also set the FILO like so:

AUTOBOOT_FILE = "hda4:/bzImage"

Now, I only needed to put the kernel in /dev/hda4 in the root directory--easy. Next, copy the romimage and flash_and_burn files to your Epia box and run flash_and_burn by itself to see if it recognizes your Flash chip. Typing flash_and_burn romimage burns it.

Now, hook a terminal up to the serial port, set to 115200-n-8-1, and boot. In 20 seconds you should hear Dexter, assuming you are using my ZipSlack image. As you can see, X works but the mouse does not--I don't know why. Type startx to start X.

I've created two disk images. One is UMSDOS, which I unzip into a MS-DOS filesystem on partition 4 of a disk. The other is a tar file that should work on an ext2 or ext3 filesystem.

The full log of our successful boot with fine-grained timings is here, and the following guided tour takes you through the highlights.

I wrote a program to dump the console out from the serial port with a timestamp. Here are some interesting things to note:

At this point sound is playing out the soundcard--19.811 seconds, or 17.431 if you skip the video BIOS call. Of this, only 6.559 seconds actually is booting Linux.

We did not get the five-second boot we were hoping for. Several of the reasons were specific to the EPIA board we used. The on-board EPIA video presented a problem, too, because its own video BIOS was part of the computer BIOS. Had it been a separate video card, its initialization could have been skipped altogether, leaving it for X to do when it launched. Similarly, if we had been using a more conventional sound card, such as an old SoundBlaster, we could have had sound much earlier in the boot process, around ten seconds.

The exercise did illustrate many useful things, however. It gave us a precise breakdown of where time was being lost, and it did give us a lower threshold--around ten seconds--in which we know we can get sound and video out to the user, given the right motherboard and hardware. And this is from a cold boot, pressing the power switch.

Although we never did get the video to display, even after X was booted, good news on this front came as this article was being completed. An implementation of FreeBIOS for the EPIA-M2 board has been completed by Nick Barker, and it seems to solve not only the video problems but a longstanding difficulty of booting directly from the CF slot on these boards. With further work, we could get our applications down to this new theoretical ten-second minimum.

But there are other ways to reduce the boot time for an in-car computer. To conclude, I'll list a few of the options we haven't covered yet.

As general-purpose computers find their way into cars in greater and greater numbers, people will keep hammering away at boot times.

"Reducing OS Boot Times for In-Car Computer Applications, Part II"

"Reducing OS Boot Times for In-Car Computer Applications, Part I"