We believe Linux is going to play a significant role in embedded applications. It is compliant with POSIX 1003.1 and supports the POSIX soft real-time extension. Theoretically, it is capable of supporting a wide range of embedded applications which require only soft real-time performance such as Internet routing. Easy customization makes it even more attractive. To investigate the possibilities of using Linux as a platform for embedded systems, we conducted an experiment in which we ported the kernel to a PowerPC-based board. The porting took a few weeks. Our result is a Linux port, called elinux here, based on the Linux kernel 2.1.132 which was the latest version at the time we began the experiment. elinux runs dozens of commands and programs, including bash and vi on the console via a serial port. Porting Linux is actually a very enjoyable experience. Besides, all the work is based on open-source software exclusively.

Linux is portable. We have already seen many Linux ports on various processors. However, few documents describe how to port Linux. The information is scattered in various documents and source code. Porting is not an easy job for any operating system. Even a minor change in the kernel code to suit a particular piece of hardware needs a considerable effort. Fortunately for Linux, all major components of the kernel are already designed to be architecture-independent. This makes the work relatively much easier.

It is harder to port Linux to a board with a new processor than with a processor Linux has already supported. In the latter case, we can reuse board-independent code; for example, the code for memory management. Only a relatively small portion of the kernel code is board-dependent. When we considered implementing elinux, we tried to avoid reinventing the wheel. We kept most of the necessary changes limited to board-dependent parts. Our experiment was done on a PowerPC-based board. Linux already has ports for PowerPC-based machines such as Power Macintosh and a few PowerPC-based embedded boards. However, due to diverse board architectures, configurations and booting methods, modifications are required when we consider a new board.

In our case, some changes in the kernel and a few small new programs were created to support elinux. In the following article, we emphasize our experience with the issues of most concern in our work rather than implementation details. These include setting up the cross-development platform, designing the booting sequence, modifying the kernel, creating the executable image and root file system image and debugging.

Our goal is not to port Linux to a particular kind of hardware. Instead, we are interested in approaches to porting the Linux kernel to potential embedded systems. Thus, it doesn't matter what kind of board we choose as long as it presents a typical situation. The board we actually used in our experiments is a PowerPC-based board built on a PCI bus. It has a PowerPC 603e processor, an MPC 106 as the memory controller and the PCI bridge, 32MB DRAM memory, a PC16552 DUART chip for two serial ports, a memory-mapped real-time clock in non-volatile memory and a simple but customized interrupt controller. It also has two flash memory slots and an Intel 82558 LAN controller to provide three LAN ports.

The board has its own bootstrap code in ROM. This code does hardware initialization and also provides a simple native file system and the TFTP support. Although we used these two services to boot elinux, our approach can support booting completely from ROM.

Simply installing binaries from any Linux distribution does not guarantee a working cross-development platform. Some people experienced difficulties in setting up a complete cross-development environment. Our experience shows that this is not only possible, but also brings us a fair amount of convenience, as we can use some of the most popular software packages. Our lessons are proper distributions, proper configurations and recompiling.

Here are our actual steps in setting up the cross-development platform for PowerPC on Pentium machines:

After the base development system is ready, we install cross-development support as follows:

Booting an operating system seems easy. You just power on the system, and after a while, you'll see a prompt on the console which indicates the system is running. If we look into the booting internals, we get a more complicated view. Booting includes hardware initialization and software startup, operations which differ from board to board. Different booting code exists for each kind of board in the Linux architecture-dependent source code directory, that is, linux/arch/. For a new board, we usually have to add a new booting sequence.

A typical embedded board has no floppy and no hard disk. Code and data are initially put in ROM or can be downloaded through a network connection. We have designed a general approach to booting such a system from ROM.

Our approach is to divide the booting into two stages supported by two separate loaders. One is called the image loader iloader, and the other is the Linux kernel loader kloader. iloader is ROM-able. That is, after the system is powered up, it starts, does necessary hardware initialization, then moves the Linux kernel loader from ROM to the proper location in RAM. kloader starts running once iloader finishes. First, it does more hardware initialization. Then, it sets up the environment for booting the Linux kernel by uncompressing the Linux kernel image. Finally, it jumps to the kernel code to begin the main Linux startup sequence.

To make things clearer, consider elinux. The final elinux image, which we call an elinux ball, is packed as a single file containing three items:

The size of an elinux ball depends on how many services and programs are included. In our experiments, it is limited to 2MB, which is big enough for most situations. If larger programs are needed, they can be downloaded after the system is up. It is good to keep the ball small. The packing is simply done by a tool called packbd (packing binaries and data images). The elinux ball is obtained using the command:

packbd iloader kloader vmlinux.bin.gz ramdisk.gz\
elinux

The implementations of iloader, kloader and packbd are straightforward for any system, except for hardware initialization which usually requires more effort.

Kernel modification is the hardest part of porting. Fortunately, Linux has been designed to be portable, with its sources well-organized into a tree structure. Once you have made considerable investigations into the kernel sources, the things to do become clear. As we mentioned earlier, changes and even new pieces of code are required when porting Linux to a new board. Basically, all board-dependent code must be modified or adjusted even if we use a Linux-supported processor. Changes are also needed when we have new requirements or any bug fixes. Most are concentrated in a few files, so hopefully this can help us conveniently catch up with new releases.

We used the experimental kernel version 2.1.132 PPC port for elinux. Almost all changes are limited to the board-dependent parts, that is, in the subdirectory linux/arch/ppc for our PowerPC board. Dozens of changes have been made; many for adapting to the new hardware, other things for bug fixes and new requirements such as new memory mapping.

Major changes for elinux include those for hardware initialization, PCI bus initialization, memory management, timer processing and interrupt processing.

Hardware initialization is the ugliest part in elinux. The loader iloader should do the most essential part of the job, such as initialization of the memory controller and PCI controller. However, our implementation of iloader simply ignores this part because we find it is unnecessary to do it again after the board's ROM code has done it. Of course, iloader has to do it if iloader is the first code to run from ROM. The kernel initialization does others such as memory protection and bus device initialization.

The original PPC port talks with PCI devices through interfaces with the BIOS. For our board, we assume there is no similar thing. We just leave the interfaces empty at the beginning. Whenever we add a new PCI device, we write code directly to set up the related base addresses, IRQs and access methods.

A few considerations are necessary for memory management. Although we don't need any changes in the major parts for memory management such as virtual memory management and paging, we do have modification requests. They are mainly for setting up the particular memory sizes and ranges, re-arranging the memory during the kernel startup, the use of PowerPC BAT (Block Address Translation) register pairs and memory mappings between physical and virtual addresses.

For timer processing, two things are modified. One is to adjust the parameters to set the PowerPC's decrementer to suit the board's bus rate. This decrementer is used to generate a timer interrupt every jiffy time (10 milliseconds). The other change is to provide the interface with direct access to the Real Time Clock (RTC) on board.

Another major change is for the interrupt controller. This controller is simple and controls only 16 IRQs through a status register, a mask register and a latch register. All the registers are 16 bits. Each bit corresponds to one IRQ. New simple code is added to handle it.

We have successfully relocated the kernel in the virtual space by redefining the symbol KERNELBASE as a Makefile macro. This involves a few changes in the kernel initialization code. Since we are able to relocate the kernel, we can reserve the space for some special purposes. For example, to load the kernel at the address 0xa0000000 instead of the default address 0xc0000000, we just define KERNELBASE in the top Makefile this way:

KERNELBASE = 0xa0000000

Minor changes are made in various Makefiles. These are necessary because we need new rules to create the elinux ball, we have a few new files to compile and link, and we found bugs in the Makefile when doing cross-compiling.

As for device drivers, we are concerned only with the serial driver for the console port. Other drivers may be added later if necessary, such as a LAN driver and drivers to control customized devices. During our experiments, we communicate with elinux using minicom over a serial port. We use the serial driver from the Linux code drivers/char/serial.c. A minor change is made for adjusting the baud rate, and another change is made in its header file since the serial port has a different IRQ number.

After all is done properly, we see through the console that elinux starts and runs happily.

ELINUX VERSION 0.001 March 1999
 Start booting Linux on Experiment Board ...
 ... (omitted long booting messages)
 #      (we start ash after the kernel is up)

As discussed previously, the elinux ball needs a statically linked kernel image and a root file image. Their preparation needs some skills.

The Linux source provides the rules to create various kernel images for different platforms. For our system, we need a compressed binary kernel image, vmlinux.bin.gz. The steps to create it are, in order, configuration, compiling and linking, transferring to binary format and compressing. In configuration, make sure the RAM disk support and the initial root file system support are selected, and that all unnecessary options are disabled. Compile and link the kernel for the ELF executable called vmlinux. Then transfer to binary format and compress it by commands such as:

(CROSS_COMPILE)objcopy -S -O binary vmlinux\
vmlinux.bin
gzip -vf9 vmlinux.bin

Because we choose to mount an initial root file system in RAM after the kernel is up, we have to prepare a root image, called ramdisk.gz, and put it into the elinux ball. We do this by creating an EXT2 file system in a 4MB RAM disk on the cross-development platform. Next, create the subdirectories such as /etc, /dev, /bin and /lib. Then, copy scripts, binaries, device nodes, etc. onto the RAM disk. Finally, compress the RAM disk image and get ramdisk.gz. For example, to create a RAM disk in /dev/ram1, type:

rdev -r /dev/ram1 4096

mke2fs -vm0 /dev/ram1 4096
mount /dev/ram1 tmp

mknod ttyS0 c 4 64

After everything in /tmp is ready, compress it by typing:

dd if=/dev/ram1 bs=1k count=4096 |
gzip -v9 > ramdisk.gz

What should be included in the initial root RAM disk will depend on our requirements. We copy a minimum number of shared libraries, plus some programs like bash and vi for tests.

Bugs are inevitable. “Given enough eyeballs, all bugs are shallow” is only partly true. Most often, we have to debug without outside help. Debugging may be painful, especially for system booting. There is a booting debugging tool supported by special hardware, but we don't want to rely on it. Other debugging mechanisms such as printk and gdb help in many cases, but they need too many system services. For example, the Linux kernel debugging support, printk, works only after the system is ready to write to a console or the file system. If the system crashes before that time, we get nothing from printk, even if printk uses a memory buffer to store information earlier in the process.

Simplicity means efficiency in this case. To help solve the problem, we add rprintf. It is a simple printing function using raw output, which writes characters directly to the console I/O port without any buffering and any other support. rprintf is like printf, but based only on this kind of raw output. It works very soon after the iloader runs, so it can be used to debug kloader and the Linux kernel as well. rprintf helps us solve most problems in the early stages of booting. We did have a few problems before rprintf is initialized, but we are not helpless. Our suggestion is to insert an operation to force a system reboot; in this way, you can locate the problem soon. We assume you know when the board starts rebooting. We provide a function called rreboot to do this job for our board by simply jumping to the system reboot entry point in ROM.

Unlike other projects, this work relies heavily on the Internet. We have learned much about Linux and obtained resources such as the kernel code and Linux-related documentation from the Web. To do something meaningful on Linux, follow proven patterns by utilizing open-source code as much as possible. Reuse pieces of code that run, even if changes are required. Don't be too ambitious in the beginning. Spend plenty of time on investigation before moving on to the next stage. Also, take the time to have a good design at the beginning and choose good debugging support. Always refer to Linux books and web sites first whenever you need help (see Resources). Readers can easily discover a huge amount of related information on the Web.

The same thing can be done in several ways. Our experiments are far from comprehensive, but we are confident in Linux's potential in some embedded systems. We hope our experience will help other people who want to port a Linux kernel to their embedded system. There are many related issues for us to research—much fun is ahead.

Resources