Late in 1999 we were developing embedded networking products (routers, terminal servers and remote access servers) with traditional proprietary technology. For a company that was a strong proponent of Linux for server-based networking and the first company in the world to support a Linux hardware product in 1994, this represented an internal contradiction.

We had been considering moving our embedded development to Linux for a couple of years. Our proprietary Cyclades Routing OS (CyROS, the Cyclades equivalent of Cisco IOS) has been developed, improved and optimized for more than ten years. It was not a trivial decision to give up CyROS in favor of an OS that is great for servers but might not have all the functionality we needed and be as good for embedded.

Linux runs well on ix86 and other standard platforms, but will it run on an embedded Flash-based PowerPC architecture? How long will it take to write the device drivers? How much memory is needed? Will it run fast in a small, embedded processor? How do I manage the fast software releases? Do I need real time? Are there tools and professional services available?

Early in 2000 we decided to find the answers to those questions. This article is about the development of an embedded networking box, the TS2000 terminal server, using embedded Linux. One year after that decision, the resulting product is a success and is being used in the field to manage some of the largest server farms in the Internet.

We have designed similar products using both proprietary and open-source software, so we can compare apples to apples. I hope this article will give you confidence to move forward and adopt Linux as your embedded operating system.

As in the server market, using Linux costs less than using a proprietary commercial kernel. Usually, there are no license/royalty fees, and you are not tied to a single vendor. Another advantage is the unlimited access to the source code, which is especially important for embedded developers.

That is not enough to convince the skeptical, though. There are relatively inexpensive commercial RTOSes out there. Some are royalty-free, and many of them give you access to source code. So, why would you change something that has been working fine?

We know that one of the main advantages of open source is the possibility of leveraging on the software available in the Open Source community. Most of this software is released under the GNU General Public License (GPL) and can be incorporated to commercial products without much overhead or cost. It becomes easy and quick to add new features and functionality.

We are in the embedded networking market, and it was difficult for us to develop and keep pace with the constant introduction of new network protocols. We were hoping that our internal R&D resources would outlast the availability of three-letter combinations to name new protocols. Then they started creating four-letter acronyms. We had to do something. Linux has great networking support, and our boxes are ready for current protocols and will quickly incorporate any future network protocol without us having to develop them all in house. We can spend more time designing better applications and products and less time maintaining kernels and protocol stacks.

Another great benefit is the possibility of letting users and external developers run their own software in your appliance. Let people scratch their own itch and find new applications for your product.

This flexibility is especially interesting for a terminal server. Many industrial and commercial automation applications require connection of multiple serial devices to the LAN. Users sometimes are forced to use PC servers with a multiport serial board to implement custom terminal server applications. Now they can use a compact and reliable embedded system and, at the same time, are free to add their own software and create customized solutions.

The same benefits apply to other types of embedded systems. Develop a PDA and maybe someone will add software and sell it as a Gameboy. Others will port automation software and incorporate it to commercial automation solution. Use of standard software architecture opens doors for new business.

We wanted to design a new console access and terminal server. This is a box that provides remote access to console ports in a server farm (for management purposes), provides dialup access through a modem bank in an ISP environment or can connect serial devices to the network in an industrial or commercial application.

The hardware specifications were 32 serial RS-232 ports, one Ethernet 10/100BT LAN port, PowerQUICC (Motorola MPC860T, PowerPC core), Flash/RAM-based (no hard disk storage), compact (1U of rack space) and low cost. When we started the project we did not know how much Flash and RAM memory were needed.

The PowerQUICC is an embedded networking chip by Motorola. It has two CPUs (a PowerPC core that runs the user application and an I/O RISC CPU) and several built-in peripherals (serial controllers, multichannel HDLC controllers, Ethernet controller, memory controller, etc.). It is the successor of the 68360 QUICC (similar functionality, but a 68000 core), the most popular processor for embedded networking ever.

The new product (called TS2000) would be an improved version (faster, smaller, less expensive and more reliable) of an existing product (PR3000) in hardware, but it would also bring a software innovation that was revolutionary rather then evolutionary. It would run embedded Linux and would be all open source.

TS2000 Bare Board

Using an embedded distribution saved us a lot of work. Hard Hat Linux 1.2 is based on Linux kernel 2.2.14 and had off-the-shelf support for the PowerQUICC (and many other platforms), with working device drivers for most of the on-chip peripherals.

With one engineer dedicated to the job full-time, the kernel was booting from Flash and running on our board in two weeks. By the second month, all interfaces were working and drivers debugged. By the fourth month, we had a functional terminal server. We had a product six months after the start of the project.

I know you have read marketing material about ``porting an RTOS in two weeks'', but we all know that, when it comes to the real world, shipping a product in six engineer-months is a very fast schedule, especially if we consider the adoption of an entirely new software platform.

That was really a good surprise to us. We expected a lot more problems than we had, and we were prepared to spend a lot more engineering time to get the product out the door.

In the proprietary software world, you can choose to design everything in-house from scratch or get a commercial RTOS kernel (VxWorks, Nucleus, PSOS, Lynx, QNX, etc.). Nowadays, almost all embedded developers choose to use a commercial product and get kernel support from specialized companies so they can focus on the application.

The same is valid in the Linux world. There are companies specializing in developing, supporting and providing professional services for embedded developers using Linux. Lineo, LynuxWorks, MontaVista and Red Hat are a few examples.

If your core business is not kernel development, I strongly recommend you use an embedded distribution and work with one of those companies. It certainly reduces your time-to-market. The cost involved is lower than in the proprietary RTOS world, and it is much lower than if you had to engineer everything from scratch.

Linux is very scalable, but it was not originally designed for embedded. So, there is a lot of work and value added in an embedded Linux distribution.

We selected MontaVista Hard Hat Linux. They had MPC860T support off-the-shelf (including board support packages for the most popular evaluation boards). The Hard Hat distribution is 100% open source (some vendors have proprietary tools and add-ons) and, other than the minimum necessary to accommodate new hardware, the kernel is standard Linux.

Lineo was focusing more on ix86-centered, Linux-based appliances than on true embedded systems. LynuxWorks was still to announce their BlueCat Linux product when we did the research. Red Hat had ECOS as a side effect of acquiring Cignus, but not real Linux nor a focus on embedded. Things are changing fast in this market, and our decision is over one year old, so you should talk to all vendors before picking the one that works best for you.

We had never done embedded Linux before, but we had both Linux kernel developers and embedded designers in-house (some people had already done both separately), so putting things together was relatively simple. With that background, we purchased a regular technical support package from MontaVista and decided to tackle the challenge based only on in-house experience. The embedded Linux vendors offer special training and porting services if you need or want external help.

Our cross-development platform is a regular ix86-based PC with Red Hat Linux. The development environment is GNU (GCC Compiler).

Bringing up new embedded hardware is always challenging and there were not many debug tools available (in-circuit emulators, target monitors, hardware monitors) with specific support for Linux. As embedded Linux becomes dominant in new embedded designs, the vendors are addressing this, but you should be aware that this is a relatively new market and not all the tools available for VxWorks or PSOS, for example, are available for Linux.

Our strategy was to validate the hardware with our existing proprietary software before trying to boot Linux in it. With the hardware validated, it took about one week to understand how to generate a Linux image bootable from Flash and get it to run in our hardware. In a few weeks, it was actually doing stuff.

We used HMI and SDS hardware monitors that work with the PowerQUICC BDM debug port to verify that the hardware was really working. But we did that in our old development environment (Windows development host, DiabData C compiler). After switching to Linux, we did not have hardware-based single step or source code-based debugging. Debugging of the kernel was done analytically and with printf and homemade software-based debug tools. KDB and GDB can be used to debug the kernel and user-space applications, but we did not use them.

If the developers are not familiar with GPL, the cultural shock of transitioning from proprietary software to open source is significant. If you are new to open source, spend some time trying to understand how it works.

In traditional embedded development, reading the hardware data sheet is usually the first step before writing a device driver. With open source, the first step is looking for the drivers already available somewhere. In most cases, you will find the exact or at least a similar driver to start from.

After modifying the driver for your specific hardware, make it available to other people. Besides being fair (since it was based on software developed by the community) and part of accepting the GPL, this benefits mostly you. If your modified driver is good and useful to other people, you will see it being improved and maintained by the community so that you can focus on your application.

In our case, the drivers for the peripherals in the PowerQUICC were already supported by the distribution. The ramdisk drivers necessary to run without a hard disk were also included. Other than minor problems and I/O mapping adjustments, those drivers worked fine.

To support the RS-232 serial ports, our product used 16654 quad-UART chips. As mentioned previously, device drivers for almost any chip have been written already and are available. After a couple of days working with the standard Linux serial driver (which supports UARTs from the 16x5x family), we could send and receive data and see the serial ports as /dev/tty in our system.

We wrote a custom Flash driver so that applications could simply open a file and write to it to save data in Flash.

Device driver development is usually a large portion of traditional embedded design. I'm sorry if three paragraphs seem too little to cover it, but that is really how easy it was.

Once the hardware is debugged and the kernel is running, it is time to port or write the application that will provide the functionality expected from the embedded system. Here lies another major difference between a traditional embedded system and a Linux-based design.

In a traditional design, the application would be implemented as a task tightly integrated to the kernel and managing resources using low-level kernel calls.

Of course you can do the same in Linux, but it is much easier and faster to develop the application as you would in a desktop environment. The application can run in user space, taking full advantage of an execution environment. It can open files and manage resources using high-level Linux system calls.

Someone without any specific knowledge of your hardware or software environment can design, implement and debug a new feature or application using a regular Linux box. It will run the same way in the embedded system. In many cases, new features don't even require C programming, just some shell scripting.

To create the functionality needed by a console access and terminal server, instead of creating specialized tasks running in kernel space, we grabbed a few user-space packages already available and modified/merged/customized them.

Size matters in embedded, and many open-source projects focus on making Linux possible in small hardware. In the networking arena, work from the Linux Router Project (http://www.linuxrouter.org/) is extremely useful, and we have much to thank them for. Besides contributing your code, you may want to become a sponsor of open-source projects related to your area.

The problem with the embedded market is that each project is different. It is difficult to answer questions such as ``how much memory do I need?'' based on someone else's experience. While we cannot give you the answer for your project, we have done it both ways, and we will give you parameters for comparison (see Table 1).

Table 1. PR3000/TS vs. TS2000

When we migrated to an architecture that uses standard high-level APIs and provides a protected execution environment for applications, we expected to need more memory and Flash. The results we got demonstrated that, but the cost of RAM and Flash use and performance is low compared to the gains in flexibility and functionality.

In addition to the penalty paid for a well-defined application environment and interfaces, a user-space application takes spaces in ramdisk and, when it is loaded, takes another portion of memory.

Our system has a kernel configured with the necessary drivers and a complete TCP/IP protocol stack. In user space, we have bash (the shell), vi (the text editor, used for configuration), BusyBox (the small footprint UNIX/Linux utilities replacement) and all basic dæmons necessary for the system to work properly.

This complete Linux system with full routing and networking capabilities runs in as little as 6MB of RAM. The kernel itself takes about 2MB; a minimum ramdisk with the user-space applications and files/utilities necessary to have fully functional Linux takes about 3MB; and you still need some buffer space for the kernel to run and load the user-space applications.

Because we were designing a terminal server, we also ran pppd (the PPP dæmon) and modified/customized versions of Portslave, Sredird, TinyLogin and OpenSSH to control the serial ports and terminal server and TCP socket functionality.

This requires about 300k of additional buffer space per serial port, which adds up to a lot for 32 ports. With some additional work, we could drastically reduce this requirement, but we preferred to pay for a little more RAM and use the packages with minimum changes to facilitate maintenance and future updates.

It is difficult to say how much memory/Flash is for data and code and compare it with a traditional embedded design. Most of the functionality that was code in the traditional design is now implemented as binary files stored in the Flash filesystem to be executed in user space.

The kernel image, the basic filesystem, configuration, user files and init/boot code are stored in Flash using data compression (about 2.5:1 compression ratio). The box has 4MB of Flash, but we are currently using between 2 and 2.5MB.

The two CPUs inside the PowerQUICC run at 50MHz, which is fast for an embedded system, but much less than a typical Pentium server. While the CPU usage is higher than with our previous proprietary OS (as expected Linux is less efficient than our tightly integrated and optimized proprietary design), the box is capable of driving all serial ports without performance bottlenecks. For our specific application, we see an increase in CPU usage (if we had a CPU bottleneck, it would translate into a decrease in performance) between 20-30%.

TS2000 Blue Box

Our vision of the future of the networking market is that more and more equipment manufacturers will have to leverage on commodity technologies both in hardware and in software to remain competitive.

With commodity hardware and open-source software available to all, the model where each company designs and maintains its own internetworking OS and TCP/IP stack and proprietary motherboards from scratch won't work in the long run. Vendors need to leverage on modules and components that benefit from volumes and economy of scale generated by the consumer market and on open-source software. The engineering time and R&D resources should be invested in each one's core, value-added technology. Energy should be geared toward innovation, not toward creating new versions of what already exists.

That is the very core of the open-source philosophy. It makes sense in the server/general-purpose applications, but it makes even more sense in the embedded market. This is just the beginning of the revolution that is making Linux the standard embedded platform for the future.

We hope this article gives you the incentive to consider moving your embedded design to Linux. This is not yet a trivial decision at first, but the benefits certainly and quickly offset the initial challenge.

Cyclades and Linux

Marcio Saito (left) and Edson Seabra (right)