Instructor Ed Skinner

MontaVista Software's training classes are available only to customers, and that makes us curious. What secret arts are revealed to these select few? So I asked nicely and got into the four-day "System Development Jump Start Using Hard Hat Linux'' class.

I signed a nondisclosure agreement before I went to the class, to protect the identities of the customers, along with any company secrets they might let slip. All the other students were discreet, though, so nothing has been nondisclosed.

Instructor Ed Skinner, who has an XYZZY license plate and years of experience training developers in pSOS and VxWorks, gave a brief "what is Linux'' introduction, then jumped right into finding your way around Hard Hat's NFS-based development environment. So by the time class was over, I was pretty comfortable with Hard Hat's layout and way of doing things.

Throughout the class, Ed alternated between general principles and specific practice. The first day he introduced the two main embedded systems development strategies, "all-at-once'', where you build a whole image, download it to the target and test, and "incremental'', where you get a few things working at a time. He did point out that the incremental approach requires a reconfiguration step, where you get the target running independently of the host. But despite this, incremental is faster.

Hard Hat Linux, like other current embedded OSes, is definitely the latter and seems to take the incremental approach to its logical conclusion, with what amounts to a complete Linux distribution, including gcc, available on the target during development.

Lab sessions started with an easy task: connecting the host and target, and running the Hard Hat install script. Ed chose an x86 laptop as the host and a PowerPC SBC as the target, so it was easy to tell if you compiled for the wrong architecture. Each pair of students got one host and one target.

We compiled a kernel for the target, which was a task straight out of regular Linux, except that we turned the final image into S records and transferred them via TFTP to the target, instead of doing a LILO install. The image included a boot loader, but we didn't get to the process of putting anything except the basics into Flash. This class is all in NFS-land. We booted the target and logged in. My lab partner and I had "hello world'' running on the target at 3:27 P.M. on day one.

The class is billed as requiring proficiency in C and in a Linux or similar programming environment, but it really only seemed to require the fundamentals of C and a little bit about make. If you're used to integrated development environments, you might want to read the GNU information pages for make and write a couple of practice Makefiles before class.

The labs gave us quite a bit of supervised practice with GNU development tools, including a cross-compiler gcc and client/server gdb for debugging over the network. And all in all, the amount of material covered was modest for a four-day class. There was plenty of time for questions and for Ed to help people during lab time. Everyone's programs ran. If you've never written anything for any kind of Linux, this would be a good way to get started.

One day was devoted to pthreads and their performance advantages, POSIX real-time priorities, semaphores and mutexes. The lab was a context-switch competition, in which pairs of students posted counts for how fast their programs could switch between two threads alternately reading and writing a block of memory.

Ed particularly was impressed with Source Navigator, a tool for browsing and searching source code, and later used it for navigating the kernel. Speaking of the kernel, he covered kernel internals quite a lot for an introductory class, including showing how to interpret /proc/*/maps to see processes' memory maps.

The final exercise was to write a kernel module that basically could be insmod-ed and rmmod-ed, and that had a /proc interface, and that's it. Drivers are the next class.

Some practical knowledge we got in this class might mess you up for a week if you didn't know it. "For embedded systems, always change panic.c'', Ed said. The kernel default is intended for systems with a console, where the operator might be able to do something to deal with the panic. In an embedded system, you might want to reboot right away, or shut down entirely and not restart. Panic your way.

Another file in the kernel tree that might not be appropriate is oom_kill.c, the out-of-memory killer. On an embedded system, the user might not be able to restart a killed memory-hog process, or it might be a really bad idea to kill a certain process. The oom_kill.c rules for determining which process to kill are pretty simple, and a custom oom_kill.c will handle the realities of your system better than the generic stock one.

Add printk()s to figure out what's happening in the kernel. Recompiling and downloading to the target is really quick if you just change source code and not configuration. And finally, "Develop in baby steps''--the incremental approach is best.

We only ran our programs from the NFS-mounted target directory on the host. There was no opportunity to get to the reconfiguration step, load a complete image into Flash and see if we could make the lights blink without the host connected.

Although many developers might spend almost all their working time with all the comforts of home via NFS, at some point you have to test without it. Currently MontaVista doesn't have a class on the reconfiguration step, but it might be the subject of a future class, Ed said.

I got a lot out of this class and would recommend it, except to practicing embedded Linux wizards who have already skipped past this article anyway.