Managing large numbers of Linux and UNIX systems takes a lot of organization, automation and careful use of technology. A significant chunk of one's time as a system administrator is spent building infrastructure to make managing those systems easier. Doing so improves flexibility, recoverability and reduces downtime. All of this hopefully results in less stress and longer vacations. This article discusses one of those simple technologies that helps accomplish all of the above, serial consoles.

Figure 1. Managing Many Servers from a Console Server

Serial consoles always have been a standard feature of enterprise UNIX hardware. Modern high-density server and cluster configurations sometimes can squeeze more than 50 servers in a 19" rack, so having monitors and keyboards for each one is unthinkable. Although KVM (keyboard/video/mouse) switches can connect many servers to a small number of monitors and keyboards, they are expensive and even more so with remote access features. Serial consoles allow you to take racks or shelves of servers and have all their consoles available all the time, from anywhere.

The console is a simple I/O device, initialized early in the kernel boot process to convey informational messages as the system comes up. Once the operating system starts running startup scripts, the console can be used to recover an ailing system or to get input from the system administrator interactively, like Red Hat's Kudzu does. One compelling feature of serial consoles is never having to drive in to work because a system is hung up at reboot asking for input for fsck. After the system is up completely, the console usually becomes a login terminal, sometimes a graphical one. The console also can be used as a last resort method of reporting problems inside the kernel. Under these panic conditions it is not possible always to write to log files or network log servers, so messages are reported to the console. For this reason and many more, consoles on servers should be a simple device, and a serial port is the simplest device included on standard system configurations. For those last-minute panic messages, one could add a console device that supports buffering and logging, so you never miss a moment of the excitement.

What we are talking about here is booting up a system without using an attached keyboard, mouse or video monitor. Some motherboards may complain without a directly attached keyboard, but this requirement usually can be changed in the BIOS configuration. In fact, with the recent popularity of USB keyboards, most BIOS versions do not care about missing keyboards. If you are using a system that was designed to be a server, you may be even more fortunate. Several vendors have started adding extra functionality to their BIOS versions to better support serial consoles from power-on. These features sometimes include power-on system test (POST) output and BIOS configuration access over the serial port. Depending on your needs, you can select your hardware accordingly by checking the vendor specifications. Even without BIOS support, you still can use serial consoles quite effectively on almost any PC system.

This is not a perfect solution, though, and your average PC hardware does not provide all of the features available in typical enterprise-class UNIX hardware. PC BIOS versions do not have the concept of a boot monitor (see the “What Is a Boot Monitor?” sidebar), nor can you perform a hardware halt of the OS as you can in enterprise UNIX hardware. For many applications this is okay, but when more functionality is needed add-on hardware options are available, and I discuss them later.

Most PC hardware BIOS versions can be configured only with a directly attached keyboard and video monitor. Luckily most come with usable default settings, so this is not normally an issue. If it is, you may need to have the system initially configured with direct attached video and keyboard and then switch to serial console. In my experience, I have rarely needed to do this; it needs to be done only once during initial hardware setup.

As packaged by most distributions, the Linux kernel and bootloader select the directly attached video controller and keyboard as console, but this is easily changed. When a PC-based system boots, the bootloader is the first program to be loaded off the disk. The three major bootloaders in popular use on Linux systems are GRUB, LILO and SYSLINUX (used on boot floppies); all of them support serial consoles. Next, the Linux kernel needs to be told to use a serial port for its console, which can be handled at compile time or by passing kernel command-line options from the bootloader configuration. Finally, if you want to be able to log in on the console, you need to configure a getty process to run after the system is up.

We discuss the kernel configuration next because it is a prerequisite to understanding the bootloader config later on. The most flexible way to configure the kernel console is with the options passed on the kernel command line. You can append arguments to the command line from the bootloader. Here is an example of the kernel command-line syntax:

console=ttyS0,9600n8

This tells the kernel to use ttyS0 (the first serial port discovered by the kernel), running at 9,600 baud, no parity and 8 bits. The kernel defaults to one stop bit. This is the most common speed and configuration for a serial console, which is why most serial ports and terminals default to 9600n8. It is possible to append more than one console= argument to the command line; kernel messages then are output to all of them, but only the last one is used for input.

GRUB is a flexible bootloader with excellent support for serial consoles. When properly configured, GRUB allows multiple devices to be used as a console. Listing 1 shows an example grub.conf file (usually /boot/grub/grub.conf and symlinked to /etc/grub.conf) as configured by the Red Hat/Fedora Core installer. Yours may be slightly different.

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub
# after making changes to this file
# NOTICE:You have a /boot partition.
#        This means that all kernel
#        and initrd paths are relative
#        to /boot/, eg.
#        root (hd0,1)
#        kernel /vmlinuz-version ro root=/dev/hda6
#        initrd /initrd-version.img
#boot=/dev/hda
default=0
timeout=10
splashimage=(hd0,1)/grub/splash.xpm.gz
title Red Hat Linux (2.4.20-8)
        root (hd0,1)
        kernel /vmlinuz-2.4.20-8 ro root=LABEL=/
        initrd /initrd-2.4.20-8.img

The first thing to do is remove all splashimage directives. In some early versions, these directives confuse GRUB and make it default to the video console. Then add a serial and terminal line. The serial line initializes the serial port to the proper baud and settings. In the terminal line, we configure GRUB to send prompts to both the serial port and to the keyboard and monitor. You can press any key on either, and it becomes the default console. The --timeout=10 argument tells GRUB to default to the first device listed in the terminal line after ten seconds. We also modified the kernel command line to include the option that tells the Linux kernel to use the serial port as console. Listing 2 shows the complete modified grub.conf file.

The LILO bootloader, although much more mature than GRUB, is less feature-rich. We must configure LILO and pass options to the kernel to use a serial port. To do this, we add:

serial=<port>,<bps><parity><bits>

where port 0 is the first serial port detected by LILO. Also, the append= line is modified to include the kernel options. After modifying the /etc/lilo.conf file, be sure to run LILO to update the bootloader. The completed lilo.conf file is shown in Listing 3.

SYSLINUX is a bootloader designed for use with DOS/FAT formatted bootable floppies. Red Hat/Fedora Core Linux uses SYSLINUX for both the install boot and rescue floppies. In order to install or recover from boot floppies over serial console, the floppies need to be modified. We have added the console= and text directive to the append line, and we have removed the extra boot selections present in Red Hat's original file. The first line initializes and directs SYSLINUX to use serial port 0 (aka /dev/ttyS0) and defaults to 9600n8. Using this modified boot floppy, we can install the OS over the serial console. Red Hat's text installation option works quite nicely this way. Using the above modifications, you can convert any SYSLINUX boot floppy to use serial consoles. This procedure also works for ISOLINUX, which is a spinoff of SYSLINUX used on bootable CD-ROMs.

As stated before, the console can become a login terminal after the system is up. For this to happen, the getty entries in /etc/inittab must be modified. The standard /etc/inittab starts mingetty on virtual consoles only. Because mingetty is not suitable for serial terminals, we must use something else. Many getty-type programs are available, but agetty is included with almost every Linux distribution, so we use it. Also, make sure the system boots to nongraphical mode, normally runlevel 3. Some Linux distributions default to an X login, usually at runlevel 5, if any X packages were installed. The default runlevel is determined on the initdefault line. To enable agetty on serial lines, you can modify the initdefault line in /etc/inittab:

id:3:initdefault:

and add a line for agetty:

co:2345:respawn:/sbin/agetty ttyS0 9600 vt100

This tells agetty to start waiting for logins on /dev/ttyS0 at 9,600bps, using vt100 terminal emulation. You may want to keep the original mingetty entries to allow a directly attached keyboard and monitor to be used for logins. If not, simply comment them out. Where root can log in from is controlled strictly; in order for root to log in from ttyS0, you must add the device to the /etc/securetty file.

Finally, if your system has created a /etc/ioctl.save file, delete or rename it. This file is used to save console settings between reboots. If the system was booted using a directly attached keyboard and monitor, this file attempts to restore improper settings. A new one is created when you reboot using the serial console.

Red Hat's bootup scripts use escape sequences, so the OK, PASS and FAIL messages show up in color. This can confuse serial consoles, so it is best to disable it. Simply modify /etc/sysconfig/init, and change the BOOTUP= line to say BOOTUP=serial. This will prevent the use of color messages.

Serial cabling can cause some confusion. Basically, there are two kinds of serial ports, DCE (Data Communication Equipment) and DTE (Data Terminal Equipment). The ports differ in how specific signals are connected to pins on the connector. Data communication with serial ports uses separate transmit and receive wires, so when connecting two pieces of equipment together, one must make sure the transmit wire on one side connects to the receive wire on the other side. As long as you are connecting a DCE device to a DTE device you can use a regular straight-through cable, where each pin is connected to the same pin on the other side of the cable. If you are connecting devices of the same type, however, you must use a special cable or adapter, called a null modem, so the signals are swapped properly. DTE devices usually are terminals, computers and printers. DCE devices are designed to connect directly to computers, such as modems and serial mice.

In addition to the data transmit and receive wires, a number of handshaking signals are used to control the flow of data, so one side is not talking too fast for the other to understand. These signals also must be swapped by the null modem. To add to the confusion, two popular connectors are in use for serial ports, the 9-pin DB9 and the 25-pin DB25. These can come in both male and female varieties. In almost every case, the devices used for serial consoles (terminals, computers and console servers) are all DTE, which means you need a null modem of some sort. These are available in the form of adapters and cables. Most off-the-shelf units work fine, but if you want to solder your own, check the on-line Resources section for links to pinouts and cable diagrams.

At this point, we have described a Linux system that can boot up without a directly attached keyboard and monitor. It uses the first serial port for all informational messages as the system boots and accepts logins from that console once the system is up. But to what should you connect that console port? There is a world of possibilities. If you have no particular need for remote-console access, you simply can leave the port unconnected until you need to maintain the system. You can use a computer or laptop connected over a null modem with the minicom program to access your system's console. Simply configure minicom to speak to an unused serial port, set the speed to 9,600 baud, 8 bits, no parity and 1-stop bit (aka 9600-8n1). Cable the systems together, then watch the system boot and eventually ask you to log in.

For remote access to a server's console, you can set up a console concentrator, which is a lot like a terminal server. It can be a homegrown Linux box with multiport serial cards, giving you as many ports as you have servers. With this kind of setup, you can access all your servers' consoles by logging in to a single dedicated Linux box.

If you like the idea of remote access to your consoles but want more of an appliance, a number of products can help. Cyclades (www.cyclades.com) makes a console concentrator called AlterPath; it is reasonably priced and comes in 1, 4, 8, 16, 32 and 48-port models. The AlterPath units run Linux internally from Flash memory. A Web interface is used for configuration, or you can modify the configuration files directly through a shell login.

The most flexible way to configure the Cyclades unit is to present the consoles using Cyclades' modified SSH dæmon. This way you can SSH directly to each connected server's console port, which is identified by a textual name you choose. So, to connect to a server identified as server hooked to a Cyclades unit with a hostname of cyclades as the user matt, the command would look like: ssh matt:server@cyclades. (The colon syntax is a Cyclades modification to sshd, allowing you to pass a port name.) This setup is easy to use, and you even can set up SSH private key authentication.

Other vendors make console concentrators or servers, including Digi (www.digi.com), Equinox (www.equinox.com) and Raritan (www.raritan.com). All of these vendors offer network-attached serial console products.

As mentioned earlier, serial consoles on standard PC hardware lack some of the features available on enterprise UNIX hardware. One solution is PC Weasel (www.realweasel.com), which comes in the form of a PCI or ISA card. This device emulates a video card and translates all output to the serial port as normal terminal escape sequences. Input from the serial port is translated into PC keyboard scan codes. Because it looks like a video card to the system, the system allows it full access to BIOS and POST. Additional features allow you to do a remote hard reset. The PC Weasel also has its own processor, so it is available even if the host into which it is plugged crashes.

If you would like to build your own console concentrator, some options are available to make it a little better than a simple box with a lot of serial ports. Conserver (www.conserver.com) is an open-source software package for managing systems connected to serial consoles. It supports SSL encryption and is highly configurable.

Resources for this article: /article/7507.