Radio E-Mail in West Africa
Editors' Note: The complete version of this article, was later published on the Linux Journal web site.
Deep inside the warm green interior of Guinea, centered in the frontal lobe of West Africa, international rescue workers in the widely scattered towns of Dabola, Kissidougou and Nzérékoré now enjoy regular internet e-mail, delivered straight to their own desktops. There isn't a telephone line or satellite dish in sight. Instead we are moving the mail over distances of hundreds of miles—over jungled mountains and high palmy savannahs—using high-frequency (HF) radio. Our project is called Radio E-mail, and here is its story.
The Republic of Guinea is a cashew-shaped nation on the Atlantic, ten degrees north of the equator in West Africa. It is a beautiful and resource-rich nation, about the size of Oregon. As far as African countries go, Guinea is a calm pocket of peace and stability and generally doesn't attract a lot of attention. But Guinea quietly has played a heroic role in the theater of world events in recent years, providing a safe and welcome refuge for as many as half a million people displaced by brutal wars and civil upheavals in the neighboring countries of Sierra Leone and Liberia.
The International Rescue Committee (IRC) has one of their largest operations in Guinea, providing services and support to a population of up to 200,000 refugees in many camps established throughout the country. I became involved with IRC when my wife accepted the position of country director for the program in the summer of 2001. Soon we were traveling on an inspection tour of the camps, making the long road trip to visit the program's three field offices “up-country”.
Traveling outside the capital city of Conakry, one immediately finds that Guinea has little infrastructure, especially in the way of electrical grid and telecommunication systems—to say nothing of broadband access to the Internet. So IRC field offices must provide their own infrastructure: diesel generators for electricity and HF radio sets to communicate with other offices and mobile units, which can be up to hundreds of miles apart.
Expecting this isolation and general lack of connectivity, I was quite astonished to find a radio operator using his equipment to make a binary file transfer from his desktop PC to another field office—wirelessly! On top of the operator's radio set, connected to the serial port of his PC, sat a dingy black box labeled “9002 HF Data Modem”. The operator used a proprietary, MS-DOS-based program to make his file transfers, but I immediately began wondering. If this device is moving binary data over the ether of radio, why couldn't we set it up with Linux and network with PPP connections as well?
Since IRC owned most of the equipment already and because we would be using Linux and other freely available software, the system could be implemented at negligible cost. I developed a design and specification for the system, and the project we call Radio E-mail has been continuously operational since January 2002.
If you have been making the move to wireless lately, most likely you are working with the microwave, high-bandwidth frequencies of 802.11b. If so, you know that on a clear day you maybe can get a line of sight connection out ten miles or so. HF radio is another animal. Its longer waves reflect off the ionosphere to follow the curvature of the earth, giving HF signals a range in the hundreds of miles. From Conakry to Nzérékoré (IRC Guinea's most distant field office), HF easily covers a straight-line distance of over 375 miles (600 kilometers).
So the great advantage of HF is it can go the distance, leaping the obstacles in its path with aplomb. Now for the bad news: where HF wins the wireless game in range, it loses its pants in data capacity. If 802.11b is considered broadband, think of HF as slim-to-none-band. The radio modems we are using here are spec'd at an anorexic 2,400 baud!
And wait, it gets worse. Two-way radio is the classic half-duplex medium of communication. That is, you are either transmitting (push to talk) or receiving, not both at the same time. This, plus the robust error-checking protocols implemented by the modem hardware, means the actual link experience is more on the order of 300 baud. Does anyone remember 300 baud? Unless you measure your patience with radio-carbon, your dreams of remote login sessions will be dashed and splattered.
However, for classic store-and-forward applications like text-based e-mail, the bandwidth limitation of HF radio is workable. We do need to pay close attention to our configuration and try to optimize as much as possible. With HF radio, every packet is precious.
Given these capabilities and limitations of HF, our design strategy for the project uses radio modems in a topology of field offices, as shown in the reference network of Figure 1. In Conakry we have a full-time internet gateway on the host coyah, and the radio modem on the host congo serves as the dial-in hub for each of the three field offices. We establish periodic PPP links between the field and Conakry and use our choice of carefully selected client/server protocols over TCP/IP. Although we might have implemented e-mail to the field using UUCP instead, our TCP/IP system is easier to configure and integrate with the existing network and e-mail system. This setup also is converted readily to use faster telecommunications linkups, such as land lines or satellites, if and when these become available in the field.
Figure 1. Radio Network Topology
For now, though, we don't have the luxury of dedicating our radios to full-time connections for data communications. In fact, voice communication continues to be the central purpose of the radio equipment. Our implementation and procedures must supplement this lifeline, not impair it. Because data sessions over the radio block voice calls at each end of the link, we have policies and configurations to hold connections to less than 15 minutes per session to keep the radios free for voice contact. In the field, radio operators have procedural protocols for making their periodic “dial-up” connections with Conakry for e-mail exchange at regular intervals throughout the day. Radio operators adjust the schedule and break sessions as necessary to accommodate urgent voice communications. For these reasons, all dial-ups are necessarily under the control of the radio operators, rather than set up with cron jobs or diald.
The equipment used in this project is made by Codan, an Australian manufacturer. Although there are other manufacturers, including Motorola, Kenwood and Yaesu, Codan seems to be the radio of choice for international aid organizations in this part of the world. Their big white Land Cruisers with official markings have substantial Codan whip antennas bolted conspicuously to every front bumper. The symbolic authoritative value of these thick black whips, such as when crossing the many military checkpoints, is certainly their most dominant feature.
The modem used in this project is their 9002 model. These modems are equipped with a basic Hayes-like AT command set, so they are easy to configure, operate and troubleshoot with any telecommunications application.
There are some significant differences between this modem and the family Sportster, however. For one thing, the Codan unit actually is built as DTE (data terminal equipment) rather than DCE (data communication equipment). To connect it to your serial port, you will need a DB-9 null modem cable, wired as diagrammed in David Lawler's Text-Terminal-HOWTO (www.tldp.org/HOWTO/Text-Terminal-HOWTO.html). Not all cables described as null modem are wired the same, so this detail is crucial. Also, the AT commands are not as extensive as, and vary slightly from, the standard command set. And, of course, this modem moves data more slowly than you can possibly imagine.
The Conakry radio host congo, aliased as radiohub, is configured as a PPP server, ready to accept dial-in connections over the radio from offices in the field. As with a conventional telephone dial-in configuration, we use mgetty to watch the serial line, initialize the modem, wait for incoming calls, answer the “ring” and fire off the PPP dæmon.
The 9002 works pretty much as mgetty expects. We start by setting the modem to a known state with the factory defaults (AT&F) and then get defensively redundant. We set all command, local and remote echo off (E0 L0 R0); ignore carrier (X0); use hardware flow control (&K3); auto-answer off (S0=0 & A=0); and reset the station address (&I=nnnn, where nnnn is like a phone number, the unique identifier that other radios call to reach this particular station).
After mgetty answers the “ring” and gets a “connect” back from the the other end, then what? This is controlled by mgetty's login.conf file. It is common in dial-in systems to let mgetty look for incoming PPP packets and then start the PPP dæmon automagically, typically using CHAP in the link negotiation process for authentication. Such a configuration is handled by the line starting with the magic string /AutoPPP/.
In our experience, though, the high latency of the radio link makes an /AutoPPP/ configuration slow to kick in. What we do instead will be shocking: we dispense with conventional authentication entirely! In our configuration, the login name provided by the chat script of the incoming connection is used to start the PPP dæmon directly. When mgetty matches a login name with an entry in the first field of the login.conf file, such as Pklogin, it then runs the program specified in the fourth field, such as /usr/local/sbin/pppd.login.kenya. In essence, then, the login name used by the remote system serves to control access. Note that Pklogin is bogus as a system account, and you can be sure I haven't told you the string we are really using! (Note also that we have an implicit system of human authentication even before a connection is made, when operators agree by voice to lock on a specific channel before starting the link.)
When mgetty gets a login name listed in the login.conf file, it then passes control to the corresponding start-up script, such as pppd.login.kenya. This in turn starts the PPP dæmon, using an options file for the particular remote host, such as options.kenya.
To spare yourself bitter trials of sweat, tears and other emotional excretions, and pay attention to the lcp-restart and ipcp-restart options. These parameters give the time, in seconds, that pppd will wait to receive a reply to that particular phase of PPP negotiation before trying again. The default value of these parameters is three seconds, which is generally more than adequate when using regular telecommunications.
Over the radio, though, if these restart defaults are not extended, you'll only snag yourself a useless snarling hair ball. Here's what happens. As pppd starts up, each peer begins a negotiation process with the other to agree on all parameters for the connection. During these initial discussions, when one end of the link doesn't hear back from the other within its restart interval, pppd repeats the transmission. In the meantime, though, the remote end has received the original transmission and sends back its reply. The local end gets this response back to its first transmission, thinking it has a proper response to its second, and proceeds to the next step of negotiation. But then the local server gets what is now the unexpected response back from its second transmission, and the negotiations break down in unresolved chaos.
By extending the lcp-restart and ipcp-restart parameters, you can delay sending out repeat transmissions for a sufficient amount of time for the peers at each end to receive a response. We have configured a generous 16-second delay and have not had any more problems.
Turning our attention up-country, each remote host in the field is configured to dial up the radio server in Conakry. After all our testing, failures and ordeals of travel, it was a happy and amazing day when we finally got our first link across radio waves spreading invisibly through the Guinean atmosphere. From kenya we actually made SSH connections with congo simply to express our exuberance over “talk” sessions with the radio operators watching their terminal screen in Conakry—“Greetings from Kissi!”
With the configuration details finally worked out, we found that the PPP link, although slow, would come up reliably and remain stable at all times of the day, even over channels that otherwise sounded fuzzy and had considerable static. Of course, all radios and antennas should be tuned for their best performance. But once you are able to establish a link, it is reassuring to find that the radio modems are quite capable of maintaining it, even when conditions are less than optimal. Because somehow, conditions always have a way of being less than optimal.
D. J. Bernstein, author of qmail, also has developed a number of special-purpose tools and applications perfectly suited to the Radio E-mail Project. Central among these, qmail includes a QMTP server, implementing Bernstein's own Quick Mail Transport Protocol. QMTP is a supplemental suite of programs designed for moving mail over slow connections.
As shown in the reference network in Figure 1, qmail runs on five hosts: the central mail server coyah, the radiohub server congo and each of the three field office hosts.
Once qmail considers the message delivered, we wait for the next PPP connection with the remote host kenya in Kissidougou. Then we can make use of the serialmail package to blast (relatively speaking) all the mail collected in the /var/qmail/qturn/kissidougou/.QMAIL.PPP/ mailbox across the link using QMTP.
Each of the radio e-mail servers in the field run headlessly, controlled from a simple command-line interface via Telnet session from the operator's desktop PC. The basic interface consists of four commands, usually run in the following sequence:
ppp.start mail.get mail.send ppp.stop
These commands are simple shell scripts that perform their respective tasks, each providing the operator with a modest amount of feedback about what is happening at the time. The functions could be further collected into a single command, such as mail.run, but we want to enable the operator to maintain some discretion over radio access, depending on the demands for voice communication.
We don't try to get and send mail simultaneously; we first do one, and then the other. This is another accommodation for the anemic, half-duplex bandwidth of the HF radio link. As far as network traffic goes, this link is like a one-lane back road—more than a little traffic creates a long skinny parking lot.
As we may have mentioned, the HF radio link is a tad on the slow side. Nevertheless, we manage to move a decent amount of mail with it. On an average day, over 300 messages travel the air waves between Conakry and field offices, over two to three brief connections per office. And as is typical with all internet technologies, every taste stimulates an even greater appetite.
Given the limitations inherent in radio e-mail, we try to maintain a usage policy that is as open as possible. For example, staff members are free to use radio e-mail for personal correspondence with friends and family anywhere in the world, and there is no limit to the number of messages any user may send. Our only explicit policy restriction is the request that users not subscribe to mailing lists.
To prevent the radio links from getting choked-up for hours on huge attachments, such as large documents and graphic files, all qmail servers connected to radios (that is, the radiohub in Conakry and each of the field office servers) are run with a message size limit of 8,000 characters. This is sufficient for three to four pages of text. Whatever can be squeezed into the 8,000-byte limit by way of attachment and file compression is free to go.
The system has proven extremely reliable. Despite the intermittent power outages typical in Conakry, we do try to keep the mailhub server coyah running at all times by using a generator and battery backup. So far, these measures have kept this machine serving flawlessly since it was first installed, with a continuous uptime at this writing of over three months without reboot.
Yet this reliability would mean nothing if the system were not sustainable over the long term. Two months before we installed the first radio server in the field, we formed a Network Users/UNIX Group (IRC-NU/UG) among interested and capable IRC staff. This group meets regularly and enthusiastically to learn Linux/UNIX and to develop network administration skills. The group now has a number of functional production systems on which to work and play, using mostly recycled hardware. The Linux servers installed for this project also host a typical range of other servers and services, including DHCP, DNS, NATD, Apache, FTP, Samba and PostgreSQL. The IRC-NU/UG provides a human network that will continue to sustain and grow the technical network in the years to come.
The successes of this project are readily duplicated anywhere in the world. Schools, government ministries and other organizations can build remote networking solutions easily over HF radio where access is otherwise not available and do it at a minimal cost. Once installed, these systems are almost trivial to administer and may be adapted quickly to alternative TCP/IP carriers. Maintenance of the e-mail system itself involves only the routine addition and deletion of user accounts and keeping the /etc/aliases files up-to-date.
We are now serving mail to over 50 desktops and 150 staff in four offices throughout Guinea. The entire wide area network is serviced behind a single public IP address, at a total ISP cost of $150 US per month. Best of all, the system has deployed standard network and internet technologies throughout the organization (and throughout Guinea) utilizing freely available technologies. Not only does this plant grassroots networking infrastructure where there is no Internet yet, it helps build the core competencies and capabilities essential for Africa's connected future.
