Linux in Embedded Industrial Applications: A Case Study
A big turbocompressor in an oil extraction plant is controlled by a 15-year-old Digital Control System (DCS), which provides all the supervision functions, plus the logging of significant events and data onto a printer. The system has been running satisfactorily for years, but now the management would like to integrate the machine into the global plant SCADA system, where data from various machines are gathered and processed in order to provide integrated management of the entire plant.
The PLC-based architecture of the existing DCS is hardly expandable and doesn't allow, from either the hardware or software point of view, for the addition of the data transmission functions which would be necessary for the project. A redesign of the DCS system to add the required functionalities was considered an excessively high-impact solution for the purpose.
The solution proposed was thus to add to the existing DCS a sort of “protocol converter gateway”, i.e., a box which captures the printer data stream, analyzes the lines of text to extract data values and makes them available to the SCADA (see sidebar) system. The latter will periodically issue data queries to receive the data values from the gateway. For this part of the connection, the standard protocol Modbus RTU was selected because of its simplicity and because it was already widely used throughout the plant.
From the hardware point of view, what is needed is a box with two serial ports: one to receive the printer data stream from the DCS and the other to receive data queries from the SCADA system. The box must run some program which decodes the printer data stream and replies to the SCADA queries.
The main challenge in this kind of application is represented by the particular environment: the plant is located in a geographically remote area and the site has all the problems which are usually found in industrial environments—dust, electrical noise, wide ambient temperature ranges, and so on. Industrial PCs are easily available on the market, but one must take into account a few peculiarities which are usually not found in other PCs:
There is no fan on the CPU. Spinning parts are the weak points in a PC and the miniaturized CPU fan is particularly unreliable with time. The main fan cannot be easily avoided, and is accompanied by suitable filters. We can do without the CPU fan, provided that the clock speed is reduced to limit the heat dissipation needs.
There are no disks. The hard disk is substituted by a solid state “Flash EPROM disk”. The floppy disk drive may be included, but it will be used only for software uploading. This means that the available disk space is very limited, usually not more than 64MB, but in our case, just 8MB.
Standard RS-232 lines are not suitable for serial data transmission in “noisy” environments. The more reliable RS-485 standard is usually adopted.
The PC will usually run without keyboard or display. They can be reconnected for maintenance needs.
As a result of some of the above constraints and due to working conditions which are typical of embedded applications, the software must also meet a number of requirements:
It must work with a small disk space.
It must require limited CPU power.
It must boot with no need of human intervention.
It must restart smoothly after power off, both programmed and due to failures.
It must cope with untrained personnel.
Why not a real-time O.S.? As it should be clear by the description of the problem given, we are not challenged by hard, real-time requirements. The limited speed of the two I/O channels puts an upper limit to the data throughput and no problem should arise from unpredictable delays in servicing the two serial lines. This makes it possible to use a comfortable standard O.S. and rules out any special purpose real-time operating system as a cost-effective solution.
Why not DOS? The problem at hand is essentially asynchronous in nature. The system must be able both to swallow the data stream as produced by the DCS printer output and to reply to queries from the SCADA system, two process which are not synchronized in any way. DOS is thus clearly not a viable solution. Although it has the advantage of being very lightweight and having a small footprint, it would require quite a lot of low-level programming to solve the problem.
Why not Windows 95/98/NT? Microsoft Windows (especially Windows NT) is gaining consideration in industrial applications due to the great number of third-party packages and tools suited to any need, and because it is backed by a major vendor. However, the hardware characteristics of the target computer makes it challenging, if not impossible, to tailor a version of any Windows brand to the limited file space and CPU power available. Windows CE was not stable or reliable enough to be worth consideration. Moreover, the peculiar (weak) multitasking capabilities of Windows would likely make the programming of concurrent applications slightly more difficult.
Why Linux? Managers are usually worried when they have to make choices which are not “industrially sound”. They will have no problems in justifying the choice of a costly product provided that it is backed by a major vendor and has a large installed base. This is not (yet) the case of Linux, so to foster this kind of choice the technical arguments must be particularly strong. In this case, I believe that the advantages provided by Linux are superior to many objections.
The choice of Linux has many pros:
Linux can be tailored to any particular set of needs down to the level of kernel configuration.
Documentation on how to tailor the system is thorough and easily available.
Linux can run from a RAM disk, so the Flash EPROM disk is used only to bootstrap. In this way the disk can even be write-protected.
Development may be made in the same environment where the target system will run.
It also obviously has some cons:
Finding drivers for non-standard devices may be difficult or even impossible.
Support from vendors may be more difficult.
In this project we needed two non-standard devices, i.e., the Flash EPROM disk and the RS-485 serial interface. This didn't cause any problem: the Flash EPROM disk was used as a standard DOS formatted device and the serial interface replaces the COM ports and is managed by the standard serial driver.
The protocol converter gateway software has a very simple structure. It consists of two independent processes: a data swallower and a Modbus server. The data swallower receives printing lines from the serial port, analyzes the strings, extracts relevant data and writes the values into a shared memory buffer. The Modbus server receives queries from the SCADA system according to the Modbus protocol specification and sends back the requested values, reading them from the shared memory buffer.
Two interactive programs have been added for debugging purposes: a memory dumper which prints out values read from the shared memory buffer, and a program to write values into the shared memory. Both programs can, obviously, run concurrently with the gateway processes.
All the code is written in C and the shared memory management was implemented by using the standard System V interprocess communication API which allows the creation and management of shared memory segments and provides semaphores for synchronizing the access to them.
Due to the very simple structure of the problem, synchronization was easily implemented by locking access to the entire shared buffer when doing a memory access. This simple-minded approach is quite suitable in this case because all the accesses to memory are performed in chunks and the low speed of the I/O operations, with respect to memory accesses, ensures that any process will wait for a memory lock release for relatively short times.
As a tool for the constant verification of the proper functioning of the gateway, a location of the shared-memory buffer has been reserved for a counter which is incremented anytime the data swallower process ends a reading cycle. The SCADA system periodically reads the variable and may thus raise an alarm if the variable is not incremented after a given period of time.
Following directions found in the Linux BootDisk-HOWTO, a small Linux system was built, starting from a standard Red Hat 6.1 installation. This is actually a trial-and-error process in that one has to find out exactly which files are needed for one's purposes.
Even though our Flash EPROM disk provided a comfortable 8MB of disk space, all the software must be transferred onto the target computer by floppy disks, thus it is desirable that the resulting system is as small as possible.
The tailored system includes the kernel, a number of standard Linux commands (we were fairly prodigal in adding commands: better to have all the needed tools at hand when doing maintenance in the future) and all the related libraries. It also includes the loadable modules which are needed to manage DOS format volumes. This may be useful to mount and access either DOS format floppy disks or the Flash EPROM disk.
Needless to say, the four programs developed for the gateway, plus a few ASCII configuration files used at startup by the two running processes, were also included.
Due to the Linux bootstrap procedure requirements, the above components were stored into two files: the compressed kernel image (450KB) and the compressed root image (2500KB). Just a tiny bit more than what would fit onto two floppy disks: we actually needed three floppies for the full distribution. In the BootDisk-HOWTO one can find a number of hints related to shrinking the size of the root image, but we felt satisfied with the size reached and did not want to work more on this aspect.
The Flash EPROM disk selected for the project (M-Systems DiskOnChip) is provided with a Linux driver and can be used as a Linux bootstrap disk. This can be done by including the DiskOnChip driver into the kernel but also requires some fiddling with a DiskOnChip configuration utility and a special version of LILO to make it bootstrappable.
After a few tests we preferred a different solution: DiskOnChip was configured as a plain DOS bootstrappable disk. This has the advantage of avoiding both rebuilding the Linux kernel and reconfiguring DiskOnChip (it is the device's shipping configuration) and moreover it was considered a more stable solution with respect to future releases of the device. The Linux image files are stored on the DOS file system and Linux is booted by the LOADLIN utility. This adds around 160KB of DOS files to the software.
The system's power-on sequence is thus:
Boot DOS.
Run LOADLIN from AUTOEXEC.BAT to boot Linux. The boot sequence creates the RAM disk containing the Linux file system and expands into it the compressed root image.
Start the protocol converter processes.
The protocol converter processes are started at boot time because they are inserted into the inittab table. This also provides automatic restart in case of a crash of any of the two processes.
After booting, if the keyboard and display are connected, the usual Linux login prompt is displayed and a root login can be done. This allows us to perform maintenance operations and notably to launch Linux commands or use either of the two interactive monitoring programs described above. If needed (e.g., in order to make modifications to the configuration files without going through the entire process of making a new root image), the Flash EPROM disk can be mounted as a DOS volume.
We have shown how the use of Linux can solve a typical problem of data acquisition in an industrial environment. Actually, we were able to build a system which can do something useful with 32MB of RAM and as little as 8MB of disk space, but the most noteworthy characteristic of the system is its robustness.
Disk access is limited to read-only at boot time; the run-time file system is supported by RAM disk. This means that restarting after a power failure will never require a file system check, which could otherwise prevent the boot process from proceeding. Moreover, because all the system files stay on a read-only device, it is most unlikely that they are inadvertently tampered with by anybody. Finally, in case of a program crash, Linux provides the capability to restart the processes.
The native support of the compressed kernel and root images was also very valuable because it allowed us to keep the entire system very small.
Luca Fini (lfini@arcetri.astro.it) has been with the Osservatorio di Arcetri for 20 years where he has worked as software developer and system manager of a LAN with more than 100 computers running UNIX, Linux, Windows 95/98/NT, MacOS, and more. He also deals with control system design and development for both astronomical instrumentation and, occasionally as an independent consultant, in the industrial field.
