An earlier article [“Memory Leak Detection in Embedded Systems”, LJ, September 2002, available at www.linuxjournal.com/article/6059] discussed the detection of memory leaks when using C as the programming language. This article discusses the problem of detecting memory leaks in C++ programs. The tools discussed here detect application program errors, not kernel memory leaks. All of these tools have been used with the MontaVista Linux Professional Edition 2.1 and 3.0 products, and one of them, dmalloc, ships with MontaVista Linux.

When developing application programs for embedded systems, designers and programmers must take great care with using system memory resources. Unlike workstations, embedded systems have a finite memory source. Typically, no swap area is available to idle programs. When the system uses up all of its resources, nothing is left to do but panic and start over or kill some programs to make room for the needed resources. Therefore, it is important to write programs that do not leak memory. Many tools aid programmers in finding these resource leaks. All of the tools discussed here come with their own test programs.

One method of testing, which I have seen used successfully by application developers, involves using a workstation to develop prototype code and debugging as much as possible on it. Using memory leak tools in this manner is strongly advised. By debugging on a workstation, the application programmer can be assured that the transition to the target processor will be easier. A major reason for using workstations is they are cheap, and everybody involved has one. Targets, on the other hand, are usually few and in great demand.

Most memory leak detection programs are available as full source. They typically have been built on an x86-based platform. Running them on non-x86 targets requires some porting. This porting effort could be as simple as a recompile, link and run, or it could require changing some assembler code from one platform to another. Some of the tools come with hints and suggestions for use in cross-compiling environments.

The author of dmalloc, a tool I covered in detail in the September 2002 article, states that his knowledge of C++ is limited, and thus the C++ detection of memory leaks also is limited. In order to use dmalloc with C++ and threads, it has been necessary to link the application as static.

The ccmalloc tool is a memory profiler with a simple usage model that supports dynamically linked libraries but not dlopen. It detects memory leaks, multiple de-allocation of the same data, underwrites and overwrites and writes to already de-allocated data. It displays allocation and de-allocation statistics. It is applicable to optimized and stripped code and supports C++. It also provides file and line number information for the whole call chain, not only for the immediate caller of malloc/free, and it supports C++. No recompilation is needed to use ccmalloc; simply link it with -lccmalloc -ldl or ccmalloc.o -ldl. ccmalloc provides efficient representation of call chains, customizable printing of call chains, selective printing of call chains, a compressed log file and a startup file called .ccmalloc. The major documentation is found in a file named ccmalloc.cfg. The test files included with the program provide more documentation. nm and gdb are required to get information about symbols and gzip or to compress log files.

NJAMD is, as the author states, “not just another malloc debugger”. As with most memory allocation debuggers, the standard allocation functions are replaced with new ones that perform various checks as memory is used. Specifically, it looks for dynamic buffer over/underflows and detects memory reuse after it is freed. The library built for NJAMD can be LD_PRELOADed, or it can be linked to the program. It creates a large memory buffer on the first memory allocation, 20MB, and it then carves this up as the program needs memory.

NJAMD can be used alone, with a front end or from within gdb. It has a utility that allows postmortem heap analysis. Another feature allows the application being debugged to skip recompilation; simply preload the library. NJAMD also is capable of tracing leaks in library functions that wrap malloc and free, GUI widget allocators and C++ new and delete. Often a memory leak is not discovered immediately but lurks, waiting to strike at the most visible moment. Tracking this down can take a long time. NJAMD has many environment variables that allow setting varying levels of detection. As with most debugging tools, performance can be an issue with NJAMD, so the tool should be used only during development. Deploying with the tool enabled can result in slower systems.

YAMD (yet another memory debugger) is another package for trapping the boundaries of allocated blocks of memory. It does this by using the paging mechanism of the processor. Read and write out-of-bound conditions are detected. The detection of the error occurs on the instruction that caused it to happen rather than later, when other accesses occur. The traps are logged with the filename and line number with trace-back information. The trace back is useful because most memory allocation is done through a limited number of routines.

The library emulates the malloc and free calls. Doing this catches many indirect malloc calls, such as those made by strdup. It also catches new and delete actions. If the new and delete operators are overloaded, however, they cannot be caught.

YAMD, like other programs of its type, needs a large amount of virtual memory or swap available to perform its magic. On an embedded system, though, this is typically not available. The earlier suggestion to use this tool on a workstation to do prototype debugging is encouraged here as well. When this debug is done, moving the application to the target can proceed with confidence that most, if not all, memory leaks have been found.

YAMD provides a script, run-yamd, that is used to make the program execute easily. It offers several options to try to recover from certain conditions. A log file can be created when the program being checked performs a core dump. A debugger can be used to debug YAMD-controlled programs. However, problems can arise using a debugger when YAMD is preloaded rather than statically linked with the program.

Valgrind is a relatively new open-source memory debugger for x86-GNU Linux systems. It has more capabilities than earlier tools, but it runs only on x86 hosts. When a program is run under the control of Valgrind, all read and writes to memory, as well as calls to malloc, free, new and delete, are checked. Valgrind can detect uninitialized memory, memory leaks, passing of uninitialized or unaddressable memory, some misuse of POSIX threads and mismatched use of malloc/free and new/delete actions.

Valgrind also can be used with gdb to catch errors and allow the programmer to use gdb at the point of error. When doing this, the programmer can look for the source of the problem and fix it much sooner. In some cases, a patch can be made and debugging can continue. Valgrind was designed to work on large as well as small applications, including KDE 3, Mozilla, OpenOffice and others.

One feature of Valgrind is its ability to provide details about cache profiling. It can create a detailed simulation of the CPU's L1-D, L1-I and unified L2 cache, and it calculates a cache hit count for every line of the program being traced. Valgrind has a well-written HOWTO with plenty of examples. Its web site contains a lot of information and is easily traversed. Many different combinations of options are available, and it is left to users to determine their favorite combinations.

Valgrind's error display contains the process ID for the program being examined, followed by the description of the error. Addresses are displayed along with line numbers and source filenames. A complete backtrace also is displayed. Valgrind reads a startup file, which can contain instructions to suppress certain error-checking messages. This allows you to focus more on the code at hand rather than pre-existing libraries that cannot be changed.

Valgrind does its checking by running the application in a simulated processor environment. It forces the dynamic linker/loader to load the simulator first, then loads the program and its libraries into the simulator. All the data is collected while the program is running. When the program terminates, all the log data is either displayed or written to log files.

The mpatrol library can be linked with your program to trace and track memory allocations. It was written and runs on several different operating system platforms. One distinct advantage of the library is it has been ported to many different target processors, including MIPS, PowerPC, x86 and by some MontaVista customers, to StrongARM targets.

mpatrol is highly configurable; instead of using the heap, it can be set to allocate memory from a fixed-size static array. It can be built as a static, shared or threadsafe library. It also can be one large object file so it can be linked to the application instead of contained in a library. This functionality provides a great deal of flexibility for the end user.

The code it creates contains replacements for 44 different memory allocation and string functions. Hooks are provided so these routines can be called from within gdb. This allows for debugging of programs that use mpatrol.

Library settings and heap usage can be displayed periodically as the program runs. All the statistics gathered during runtime are displayed at program termination. The program has built-in defaults that can be overridden by environment variables. By changing these environment variables at runtime, it becomes unnecessary to rebuild the library. Tuning of the various tests can be done dynamically. All logging is done to files in the current working directory; these can be overridden to go to stdout and stderr or to other files.

As the program is running, call stack trace-back information can be gathered and logged. If the program and associated libraries are built with debug information about symbols and line numbers, this information can be displayed in the log file.

If at some point the programmer wants to simulate a stress test on a smaller memory footprint, mpatrol can be instructed to limit the memory footprint. This allows for testing conditions that may not be readily available in the lab environment. Stress testing in emulating a customer environment or setting up a harsh test harness is made easier with this feature. In addition, the test program can be made to fail a random set of memory allocations to test error-recovery routines. This ability can be useful for exception handling in C++. Snapshots of the heap can be taken to allow the measuring of high and low watermarks of memory use.

The Insure++ product by Parasoft is not GPLed or free software, but it is a good tool for memory leak detection and code coverage, very similar to mpatrol. Insure++ does do more than mpatrol in the area of code coverage and provides tools that collect and display data. Trial copies of the software can be downloaded and tried for a specified time period on non-Linux workstations.

The product installs easily under Linux but is node-locked to the computer on which it is installed. Insure++ comes with a comprehensive set of documentation and several options. The code coverage tool is separate but comes with the initial package.

Insure++ provides a lot of information about the problems it finds. To use Insure++, it is necessary to compile it with the Insure++ front end, which passes it to the normal compiler. This front end instruments the code to use the Insure++ library routines. During the compiler phase, illegal typecasts are detected as well as incorrect parameter passing. Obvious memory corruption errors are reported. During runtime, errors are reported to stderr but can be displayed by a graphical tool. When building an application, either the command line or makefiles can be used, facilitating the building of projects and large applications.

Execution of the program is simple. Insure++ does not require any special commands to execute; the program is run as if it were a normal program. All the debug and error-trapping code is contained in the Insure++ libraries that were linked with the program.

An add-on tool, called Inuse, displays in real time how the program uses memory. It can give an accurate picture of how memory is used, how fragmented it gets and subtle leaks that seem small but could add up over time. I had an experience with a client who found that a particular C++ class was leaking a small amount of memory that, on a workstation, was seen to be quite small. For an embedded system that was expected to be running for months and possibly years, the leak could become quite large. With this tool, the leak was easily traced, found and fixed. Other available tools did not catch this leak.

Code coverage is analyzed by another tool, TCA. As the program is run with Insure++ turned on, data can be collected that, when analyzed by TCA, paints an accurate picture of what code was executed. TCA has a GUI that enhances the display of code coverage.

Resources