As impregnable as we hope our hardened systems are, security isn't a game of absolutes: the potential for system breaches must be recognized. Tripwire Open Source is a free and open-source software package that gives us a reasonable chance of being notified of possible breaches as soon as they occur.

Integrity checkers such as Tripwire create cryptographic “fingerprints” of system binaries, configuration files and other things likely to be tampered with in the course of, or subsequent to, a security breach. They then periodically check those files against the stored fingerprints and e-mail discrepancies back to you.

Tripwire is the most well known and mature integrity-checking system, and the one we're about to discuss in depth. You may also be interested in AIDE, which runs on more platforms than Tripwire Open Source, and FCheck, which is written 100% in Perl and, thus, even less platform-dependent than AIDE or Tripwire (it even runs on Windows). See the Resources section at the end of this article for links to AIDE's and FCheck's web sites.

Integrity checking mechanisms are like system backups; we hope we'll never need them, but heaven help us if we do and they're not there. Also, like system backups, integrity checking is an important component of a larger plan. If a system has been hardened, patched and maintained according to the industry's highest standards (or at least common sense), an integrity checker will provide a final safety net that helps minimize the damage done by whatever brilliant cracker manages to sneak in.

The principle on which integrity checkers operate is simple: if a file changes unexpectedly, there's a good chance it's been altered by an intruder. For example, one of the first things a system cracker will often do after “rooting” a system is replace common system utilities such as ls, ps and netstat with “rootkit”, which makes them appear to work normally but conveniently fail to list files, processes and connections (respectively) that might betray the cracker's presence.

Integrity checkers can be used to create a database of hashes (checksums) of important system binaries, configuration files or anything else we don't expect or want to have changed. By periodically checking those files against the integrity checker's database, we can minimize the chances of our system being compromised without ever knowing it. The less time between a system's compromise and the administrator's learning of it, the greater the chance administrators can catch, or at least evict, the intruders.

One caveat: any integrity-checker with an untrustworty database is worthless. It is imperative to create this database as soon as possible after installing the host's operating system from trusted media. I repeat: installing, configuring and maintaining an integrity-checker is not worth the effort unless its database was initialized on a clean system.

Among the most celebrated and useful things to come out of Purdue's COAST project (http://www.cerias.purdue.edu/coast/) is Tripwire, created by Dr. Eugene Spafford and Gene Kim. Originally both open source and free, Tripwire went commercial in 1997, and fee-free use was restricted to academic and other non-commercial settings.

Happily, last October Tripwire, Inc. released Tripwire Open Source, Linux Edition. Commercial versions of Tripwire until then had included features not available in the older Academic Source Release. In contrast, Tripwire Open Source is a more-or-less current version of the commercial product. Other than lacking enterprise features such as centralized management of multiple systems, it is very similar to the Tripwire for Servers product.

Note that Tripwire Open Source is free for use only on non-commercial Unices, i.e., Linux and Free/Net/OpenBSD. In fact, it's only officially supported on Red Hat Linux and FreeBSD, although there's no reason it shouldn't compile and run equally well on other Linux and BSD distributions. Only the older Academic Source Release is free for use on commercial Unices such as Sun Solaris and IBM AIX; the proprietary version must be purchased for these systems.

But we're all Linux geeks here, right? For the remainder of this discussion I'll focus on Tripwire Open Source, Linux Edition.

As of this writing, the most current version of Tripwire Open Source is 2.3.1-2. It can be downloaded as a source-code tarball at http://sourceforge.net/projects/tripwire/. I strongly recommend that you obtain, compile and install this version. While Tripwire has had only one significant security problem (and only a denial-of-service risk, at that) in its history, we use Tripwire because we're paranoid. For paranoiacs, only the latest (stable) version is good enough.

Having said that, the binary version included with Red Hat 7.0 is reasonably up-to-date. As far as I can tell, the differences between Red Hat's v2.3-55 RPM and the official source-release v2.3.1-2 involve non-security-related bugfixes; therefore you're probably taking no huge risk in using your stock RH 7.0 RPM. But don't say I told you to!

To compile Tripwire Open Source, move the archive to /usr/src and un-tar it, e.g., tar -xzvf ./tripwire-2.3.1-2.tar.gz. Next, check whether you have a symbolic link from /usr/bin/gmake to /usr/bin/make (non-Linux Unices don't all come with GNU make, so Tripwire explicitly looks for gmake--of course, on most Linux systems this is simply called make). If you don't have it, the command to create this link is ln -s /usr/bin/make /usr/bin/gmake.

Another thing to check for is a full set of subdirectories in /usr/share/man—Tripwire will need to place man pages in man4, man5 and man8. On my Debian system /usr/man/man4 was missing, and as a result the installer created a file called /usr/share/man/man4 that, of course, was actually a man page incorrectly copied to that name rather within it.

Finally, read the source's README and INSTALL files, change to the source-tree's src directory (e.g., /usr/src/tripwire-2.3.1-2/src), and make any changes you deem necessary to the variable-definitions in src/Makefile. Be sure to verify that the appropriate SYSPRE definition is uncommented (SYSPRE = i386-pc-linux, SYSPRE = sparc-linux, etc.).

Now we're ready to compile, type make release. This will take awhile, so now is a good time to grab a sandwich. When the build is done, navigate up one directory level, e.g., /usr/src/tripwire-2.3.1-2, and execute these two commands:

cp ./install/install.cfg .
cp ./install/install.sh .

Now open install.cfg with your favorite text editor; while the default paths are probably fine, you should at the very least examine the Mail Options section. This is where we initially tell Tripwire how to route its logs. If we set TWMAILMETHOD=SENDMAIL and specify a value for TWMAILPROGRAM, Tripwire will use the specified local mailer (sendmail by default) to deliver its reports to a local user or group.

If instead we set TWMAILMETHOD=SMTP and specify values for TWSMTPHOST and TWSMTPPORT, Tripwire will mail its reports to an external e-mail address via the specified SMTP server and port. Note that if you change your mind later, Mail Options settings can be changed in Tripwire's configuration file at any time.

If the system on which you're installing Tripwire is a multiuser system, and one that you or other system administrators routinely log on to and read e-mail, the SENDMAIL method is probably preferable. If the system is a host you typically administer remotely from other systems, the SMTP method is probably better.

Once install.cfg is set to your liking, it's time to install Tripwire. Simply enter sh ./install.sh. You will be prompted for site and local passwords; the site password protects Tripwire's configuration and policy files, whereas the local password protects Tripwire databases and reports. This allows the use of a single policy across multiple hosts in such a way as to centralize control of Tripwire policies but distribute responsibility for database management and report generation.

Needless to say, it's simpler and faster to install RPMs (but again, note that the most up-to-date version of Tripwire may not be available in this format). The only thing you need to know is that after you run rpm, you'll need to enter /etc/tripwire/twinstall.sh to generate site and local passwords. This script behaves much like the end of the source distribution's install.sh script—see the previous paragraph.

As useful as Tripwire is, it has a reputation for being difficult to configure (which is, of course, true of most powerful and flexible tools). But it's really not as bad as all that, and by following the simple instructions I'm about to set forth, you can use Tripwire effectively enough to catch yourself some bad guys. So now, let's enter the elite ranks of users who have not only installed, but actually used, Tripwire!

The first thing we need to do is double-check our configuration file, tw.cfg. Actually, this file was just encrypted by the installation script, but for our convenience a clear-text copy called twcfg.txt should reside in /etc/tripwire. This is the place to change any settings you've had second thoughts about since running the installation script, and the variables are named accordingly.

If you make any changes, re-encrypt the configuration file with the command:

twadmin --create-cfgfile --site-keyfile ./site.key twcfg.txt

where site.key is the name of the site-key created at installation time and twcft.txt is the name of the clear-text configuration file you just edited and wish to encrypt. That may seem obvious given that these are the default names for these files, but you can name them whatever you like. Regardless, don't forget to specify the keyfile, or twadmin will return an error (remember, the point of this exercise is to encrypt the configuration file).

Warning: you should not, as a matter of practice, leave clear-text copies of your Tripwire configuration (tw.cfg) or policy (tw.pol) files on your hard drive. After editing and encrypting them, delete the clear-text versions. You can always retrieve them later with the command:

twadmin --print-cfgfile > mycfg.txt

where, predictably, you can substitute mycfg.txt with whatever you like.

Although I haven't yet described Tripwire binaries in any detail (it's more useful to explain them in context), you've no doubt guessed by now that twadmin is used to manage Tripwire's configuration, key and (initially, at least) policy files.

Like the Tripwire configuration file, policies are edited as text files but are encrypted and signed before being installed. Unlike the configuration file, however, we only use the twadmin command to install a policy file for the first time on a given system; subsequently we'll use the tripwire command in policy-update mode.

In any event, the command to install a policy the first time after installing Tripwire is:

twadmin --create-polfile twpol.txt

where twpol.txt is the name of the clear-text policy file you wish to install.

As with configuration files, you shouldn't leave clear-text policy files on your system. If you need to refer to or edit the policy later, you can retrieve it by typing:

twadmin --print-polfile > mypol.txt

mypol.txt can be whatever you wish to call the clear-text copy of the policy. (See a pattern here?)

And now we begin the serious voodoo. Tripwire's policy file is its brain: it specifies what to look at, what to look for and what to do about it. It's also a little on the user-hostile side, though not nearly so bad in this regard as, say, sendmail.cf (but prepare to memorize some abbreviations).

Naturally Tripwire Open Source comes with a default policy file, and naturally you may, if you like, use this as your very own personal Tripwire policy. But since the default policy was created for a Red Hat system running nearly everything in the distribution, you should probably edit this policy heavily rather than use it as is.

First, a word about tuning. If your policy doesn't check enough files or doesn't look closely enough at the ones it does check, Tripwire's purpose is defeated and shenanigans will go undetected. Conversely, if the policy looks too critically at files you expect to change anyhow, Tripwire will generate “false positives” that serve no purpose other than to distract your attention from actual discrepancies.

It's doubtful you'll create a sane baseline the first time around. You'll almost certainly need to adjust your policy on an ongoing basis and especially after the first time you run an integrity-check. Thus, even if you do have a Red Hat system with exactly the same configuration as the one for which the default Tripwire Open Source policy was designed, you still need to learn proper Tripwire policy syntax. Let's get cracking.

I'm going to explain policy file structure and syntax by dissecting a working policy file into manageable chunks. The first chunk we'll look at is from the very beginning of our sample policy file and lists some variable definitions:

WEBROOT=/home/mick/www;
CGIBINS=/home/mick/www/cgi-bin;
TWPOL="/etc/tripwire";
TWDB="/var/lib/tripwire";

We can use these variables to save valuable touch-typing energy. On to the next chunk, some fancier variable definitions:

BINS          = $(ReadOnly) ; # Binaries that should
                                not change
SEC_INVARIANT = +tpug ;       # Dir.s that shouldn't
                                change perms/ownership
SIG_MED       = 66 ;          # Important but not
                                system-critical files

The second line defines a “property mask”; property masks are abbreviations of the file properties Tripwire examines. Since property mask strings can be cryptic and unwieldy, most people prefer to use variables to refer to them. In fact, Tripwire comes with a number of predeclared variables set to common property masks, and the first line actually refers to one of these, ReadOnly, a property mask for files that shouldn't change in any way, like binaries. We'll discuss property masks in detail, but all in good time.

The third line creates a name for a severity level. Severity levels can be used to differentiate between rules of varying importance. When the tripwire command is invoked with the --severity N parameter, only rules with assigned severity levels equal to or greater than N will be parsed. If this parameter is not used, all rules will be parsed. Also note that if a rule has no severity level associated with it, the level will be set to zero by default. That is, that rule will only be parsed when the --severity parameter isn't specified.

Now that we've got a feel for policy variables and what they're used for, let's start looking at actual rules:

# Mick's Web Junk
(
  rulename = "MickWeb",
  severity = $(SIG_MED),
  emailto = mick@uselesswebjunk.com
)
{
  $(WEBROOT)             -> $(ReadOnly) (recurse=1) ;
  !$(WEBROOT)/guestbook.html ;
  $(CGIBINS)             -> $(BINS)     ;
  /var/log/httpd         -> $(Growing)  ;
}

This is a very rich chunk, so we'll begin with rule structure. Rules may either stand alone or be grouped together based on common attributes; this listing shows a group of rules (contained within curly brackets) with several shared attributes (in parentheses, above the rules). This group's rulename is “MickWeb”, the group's severity is 66 and reports involving this group will be e-mailed to mick@uselesswebjunk.com. Note that attributes are separated by commas, whereas each rule ends with a semicolon.

Attributes can also be assigned to individual rules. The first rule in this group has the attribute recurse set to 1, which means that the directory /home/mick/www will be checked down one level (i.e., the directory itself plus everything immediately “below”, but no further). Note that by default, directories will be recursed as far down as they go; in effect, the recurse attribute has a default value of “True”.

Attributes listed in rule statements usually override those listed in parentheses above such rules' group. The exception is the attribute “emailto”, which is cumulative: if a group has a shared emailto string and one of that group's rules has a different emailto string, reports relevant to that rule will be e-mailed to all e-mail addresses in those two strings.

By the way, there are only four attributes: rulename, severity, emailto and recurse. For more detailed information on these see the Resources section.

After the group attributes for MickWeb we have some actual rules. Note the use of variables to specify both objects (the Tripwire term for files and directories) and property masks. In fact, none of the rules uses a “longhand” property mask. This is common practice and perfectly acceptable.

Immediately below the first rule, which tells Tripwire to treat the first level of my WWW directory as read-only, we have a statement beginning with an exclamation point. This statement is called a stop point, and it defines an exception to a rule. In this case, the stop point tells Tripwire to ignore changes to the file /home/mick/www/guestbook.html. Attributes do not apply to (nor may they be assigned to) stop points.

There, that's a complete policy file (technically, at least—it doesn't check any system binaries or configuration files at all—real policies are much longer). Listing 1 shows it in all its non-dissected glory.

Listing 1. Sample Policy File

You may have noticed this entire file only contains one explicit reference to a property mask: the variable declaration in which SEC_INVARIANT is set to “+tpug”. What does that mean?

A property mask is a series of file/directory properties that should be checked or ignored for a given object. Properties following a plus sign are checked; those following a minus sign are ignored. The properties are abbreviated as outlined in Table 1.

Table 1. Property Mask Values

Tripwire's own documentation describes these properties in depth. If you're unfamiliar with some of the more arcane file attributes (e.g., inode reference count) I recommend the paper “Design and Implementation of the Second Extended Filesystem” by Card, Ts'o and Tweedie (see Resources, below). As for hash-types, note that you generally won't want to use more than one or two cryptographic hashes per rule: these are CPU-intensive. On the other hand, do not rely solely on CRC-32 hashes, which are fast but much easier to subvert.

As I mentioned earlier, Tripwire has a number of predefined (hard coded) variables that describe property masks, shown in Table 2.

Table 2. Predefined Tripwire Property Mask Variables

In most cases it's simpler to use these predefined masks than to “roll your own”. Note, however, that you can combine these variables with additional properties, e.g.,

/dev/console  -> $(Dynamic) -u ;
# Dynamic, but UID can change

is the same thing as

/dev/console  -> +pingutd-srlbamcCMSH-u

twadmin --create-polfile policyfile.txt

To initialize the database, we now use the tripwire command: tripwire --init. Doesn't get much simpler than that, does it? But use the --init directive only when creating a new database. If you need to change your Tripwire policy later, it's better to use the following commands:

twadmin --print-polfile > mypolicy.txt
  # dump current installed plcy
vi mypolicy.txt
  # make changes to policy
tripwire --update-policy mypolicy.txt
  # install new policy --
  # DON'T USE TWADMIN FOR THIS!

Once we've got a database installed, we can run periodic checks against it. At its simplest, the command to do so is: tripwire --check. This compares all protected files against the hash-database and prints a report both to the screen and a binary file. The report can be viewed again with the command:

twprint --print-report --report-level N --twrfile /path/file

where N is a number from 0 to 4, 0 being a one-line summary and 4 being a full report with full details. /path/file is the full path and name of the latest report (by default it should reside in /var/lib/tripwire/report).

To have Tripwire automatically e-mail the report to all recipients specified in the policy, we could have run our check like this instead:

tripwire --check --email-report

Note that the report is still printed to standard output and saved in /var/lib/tripwire/report as well.

If you've installed the Tripwire RPM on a Red Hat 7 system, your system is already set up to run Tripwire periodically in check mode. The RPM includes the script /etc/cron.daily/tripwire-check. If you've used the emailto attribute in your Tripwire policy, however, you may wish to edit the second-to-last line of this script to read:

test -f /etc/tripwire/tw.cfg && /usr/sbin/tripwire --check --email-report

(This line by default lacks the --email-report flag.)

Tripwire won't tell you much unless you run regular checks, either manually, via cron/anacron or some combination thereof.

So, what happens when Tripwire reports violations? That's up to you. Often, violations will be the result of a too-restrictive Tripwire policy rather than actual skullduggery. You'll need to decide which are which and what to do about them.

Either way, you'll probably want to update the Tripwire database after violations are found so that it reflects any legitimate changes to the files and directories being monitored. There are two ways to do this. The first is to run the tripwire command in update mode:

tripwire --update --twrfile /var/lib/tripwire/report/myhost-date.twr

The last argument is the absolute path to the report you wish to use as the basis for this update. This opens the report with the editor specified in tw.cfg so you can indicate which, if any, of the changed files/directories you wish Tripwire not to update in its database. In other words, when you exit the editing session, Tripwire will update the attributes and hashes in its database only for those report entries with an X next to them (they all are by default).

Here's an excerpt from a tripwire-update session:

Remove the "x" from the adjacent box to prevent
updating the database with the new values for this
object.
Modified:
[x] "/home/mick/www"

If I delete the “x” from this entry, exit the editor and run a check, the /home/mick/www change will be reported again; the database will not have updated to reflect this change. In short, if the change is legitimate, leave the “x” there. If it isn't or you're not sure, remove the “x”.

The second way to update the Tripwire database is to do the actual check in “interactive” mode, which immediately triggers an update session after the check finishes. Thus,

tripwire --check --interactive

is the same thing as

tripwire --check
tripwire --twrfile /var/lib/tripwire/report/reportname.twr

When you get false positives, it will often make sense to fine-tune your policy. Remember to do this in the manner described at the end of the Editing or Creating a Policy section above.

Before we sign off for this month, I leave you with two excellent tips I learned from Ron Forrester, Project Manager for Tripwire Open Source:

I hope that's enough to get you started; there's much I haven't covered here or have only touched on. Believe you me, this tool's power is worth its learning curve, and the Tripwire Open Source Manual (see below) is both comprehensive and extremely well-written. Good luck!

Resources

Sidebar