Many years ago, I had a small revelation that I'm sure many of you have experienced yourselves. I realized that maintaining 10 systems requires a good bit more work than administering a single computer. But, it doesn't have to take that much more work--assuming the proper tools and methodologies are used.

When you want to make a change to a single system, you simply decide what to change and poke around until everything works properly. Three months later, you may not even remember what it was you did or why you did it. Does that matter? Not usually.

But when you have to make those changes to several systems, do you really want to perform manually the same task numerous times? If you had 10 systems but now have 11, will you remember to make all of the same changes to the new arrival? Maybe the new system is slightly different--or maybe none of your systems are the same. Wouldn't it be nice to know exactly what you did and why you did it?

This is where GNU cfengine, by Mark Burgess, comes into play. It allows you to affect changes effortlessly across any number of dissimilar systems. Perhaps even more important, it provides automatic documentation of exactly what you did. You even can use a few comments to explain why you did it. Each of your systems become a member of one or more classes, and changes are made on a per-class basis. If a new system arrives, it automatically acquires the changes previously made to other members of its class.

Once cfengine is installed (from www.cfengine.org) and running, making changes to your group of systems becomes almost as easy as changing a single system. This gives you more time to decide what to do and how to do it, something that remains the primary responsibility of an administrator to this day.

As with many programs, the most difficult part about using cfengine is getting started. For this reason, we are going to set up a basic environment, with one client and one server. You should perform these steps on a couple of test systems, if possible, to follow along with the examples. Later, you can expand this framework to build your own cfengine environment.

I believe you always should start a new adventure with the simplest practical configuration. In this case, the systems are assumed to each have a single, static IP address and the proper DNS entries (forward and reverse). The setup process can be summarized as follows:

Let's start with the master server. It should have the directory /usr/local/var/cfengine/inputs containing the master set of configuration files. It must run the cfservd dæmon, and it must have a valid cfservd.conf configuration file, as follows:

control:
   domain = ( mydomain.com )
   AllowUsers = ( root )
   cfrunCommand = ( "/var/cfagent/bin/cfagent" )

admit:
   /usr/local/var/cfengine/inputs *.mydomain.com
   /var/cfagent/bin/cfagent       *.mydomain.com

This simple configuration file does little more than allow all hosts from mydomain.com to download the master set of configuration files. It also allows remote systems to execute the cfagent command using cfrun, a useful feature you should explore once you have things up and running.

We now can move on to update.conf, which is processed and executed first when cfagent is run on any system. This file's primary function is to transfer the master configuration files to the local system. It must be kept simple and reliable, as any errors in this file have to be repaired manually on each system.

Listing 1. Sample update.conf

control:
   actionsequence = ( copy tidy )
   domain         = ( mydomain.com )
   workdir        = ( /var/cfengine )
   policyhost     = ( server.mydomain.com )
   master_cfinput = 
                  ( /usr/local/var/cfengine/inputs )
   cf_install_dir = ( /usr/local/sbin )

copy:
   $(master_cfinput)     dest=$(workdir)/inputs
                         r=inf
                         mode=644
                         type=binary
                         exclude=*.lst
                         exclude=*~
                         exclude=#*
                         server=$(policyhost)

   $(cf_install_dir)/cfagent    
                         dest=$(workdir)/bin/cfagent
                         mode=755
                         type=checksum

   $(cf_install_dir)/cfservd    
                         dest=$(workdir)/bin/cfservd
                         mode=755
                         type=checksum

   $(cf_install_dir)/cfexecd    
                         dest=$(workdir)/bin/cfexecd
                         mode=755
                         type=checksum

tidy:
   $(workdir)/outputs pattern=* age=7

The file shown in Listing 1 should be all you need in most environments; simply replace server.mydomain.com with the hostname of your cfengine server. When executed, this update.conf file creates a local copy of the required cfengine binaries in /var/cfengine/bin (from /usr/local/sbin, which is assumed to be an NFS filesystem, or similar).

More importantly, the configuration files are copied from the server that runs cfservd to /var/cfengine/inputs on every system, including the server itself. The configuration files are compared bit-by-bit with the master file (type=binary), while the binaries are updated if their checksums don't match the master copies.

Finally, the tidy section removes output logfiles that are more than seven days old to keep your drive from filling up. The tidy section also can be used in the main cfagent.conf to clean up a variety of other files on your systems.

Listing 2. Sample cfagent.conf

control:
   actionsequence  = ( processes editfiles )
   domain          = ( mydomain.com )
   access          = ( root )
   # Where cfexecd sends reports
   smtpserver      = ( mail.mydomain.com )
   sysadm          = ( root@mydomain.com )

processes:
   # Make sure these processes are always running
   "cfservd" restart "/var/cfengine/bin/cfservd"
   "cfexecd" restart "/var/cfengine/bin/cfexecd"

editfiles:
   # Make sure cfexecd runs hourly from cron too
   {  /etc/crontab
      AppendIfNoSuchLine 
         "0 * * * * root /usr/local/sbin/cfexecd -F"
   }

The meat of your cfengine installation is found in cfengine.conf. For now, let's use a basic one that keeps cfengine running; actual security-enhancing code is added later in the article. This basic file is shown in Listing 2. The control section is required and sets a variety of global parameters. Most of these are self-explanatory. The exceptions are access, which specifies which users are allowed to run cfagent, and the actionsequence items, which defines which sections are to be executed and in what order. As you add sections, it is important to add them to this list as well, if you want them to have any effect.

In this case, the first section to be executed is processes. All this section does is check to make sure cfservd (mandatory on the server, but useful on all systems) and cfexecd are running. If a process is not running, it is restarted by executing the specified command.

The cfexecd dæmon executes cfagent once per hour, by default. cfagent in turn processes and executes all of these configuration files. But how would cfagent be able to run and start cfexecd if cfexecd is not running to start the entire sequence?

Well, it can happen by running cfagent manually, such as the first time a system is configured. You also can have cfexecd executed from the system cron scheduler on a regular basis. This way, if one method fails, the other method still should work and repair the problem. The editfiles section does exactly that--it makes sure cfexecd is run once per day from cron by adding an entry to /etc/crontab if no such entry exists. The format and location of this file may be different on your system, so make changes accordingly. You also may want to add an entry to the processes section to make sure the cron dæmon is running. If you have hosts of mixed types, you may have to use classes to operate differently on different systems, as described later in this article.

We are done with the basic configuration files, so let's move on to network communications. All network communications within cfengine utilize a public and private key pair for each host. This helps ensure that one system is talking to the remote system it thinks it is. What this means is you need to generate a key pair for each host by running the cfkey command. This creates the files localhost.pub and localhost.priv in the /var/cfengine/ppkeys/ directory. You then need to place the master's public key in this directory, named root-10.1.1.1.pub, assuming that is the IP address of the server running cfservd. Finally, you need to copy each client system's public key onto the master server, named root-10.2.2.2.pub, for example.

Now cfexecd executes cfagent for you every hour. You also can run cfagent manually on any machine whenever you desire. This command updates the configuration files from the server (as defined in update.conf). It also checks and makes changes to the system (as defined in cfagent.conf). For testing purposes, you can execute cfagent --dry-run to see what actions would have been taken by cfagent. While testing and debugging, you also may want to add the line IfElapsed = ( 0 ) in the control section of cfagent.conf. This disables a denial-of-service prevention feature that restricts how often actions are taken.

You may ask, what does any of this have to do with security? Well, now you can make almost any change to any of your systems simply by modifying the master cfagent.conf and then waiting for the machines to pull and execute this file from the configuration server. Even if a machine does not yet exist or the machine is a laptop that has been disconnected for a month, each system picks up each appropriate change as soon as possible.

So, what are some changes you can make to your systems to increase security? For starters, cfengine can report automatically on certain suspicious files and directories. Any or all of the following entries can be added to the control section of cfagent.conf:

These directives apply only to files and directories scanned by cfengine for other reasons, such as the files, tidy or copy sections. Unless you direct it to do so, cfengine does not scan your entire filesystem in order to look for these files.

You also can use cfengine to disable certain network services you do not want running on your systems. As an example, let's disable the rshd and rlogind services on all systems, as well as any files related to these services:

editfiles:
   { /etc/inetd.conf
      HashCommentLinesContaining "rshd"
      HashCommentLinesContaining "rlogind"
      DefineClasses "modified_inetd"
   }

processes:
   modified_inetd::
      "inetd" signal=hup

disable:
   /root/.rhosts
   /etc/hosts.equiv

The code above accomplishes this task on a standard Linux system when added to the master cfagent.conf, and the actionsequence list is expanded. First of all, any lines containing the strings rshd or rlogind in /etc/inetd.conf are commented out. If such a thing occurs, the class modified_inetd is defined. Then, when the processes section is executed and the class is defined, the HUP signal is sent to the inetd process to cause it to reload its configuration.

We finally tell cfengine to disable the files /root/.rhosts and /etc/hosts.equiv if they exist. They are made unreadable for anybody but root and renamed with a .cfdisabled extension. You also could disable this file in your users' home directories, but that takes a few more setup tasks that are beyond the scope of this article.

You can use cfengine to check and repair the permissions of important system files:

files:
   /etc/passwd mode=644 owner=root group=root
      action=fixall
   /etc/shadow mode=600 owner=root group=root
      action=fixall
   /etc/group  mode=644 owner=root group=root
      action=fixall

directories:
   /etc mode=0755 owner=root group=root inform=true
   /tmp mode=1777 owner=root group=root inform=true

In this case, we are concerned with certain system account and password files. We specify the desired ownership and permissions and direct cfengine to repair any problems (action=fixall) it may find. We also list a couple of important directories that should be created if they don't exist and whose permissions should be set appropriately.

You may be familiar with the Tripwire program, a wonderful security tool that monitors the files on your systems for unwanted changes. You can do similar file monitoring with cfengine. Although not as secure as the method used by Tripwire, this method is easy and convenient if you already are using cfengine for other tasks.

At the very least, I like to monitor important system files, especially setuid root files, for the proper permissions and md5 checksum:

files:
   /bin/mount mode=4755 owner=root group=root
      action=fixall checksum=md5
   /bin/mount mode=4755 owner=root group=root
      action=fixall checksum=md5

I have listed only two files in the example, but you obviously could list many more. As in the previous example, the files' permissions are checked and fixed, if necessary. But, in addition, each file's md5 checksum is compared with its previously stored value and any anomalies are reported. Cfengine updates only the stored checksums when the ChecksumUpdates = ( on ) line is present in the control section.

You also can have cfengine find and eliminate the setuid bit on files that should not be setuid. In Listing 3, we have an entry in the files section looks through the entire filesystem and turns off the setuid bit on any file owned by root and not explicitly itemized as a file to be ignored. You obviously should expand the list of files to be ignored before executing this code, as there probably are many other setuid root files on your systems. You can determine what files currently are setuid root on your system by running the command find / -perm +4000 -user root.

Listing 3: Eliminating Unwanted setuid Root Files

filters:
   { root_owned_files
      Owner:     "root"
      Result:    "Owner"
   }

files:
   /
      filter=root_owned_files
      mode=u-s     # no SUID bit may be set
      recurse=inf
      action=fixall
      inform=true
      # And the full paths of files that *should* be SUID
      ignore=/proc
      ignore=/bin/ping
      ignore=/usr/bin/su
      ignore=/usr/bin/passwd
      ignore=/usr/bin/at
      ignore=/usr/bin/crontab
      ignore=/usr/bin/rsh
      ignore=/usr/sbin/traceroute
      ...

One powerful feature of cfengine we have not used much in these examples is its support of classes. Each system automatically is a part of numerous classes, and custom classes can be defined in many ways--when a certain action is taken or based on certain files found on the system. You then can perform sets of commands only on systems of specific classes. Some examples of classes automatically defined for one of my systems are: redhat, linux, redhat_7_2 and www_kaybee_org. There also are classes defined based on the current system time. Listing 4 shows a few examples of how these classes can be used; I'm sure you can find many more uses for this feature.

Listing 4. Class Examples

classes:
   # Assume systems with this file are web servers
   web_server = ( 
      '/usr/bin/test -f /etc/httpd/conf/httpd.conf' 
   )

files:
   web_server::
      # Fix permissions on web server config files
      /etc/httpd/conf owner=root group=root
         mode=0644 action=fixall recurse=inf

   solaris::
      # On all Solaris systems
      /etc/inet/inetd.conf owner=root group=root
         mode=0644 action=fixall

   linux.!(redhat_7|redhat_8|redhat_9)::
      # On all Linux systems not running Red Hat
      # versions 7.X, 8.X, or 9.X.
      /etc/inetd.conf owner=root group=root
         mode=0644 action=fixall

   redhat_7|redhat_8|redhat_9::
      # On Red Hat Linux 7.X, 8.X, or 9.X
      /etc/xinetd.conf owner=root group=root
         mode=0644 action=fixall

   any::
      # This file exists on all systems
      /etc/passwd mode=644 owner=root group=root
         action=fixall

tidy:
   Sunday::
      # Clean out files more than 14 days old
      # in /tmp on Sunday only
      /tmp recurse=inf age=14 rmdirs=sub

GNU cfengine is much more than a system security tool. It can be used to distribute files, maintain your systems and automatically perform just about any configuration task on any system. It operates on a wide variety of platforms and, once you get started, is quite easy to use. I have shown you a few examples of the kinds of things cfengine can do for you. I hope you decide that cfengine can enhance your systems' security and use this wonderful tool to automate many of your other system administration tasks.