What do you do once you realize one of your servers has been compromised? I recently had the opportunity to hear Linux Journal's own Kyle Rankin give a very impressive talk covering this situation at the Southwest Drupal Summit in Houston, Texas.

The actions you choose to take are very important and should be prepared before the fateful event. Most people will spend time on the server trying to figure out how the intruder gained access to the machine, and what they have been doing. Not only is this problematic in that the intruder has more time to do his damage, but the longer the server is up, the more likely critical forensic data will be lost.

Kyle argued that the best first step is to immediately pull the plug on the box. Do not diagnose the situation and do not shut the machine down gracefully. We use journaling file systems for a reason and the machine will probably be rebuilt from scratch, so the danger of corrupted data from killing the power is small. Once the machine is off, you should image the compromised drive with something like 'dd' and make a copy of the image to do your work on to protect you from accidentally contaminating the evidence.

The issue with poking around on the live system is that you will destroy any information you could have learned from the meta data stored on every file on the computer. Linux uses MAC times to record when certain events occurred most recently. The events that change MAC times on a file are “modification” (the data in the file was modified), “access” (some part of the file was read or executed), and “metadata change” (the file's permissions or ownership were changed). By pulling power from the server at the earliest possible moment, you decrease the likelihood that MAC times recorded by the intruder's action will have been updated by another user.

Kyle then did a live demo on a compromised image showing how to use The Sleuth Kit and Autopsy Browser to perform the investigation. These tools can be used to view log files, recover deleted files, and to order the files on the file system by MAC times. With this information, he was able to paint an interesting picture of not only how an intruder gained access to one of his machines, but what they did once they once they had access.

Kyle is a great public speaker and I highly recommend seeing him if you get the chance. His slides for the Southwest Drupal Summit presentation are available online as is his Linux Journal article Introduction to Forensics where he goes into great detail on how to use these tools.