Hello, everyone. Nice to see your smiling faces back here on the corner, the SysAdmin's Corner, that is. This week, we're going to wrap up the "Tell-Tale Heart" series with something so cool, you may never look at your Linux system in the same way. You might not know it, but there is a phantom living on your computer: a filesystem with lots of directories and files, much like any other filesystem, except that this one isn't really there at all. In fact, it disappears every time you shut down, and is rebuilt every time you reboot.

While I was writing this article, I realized I was having almost too much fun. Some things in life are so cool that you just want to keep fiddling with them. In the last few articles, I've been comparing your Linux system logs to a kind of digital diary kept by your system. If logs are a peek at your system's private diaries, then the /proc filesystem (or pseudo-filesystem, since it doesn't really exist) is a means of looking into the very soul of your system.

Okay, I've said it twice now. How can the mysterious /proc filesystem not really exist? If I do an ls -l on the root directory, I get a directory of size zero. Try it out yourself.

   # ls -l /
   (note: only showing partial listing)
   drwxr-xr-x    2 root     root         4096 Aug  1 15:07 patch
   dr-xr-xr-x   90 root     root            0 Aug 12 19:57 proc
   drwxr-x---  113 root     root        12288 Aug 14 14:55 root
   drwxr-xr-x    3 root     root         4096 Jul 12 16:40 sbin
   drwxrwxrwt    8 root     root         4096 Aug 14 15:21 tmp
   drwxr-xr-x   21 root     root         4096 Mar 13 17:07 usr
   drwxr-xr-x   18 root     root         4096 Jun 22 18:59 var

Notice that /proc does indeed have a zero size. Yet when I change to the /proc directory with cd /proc and do an ls there, I see tons of information.

# cd /proc
# ls
1 10102 10105 10311 10315 10702 10710 10712 10714 10715
10716 10717 10718 10719 10720 10721 10722 10723 10755 10766
1145 1146 1147 1148 1149 1150 1151 1152 2 3 330 345 346 355
369 4 420 429 443 446 447 449 450 461 475 490 5 508 528 583
6 612 626 675 715 716 717 718 719 720 723 745 752 753 757 
771 773 775 776 7922 808 811 8784 8786 8788 8789 8936 9008
9088 911 914 9957 apm bus cmdline cpuinfo devices dma fb
filesystems fs ide interrupts ioports kcore kmsg ksyms loadavg
locks mdstat meminfo misc modules mounts net partitions pci
rtc scsi self slabinfo sound stat swaps sys tty uptime version

For an empty directory, there's quite a bit here. Let's start with some easy stuff.

Oops. Before I move on, let me offer a word of WARNING!!! <begin weasel words> Unless you know exactly what you are doing, do not go changing the contents of your /proc filesystem. There are some pretty neat things you can do with several of the entries and you can tweak your actual running system by changing various things here, but be warned. <end weasel words> Ahem, where was I? Oh, yeah...

Easy stuff. Look at the file called "cpuinfo", for starters. Another zero-size file. Now, cat the file and have a look at the results. This is what mine shows:

     # cat /proc/cpuinfo
     processor  : 0
     vendor_id  : GenuineIntel
     cpu family : 5
     model              : 2
     model name : Pentium 75 - 200
     stepping   : 12
     cpu MHz            : 150.342254
     fdiv_bug   : no
     hlt_bug            : no
     sep_bug            : no
     f00f_bug   : yes
     coma_bug   : no
     fpu                : yes
     fpu_exception      : yes
     cpuid level        : 1
     wp         : yes
     flags              : fpu vme de pse tsc msr mce cx8
     bogomips   : 59.80

Actually it's a Pentium 150, but it is still kind of fascinating to read what my system thinks of itself. Let's see what happens when we look at the file called "interrupts". Guesses, anyone?

     # cat /proc/interrupts
                CPU0       
       0:    6422178          XT-PIC  timer
       1:      43093          XT-PIC  keyboard
       2:          0          XT-PIC  cascade
       5:          3          XT-PIC  soundblaster
       8:          1          XT-PIC  rtc
      10:      34143          XT-PIC  eth0
      12:     541052          XT-PIC  PS/2 Mouse
      13:          1          XT-PIC  fpu
      14:    1622951          XT-PIC  ide0
      15:          6          XT-PIC  ide1
     NMI:          0

Similarly, doing a cat on the meminfo file will give you current stats about your memory, both physical and swap. Looking into /proc/scsi will reveal your SCSI assignments, while a look at the sound file will show you your sound card configuration. A cat of your /proc/partitions file will display your disk partition information, and "version" is your running kernel and how it was compiled. Keep looking. There's a lot here, and it's balls of fun.

By now, you may have noticed I did not tell you the whole truth. When you did an ls -l on /proc, you actually found two non-zero-size files. One of them is called "kcore" and it can be a fairly good size for something that takes up no space. If you really want to look into the heart of your Linux system, listen closely; you can almost hear it beating when you look here. This is a "mirror" of sorts for your kernel. Perhaps the lens of a microscope would be a better analogy. Notice how the file takes up about as much space as you have physical memory. That is no coincidence. kcore is your system's memory - its RAM, if you prefer. In fact, it is everything currently in RAM, and as such, it is a dynamic beast. The same can be said for much of what you see in /proc. Looking at "meminfo" from one minute to the next will yield a different set of numbers as the demands of real memory vs swap are handled.

The other "big" file (though much smaller) is something called "self", and if you do an ls -l on /proc, you'll notice that self is actually a pointer to a number. Here's what I get when I do that ls -l right now:

   lr>wxrwxrwx    1 root   root     64 Aug 16 13:28 self -> 2183

The reason I asked you to ls -l the whole /proc directory rather than just "self" is this: I wanted you to see that the number (in my case, 2183) is also a directory under the /proc directory. Now, do another ls -l. It's gone, isn't it?

What about all those numbers? What are they for? Each one seems to be a directory. What will you find there? Do this: try a ps ax to get a list of the running processes on your system. Even better, do it this way:

     ps ax | cut -c1-5

That will get you just the process numbers without the long display. Now, compare that list to the numbers sitting in /proc. Look familiar? Every process currently running on your system has an analog sitting in the /proc filesystem. Let's have a look at what we can discover hidden in there. Just for fun, let's take one of the processes on my system. For this little exploring about, I am using a terminal emulator called Eterm. If you don't have it, get it from http://www.eterm.org. You'll thank me later. Sure, it's eye candy, but a little bit of eye candy isn't all bad, right? (There I go, wandering off again. Sorry.)

The process ID for one of my Eterm sessions is 834 (the result of ps ax | grep Eterm). If I cd to /proc/834 and do an ls -l, this is what I see:

total 0
-r--r--r--    1 mgagne   mgagne          0 Aug 16 13:59 cmdline
lrwx------    1 mgagne   mgagne          0 Aug 16 13:59 cwd ->
r/share/Eterm/themes/Eterm
-r--------    1 mgagne   mgagne          0 Aug 16 13:59 environ
lrwx------    1 mgagne   mgagne          0 Aug 16 13:59 exe ->
r/bin/Eterm-0.8.10
dr-x------    2 mgagne   mgagne          0 Aug 16 13:59 fd
pr--r--r--    1 mgagne   mgagne          0 Aug 16 13:59 maps
-rw-------    1 mgagne   mgagne          0 Aug 16 13:59 mem
lrwx------    1 mgagne   mgagne          0 Aug 16 13:59 root -> /
-r--r--r--    1 mgagne   mgagne          0 Aug 16 13:59 stat
-r--r--r--    1 mgagne   mgagne          0 Aug 16 13:59 statm
-r--r--r--    1 mgagne   mgagne          0 Aug 16 13:59 status

Once again, a number of these can be viewed with cat or more or less. The cmdline file shows the command that executed Eterm, while environ displays the environment variables at work here. Check out maps for a list of all the libraries at work with the current executable. Look inside that fd directory, and see if you can figure out what those numbers represent.

Sure, there are system commands that do a fine job of showing what is happening in /proc. The command top, for instance, is just such a snapshot for your processes. Files such as meminfo and pci can be graphically unwound with the help of KDE's kcontrol program. Just check out the Information tab. If you look long and hard enough, you will likely find a tool to display all this stuff in a nice, graphical format. In some ways, that's part of the reasoning behind the /proc filesystem: to give programmers/administrators/etc. an easier means of accessing core system information and modifying a running kernel. Get the tools later. Look around now. It's a kind of techno adventure, where the prize is a deeper understanding of what happens under your system's "hood". Yes, Bobby, that's the same hood that's not welded shut.

Did I just say modifying a running kernel? Hmmm... that sounds like a good place to stop and get you ready for something brand-new when next we meet here at the SysAdmin's Corner. (Ain't I a stinker?) Until next time, always remember: your Linux system is talking to you. Are you listening?