How would you find out how much RAM is free on your Linux desktop? That's a really easy question with a lot of answers—free, any of the implementations of top and Glances all are valid responses. How would you find out how much RAM is free on 200 Linux instances, which are running on a mixture of real and virtual hardware, in dozens of physical locations spread out around the globe? That's a much bigger problem, and there is a tool to make the job easier. However, the lack of upkeep on the standards and lack of development support for the Linux implementation are resulting in proprietary standards creeping in where there once was a more open standard.

SNMP (Simple Network Management Protocol) was designed in 1990 to read and write structured data on devices attached to a network, such as how much free RAM there is. Yes, and this is important, the M in SNMP really does stand for "Management", not "Monitoring". Although SNMP is usually used to request operational status information, the SNMP "write" functionality can be used to change the configuration on remote devices. Given the lack of security and authentication in the SNMP protocol, SNMP "write" functionality almost always is disabled on the modern internet, and I will not be discussing it here.

History of SNMP

The original IETF (Internet Engineering Task Force) RFC (Request for Comments) standard for SNMP v1 was published by the IETF in 1990. SNMP v2 was published in 1994–1996 as a series of RFCs and included the first effort to secure SNMP. This effort proved unpopular due to the load it placed on network hardware, which, at the time, had very low performance CPUs. This performance issue exists today and still can cause problems for administrators attempting to secure SNMP. Due to the performance problems, SNMP v2c (SNMP v2 with SNMP v1 communities) became the standard. Concurrently with the release of SNMPv2c, the public began to access the internet, and during the next decade, security would become a serious problem with SNMP since SNMP v2c was entirely unencrypted. SNMPv3 came along in 2003 and added TLS to the previous implementation of SNMP v2c. If all of this seems a bit complicated and unnecessary, it's important to know that many implementations of SNMP still ship with support for SNMP v1, v2c and SNMP v3. This means you're likely to see all of them in the wild.

How Is SNMP Used?

One of the challenges on a modern network is scale, and achieving scale requires managing resources. SNMP provides an agent, which listens for incoming SNMP requests on each host, and a standard communications protocol allowing a central collection system called a Network Management System (NMS) to collect data. NMS is outside the scope of this article, but there are many good open-source NMSes, including Zabbix, OpenNMS, Nagios and Zenoss. The data collected by each NMS is pretty standard, and it includes basic systems information like CPU, memory, network and storage utilization.

SNMP Data Structure

SNMP isn't just an agent, it's also a data structure. Each object in the data structure has an Object IDentifier, or OID. Each OID belongs to an MIB, or Management Information Base. These object identifiers and the hierarchical structure function as a tree. Each sequential number is a branch and has a meaning, and each branch is separated by periods (.), somewhat like an IPv4 address. This means that the meaning of an OID can be decoded very simply.

Given an example OID, 1.3.6.1.2.1.1.1.0, each number has the following meaning:

• 1 = iso

• 3 = org

• 6 = dod

• 1 = internet

• 2 = IETF Management

• 1.1 = SNMP MIB-2 System

• 0 = sysDescr

From the decoded values, it can be determined that this OID is from the IETF standard MIB (more on MIBs later in the article), and it provides a system description of some sort. Let's look at a real-world example from a CentOS 6 box:

1.3.6.1.2.1.1.1.0 = STRING: "Linux foo.example.lan
 ↪2.6.32-573.1.1.v6.i686 #1 SMP Fri Aug 21 14:37:07 MDT 2015 i686"

From this description, you can determine that the system this agent is running on is running Linux, 2.6.32, and is 32-bit.

Nearly every OID starts with "1.3.6.1", and the reason for this should be obvious. The modern public internet originally was created by the United States Department of Defense, and at one time, TCP/IP was called the "DOD Model". Since these values are in every OID, they aren't all that useful for identifying what that OID does, and they generally can be ignored.

After 1.3.6.1, there are more types of OID. If the MIB continues with 1.2, as with the example above, the description of the OID can be found in the standard IETF MIB. If it continues with 1.4, the MIB is "private", and you will need to get the MIB from your hardware vendor. Despite being called "private", these MIBs are almost always available.

What Types of OIDs Are There and How Is Each Used?

There are many different types of OIDs so that SNMP can provide an extensive and extensible variety of information. The example from the previous section, 1.3.6.1.2.1.1.1.0, is a STRING. You can tell because SNMP tells you the type of OID when you retrieve it:

1.3.6.1.2.1.1.1.0 = STRING: "Linux foo.example.lan
 ↪2.6.32-573.1.1.v6.i686 #1 SMP Fri Aug 21 14:37:07 MDT 2015 i686"

Other types of OIDs exist, and each has a use. The following is a list of common types of SNMP OID:

• Integer/Integer32: signed 32-bit integer—these are commonly used for storing values, such as the amount of available memory and the amount of free memory.

• Uinteger32: unsigned 32-bit integer (fairly rare).

• Octet String: this is a short (255-character) length of binary or text data.

• IP Address: this returns an IP address.

• Counter32: this returns a 32-bit counter that counts up, then wraps around to 0 when it reaches 32 bits in length minus 1 (4294967295). This is important, because gigabit Ethernet can send far more than that many bits in five minutes, which is a common NMS polling period.

• Counter64: this has a maximum value of 64 bits – 1, which allows for higher speed Ethernet traffic counting and counting of other large numbers.

• Object Identifier: this returns a different OID and functions like a GOTO, if that data is in another MIB.

• Bit String: this is the type of string above, and it returns text information.

• Gauge32: this goes up and down, but it never exceeds a maximum value.

• TimeTicks: represents an unsigned integer of time since another time (often used for uptime).

What Is an MIB, and Isn't a Name Better Than a Bunch of Numbers Anyway?

Earlier I looked at an OID with the ID 1.3.6.1.2.1.1.1.0. It's a pain to remember that every single time a system description is required. The good news is that SNMP avoids having to memorize or even deal with long strings of numbers by using Management Information Bases, or MIBs. MIBs decode the OID's purpose for you, so you don't have to remember all the values.

By installing MIBs, the previous difficult-to-read output:

1.3.6.1.2.1.1.1.0 = STRING: "Linux foo.example.lan
↪2.6.32-573.1.1.v6.i686 #1 SMP Fri Aug 21 14:37:07 MDT 2015 i686"

becomes much easier to read:

SNMPv2-MIB::sysDescr.0 = STRING: Linux foo.example.lan
 ↪2.6.32-573.1.1.v6.i686 #1 SMP Fri Aug 21 14:37:07 MDT 2015 i686

The quotation marks also disappear. The MIB not only translates the OID, but the value as well. The MIB already knows that that OID is a string, so the quotation marks go away.

How do MIBs know how to do this? MIBs are human-readable plain-text files, often found in /usr/share/snmp/mibs. For sysDescr, the SNMP client looks up the value in the SNMPv2 MIBs and is able to learn the type of OID, the purpose of the OID and whether it can be written to (from NET-SNMP's SNMPv2-MIB.txt):

sysDescr OBJECT-TYPE
    SYNTAX      DisplayString (SIZE (0..255))
    MAX-ACCESS  read-only
    STATUS      current
    DESCRIPTION
        "A textual description of the entity. This value should
        include the full name and version identification of
        the system's hardware type, software operating-system,
        and networking software."
    ::= { system 1 }

How Does SNMP v1/v2c Work in Linux?

Getting started with SNMP v1 and v2c in Linux is quite simple. The information will be transmitted in plain text, including the SNMP "Community", which is sort of like a password. Using your package manager, install net-snmp. Edit /etc/snmp/snmpd.conf, remove everything in the file, add the following lines, then save and exit:

rocommunity public
syslocation Somewhere (In the World)
syscontact Overworked Admin <admin@paymemore.com>

Restart snmpd, run the following command from the same system, and you'll again see the example OID this article has used since the beginning:

[user@foo mibs]$ snmpget -v2c -c public localhost SNMPv2-MIB::sysDescr.0
SNMPv2-MIB::sysDescr.0 = STRING: Linux foo.example.lan
 ↪2.6.32-573.1.1.v6.i686 #1 SMP Fri Aug 21 14:37:07 MDT 2015 i686

If you don't know the specific OID you're looking for, you can use snmpwalk, which will "walk" the entire MIB and print the value for each OID. This tends to produce a lot of output, and you can shorten it with head:

[user@foo mibs]$ snmpwalk -v2c -c public localhost | head
SNMPv2-MIB::sysDescr.0 = STRING: Linux foo.example.lan
 ↪2.6.32-573.1.1.v6.i686 #1 SMP Fri Aug 21 14:37:07 MDT 2015 i686
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (154) 0:00:01.54
SNMPv2-MIB::sysContact.0 = STRING: Overworked Admin <overworked@admin.com>
SNMPv2-MIB::sysName.0 = STRING: foo.example.lan
SNMPv2-MIB::sysLocation.0 = STRING: Somewhere out there

As snmpwalk runs, sysDescr.0 shows up again, then another OID called SysObjectID, which refers to yet another OID, NET-SNMP-MIB::netSnmpAgentOIDs.10. snmpwalk will look up that OID and display its type and value before continuing through the rest of the SNMPv2-MIB tree.

A lot of the information that SNMP can provide is very sensitive, and it really shouldn't be transferred over the LAN or, worse, the public internet unencrypted.

How Does SNMPv3 Work in Linux?

SNMPv3 is very complex compared to SNMPv2, and it requires several steps to set up. If you're curious about your Linux router at home, the above SNMPv2 example probably will suffice, but in almost any other environment, SNMPv3 is a must. To set it up, first create a read-only SNMPv3 user name, with a local password that is encrypted with SHA and that uses AES. This is more secure than the default values of MD5 and DES, but it's still far from perfect (both MD5 and DES can be broken trivially):

[root@foo mibs] # service snmpd stop
Stopping snmpd:                                     [  OK  ]
[root@foo mibs] # net-snmp-create-v3-user -ro -A snmpv3authPass
 ↪-a SHA -X userpass -x AES user
adding the following line to /var/lib/net-snmp/snmpd.conf:
    createUser user SHA "snmpv3authPass" AES userpass
adding the following line to /etc/snmp/snmpd.conf:
    rouser user

Now edit /etc/snmp/snmpd.conf as root, and comment out the rocommunity line you added earlier:

#rocommunity public

Restart snmpd, and run snmpwalk with your new SNMPv3 credentials:

[user@foo mibs]$ snmpwalk -u user -A snmpv3authPass -a SHA -X
 ↪userpass -x AES -l authPriv 127.0.0.1 -v3 | head
SNMPv2-MIB::sysDescr.0 = STRING: Linux clearos65.trelane.lan
 ↪2.6.32-573.1.1.v6.i686 #1 SMP Fri Aug 21 14:37:07 MDT 2015 i686
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (12756) 0:02:07.56
SNMPv2-MIB::sysContact.0 = STRING: Overworked Admin
 ↪<overworked@admin.com>
SNMPv2-MIB::sysName.0 = STRING: clearos65.trelane.lan
SNMPv2-MIB::sysLocation.0 = STRING: Somewhere out there
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (42) 0:00:00.42
SNMPv2-MIB::sysORID.1 = OID: SNMP-MPD-MIB::snmpMPDMIBObjects.3.1.1
SNMPv2-MIB::sysORID.2 = OID: SNMP-USER-BASED-SM-MIB::usmMIBCompliance
SNMPv2-MIB::sysORID.3 = OID:
 ↪SNMP-FRAMEWORK-MIB::snmpFrameworkMIBCompliance

You'll notice that you still get the same information, but now it's being transferred via user name/password authentication and 128-bit AES. If you try again with SNMPv2, you'll get a timeout now:

[user@foo mibs]$snmpwalk -v2c -c public localhost
Timeout: No Response from localhost

This is the most secure SNMP agent configuration currently possible. To increase security, the SNMP port should be firewalled to accept only connections from your NMS.

What Is Happening to SNMP Due to the Lack of Implemented Updates to the Standard Since 2004?

Despite being widely adopted, important and incredibly flexible, SNMP is falling by the wayside. SNMPS, SNMP datagrams over TLS, standardized in 2010, has gone mostly unimplemented. SNMPv3 is difficult to use and troubleshoot on devices other than Linux. Microsoft has dropped SNMP support entirely from Windows, replacing it with WMI and then WinRM. Other vendors, and products providing monitoring interfaces, are using an often proprietary API over HTTPS or, worse, unencrypted HTTP that listens to and replies in JSON (Javascript Object Notation) or XML. This balkanization from a single standard has made cohesive monitoring of large networks containing diverse devices more difficult and time consuming.

Linux's NET-SNMP is in even worse shape. With only two contributors since January 1, 2015 (both from VMware) and one project manager, there have been fewer than 30 commits since January 1, 2015. The last stable release of NET-SNMP was in 2014. NET-SNMP has not implemented SNMPS, and there are no apparent plans to do so, leaving that standard dead in the water.

Returning a Value

It's unfortunate to see a standard balkanized into hundreds of different proprietary implementations. This wastes time and money and causes aggravation for systems administrators. SNMP and the various standards-compliant implementations of SNMP are still relevant and are in production nearly everywhere. That ubiquity and relevance is on the brink of changing as replacements utilizing proprietary data structures in JSON and XML instead of MIBs and OIDs begin to take over. NET-SNMP and the SNMP standard itself seem to be solid candidates for a rescue. Expanding the IETF MIBs to support newer networked devices, like network attached storage, storage area networks, software-defined networking, containers, cloud, converged and hyper-converged infrastructure will be a must if SNMP is to continue to be relevant.