This column is the second in a series dedicated to one of my favorite subjects: troubleshooting. Because my column is generally aimed more at tips and tricks and less on philosophy and design, I'm not going to talk much about overall approaches to problem solving. Instead, in this series, I describe some general classes of problems you might find on a Linux system, and then I discuss how to use common tools, most of which probably already are on your system, to isolate and resolve each class of problem.

In the first column, I talked about how to diagnose high-load issues on a server, but the fact is that these days, just about every Linux computer is connected to a network, and a large number of the problems you have are based in the network. This month, I focus on local network troubleshooting, and although I am writing from the perspective of servers, most of these steps will apply to any Linux machine on a network. Also, because the goal of this article is to show how to become better at troubleshooting, I list each step from the lowest level on up. In real life, I'd probably skip ahead here and there to make the troubleshooting process faster.

The generic problem I cover here is how to track down the root cause when one machine can't communicate with another machine on the same network. For this example, let's assume I have two servers named bill and shawn. The server shawn is trying to communicate with bill over port 25 (port 25 is used for sending e-mail over SMTP), but wouldn't you know it, bill isn't responding.

One of the first things I might do in a scenario like this is find another machine on the same network and try to connect with bill from there. If I can talk to bill from another machine on the same network, the problem is most likely with shawn or with the network in between shawn and bill. If I have the same problem from another machine on the same network, it's more likely that the problem is with bill, so I would start troubleshooting from there. Just so I can discuss more troubleshooting steps, let's start troubleshooting from shawn.

One of the most embarrassing things in troubleshooting is to waste an hour only to find out that something wasn't plugged in. So the first step I perform is to make sure that shawn is plugged in to the network. Although I could inspect the port physically on the server, if the server were in a different city, I might run a program like ethtool. ethtool gives you a lot of different diagnostics on your Ethernet devices. By default, all you have to do is run ethtool as root and pass the Ethernet device you want to check as an argument. In many cases this will be eth0:

$ sudo ethtool eth0
Settings for eth0:
     Supported ports: [ TP ]
     Supported link modes:   10baseT/Half 10baseT/Full 
                               100baseT/Half 100baseT/Full 
                               1000baseT/Half 1000baseT/Full 
     Supports auto-negotiation: Yes
     Advertised link modes:  10baseT/Half 10baseT/Full 
                               100baseT/Half 100baseT/Full 
                               1000baseT/Half 1000baseT/Full 
     Advertised auto-negotiation: Yes
     Speed: 100Mb/s
     Duplex: Full
     Port: Twisted Pair
     PHYAD: 0
     Transceiver: internal
     Auto-negotiation: on
     Supports Wake-on: pg
     Wake-on: d
     Current message level: 0x000000ff (255)
     Link detected: yes

As you can see, ethtool gives all sorts of information, including the fact that this machine supports 10 base T, 100 base T and gigabit networking speeds, but it currently communicates at 100 base T, full duplex. To check for a link, just look at the very last line that says “Link detected”. As you can see in my example, link is detected, so my cable is plugged in and I can move on.

Before I move past ethtool completely, it's worth mentioning that it does a lot more than just diagnose link problems. A common problem I've found on networks is a host with slower-than-normal network speeds. Often you'll see this crop up after a reboot or a power outage. What often happens is that when the interface connects to the network, it will try to auto-negotiate the fastest speed it can. Sometimes auto-negotiation doesn't work correctly, in which case the interface might fail back to half duplex mode or might even fail back to 10 base T! If you know that your network can support 100 base T at full duplex, you can use ethtool to disable auto-negotiation and force full duplex. To do this for eth0, you would type:

$ sudo ethtool -s eth0 autoneg off duplex full

After we have confirmed that shawn is plugged in, the next step is to confirm that eth0 on shawn is configured correctly. To do that, I would use the ifconfig command with eth0 as an argument. I should get back all of the network information I need to determine whether eth0 is set up correctly on shawn:

$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:17:42:c0:ff:ee  
          inet addr:10.1.1.9  Bcast:10.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::217:42ff:fe1f:18be/64 Scope:Link
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:1 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:229 (229.0 B)  TX bytes:2178 (2.1 KB)

There is a lot of output in that command, but the first line I would look at is the second line of output. There I can see that eth0's IP address is 10.1.1.9 and that its subnet mask is 255.255.255.0. If the machine were supposed to have a different IP or subnet mask from what I see here, that potentially could be the cause of the problem. If eth0 didn't have an IP or subnet mask configured at all, I might run ifup eth0 to bring up the interface, or I might look into the local network settings (/etc/network/interfaces on a Debian or Ubuntu machine, /etc/sysconfig/network-scripts/ifcfg-eth0 on a Red Hat-based machine) to see if anything is set incorrectly. If I can't seem to get the interface to come up, and this host gets its IP from DHCP, I might have to move my troubleshooting focus to the DHCP server.

After you have confirmed that the interface is on the network and should be able to communicate, the next step is to test whether you can access another host on the same subnet—specifically the gateway if you have one configured. Why? Well, if you can't talk to a host on the same subnet, especially if you can't talk to the gateway, there's no point in testing communications with hosts outside of your local subnet. First, I will use the route command to see what gateway is configured, and then I will use ping to see whether I can access the gateway:

$ sudo route -n
Kernel IP routing table
Destination  Gateway   Genmask         Flags Metric Ref  Use Iface
10.1.1.0     *          255.255.255.0   U     0      0     0 eth0
default      10.1.1.1  0.0.0.0          UG    100    0     0 eth0

In this example, I have a very basic routing table, and the line that begins with the word default defines my default gateway: 10.1.1.1. Be sure to use the -n option with route in this step. Without the -n option, route will try to resolve any IP addresses it lists into hostnames. Besides the fact that route will execute faster with -n, if you have network problems, you might not even be able to talk to your DNS server, plus DNS troubleshooting is a topic for another column.

Because I see that the gateway is 10.1.1.1, I would use the ping command to confirm that I can communicate with that gateway:

$ ping -c 5 10.1.1.1
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_seq=1 ttl=64 time=3.13 ms
64 bytes from 10.1.1.1: icmp_seq=2 ttl=64 time=1.43 ms
64 bytes from 10.1.1.1: icmp_seq=3 ttl=64 time=1.79 ms
64 bytes from 10.1.1.1: icmp_seq=5 ttl=64 time=1.50 ms

--- 10.1.1.1 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4020ms
rtt min/avg/max/mdev = 1.436/1.966/3.132/0.686 ms

This output tells me that my machine can at least talk with the gateway and presumably with the rest of the 10.1.1.x network. Now, if I couldn't talk to the gateway, that could mean my network administrator is being annoying and blocking ICMP packets. If that's the case, I would just choose another machine on the same subnet (10.1.1.2-10.1.1.254) and try to ping it instead. If I am the network administrator (and therefore not blocking ICMP), or if ICMP isn't being blocked for some other reason, the problem at this phase could be some sort of VLAN issue that I would have to resolve on the network switch itself.

If you run the route command and don't find a default gateway set, you might be tempted to conclude that's the source of the problem. Be careful! That conclusion might be premature. See, if shawn and bill are on the same subnet, I don't need a default gateway configured for those servers to communicate. I'm not going to get into how to calculate subnets in this column, but suffice it to say in my example, if shawn has an IP of 10.1.1.9 and a subnet mask of 255.255.255.0, bill could have an IP of 10.1.1.1 through 10.1.1.254 and be on the same subnet. In that case, I might just ping bill directly. Ideally, I would have a third host on the same subnet I also could ping. That way if bill doesn't respond, but another host on the same subnet responds, I can narrow in on bill as the likely source of the problem.

If bill is responding to ping, the next step is to test whether port 25 is even open on bill. There are a few different methods for doing this, but telnet is one of the easiest and is likely already to be installed on your machines. Let's assume bill has an IP of 10.1.1.17; I would type:

$ telnet 10.1.1.17 25
Trying 10.1.1.17...
telnet: Unable to connect to remote host: Connection refused

If telnet doesn't complain about Connection refused, but instead starts outputting SMTP commands, then congratulations, you don't have a networking problem! On the downside, this means you probably have some sort of SMTP problem, which might be more of a pain to troubleshoot. If telnet complains with Connection refused, either port 25 is down on the remote machine (possibly the SMTP service on bill isn't running or isn't listening on that port), or a firewall is blocking you. This is where a tool like nmap can be handy, and it's one of the reasons I often use nmap instead of telnet when I want to test whether a port is available.

You see, many firewalls are configured to block ports by dropping packets with no reply. Because normally a server would send a basic reply back to let you know the port is closed, if the packet is dropped instead, nmap will flag it as filtered instead of closed:

$ nmap -p 25 10.1.1.17

Starting Nmap 5.00 ( http://nmap.org ) at 2010-01-04 20:20 PST
Interesting ports on 10.1.1.17:
PORT   STATE  SERVICE
25/tcp filtered smtp

In this case, nmap says the port is filtered, which tells me there is a firewall blocking this port. If these machines were on different subnets, there might be a firewall in between the networks restricting access. Because I know these machines are on the same subnet, I would assume that there is some iptables firewall configured on bill that needs to be checked.

Let's assume we think the problem is on bill. After I've performed the same network troubleshooting on bill that I have on shawn, the next step is to log in to bill and test whether port 25 is open and listening for connections. For this, I will use the netstat tool. netstat can be used to output all sorts of information about network connections on the machine. In this case though, I will just use the -lnp options to list listening ports and the processes that have the ports open, then I will grep for the port I'm interested in, port 25:

$ sudo netstat -lnp | grep :25
tcp        0      0 0.0.0.0:25    0.0.0.0:*     LISTEN   1878/master

The column I want to pay the most attention to here is the fourth column that lists what local address is open on port 25. In this case, I can see it is set to 0.0.0.0:25, which means bill is listening to port 25 connections on all available interfaces. If I had set up the mail server to listen only on eth0, this would be set to 10.1.1.17:25. If, on the other hand, I saw this was set to 127.0.0.1:25, I might have found the cause of the problem: the mail server was set to listen only to the localhost address (127.0.0.1) and isn't listening for any connections from the outside network. In that situation, I would reconfigure my mail server so that it listens on eth0. If I got no output from the above command, I would know my problem is that my server isn't running at all (or isn't set to listen on port 25). Then, I'd need to start my mail server and troubleshoot why it stopped running to begin with, or why it isn't listening on the right port.

As you can see, network troubleshooting can lead you in all sorts of interesting directions. Even now I've barely scratched the surface. In my next column, I'll extend network troubleshooting beyond the local network and touch on how to track down routing and DNS problems from your local networks to the Internet itself.