Recently at our local (Phoenix) Free Software Stammtisch, we were talking about security. I was surprised to find that no one else realized you can ssh safely across a compromised host to another machine behind it. A common example of “crossing” a host to get to another machine is accessing a machine on a private network protected by a firewall. The firewall is on the edge of the private network and provides the only means to get to the private network. So, you have to pass through the firewall to get to the private network.

But, what happens if the firewall has (or you believe it has) been compromised? How do you get to the private network without more security problems? It turns out that the answer is the same. You go through the firewall, but you do it in such a way that your connection remains secure even when the firewall itself may no longer be. The connection to the machine on the private network still can be made securely, even if you know the host you're passing through has been compromised and some rogue element has taken control of it.

As an example, let's say you need to connect from your Netbook to a server on your work's network. Let's also say the only way to get to your server is via a connection to your work's SSH/VPN gateway, but you think someone has broken into the gateway (yes, yes, get it off-line as soon as possible, but let's do that, and continue to be able to work and recover the gateway).

Now, let's consider an example scenario with three machines, named corresponding to what they are: Netbook, Gateway and Server. If you already understand SSH tunneling and want the short story, see the Short Story sidebar.

-L $local_port:Server:22

For the long story, let's start with a description of some simple tunneling. The -L option on the command line allows you to create a tunnel:

ssh -N -f -L 2222:localhost:22 Gateway

The -L option allows you to specify the entry and exit points for a secure tunnel created by SSH. In the examples used in this article, it gets an argument that has three fields. The fields are separated by colons. The first field is the local port to bind (the local port to listen on). The next field is the host to connect to. This field is interpreted by the remote machine in the SSH connection, not by the local machine. The computer initiating the SSH connection doesn't need to know how to get to it. The third field is the port to connect to on the far end of the tunnel.

If you run this first command from Netbook, the command creates an SSH connection to Gateway and builds a tunnel that forwards port 2222 on Netbook to port 22 (the standard SSH port) on Gateway. The local port, 2222, can be almost any value as long as it's not already in use, although it must be above 1023 if you're not connecting as root. Now you can connect to SSH on Gateway by using that tunnel. Again, it's important to note that the tunnel destination hostname (the second field) is interpreted by Gateway, so “localhost” is instructing Gateway to connect to itself. localhost is not Netbook; it's Gateway. For more examples of using -L to create tunnels, see the SSH -L Examples sidebar.

The -N and -f options are useful when just creating a tunnel and not needing a remote shell. The -N option tells SSH not to run a remote command. The -f option puts the ssh command in the background, so you can regain control of the shell that ran the ssh command.

ssh -L 8888:www.LuftHans.com:80 remote_host

ssh -L 9993:mail_server:993 remote_host

Now, run the following command on Netbook:

ssh -p 2222 localhost

This second command creates an SSH connection to port 22 on Gateway by using the tunnel created in the first command. In the second command, localhost is the destination machine, so it's interpreted by the SSH client on Netbook, which means the ssh command running on Netbook connects to Netbook but on a nonstandard port. The tunnel is entered at port 2222 on Netbook and comes out at port 22 on Gateway. Because sshd is listening on port 22, the tunnel connects to the SSH dæmon on Gateway.

Presuming Gateway can connect to Server even though Netbook can't (remember Server is “firewalled” behind Gateway), it's also possible to create a tunnel to Server using a command such as:

ssh -a -N -f -L 3333:Server:22 Gateway

In this case, the tunnel from port 3333 on Netbook doesn't connect to Gateway, rather it connects to port 22 on Server. Gateway is essentially forwarding packets from Netbook to Server. Like the first command, the tunnel destination (the second field, Server) is interpreted by Gateway, so this tunnel connects to Server. Netbook doesn't need to know how to get to Server; Gateway handles the routing.

The -a option here makes sure authentication agent connections are not forwarded to Gateway. If you are concerned that Gateway is compromised, you don't want the attacker to gain control of your authentication agent connections. For more, see the Authentication Agents sidebar.

ssh -A hostB

Figure 1 displays the example graphically. The initial ssh command builds the SSH connection for the tunnel, the tunneled connection and the forwarded connection. The second ssh command initiates a new connection through the tunnel and forwards the connection directly to the machine you are trying to reach. The forwarded connection is in red to show that it is unencrypted, but the blue SSH connection going through it is secure.

Figure 1. Tunneling across a Compromised Host (Brian Cluff, LJ2009@Macrosift.com, created the image for this article.)

This tunnel could work for any TCP protocol, but the packets from Gateway to Server and back to Gateway are unencrypted. That means unencrypted services would not be secure between Gateway and Server, even though you're using an SSH tunnel. You may want to review the -L option section of the SSH man page and consider this a bit to convince yourself that the Server/Gateway connection (the red part in Figure 1) is unencrypted. Anything that isn't secure or that gets decrypted on Gateway can be read by the attacker if the attacker has root access on Gateway. The attacker also can read much of that simply by having access to your account on Gateway even without root access on Gateway.

Before connecting to the tunnel, you need to make sure you have Server's host public key registered as a key for localhost. If you tried to pull the public key out of Gateway's known_hosts file, the attacker can give you a bogus key, so you need to get the key another way.

Therefore, it's best to have Server's public key already. If you don't already have the public key, make sure to acquire it or the fingerprint for the key from a secure, trusted source. For instance, you could list the SSH key fingerprints for all of your servers on a secure Web page.

I suggest registering Server's public key in your known_hosts file under the real server name as well as under localhost. By registering under the real server name, you can figure out to which server a key belongs with the ssh-keygen command.

The following command checks your $HOME/.ssh/known_hosts file for a public key for Server. The command also reports the line number for the entry:

ssh-keygen -F Server

The entry then can be copied. By replacing the hostname portion of the copied entry with localhost, you create an entry for that key on localhost. See the Obtaining the Public Key sidebar for information on how to obtain a server's public key securely.

If your known_hosts file has hostnames that look like the keys, you have hashed hostnames. Use the -H option to ssh-keygen to hash your new localhost entry. The following command translates plain-text hostnames in your known_hosts file to hashed hostname entries:

ssh-keygen -H

Now with the tunnel set up, you can connect to SSH on Server, and because it's an SSH connection, it's encrypted:

ssh -p 3333 localhost

Again, localhost here is from Netbook's perspective, so that command connects to port 3333 on Netbook. The last SSH “tunnel” command created a tunnel from that port over to the SSH port on Server via Gateway—meaning that this command uses the tunnel to connect to Server's SSH server via port 3333 on Netbook.

Even though the tunnel passes through Gateway, it is unreadable by Gateway. Just as SSH connections normally travel across untrustable routers, yet are secure, the connection through Gateway to Server is secure. The SSH connection via the tunnel is never unencrypted on Gateway, so the attacker can't compromise it. But again, remember to have verified the host key for Server already, or the attacker can compromise the connection with a man-in-the-middle attack.

With the connection to Server, you now could set up a SOCKS proxy to allow applications running on Netbook to be effectively on your work's network. See the SOCKS Proxy sidebar for more information.

ssh -p 3333 -D 1080 localhost

If Netbook has a public IP, you also can set up a reverse tunnel from Server to Netbook. The reverse tunnel allows you to connect to Netbook from Server and then connect back to Server via the reverse tunnel. If the reverse tunnel doesn't need to go through Gateway, you then could take Gateway down for investigation and repair while still being connected to the internal network. See the Reverse SSH Tunnel sidebar for more information about reverse tunnels.

Connect to your possibly compromised machine, Gateway, and create a tunnel to the machine you ultimately want to reach, Server. Once the tunnel is up, use it to connect to SSH on Server:

ssh -a -N -f -L 3333:Server:22 Gateway
ssh -P 3333 localhost

Reminders:

Any SSH connection across the Internet is crossing questionable hosts/routers. If those were safe, you wouldn't need a secured connection. The tunnel is the same scenario, because it's just enabling a normal SSH connection.