Every year or so, I like to write a column I title “Lightning Hacks”. This column is inspired by the lightning talks common at most conferences. In a lightning talk, instead of having one speaker give a 60-minute presentation, multiple speakers give short 5–10-minute presentations. By the end of a lightning talk, you end up hearing about all sorts of cool topics that wouldn't have gotten their own time slot. In this column, I get a chance to talk about a few cool “hacks” I've run across that wouldn't fill an entire column by themselves.

In prior Lightning Hacks columns, I've covered a number of different topics, but this time I've decided to focus on only one: SSH. Like many system administrators, I spend a great deal of my day within SSH sessions, and over the years, I've found a few shortcuts and handy tips that I save in shell scripts so I don't forget them.

This first hack seems really simple—after all, I am adding only one extra flag to SSH. Normally, if you want to ssh into a machine and run a program, you simply pass the program at the end of your SSH command:

$ ssh user@remotehost.example.org df

Yet, if you ever have tried to write a shell script that would automatically ssh in to a remote machine and launch mutt or screen or similar programs, you have seen the session either sit there or exit with some message like “Must be connected to a terminal.” I ran into this problem on my N900 palmtop when I wanted to launch two special terminal sessions: one that automatically reconnected to a remote screen session and another that loaded mutt. Yeah, that's right. I still prefer mutt and irssi, even on a palmtop. Neither worked though until I added the -t flag:

$ ssh -t user@remotehost.example.org screen -dr
$ ssh -t user@remotehost.example.org mutt

The first example connects to the remote host and re-attaches my remote screen session (I run only a single screen session on my host and then use Ctrl-a c to create windows within that session). The second example simply runs mutt. The -t flag forces pseudo-tty allocation. It turns out that when you run programs like screen or mutt, you need to force SSH to create a pseudo-tty.

I know a million articles have been written about SSH tunneling, but this particular type of tunneling is so useful; however, I use it infrequently and forget the proper syntax. A problem you often may run into is needing to scp a large number of files between two servers (let's say londonweb1 and seattleweb1), but for some reason, the two machines are firewalled off from each other. Usually, you have one server that is able to ssh into both machines (let's call that server admin1), and if it were just one or two files that needed to be transferred, you could copy the files first from londonweb1 to admin1, then from admin1 to seattleweb1.

When you need to transfer multiple files (or perhaps pipe dd traffic) between the two sites, it can be impractical, if not impossible, to move data to an intermediary server first. That's where SSH reverse tunnels come in handy. With a reverse tunnel, you launch an SSH session from your intermediary server (admin1 in this case) to the first server (londonweb1) and open up a local high port that is unused, such as 2222. Then, you tell SSH to tunnel all traffic on that port over to the remote server (seattleweb1). Once the tunnel is set up, you can use scp as you normally would, except you point it to localhost port 2222.

To set up the tunnel, I would run the following command from admin1:

kyle@admin1:~$ ssh -R 2222:seattleweb1:22 londonweb1

The arguments to -R can be easy to mix up. Note that the last server in the command (londonweb1) is the server to which I log in. The first argument to -R is the port to open up on that server (2222). The next two arguments list to which server and port to forward any traffic (seattleweb1 and 22, respectively).

Once I log in to londonweb1, I can use scp (or rsync) like I normally would, but I point it to localhost port 2222:

kyle@londonweb1:~$ scp -r -P 2222 /var/www/mysite localhost:/var/www/

When I initiate this scp command, all the traffic enters the tunnel and goes to admin1, and then from there, it is forwarded to port 22 on seattleweb1. Keep in mind that this means if these machines are far apart, your bottleneck will be the slowest link between the servers.

If you are a security-minded individual in charge of a network, you may not like how easy it is to route around your basic firewall rules. It's important to realize that reverse tunnels also can be used to connect from inside your network to a person's home machine, so even with incoming firewall rules set, a user still could tunnel in.

A lesser-known feature of SSH is that you can enter an internal command-line mode in an existing session and add extra tunnels. Let's say you already have an SSH session open from admin1 to londonweb1, and now you want to add the reverse tunnel without having to log out. First, press ~C (that's the ~ character and then a capital C) to open the SSH command line. Then, you can add extra port-forwarding commands as though they were part of the original SSH command line. When you are done, simply press Enter to return to the regular shell:

kyle@londonweb1:~$ 
ssh> -R 2222:seattleweb1:22
Forwarding port.

kyle@londonweb1:~$

This also could be useful if you use regular SSH tunnels (the -L option) as a poor-man's VPN and realize that, for instance, you need to set up an extra VNC or RDP tunnel to a new server. When you use the SSH command line, you won't have to close and break any existing sessions you have.