Introduction

OpenBSD has recently stressed to us the value of key rotation by their use of “Signify” distribution release signatures. We have realized that SSH keys should also rotate, to reduce the risk of powerful keys that fall into the wrong hands which become “the gift that keeps on giving.” There have always been open questions on the retirement of SSH keys. These questions have grown in volume and many are joining the advocacy for SSH certificate authorities.

To “rotate” an SSH key is to replace it, in such a way that it is no longer recognized, requiring removal from the authorized_keys file. SSH rotation is commonly addressed with Ansible, but this leaves many users on smaller systems or lacking privilege without recourse. A more basic and accessible method to migrate SSH keys is sorely lacking.

Below is presented an SSH key rotation script written in nothing more than the POSIX shell.

There is palpable danger in the misuse of such a tool. Many administrators control inaccessible systems that entail massive inconvenience in a loss of control. Demonstrated here are rotation schemes of increasing risk, for any holder of a key to choose, to their own tolerance. Hopefully, I have not made grave mistakes in the design.

The most conservative users of this approach should tread with extreme caution, test carefully, and ensure alternate means of access prior to any deployment. As the author, I have no desire to assume any responsibility for a failed rotation, and its consequences. I especially disavow the “wipe” option below to remove entries from authorized_keys. It is presented as commentary, not working code.

In any case, we foolishly rush in where the more prudent fear to tread.

SSH Rotation Script

Below, an SSH key rotation script is presented. It is designed to be used in several phases, as keys are sent, tested, remotely wiped, and migrated. It is intentionally prone to error, brittle, and quick to terminate. It will immediately fail if an ssh-agent is not running (if you are not familiar with agent usage, then you are not ready for SSH key rotation). The agent should hold a working key for all the target hosts before the script is called (this can be the key targeted for rotation, but not required). It will verify the password of a new distribution key on every run, partly so the user takes a final pause before potentially irrevocable action. The script below is the complete deployment solution for the management of SSH authentication key rotation (host key rotation is not addressed).

There are a few critical points to consider before we begin:

• Rotated keys must be removed from the authorized_keys file on all target hosts. There is no point in generating and deploying new keys when those that are replaced are still honored for authentication.

• New keys generated for rotation should have different passwords from any previous keys. If a private key falls into the hands of an adversary, a successful password crack should not compromise newer keys.

With these thoughts in mind, here is the script:

$ cat ssh-rotate
#!/bin/dash
 
alias p=printf
 
usage () { p "%s - SSH key rotation
  -c             - Use ssh-copy-id to distribute SSH public keys of any type
  -d             - Use ssh-copy-id in dry-run mode
  -e             - Edit/copy existing ed25519 entry to a new key
  -h hostfile    - Target hosts [host port username]
  -k keyfile.pub - Deploy a specific public key
  -m             - Migrate id_ed25519 keypair - archive old, place new
  -r rounds      - Set rounds for auto-generated ed25519
  -w             - Wipe existing id_ed25519.pub from authorized_keys
" "$0"; }
 
err () { usage; p '\nerror:'; for x; do p ' %s' "$x"; done; p '\n'; exit; }
 
[ -z "$SSH_AUTH_SOCK" -o -z "$SSH_AGENT_PID" ] && err 'no agent'
 
set -eu; unset IFS  # http://redsymbol.net/articles/unofficial-bash-strict-mode/
copyid= dryrun= edit= hf= kprv= migrate= rounds=100 wipe= format=\
'=%s=%s=%d====================================================================='
 
while getopts cdeh:k:mr:uw arg
do case "$arg" in
        c) copyid=1                ;;
        d) dryrun=1                ;;
        e) edit=1                  ;;
        h) hf="$OPTARG"            ;;
        k) kprv="${OPTARG%[.]pub}" ;;
        m) migrate=1               ;;
        r) rounds="$OPTARG"        ;;
        u) usage; exit             ;;
        w) wipe=1                  ;;
   esac
done
 
[ "$wipe" -a -z "$kprv" ] && err 'You must specify a key to wipe'
[ -z "$hf"   ]  && err 'no hosts'
[ -z "$kprv" ]  && kprv=~/.ssh/id_ed25519_pre$(date +%Y%m%d); kpub="${kprv}.pub"
[ -f "$kprv" ]  || ssh-keygen -t ed25519 -o -a "$rounds" -f "$kprv"
[ -f "$kpub" ]  || err 'public key not found'                    # ssh-keygen -y
ssh-add "$kprv" || err 'key rejected by agent'
 
ktyp="$(sed 's/[ ].*$//' $kpub)"
 
nrip="$(sed 's_^[^ ]*[ ]__; s_[ ][^ ]*$__' "$kpub")"
[ ssh-ed25519 = "$ktyp" ] &&
  orip="$(sed 's_^[^ ]*[ ]__; s_[ ][^ ]*$__' ~/.ssh/id_ed25519.pub)"

chk255 () { [ ssh-ed25519 != "$ktyp" ] && err 'requires id_ed25519'; true; }
 
while read host port user <&9
do case "$host" in [#]*) continue;; esac

   [ -z "$host" -o -z "$port" -o -z "$user" ] && err "bad host file"
 
   printf "$format" "$user" "$host" "$port" | cut -c-80

   [ "$copyid" ] && ssh-copy-id    -i "$kpub" -p "$port" "$user@$host"
 
   [ "$dryrun" ] && ssh-copy-id -n -i "$kpub" -p "$port" "$user@$host"
 
   [ "$edit" ]   && { chk255
                      ssh -p "$port" "$user@$host" \
                        sed -i.rotate "'\_${orip}_{p;s__${nrip}_;}'" \
                          '~/.ssh/authorized_keys'; }

#  [ "$wipe" ]   && { chk255
#                     prmpt="$(ssh -p "$port" "$user@$host" sed -n \
#                       "'\_${orip}_p'" '~/.ssh/authorized_keys')"
#                     [ -z "$prmpt" ] && continue
#                     p '%s\n\n%s - Confirm key wipe (Y/n)? ' "$prmpt" "$host"
#                     read check
#                     [ n = "$check" ] && continue
#                     [ Y = "$check" ] &&
#                       ssh -p "$port" "$user@$host" \
#                         sed -i.rotate "'\_${orip}_d'" \
#                           '~/.ssh/authorized_keys' || err 'Wipe aborted'; }

done 9< "$hf"
 
if [ "$migrate" ]
then chk255
     mv -iv ~/.ssh/id_ed25519 ~/.ssh/id_ed25519_$(date +%Y%m%d)
     mv -iv ~/.ssh/id_ed25519.pub ~/.ssh/id_ed25519.pub_$(date +%Y%m%d)
     mv -iv "$kprv" ~/.ssh/id_ed25519
     mv -iv "$kpub" ~/.ssh/id_ed25519.pub
fi                                # ssh-add -l; ssh-add -d ~/.ssh/id_ed25519.pub

This script uses the Debian DASH shell, which (mostly) enforces POSIX compliance, and allows no “bashisms” or other advanced shell features. This should run unaltered in bash, Korn shell variants, and Busybox. The infrastructure prerequisites are minimal, allowing the script to run on a wide array of platforms.

The “rounds” option above makes the private key more resistant to password cracking attacks (in the event that it falls into the hands of an adversary), at the expense of delayed decryption (either by the agent, or when used with the -i option). OpenBSD's default is 16 rounds. The default above is 100 rounds. Adjust this parameter to your taste.

Notice above that the “wipe” operation is commented. Under no circumstances should conservative users enable this functionality.

The script has a preference for ed25519 keys. Other public key schemes that are supported by OpenSSH can be used, but they will operate at reduced functionality. Backporting this script's functions to NIST ECDSA or RSA should not be attempted. The tinyssh server can bring modern and safe cryptography to older systems, including ed25519.

ssh-copy-id en masse

The safest rotation method, using the ssh-copy-id script included in OpenSSH, has been carefully designed by skillful developers to cope with commercial UNIX. The target accounts must be able to run a UNIX shell, so “exotic” platforms (such as VMS or native Windows) may not work at all well, although these scripts do work reasonably under Cygwin. SFTP-only accounts, or keys locked with the command= option, similarly cannot be rotated with this approach, and will require administrator intervention to change an authorized_key entry. The ssh-copy-id script is written with great consideration for elderly systems that go no farther than POSIX compliance (on a good day). This should work reasonably well on most UNIXish systems with shell access.

First to be performed is a dry run, which will generate a new key for in-bound rotation. The new key password must be entered a total of three times on any type of initial run (in an effort to prevent this script from locking thousands of people out of their servers). The new key is always confirmed on each run, and the pause is a good thing (do not remove it).

The host list below is short, but could easily be much, much longer. If an NFS server holds your home directory (and the ~/.ssh/authorized_keys file), only include a single host among any servers that mount it.

Use a new password on the newly-generated private key. If an older key falls into the hands of an adversary, a successful password crack upon it should not compromise the new key.

$ cat shosts
bsd.myhost.com 22 cfisher
ubuntu.myhost.com 22 cfisher

$ ./ssh-rotate -h shosts -d
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/cfisher/.ssh/id_ed25519_pre20211130.
Your public key has been saved in /home/cfisher/.ssh/id_ed25519_pre20211130.pub.
The key fingerprint is:
SHA256:ZU5y5lOaz3i7000+v1gE87Q9HJX8I92AC02Y/ayu6II cfisher@centos
The key's randomart image is:
+--[ED25519 256]--+
|           *... o|
|          + + .o.|
|        . *..*.+o|
|         X +..Oo*|
|        S *  ..*o|
|           =. . o|
|     .    ..+. = |
|    E .  . .o.+.o|
|       oo ..o+ .=|
+----[SHA256]-----+
Enter passphrase for /home/cfisher/.ssh/id_ed25519_pre20211130: 
Identity added: /home/cfisher/.ssh/id_ed25519_pre20211130 (cfisher@centos)
=cfisher=bsd.myhost.com=22======================================================
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cfisher/.ssh/id_ed25519_pre20211130.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
=-=-=-=-=-=-=-=
Would have added the following key(s):

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINm3PZqO1Y+4uEjDjsftIG5+O7OWV/uvFI1JjkL3V6Bg cfisher@centos
=-=-=-=-=-=-=-=
=cfisher=ubuntu.myhost.com=22===================================================
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cfisher/.ssh/id_ed25519_pre20211130.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
=-=-=-=-=-=-=-=
Would have added the following key(s):

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINm3PZqO1Y+4uEjDjsftIG5+O7OWV/uvFI1JjkL3V6Bg cfisher@centos
=-=-=-=-=-=-=-=

The ssh-copy-id command did not prompt for remote passwords, because the agent had cached a working key, which is the intended use. This output appears correct, so we will perform the copy:

$ ./ssh-rotate -h shosts -c
Enter passphrase for /home/cfisher/.ssh/id_ed25519_pre20211130: 
Identity added: /home/cfisher/.ssh/id_ed25519_pre20211130 (cfisher@centos)
=cfisher=bsd.myhost.com=22======================================================
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cfisher/.ssh/id_ed25519_pre20211130.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -p '22' 'cfisher@bsd'"
and check to make sure that only the key(s) you wanted were added.

=cfisher=ubuntu.myhost.com=22===================================================
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cfisher/.ssh/id_ed25519_pre20211130.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh -p '22' 'cfisher@ubuntu'"
and check to make sure that only the key(s) you wanted were added.

SSH Commands en masse

Now we should examine the authorized_keys files on our collection of hosts. To do this, the script below can be used to send SSH agent-authenticated commands to each of our servers:

$ cat ssh-run-minimal
#!/bin/dash

alias p=printf

usage(){ p "%s - SSH run\n -h hostfile: Targets [host port username]\n" "$0"; }

err () { usage; p '\nerror:'; for x; do p ' %s' "$x"; done; p '\n'; exit; }
 
[ -z "$SSH_AUTH_SOCK" -o -z "$SSH_AGENT_PID" ] && err 'no agent'
 
set -eu; unset IFS  # http://redsymbol.net/articles/unofficial-bash-strict-mode/

hf= format=\
'=%s=%s=%d====================================================================='

while getopts h:u arg
do case "$arg" in h) hf="$OPTARG"; shift 2;; u) usage; exit;; esac; done
 
[ -z "$hf" ] && err 'no hosts'

while read host port user <&9
do case "$host" in [#]*) continue;; esac
   p "$format" "$user" "$host" "$port" | cut -c-80
   ssh -p "$port" "$user@$host" "$@"
done 9< "$hf"              # This feeds into ssh's stdin without alternate #< fd

Now we can examine the authorized_keys on the target servers (with some careful quoting):

$ ./ssh-run-minimal -h shosts cat '~/.ssh/authorized_keys'
=cfisher=bsd.myhost.com=22======================================================
no-agent-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAWnRQaFXxeBM1WRyS8Ncm42bLrQ4S88JKomEbtvKWzX cfisher@slackware
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINm3PZqO1Y+4uEjDjsftIG5+O7OWV/uvFI1JjkL3V6Bg cfisher@centos
=cfisher=ubuntu.myhost.com=22===================================================
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAWnRQaFXxeBM1WRyS8Ncm42bLrQ4S88JKomEbtvKWzX cfisher@slackware
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINm3PZqO1Y+4uEjDjsftIG5+O7OWV/uvFI1JjkL3V6Bg cfisher@centos

The solution thus far is not ideal, but for the most conservative users, this is all the risk that should be taken. For such keyholders, responsibility falls upon you to purge authorized_keys by some method. This will be discussed below, so please read on.

Replicating Key Metadata

There are several problems with new keys installed via ssh-copy-id:

• The key option above, no-agent-forwarding, is not applied to the new key. That would be inconvenient to fix manually if the collection of hosts is large.

• It might also be preferable if the email/comment on the previous key was retained.

The above functionality can be achieved with “in place editing” via the stream editor utility, “sed -i” which is not within the POSIX sed standard, so this will be (violently) incompatible with commercial UNIX. In-place editing is such a seductive tool for this task that it is impossible to resist, but we must leave POSIX behind to adapt it, which will reduce the number of compatible platforms.

As the new authorized key doesn't meet the stated needs, we will remotely erase it with sed. Ladies and gentlemen, please don’t try this at home.

$ ./ssh-run-minimal -h shosts sed -i.rotate '/centos/d' '~/.ssh/authorized_keys'
=cfisher=bsd.myhost.com=22======================================================
=cfisher=ubuntu.myhost.com=22===================================================

$ ./ssh-run-minimal -h shosts cat '~/.ssh/authorized_keys'
=cfisher=bsd.myhost.com=22======================================================
no-agent-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAWnRQaFXxeBM1WRyS8Ncm42bLrQ4S88JKomEbtvKWzX cfisher@slackware
=cfisher=ubuntu.myhost.com=22===================================================
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAWnRQaFXxeBM1WRyS8Ncm42bLrQ4S88JKomEbtvKWzX cfisher@slackware

Before attempting this merge of old key metadata with a new key, it is useful to always perform the above dry run to confirm that all the hosts are accessible, and their known_hosts entries are present and correct. I am omitting the dry run here, but it is better to execute it repeatedly than encounter an error because it was omitted.

Assuming the host list is correct, we will pull all the old key’s metadata, and wrap it around the new key:

$ ./ssh-rotate -h shosts -e
Enter passphrase for /home/cfisher/.ssh/id_ed25519_pre20211130: 
Identity added: /home/cfisher/.ssh/id_ed25519_pre20211130 (cfisher@centos)
=cfisher=bsd.myhost.com=22======================================================
=cfisher=ubuntu.myhost.com=22===================================================

$ ./ssh-run-minimal -h shosts cat '~/.ssh/authorized_keys'
=cfisher=bsd.myhost.com=22======================================================
no-agent-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAWnRQaFXxeBM1WRyS8Ncm42bLrQ4S88JKomEbtvKWzX cfisher@slackware
no-agent-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINm3PZqO1Y+4uEjDjsftIG5+O7OWV/uvFI1JjkL3V6Bg cfisher@slackware
=cfisher=ubuntu.myhost.com=22===================================================
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAWnRQaFXxeBM1WRyS8Ncm42bLrQ4S88JKomEbtvKWzX cfisher@slackware
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINm3PZqO1Y+4uEjDjsftIG5+O7OWV/uvFI1JjkL3V6Bg cfisher@slackware

Now the new key more closely mimics the old.

For an acid test, we could remove the original key from the agent:

$ ssh-add -l
256 SHA256:zvvXqCNs5V2xYx8tmXZtu8FYwJgU66sH8tooP0qgK2I cfisher@centos (ED25519)
256 SHA256:ZU5y5lOaz3i7000+v1gE87Q9HJX8I92AC02Y/ayu6II cfisher@centos (ED25519)

$ ssh-add -d ~/.ssh/id_ed25519
Identity removed: /home/cfisher/.ssh/id_ed25519 (cfisher@centos)

Notice below that sed has created backup files each time it was executed, and our ability to run commands after the removal of the agent's old key confirm that the new key is functional:

$ ./ssh-run-minimal -h shosts ls -li '~/.ssh'
=cfisher=bsd.myhost.com=22======================================================
total 24
39079171 -rw-------  1 cfisher  fishecj  238 Nov 30 20:13 authorized_keys
39079172 -rw-------  1 cfisher  fishecj  119 Nov 30 20:00 authorized_keys.rotate
41196027 -rw-r--r--  1 cfisher  fishecj   95 Mar  1  2021 known_hosts
=cfisher=ubuntu.myhost.com=22===================================================
total 28
540964721 -rw------- 1 cfisher cfisher  198 Nov 30 20:13 authorized_keys
536871207 -rw------- 1 cfisher cfisher   99 Nov 30 20:00 authorized_keys.rotate
537003647 -rw-r--r-- 1 cfisher cfisher 1792 Mar 30  2020 known_hosts

It appears that systems running SELinux preserve the ssh_home_t context in the sed in-place edit operation.

The inode of the post-edit authorized_keys file is different after processing, indicating that sed creates a new file. The backup file has been observed to have the original inode number.

Use caution in proceeding beyond this step, as any further sed actions will erase the backup generated with these commands.

Wipe

You should purge old content from your authorized_keys files. There is no point in deploying new private keys if any authorized_keys file retains previous public keys. Restricting access by proper key rotation requires all authentication-related portions of older keys to be removed.

If your private keys have been stolen, they must be removed from every instance of authorized_keys, with all due haste. Even if they are encrypted, an adversary will load them into cracking software if there is the slightest interest.

Before attempting any means of wipe, test and verify alternate remote access to your target servers via separate accounts on distinct keys. Restricting logins to the console on a server in another country can be a very costly mistake.

If I enable the “wipe” content above, I can remove these keys interactively. Note that I do not recommend that any reader enable this functionality, and I cannot assume responsibility for catastrophe.

$ ./ssh-rotate -h shosts -k ~/.ssh/id_ed25519 -w
Enter passphrase for /home/cfisher/.ssh/id_ed25519: 
Identity added: /home/cfisher/.ssh/id_ed25519 (cfisher@centos)
=cfisher=bsd.myhost.com=22======================================================
no-agent-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAWnRQaFXxeBM1WRyS8Ncm42bLrQ4S88JKomEbtvKWzX cfisher@slackware

bsd.myhost.com - Confirm key wipe (Y/n)? Y
=cfisher=ubuntu.myhost.com=22===================================================
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAWnRQaFXxeBM1WRyS8Ncm42bLrQ4S88JKomEbtvKWzX cfisher@slackware

ubuntu.myhost.com - Confirm key wipe (Y/n)? Y

$ ./ssh-run-minimal -h shosts cat '~/.ssh/authorized_keys'
=cfisher=bsd.myhost.com=22======================================================
no-agent-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINm3PZqO1Y+4uEjDjsftIG5+O7OWV/uvFI1JjkL3V6Bg cfisher@slackware
=cfisher=ubuntu.myhost.com=22===================================================
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINm3PZqO1Y+4uEjDjsftIG5+O7OWV/uvFI1JjkL3V6Bg cfisher@slackware

Note above that a lower-case “y” will cause the script to fail immediately. When complete, the old key will no longer be accepted by any of the target hosts.

I pray you, reader, do not use this feature. Find your own path to safely purge authorized_keys of stale tokens.

An example above of in-place sed edits to remove a key is one option worthy of study, and preparatory sed processing for cleanup of authorized_keys might involve the /^$/d pattern to remove blank lines, and the /ssh-rsa/d pattern to remove RSA keys that are critically weakened due to their reliability on SHA-1.

Rather than a sed delete operator, some might be more pleased by prefixing a # comment marker, and suffixing the current date and time. This is not conducive of a neat and orderly authorized_keys, but could make an emergency revert somewhat more simple.

Keeping two keys in place on all systems would reduce the risk of inadvertent lockout of a single key.

If uniform authorized_keys files on all targets are an option, then a mass sftp (or scp, although it is deprecated) can be performed in lieu of any in-place editing with sed. This is a better cleanup approach for commercial UNIX.

In any case, tread carefully in whatever you decide.

Private Key Migration

At this point, we are ready to archive the old private key, and move the new one into position:

$ ./ssh-rotate -h shosts -m
Enter passphrase for /home/cfisher/.ssh/id_ed25519_pre20211130: 
Identity added: /home/cfisher/.ssh/id_ed25519_pre20211130 (cfisher@centos)
=cfisher=bsd.myhost.com=22======================================================
=cfisher=ubuntu.myhost.com=22===================================================
‘/home/cfisher/.ssh/id_ed25519’ -> ‘/home/cfisher/.ssh/id_ed25519_20211130’
‘/home/cfisher/.ssh/id_ed25519.pub’ -> ‘/home/cfisher/.ssh/id_ed25519.pub_20211130’
‘/home/cfisher/.ssh/id_ed25519_pre20211130’ -> ‘/home/cfisher/.ssh/id_ed25519’
‘/home/cfisher/.ssh/id_ed25519_pre20211130.pub’ -> ‘/home/cfisher/.ssh/id_ed25519.pub’

For improved security, it would be preferable to add the old key to an encrypted archive.

The agent should likely be recycled with the new key file (with ssh-add -D; ssh-add, or ssh-agent -k).

An Improved ssh-run Script

As a final gift, below is a cleaner implementation of ssh-run that uses color and Unicode line-drawing characters.

The printf builtin bundled into the dash shell does not display Unicode on any tested platform, so the alias must be set to a capable external printf binary. On Cygwin, which forks processes very slowly, there will be visible delays in using dash that can be corrected by instead using bash set to the builtin printf.

Due to some bash weirdness in POSIX compliance, the alias to the printf builtin fails when bash is invoked as #!/bin/bash, but functions normally when bash is called as #!/bin/sh.

Some experimentation might be necessary for best results.

$ cat ssh-run
#!/bin/dash
 
alias p=/usr/bin/printf                         # must be Unicode-capable printf
usage () { p "%s - SSH run
  -h hostfile - Target hosts [host port username]
  -u usage
" "$0"; }
 
err () { usage; p '\nerror:'; for x; do p ' %s' "$x"; done; p '\n'; exit; }
 
[ -z "$SSH_AUTH_SOCK" -o -z "$SSH_AGENT_PID" ] && err "error: no agent"
 
set -eu; unset IFS  # http://redsymbol.net/articles/unofficial-bash-strict-mode/
N="$(p '\033[')" x=30 maxlen=0 hf=
 
for a in Bl R G Y B M C W # 4-bit Black Red Green Yellow Blue Magenta Cyan White
do eval $a='$N'"'"$((   x))"m'" \
       b$a='$N'"'"$((60+x))"m'" \
    ${a}bg='$N'"'"$((10+x))"m'" \
   b${a}bg='$N'"'"$((70+x))"m'"        # bX=bright Xbg=background bXbg=brgt bgnd
   x=$((x+1))
done
 
while getopts h:u arg
do case "$arg" in h) hf="$OPTARG"; shift 2 ;; u) usage; exit ;; esac; done
 
[ -z "$hf" ] && err 'no hosts'; N="${N}0m"
hl="$(p '\u2550')" vl="$(p '\u2551')" ll="$(p '\u2554')" bl="$(p '\u255A')"
 
while read host port user <&9
do case "$host" in [#]*) continue;; esac
   p "$B$ll$hl$G${user}$B$hl$M${host}$B$hl$C(${port})$N:$B\\n"
   ssh -p "$port" "$user@$host" "$@" |
     while read t
     do p "$B$vl$Y$t\\n"
     done
   x=$((${#host} + ${#port} + ${#user}))
   maxlen=$((x > maxlen ? x : maxlen))
   ll="$(p '\u2560')"
done 9< "$hf"              # This feeds into ssh's stdin without alternate #< fd

p "$B$bl"

x=0 maxlen=$((maxlen + 6))

while [ $x -lt $maxlen ]
do p "$hl"
   x=$((x+1))
done

p "$N\\n"


$ ./ssh-run -h shosts cat '~/.ssh/authorized_keys'
╔═cfisher═bsd.myhost.com═(22):
║no-agent-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINm3PZqO1Y+4uEjDjsftIG5+O7OWV/uvFI1JjkL3V6Bg cfisher@slackware
╠═cfisher═ubuntu.myhost.com═(22):
║ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINm3PZqO1Y+4uEjDjsftIG5+O7OWV/uvFI1JjkL3V6Bg cfisher@slackware
╚════════════════════════════════

This script also does not appear to display properly on OpenBSD, with any shell or version of printf, due to Unicode issues (the color does work, though). OpenBSD is generally regarded as a more secure platform than many common Linux distributions, so a version of ssh-run that functions OpenBSD as a secure bastion has some value.

As a crude fix for OpenBSD, adjust all assignments of the following variables to these values:

hl=- vl='|' ll=+ bl=+

Perhaps there is some better UTF-8 solution on the OpenBSD platform, but it is not immediately obvious.

Conclusion

SSH has been stretched far beyond its original design goals, and there are many things that it does not do well. The poster child of the foibles of SSH is surely SFTP, which has terrible performance compared to the FTP protocol that it replaced. The fallout from that debacle continues to unfold.

Likewise, none of the original SSH designers imagined the need for key rotation. This new day has dawned upon us, and we see at last that the care that we have taken of these authentication tools has been found wanting.

We have played fast and loose with private keys, omitting passwords at their creation, pushing them onto Github or otherwise exposing them, and neglecting the command= and from= options in authorized_keys to limit the damage that they can inflict. And we wonder at the increasing success of ransomware attacks.

This script is a flawed and humble effort that is masquerading as system software addressing a critical issue. Many keyflips and corrected scripting errors do not make best practice.

Worryingly, the methods presented here can facilitate account takeover by anyone possessing an authorized (potentially shared) private key. A disgruntled employee could easily revoke group access to a large set of accounts with this tool, and knowledge of these techniques may impact those who had no intention of using it.

It will be painful to implement good habits with tools not designed for modern security needs. Hopefully, this is a start.