1. Introduction
The SSH protocol supports public key authentication. Thus, an SSH client can log in to an SSH server with a private key, also known as an identity. This mechanism is more secure than password authentication. However, these keys generally don’t expire. Over time, we may lose track of when we last used a key. Moreover, the server may have old, inactive, yet authorized public keys. In the case of SSH key management tools like ssh-agent, some keys could remain active unbeknownst to us.
In this tutorial, we’ll learn how to check active SSH keys based on recent usage. Initially, we’ll explore SSH key fingerprints and sshd logging configuration for tracking key usage on the remote host. Then, we’ll study ssh-agent and managing keys with ssh-add.
We’ll use OpenSSH version 8.4 in Debian 11 (Bullseye) OS and the Bash shell.
2. Checking Key Usage in sshd Logs
SSH comprises a server and a client. We run the ssh command on the client. Similarly, a daemon service called sshd runs on the server. When a client attempts to connect to the server with a private key, the daemon verifies whether the key is an authorized public key listed in the ~/**.ssh/authorized_keys file of the user logging in.
This daemon logs certain events in the system’s authentication logs. Thus, we can use this in-built logging capability to evaluate key usage on the server. These logs accumulate in the /var/log/auth.log file for Debian-based OS and /var/log/secure for Fedora-based OS. Alternatively, we can access these logs through journalctl.
On the client, the ssh command logs only to stderr by default. The ssh LogLevel configuration must be at least DEBUG to log the authenticated key fingerprint. However, the DEBUG level isn’t recommended due to privacy and verbosity concerns.
2.1. SSH Key Fingerprint
A fingerprint is a hash generated from an SSH key’s public component. SHA-256 is the current default hashing algorithm, even though MD5 is still supported. For a given key pair, the fingerprint of both the public and private keys is the same.
A fingerprint is accepted as uniquely identifying a key, as it’s computationally infeasible to find another key having the same fingerprint. Hence, this fingerprint is used to identify a server’s host key, match a public and private key of the same pair, and log keys.
Let’s create a sample key pair using ssh-keygen:
$ ssh-keygen -t ed25519 -C "baeldung" -f ~/baeldung_id_ed25519
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/baeldung/baeldung_id_ed25519
Your public key has been saved in /home/baeldung/baeldung_id_ed25519.pub
The key fingerprint is:
SHA256:brKIyoYyeLQpOsyx1lprdt8R7yRr3ic8393pyLzofxg baeldung
The key's randomart image is:
+--[ED25519 256]--+
| |
| |
| |
| |
| . S |
| .. . o. E |
|=.+* oooo o |
|O*Bo.ooB .. =.o..|
|BX=o++.o=..+.B++ |
+----[SHA256]-----+
This creates two files: the private key baeldung_id_ed25519 and the public key baeldung_id_ed25519.pub. The SHA-256 hash and the visual fingerprints of the key are shown above.
Now, let’s check the fingerprint of the created private key by passing the options -l for calculating the fingerprint and -f for specifying the file path:
$ ssh-keygen -lf ~/baeldung_id_ed25519
256 SHA256:brKIyoYyeLQpOsyx1lprdt8R7yRr3ic8393pyLzofxg baeldung (ED25519)
As shown above, this fingerprint “SHA256:brKIyoYyeLQpOsyx1lprdt8R7yRr3ic8393pyLzofxg” matches the one in the previous output.
Let’s try the same command with the public key file:
$ ssh-keygen -lf ~/baeldung_id_ed25519.pub
256 SHA256:brKIyoYyeLQpOsyx1lprdt8R7yRr3ic8393pyLzofxg baeldung (ED25519)
Thus, we see that both fingerprints are the same. Among these two, the client stores the private key, and the server, the public key.
2.2. Configuring Logging for Fingerprint
When an SSH login is successful, sshd logs details on the server side, such as the client IP address, port, protocol, key algorithm, and the authenticated key fingerprint.
Let’s query journalctl for the sshd logs. We’ll pass filters for the systemd unit as -u ssh, the ISO-8601 timestamp format as -o short-iso, and sorting to view the newest log first using the -r, i.e., reverse option:
$ journalctl -u ssh -o short-iso -r | head
-- Journal begins at Tue 2023-07-25 00:00:00 UTC, ends at Tue 2023-07-25 12:20:43 UTC. --
2023-07-25T12:20:43+0000 server-b sshd[937]: pam_unix(sshd:session): session opened for user baeldung(uid=1000) by (uid=0)
2023-07-25T12:20:43+0000 server-b sshd[937]: Accepted publickey for baeldung from 10.0.0.14 port 60310 ssh2: ED25519 SHA256:brKIyoYyeLQpOsyx1lprdt8R7yRr3ic8393pyLzofxg
...
We can see successful login details, including the fingerprint. The sshd requires a LogLevel of INFO or higher (VERBOSE, DEBUG, DEBUG2, DEBUG3) to log the key fingerprint. This configuration must be set in /etc/ssh/sshd_config. Let’s view its current setting:
$ cat /etc/ssh/sshd_config
...
# Logging
#SyslogFacility AUTH
#LogLevel INFO
...
INFO is the default value for both ssh_config and sshd_config. So, it’s visible as a commented line. If we modify this setting, the sshd service requires a restart to apply the new changes. For this tutorial, we’ll proceed with LogLevel INFO.
2.3. Evaluating Last Usage Information From Logs
Let’s run a script to evaluate when our authorized public keys were last used. We’ll follow these steps:
- calculate the fingerprint of all keys in the ~/.ssh/authorized_keys file
- track the most recent log of a particular fingerprint in the journalctl logs
- list down keys having no logs
Firstly, we’ll use ssh-keygen to fetch the fingerprints. From the output, we’ll extract the second column for the hash. Let’s store the result in a variable:
$ MY_AUTHORIZED_PUBLIC_KEYS=$(ssh-keygen -lf ~/.ssh/authorized_keys | cut -d ' ' -f2)
$ echo $MY_AUTHORIZED_PUBLIC_KEYS
SHA256:brKIyoYyeLQpOsyx1lprdt8R7yRr3ic8393pyLzofxg SHA256:xbPA+pnHGFHkjpH3sXqGcXkaZNpK/59a7Dl1Mngj8PU
We can see the fingerprints in a space-delimited string. Now, we’ll run an awk script that traverses the journalctl logs until it finds a matching record for each fingerprint:
$ journalctl -u ssh -o short-iso -r | awk -v MY_AUTHORIZED_PUBLIC_KEYS="$MY_AUTHORIZED_PUBLIC_KEYS" '
BEGIN {
key_count = 0
split(MY_AUTHORIZED_PUBLIC_KEYS, keys, "[ \t\n]+")
for (i = 1; i <= length(keys); i++) {
map[keys[i]] = 0
key_count++
}
}
{
for (auth_pub_key in map) {
if (auth_pub_key == $NF) {
print "The key \"" auth_pub_key "\" was last used on " $1 "."
delete map[auth_pub_key]
key_count--
}
}
if (key_count == 0) {
exit 0
}
}
END {
for (auth_pub_key in map) {
print "No logs found for the key \"" auth_pub_key "\"."
}
}
'
The key "SHA256:brKIyoYyeLQpOsyx1lprdt8R7yRr3ic8393pyLzofxg" was last used on 2023-07-25T12:32:02+0000.
No logs found for the key "SHA256:xbPA+pnHGFHkjpH3sXqGcXkaZNpK/59a7Dl1Mngj8PU".
In this script, we first initialize an associative array named map for keeping track of fingerprints. Once we find a corresponding log, we can remove that fingerprint from map. In the end, map contains the remaining fingerprints. Thus, we come to know the recently used and unused public keys.
We can also check the last usage of a private key stored on the client by passing a command to ssh. For simplification, we’ll find up to 10 matches of our private key fingerprint within the server logs using grep with –max-count (-m) and –fixed-strings (-F) options:
$ MY_KEY=$(ssh-keygen -lf ~/baeldung_id_ed25519 | cut -d ' ' -f2)
$ ssh -i ~/baeldung_id_ed25519 baeldung@server-b "journalctl -u ssh -o short-iso -r | grep -m10 -F $MY_KEY"
2023-07-25T14:10:52+0000 server-b sshd[1908]: Accepted publickey for baeldung from 10.0.0.14 port 59982 ssh2: ED25519 SHA256:brKIyoYyeLQpOsyx1lprdt8R7yRr3ic8393pyLzofxg
2023-07-25T12:20:43+0000 server-b sshd[937]: Accepted publickey for baeldung from 10.0.0.14 port 60310 ssh2: ED25519 SHA256:brKIyoYyeLQpOsyx1lprdt8R7yRr3ic8393pyLzofxg
Thus, we traced two logins on the server authenticated with our private key. The first record is for this login itself.
3. Checking ssh-agent Active Keys
ssh-agent is a program for managing SSH private keys. The agent enables us to login to remote hosts with SSH without manually providing the private key or its passphrase each time. These keys remain active until one of these events:
- the agent stops running or becomes unreachable
- the key exceeds its lifetime within the agent, if any
- we manually remove the key
Let’s start the ssh-agent in the background:
$ eval "$(ssh-agent -s)"
Agent pid 5020
The agent started with a PID of 5020. Alternatively, we can configure ssh to automatically add keys to the agent by setting a property in its file, i.e., ~/.ssh/config. We can create this file if it’s absent. We can also configure this for all users in the /etc/ssh/ssh_config file. For now, let’s configure at the user level by adding this line:
$ cat ~/.ssh/config
...
AddKeysToAgent yes
...
As per this setting, if the ssh-agent is running and we login with ssh, the command automatically adds the identity to the agent. This setting is available in OpenSSH versions 7.2 and above.
3.1. Using ssh-add
ssh-add acts as the entry point to ssh-agent. It allows operations like adding or removing a key, listing active keys, and locking or unlocking the ssh-agent with a password.
Let’s add the key we created by providing its path to ssh-add:
$ ssh-add ~/baeldung_id_ed25519
Enter passphrase for /home/baeldung/baeldung_id_ed25519:
Identity added: /home/baeldung/baeldung_id_ed25519 (baeldung)
The command asked us to enter the passphrase, if any. The agent is now ready for ssh login, as it has loaded this key into the RAM.
Let’s check the active keys by passing the -l option to ssh-add:
$ ssh-add -l
256 SHA256:brKIyoYyeLQpOsyx1lprdt8R7yRr3ic8393pyLzofxg baeldung (ED25519)
We can see the fingerprint of the key we just added. Furthermore, it shows the key’s comment and public key algorithm.
3.2. Removing a Key From ssh-agent
We can delete a key from the agent by passing the -d option to ssh-add along with the identity file path:
$ ssh-add -d ~/baeldung_id_ed25519
Identity removed: /home/baeldung/baeldung_id_ed25519 ED25519 (baeldung)
Hereafter, the agent can’t authenticate us with this key until it gets loaded again.
4. Conclusion
In this article, we learned to check active SSH keys on the client and server sides.
On the server side, we examined sshd logging. Based on the authorized public key fingerprints, we traced the most recent usage of each key. We also queried for matching logs by passing an ssh command from the client to the server.
On the client side, we inspected the active keys loaded in the ssh-agent using ssh-add. Additionally, we ran commands to remove such a key.