1. Overview
Public key authentication is a comfortable way to access servers over the internet. Consequently, we can automate the login process, making it passwordless by offering the identity file instead. However, this way may lead to problems when too many keys are handed one by one.
In this tutorial, we’ll learn how to recover from the Too many authentication failures error and its possible aftermath.
2. The Error
Let’s notice that we’re going to face this error when ssh offers too many not matching private keys. In detail, the number of attempts is limited by the MaxAuthTries in the configuration of the sshd service with the default value 6.
Now let’s look through ways to come to such a situation. Throughout this tutorial, we’ll use the Fedora 35 system with sshd of version OpenSSH_8.7p1, OpenSSL 1.1.1q FIPS. Additionally, we’ve reduced MaxAuthTries to 3.
2.1. ssh Default Key Check
Let’s find out that by default, ssh looks for a number of default keys in the user’s .ssh folder. So, let’s assume that we have a proper key of a non-default name id_rsa4 and a bunch of passwordless default keys:
$ ls -A1 id_*
id_dsa
id_ecdsa
id_ed25519
id_rsa
id_rsa4
Now let’s start ssh in a more verbose way to find out the failure details:
$ ssh -v [email protected]
# ... the default checks:
debug1: Skipping ssh-dss key /home/joe/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes
debug1: Will attempt key: /home/joe/.ssh/id_rsa RSA SHA256:3jDUBtuxfpaGnT8875C6G1Oj6tZVL9+kiav/LGsggro
debug1: Will attempt key: /home/joe/.ssh/id_ecdsa
debug1: Will attempt key: /home/joe/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/joe/.ssh/id_ed25519
debug1: Will attempt key: /home/joe/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/joe/.ssh/id_xmss
# ...
debug1: Trying private key: /home/joe/.ssh/id_ed25519
Received disconnect from 192.168.56.105 port 22:2: Too many authentication failures
Disconnected from 192.168.56.105 port 22
Consequently, we’ve exhausted the number of tries by offering existing but unused keys.
2.2. Issue With ssh-agent
ssh-agent is a popular keys manager, as it comes in the openssh-server package. So, let’s start it:
$ eval `ssh-agent`
Next, let’s add some identities with ssh-add:
$ ssh-add id_rsa
Enter passphrase for id_rsa:
Identity added: id_rsa (joe@joe-virtualbox)
Now let’s try to connect, assuming that the proper key is the fourth one:
$ ssh -v [email protected]
# ...
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: joe@joe-virtualbox RSA SHA256:DNrIKSlPWJZdDmpfQAT27eo3z4xa6Ki1/HFIkg7P7zk agent
Received disconnect from 192.168.56.105 port 22:2: Too many authentication failures
Disconnected from 192.168.56.105 port 22
So once again, we’d run out of tries before we came across the proper key.
3. Resolving the Situation
Now let’s solve the problem by passing the correct key to ssh.
3.1. Immediate Solution
First, let’s use the i option to ssh. We should set it to the identity file we want to use:
$ ssh -i ~/.ssh/id_rsa4 [email protected]
Further, for this method to work properly, we need to kill the ssh-agent first:
$ eval "$(ssh-agent -k)"
Agent pid 844 killed
Otherwise, its identities would still have priority.
3.2. ssh Configuration
Now let’s bind hosts and their identities permanently. So, we’re going to configure the ssh client. Thus, let’s edit ssh_config:
$ sudo joe /etc/ssh/ssh_config
Then, let’s add the Host entry:
Host 192.168.56.105
User john
IdentityFile ~/.ssh/id_rsa4
IdentitiesOnly=yes
Let’s notice that we identify the host by its IP address. Moreover, Host accepts patterns with wildcards. In this way, we can store configuration for multiple servers.
Then comes the name of the remote user John. Next, we add the IdentityFile entry pointing to the key file. Finally, we dismiss the ssh-agent identities with the IdentitiesOnly=yes entry. So we don’t need to disable the agent as before.
4. More Troubles Looming – Getting Blacklisted
As we’ve found out, we can cope with the Too many authentication failures error rather easily on the ssh level. However, during attempts to resolve it, our repeated ssh requests might be classified as a security threat and cause our IP to be banned.
4.1. iptables and ipset
The iptables service filters traffic and checks packets against a set of rules. Further, it undertakes appropriate action on the packet. Moreover, we’re going to use a combo of iptables and ipset. The latter is to handle a list of banned IPs.
So first, let’s define the ssh_in_blacklist set for banned IPs. The IP will stay on the list for one hour:
$ sudo ipset create ssh_in_blacklist hash:ip timeout 3600
Next, let’s add a bunch of iptables rules. First, let’s drop any packet from the blacklisted IP:
$ sudo iptables -A INPUT -m set --match-set ssh_in_blacklist src -j DROP
Then we accept any not new connection:
$ sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
Now we set the rule to filter out the new traffic:
$ sudo iptables -A INPUT -p tcp -m conntrack --ctstate NEW -m tcp --dport 22
-m recent --set --name newcomers --rsource
Then we check if, within one minute, two connection attempts repeat, with seconds 60 and hitcount 2. If it’s a case, we first log this event, then add the source IP to ssh_in_blacklist, and finally drop the packet:
$ sudo iptables -A INPUT -m recent --update --seconds 60 --hitcount 2
--name newcomers --rsource -j LOG --log-prefix "SSH flooder: "
$ sudo iptables -A INPUT -m recent --update --seconds 60 --hitcount 2
--name newcomers --rsource -j SET --add-set ssh_in_blacklist src
$ sudo iptables -A INPUT -m recent --update --seconds 60 --hitcount 2
--name newcomers --rsource -j DROP
And at the very end, let’s allow the remaining traffic:
$ sudo iptables -A INPUT -j ACCEPT
Now let’s check the content of the blacklist after a few failed attempts:
$ sudo ipset list
Name: ssh_in_blacklist
# ...
Members:
192.168.56.107 timeout 3556
So, let’s free our IP:
$ sudo ipset del ssh_in_blacklist 192.168.56.10
4.2. fail2ban
fail2ban is a utility to reduce the brute force authentication tries. Let’s notice that it scans logs to find the track of failed attempts and dynamically adds iptables rules to ban the offender.
First, let’s set the fail2ban aggressive mode to detect the public key authentication failure:
$ sudo cat /etc/fail2ban/jail.local
# ...
[sshd]
mode = aggressive
Then let’s set a ban for 60 minutes. If during 2 minutes 2 tries take place, with bantime, findtime, and maxretry:
$ sudo cat /etc/fail2ban/jail.local
# ...
[DEFAULT]
bantime = 60m
findtime = 2m
maxretry = 2
# ...
Now let’s check the banned IPs after some login tries:
$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
| |- Currently failed: 0
| |- Total failed: 2
| `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: 192.168.56.107
And finally, let’s unban our IP:
$ sudo fail2ban-client set sshd unbanip 192.168.56.107
5. Conclusion
In this tutorial, we learned how to get rid of the Too many authentication failures error when we used a public key identification. First, we studied situations when this error occurred. Then we made it go by specifying the exact key to use.
Finally, we examined the iptables and fail2ban firewall configuration and learned how to unban our IP if necessary.