1. Overview
Transport Layer Security (TLS) handshake is part of the TLS authentication mode of OpenVPN. As this handshake is the initial stage of the Virtual Private Network (VPN) connection setup, multiple factors could cause a failure.
In this tutorial, we’ll explore some of the causes behind a TLS handshake failure with the OpenVPN client and learn how to resolve them. Initially, we’ll verify the server host address configuration. Then, we’ll examine the firewall and port configurations. Then, we’ll study issues concerning the client or server’s certificates. Lastly, we’ll see how discrepancies in the TLS configuration could lead to a handshake failure.
We tested all scripts in this tutorial in Bash shell on Debian 12 (Bookworm) OS configured with OpenVPN community version 2.6.3. The scripts should work on POSIX-compliant machines. In our case, the server has the default port and protocol, port 1194 with UDP. Moreover, this tutorial uses 30.100.10.55 as the client’s IP address and 140.20.11.30 as the server’s.
2. TLS Handshake on OpenVPN Client
During the TLS handshake, the client and the server mutually authenticate and agree on parameters for further communication.
A successful handshake requires five criteria to be met:
- The OpenVPN daemon is running on the server.
- The server’s port 1194 is exposed to allow UDP protocol requests from the client’s IP address.
- The client can connect to the server machine.
- Both client and server certificates are valid.
- Both the client’s .ovpn file and the server’s configuration file /etc/openvpn/server.conf have a set of common directives with compatible values.
Moreover, OpenVPN recommends TLS authentication mode with an extra HMAC signature enabled for each packet via the tls-auth or tls-crypt directive. Thus, a client can initiate a handshake only if it has the same pre-shared key (PSK) as the server.
Let’s observe a few lines from the openvpn command logs for a failed handshake:
$ sudo openvpn --config client.ovpn
...
2023-12-24 19:00:31 UDPv4 link remote: [AF_INET]140.20.11.30:1194
2023-12-24 19:00:31 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2023-12-24 19:01:31 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
2023-12-24 19:01:31 TLS Error: TLS handshake failed
...
Above, we saw that the client was trying to connect to the server at IP 140.20.11.30 on port 1194.
Now, let’s examine the causes of a handshake failure. At the end of each section, we can test the openvpn command to check whether the handshake succeeds.
3. Server Host Address Configuration
Let’s ensure the client has the correct server host address, port, and protocol.
3.1. Server’s server.conf File
The /etc/openvpn/server.conf file has a local directive for defining which host address to bind to. Let’s comment out this directive if enabled, as the server listens on all interfaces by default:
;local a.b.c.d
Here, the semi-colon marks the line as a comment.
This file also defines the directives for the port as port and protocol as proto in two distinct lines:
port 1194
proto udp
Let’s restart the server daemon to apply the modifications made, if any, in server.conf:
$ sudo systemctl restart openvpn@server
Thus, we restarted the profile associated with server.conf.
3.2. Client’s .ovpn File
The client’s .ovpn file has a remote directive that defines the server’s host address. Optionally, let’s add the port and protocol if these aren’t already specified using port and proto, respectively:
remote 140.20.11.30 1194 udp
Thus, we verified that the client points to the server correctly.
4. Firewall and Port Configuration
Any firewall at the server end must allow ingress for the OpenVPN port and protocol to process client requests. Similarly, servers behind middleware layers, like a gateway, need port forwarding rules to enable external access.
There could also be multiple layers of security, such as a cloud virtual machine with a cloud-based firewall and another firewall running inside the machine. If so, exposing the port and protocol at all levels is necessary.
Let’s check how to expose the port and protocol in iptables, ufw, firewalld, and on the cloud. Let’s run the commands based on the firewall running on the server.
4.1. Configuring iptables
Let’s configure iptables for enabling client access by inserting a rule at the head of the INPUT chain:
$ sudo iptables -I INPUT -p udp -s 30.100.10.55 --dport 1194 -j ACCEPT
$ sudo iptables -I OUTPUT -p udp --sport 1194 -j ACCEPT
Here, the second line allows egress on the same port.
Now, let’s list the applied rules for the INPUT chain:
$ sudo iptables -L INPUT -n
Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT 17 -- 30.100.10.55 0.0.0.0/0 udp dpt:1194
Thus, the server will now allow UDP traffic on port 1194.
4.2. Configuring ufw
Let’s add a rule to ufw for allowing client connections:
$ sudo ufw allow proto udp from 30.100.10.55 to any port 1194
Rule added
$ sudo ufw allow out 1194
Rule added
Rule added (v6)
$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip
To Action From
-- ------ ----
1194/udp ALLOW IN 30.100.10.55
1194 ALLOW OUT Anywhere
1194 (v6) ALLOW OUT Anywhere (v6)
Now, UDP connections on the above port will work.
4.3. Configuring firewalld
Let’s create a new zone in firewalld and configure it for OpenVPN clients:
$ sudo firewall-cmd --permanent --new-zone=openvpn-clients
success
$ sudo firewall-cmd --permanent --zone=openvpn-clients --add-source=30.100.10.55
success
$ sudo firewall-cmd --permanent --zone=openvpn-clients --add-port=1194/udp
success
$ sudo firewall-cmd --reload
success
$ sudo firewall-cmd --zone=openvpn-clients --list-all
openvpn-clients (active)
target: default
icmp-block-inversion: no
interfaces:
sources: 30.100.10.55
services:
ports: 1194/udp
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:
Thus, we exposed port 1194 to the client.
4.4. Configuring Cloud Firewalls
Firewalls on the cloud generally run outside the respective cloud virtual instance. For example, AWS EC2 has Security Groups, GCP Compute Engine has Firewalls, and Azure Virtual Machine has Network Security Groups. If the ingress is restricted from the client, these firewalls require a new rule with higher precedence to allow incoming UDP traffic on port 1194 for the client.
Additionally, if the default settings don’t permit outgoing traffic to the client, then allowing egress is also necessary.
5. Invalid Certificate
OpenVPN uses a Certificate Authority (CA) for the client’s and server’s certificate issuance and verification.
Four directives primarily determine the certificates and keys:
- ca refers to the CA’s root certificate, ca.crt
- cert refers to the certificate issued by the CA
- key refers to the private key
- tls-auth refers to the PSK in TLS authentication mode
Among these, the CA root certificate validity period generally ranges from 10-20 years. However, CA-issued certificates expire sooner.
Conversely, the tls-auth PSK doesn’t expire but must be the same as the server’s tls-auth key.
Let’s extract the expiry time of ca.crt using the openssl command:
$ openssl x509 -enddate -in ca.crt -noout
notAfter=Dec 20 15:51:17 2033 GMT
Thus, we came to know when this certificate expires.
5.1. Replacing an Expired Certificate With a New One
If any certificate has expired, we’ll need to generate a new certificate from our CA by making a Certificate Signing Request (CSR).
OpenVPN supports inline file configuration for the directives mentioned above, such as adding the contents of ca.crt between
Once both ends have valid certificates, they can continue with the handshake.
5.2. TLS Authentication Key
In the server.conf file on the server, the following line defines the tls-auth directive:
tls-auth ta.key 0
Here, ta.key is the PSK file stored in the same directory as server.conf. Furthermore, 0 is the key direction.
The client .ovpn file must contain the same key but with the opposite direction, which is 1:
key-direction 1
Additionally, let’s compare the existing tls-auth directive with ta.key from the server. If missing, let’s add it anew in the .ovpn file with the latest value:
<tls-auth>
#
# 2048 bit OpenVPN static key
#
-----BEGIN OpenVPN Static key V1-----
...
-----END OpenVPN Static key V1-----
</tls-auth>
If there’s a separate ta.key file on the client instead of an inline file, let’s directly pass the tls-auth directive in the same format as in server.conf:
tls-auth ta.key 1
Now, both ends can use the same tls-auth PSK correctly.
5.3. Incorrect System Time
Another cause behind an invalid certificate error is incorrect system time. If the client’s or server’s system time falls outside their corresponding certificate validity period, the TLS handshake will fail.
Let’s check the current system time on both ends with the date command:
$ date
Fri Dec 29 12:25:49 UTC 2023
If the above output doesn’t match the actual time, we’ll have to synchronize the corresponding system time with its network. Once we do this, the certificate will be valid as per the system time.
6. TLS Configuration
Finally, the handshake fails if there’s any mismatch between the client’s and the server’s supported TLS version or cipher. We define these parameters in the server.conf and the client’s .ovpn file.
Using the same OpenVPN version on both machines ensures that both ends support the same TLS parameters.
Now, let’s see how to configure these parameters to match.
6.1. TLS Version
Both ends need a common TLS version to proceed. OpenVPN versions before 2.3.3 supported only TLS 1.0. However, version negotiation is possible since version 2.3.3.
Two optional directives affect this configuration parameter:
- tls-version-min – defaults to 1.2 since version 2.6.0
- tls-version-max – the highest available version by default
Let’s set these in the files on both ends to guarantee that they use the same version:
tls-version-min 1.3
tls-version-max 1.3
As a result, both ends will use TLS 1.3.
6.2. TLS Cipher
OpenVPN provides two optional directives for specifying the TLS cipher list:
- tls-cipher – for TLS version 1.2 and below
- tls-ciphersuites – for TLS version 1.3
If these directives are absent in both files, OpenVPN will take the default value based on the underlying crypto library.
However, if these are present in server.conf or the client’s .ovpn file, but not both, we’ll need to copy it into the other file so that both values match.
Let’s verify that another directive named auth, if present, is the same in server.conf and the client’s .ovpn file. This directive specifies the HMAC algorithm for the message digest.
If we make any changes to server.conf, we must restart the server daemon. Once these directives match, the client and server can determine a set of common TLS parameters.
7. Conclusion
In this article, we studied different causes of and ways to fix TLS handshake failure on the OpenVPN client.
First, we verified that the client has the correct server details. Then, we checked how to enable the firewall and port configuration to expose a server UDP port to the client’s IP address.
Next, we understood the implications of an invalid certificate at the client or server end. We learned where to update new certificates along with the tls-auth PSK and how the system time could affect the certificate validity.
Finally, we explored how TLS configuration mismatch of the version or cipher could cause TLS handshake failure.