I am configuring OpenVPN 2.3.6-1 on my Arch Linux server in order to encrypt SMB traffic over the public Internet. When I test the setup on one of my Linux virtual machine clients, I get the error: TLS Error: TLS handshake failed .

I quickly read (openvpn on openvz tls error: tls handshake failed (google suggested solutions not helping)) and tried to switch from the default udp to tcp, but that only caused the client to repeatedly report that the connection timed out. I also tried disabling the cipher and TLS authentication, but that caused the server to fail with Assertion failed at crypto_openssl.c:523 . In both instances the necessary changes were made to the client and server configurations

I have been following the instructions at ( https://wiki.archlinux.org/index.php/OpenVPN ) to set up OpenVPN and the instructions at ( https://wiki.archlinux.org/index.php/Create_a_Public_Key_Infrastructure_Using_the_easy-rsa_Scripts ) to create the keys and certificates. The only deviations i've made from these instructions have been specifying the names of my own computer and the corresponding keycertificate file names

See also my original question about securing SMB traffic over the Internet: (Simple encryption for Samba shares)

Tell me the most effective solution for this issue?

Details.

Server: arch linux (up to date) connected directly to gateway via ethernet cable. No iptables.

Client: Arch Linux (up to date) virtual machine on VirtualBox 4.3.28r100309 Windows 8.1 host, bridged network adapter. No iptables. Windows firewall disabled.

Gateway forwarding for port 1194 enabled no firewall restrictions

Here are the configuration files on the server and client respectively They were created in accordance with the instructions on the arch wiki

/etc/openvpn/server.conf (Non-comment lines only).

port 1194
proto udp
dev tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/server-name.crt
key /etc/openvpn/server-name.key
dh /etc/openvpn/dh2048.pem
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
tls-auth /etc/openvpn/ta.key 0
comp-lzo
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
verb 3

/etc/openvpn/client.conf (Non-comment lines only).

client
dev tun
proto udp
remote [my public IP here] 1194
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca /etc/openvpn/ca.crt
cert /etc/openvpn/client-name.crt
key /etc/openvpn/client-name.key
remote-cert-tls server
tls-auth /etc/openvpn/ta.key 1
comp-lzo
verb 3

Here are the outputs of running openvpn on the machines with the above configurations I started the server first, then the client.

The output of openvpn /etc/openvpn/server.conf on the server.

Thu Jul 30 17:02:53 2015 OpenVPN 2.3.6 x86_64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Dec  2 2014
Thu Jul 30 17:02:53 2015 library versions: OpenSSL 1.0.2d 9 Jul 2015, LZO 2.09
Thu Jul 30 17:02:53 2015 NOTE: your local LAN uses the extremely common subnet address 192.168.0.x or 192.168.1.x.  Be aware that this might create routing conflicts if you connect to the VPN server from public locations such as internet cafes that use the same subnet.
Thu Jul 30 17:02:53 2015 Diffie-Hellman initialized with 2048 bit key
Thu Jul 30 17:02:53 2015 Control Channel Authentication: using '/etc/openvpn/ta.key' as a OpenVPN static key file
Thu Jul 30 17:02:53 2015 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 17:02:53 2015 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 17:02:53 2015 Socket Buffers: R=[212992->131072] S=[212992->131072]
Thu Jul 30 17:02:53 2015 ROUTE_GATEWAY 192.168.0.1/255.255.255.0 IFACE=enp5s0 HWADDR=##:##:##:##:##:##
Thu Jul 30 17:02:53 2015 TUN/TAP device tun0 opened
Thu Jul 30 17:02:53 2015 TUN/TAP TX queue length set to 100
Thu Jul 30 17:02:53 2015 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Thu Jul 30 17:02:53 2015 /usr/bin/ip link set dev tun0 up mtu 1500
Thu Jul 30 17:02:53 2015 /usr/bin/ip addr add dev tun0 local 10.8.0.1 peer 10.8.0.2
Thu Jul 30 17:02:53 2015 /usr/bin/ip route add 10.8.0.0/24 via 10.8.0.2
Thu Jul 30 17:02:53 2015 GID set to nobody
Thu Jul 30 17:02:53 2015 UID set to nobody
Thu Jul 30 17:02:53 2015 UDPv4 link local (bound): [undef]
Thu Jul 30 17:02:53 2015 UDPv4 link remote: [undef]
Thu Jul 30 17:02:53 2015 MULTI: multi_init called, r=256 v=256
Thu Jul 30 17:02:53 2015 IFCONFIG POOL: base=10.8.0.4 size=62, ipv6=0
Thu Jul 30 17:02:53 2015 IFCONFIG POOL LIST
Thu Jul 30 17:02:53 2015 Initialization Sequence Completed

The output of openvpn /etc/openvpn/client.conf on the client.

Thu Jul 30 21:03:02 2015 OpenVPN 2.3.6 x86_64-unknown-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Dec  2 2014
Thu Jul 30 21:03:02 2015 library versions: OpenSSL 1.0.2d 9 Jul 2015, LZO 2.09
Thu Jul 30 21:03:02 2015 WARNING: file '/etc/openvpn/client-name.key' is group or others accessible
Thu Jul 30 21:03:02 2015 WARNING: file '/etc/openvpn/ta.key' is group or others accessible
Thu Jul 30 21:03:02 2015 Control Channel Authentication: using '/etc/openvpn/ta.key' as a OpenVPN static key file
Thu Jul 30 21:03:02 2015 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 21:03:02 2015 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Thu Jul 30 21:03:02 2015 Socket Buffers: R=[212992->131072] S=[212992->131072]
Thu Jul 30 21:03:02 2015 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Thu Jul 30 21:03:02 2015 UDPv4 link local: [undef]
Thu Jul 30 21:03:02 2015 UDPv4 link remote: [AF_INET][my public IP here]:1194
Thu Jul 30 21:04:02 2015 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Thu Jul 30 21:04:02 2015 TLS Error: TLS handshake failed
Thu Jul 30 21:04:02 2015 SIGUSR1[soft,tls-error] received, process restarting
Thu Jul 30 21:04:02 2015 Restart pause, 2 second(s)

Best Answer

I had this problem too

Am using digitalocean provider for my server and the problem was with floating ip feature.

To fix this you have to update openvpn config settings

local <ip anchor>

ip anchor should be an ip adress gathered from ip addr command, see example:

Credits to this post