Random openVPN interruptions

My users started reporting frequent interruptions to their RDP/SSH sessions when running over OpenVPN to our building. This was of course caused by the massive uptick in WFH during COVID (and some expanding teams).

Running WireShark and PingPlotter revealed that the issue started on the server. Further investigation and correlation showed that every initial login, and rekey would cause a brief traffic interruption.

To handle the re-keys (every hour or 3600s by default) you can add reneg-sec 0 to both the client and server configuration. This will cause the server to never re-negotiate the connection. This is also a fix to getting a new smart card or MFA prompt every 60 minutes.

To properly resolve this though we want to know why it hangs, as the above does not resolve the issue at initial login. The root issue was found in #openvpn on freenode, OpenVPN is single threaded and if you use a script to authenticate users then the server has to wait for its execution.

Seen in this log, the network traffic stops between 14:22:55 and 14:23:22

Thu Mar 25 14:22:55 2021 us=791373 99.99.99.99:49341 peer info: IV_GUI_VER=OpenVPN_GUI_11
AUTH-PAM: BACKGROUND: received command code: 0
AUTH-PAM: BACKGROUND: USER: USERNAME
AUTH-PAM: BACKGROUND: my_conv[0] query='login:' style=2
AUTH-PAM: BACKGROUND: name match found, query/match-string ['login:', 'login'] = 'USERNAME'
AUTH-PAM: BACKGROUND: my_conv[0] query='Password: ' style=1
AUTH-PAM: BACKGROUND: name match found, query/match-string ['Password: ', 'password'] = 'PASSWORD'
Thu Mar 25 14:23:22 2021 us=986516 99.99.99.99:49341 PLUGIN_CALL: POST /usr/lib/openvpn/openvpn-plugin-auth-pam.so/PLUGIN_AUTH_USER_PASS_VERIFY status=0

Resolution

To resolve this we will be building a plugin that forks out the auth process, based on this sourceforge post that links to this github repo.

NOTE: When building on Debian/Ubuntu per this issue change line 20 of openvpn-plugin-auth-script.c to #include <openvpn/openvpn-plugin.h>

For Debian you need to install make and build-essential before running make plugin while in the repo. Put the resultant file in /usr/lib/openvpn/ with the auth-pam plugin. I set permissions to 644 root:root to match the auth-pam plugin as well.

No deference plugins are able to work with Auth-PAM instead you need to add setenv deferred_auth_pam 1 to your config to use the deference native to the auth-pam plugin. This is much simpler than compiling a new plugin yourself but it requires openvpn server 2.5. See here