Book Image

Mastering OpenVPN

By : Jan Just Keijser, Eric F Crist
Book Image

Mastering OpenVPN

By: Jan Just Keijser, Eric F Crist

Overview of this book

Security on the internet is increasingly vital to both businesses and individuals. Encrypting network traffic using Virtual Private Networks is one method to enhance security. The internet, corporate, and “free internet” networks grow more hostile every day. OpenVPN, the most widely used open source VPN package, allows you to create a secure network across these systems, keeping your private data secure. The main advantage of using OpenVPN is its portability, which allows it to be embedded into several systems. This book is an advanced guide that will help you build secure Virtual Private Networks using OpenVPN. You will begin your journey with an exploration of OpenVPN, while discussing its modes of operation, its clients, its secret keys, and their format types. You will explore PKI: its setting up and working, PAM authentication, and MTU troubleshooting. Next, client-server mode is discussed, the most commonly used deployment model, and you will learn about the two modes of operation using "tun" and "tap" devices. The book then progresses to more advanced concepts, such as deployment scenarios in tun devices which will include integration with back-end authentication, and securing your OpenVPN server using iptables, scripting, plugins, and using OpenVPN on mobile devices and networks. Finally, you will discover the strengths and weaknesses of the current OpenVPN implementation, understand the future directions of OpenVPN, and delve into the troubleshooting techniques for OpenVPN. By the end of the book, you will be able to build secure private networks across the internet and hostile networks with confidence.
Table of Contents (17 chapters)
Mastering OpenVPN
About the Authors
About the Reviewers

OpenVPN internals

The design of OpenVPN is not extensively documented, but most of the internals of OpenVPN can be discovered by looking at the source code.

The tun/tap driver

One of the basic building blocks of OpenVPN is the tun/tap driver. The concept of the tun/tap driver comes from the Unix/Linux world, where it is often natively available as part of the operating system. This is a virtual network adapter that is treated by the operating system as either a point-to-point adapter (tun-style) for IP-only traffic or as a full virtual Ethernet adapter for all types of traffic (tap-style). At the backend of this adapter is an application, such as OpenVPN, to process the incoming and outgoing traffic. Linux, Free/Open/NetBSD, Solaris and Mac OS include a tun kernel driver, which is capable of both tun-style and tap-style operations. Recently, a similar driver was added to AIX, which is IBM's Unix derivative.

For Microsoft Windows, a special NDIS driver was written by James Yonan, called the TAP-WIN32 adapter. At the moment, the NDIS5 and NDIS6 versions of the driver are available, supporting Windows XP through Windows 8.1. The development of this adapter is now officially separated from the main OpenVPN development, but OpenVPN continues to rely heavily on it.

The flow of traffic from a user application via OpenVPN is depicted in the preceding diagram. In the diagram, the application is sending traffic to an address that is reachable via the OpenVPN tunnel. The steps are as follows:

  1. The application hands over the packet to the operating system.

  2. The OS decides using normal routing rules that the packet needs to be routed via the VPN.

  3. The packet is then forwarded to the kernel tun device.

  4. The kernel tun device forwards the packets to the (user-space) OpenVPN process.

  5. The OpenVPN process encrypts and signs the packet, fragments it if necessary, and then hands it over to the kernel again to send it to the address of the remote VPN endpoint.

  6. The kernel picks up the encrypted packet and forwards it to the remote VPN endpoint, where the same process is reversed.

It can also be seen in this diagram that the performance of OpenVPN will always be less than that of a regular network connection. For most applications, the performance loss is minimal and/or acceptable. However, for speeds greater than 1GBps, there is a performance bottleneck, both in terms of bandwidth and latency.

It should be noted that the performance of the Windows driver is much lower than the performance of the native tun/tap adapters found in other operating systems. This is true even with the most recent NDIS6 implementation of the TAP-Win32 driver. For a single OpenVPN client, the impact is fairly small. For a large-scale OpenVPN server that serves many clients, this can easily cause performance issues. This is one of the main reasons that the open source community normally recommends the use of a Unix- or Linux-based host as the OpenVPN server.

The UDP and TCP modes

OpenVPN currently supports two ways to communicate between endpoints: using UDP packets or using TCP packets. UDP is a connectionless or lossy protocol; if a packet is dropped in transit, then the network stack does not transparently correct this. TCP packets are a connection-oriented protocol; packets are sent and delivered using a handshake protocol, ensuring the delivery of each packet to the other side.

Both modes of communication have their advantages and disadvantages. It actually depends on the type of traffic that is sent over the VPN tunnel to determine which mode of communication is best. Using a TCP-based application over a TCP-based VPN can result in double performance loss, especially if the underlying network connection is bad. In that case, a re-transmittance of lost packets is done for packets lost both inside and outside the tunnel, leading to a double performance hit. This is explained nicely in the article Why TCP over TCP is a Bad Idea at

However, it can be similarly argued that sending UDP over UDP is also not a good idea. If an application using UDP for its traffic is susceptible to message deletion or packet reordering attacks, then an underlying encrypted TCP connection will enhance the security of such applications even more than an underlying UDP-based VPN. If the bulk of traffic over the VPN is UDP-based then it is sometimes better to use a TCP connection between VPN endpoints.

When choosing between UDP or TCP transport, the general rule of thumb is as follows: if UDP (mode udp) works for you, then use it; if not, then try TCP (mode tcp-server and mode tcp-client). Some switches and routers do not forward UDP traffic correctly, which can be an issue especially if multiple OpenVPN clients are connected to the same switch or router. Similarly, the performance of OpenVPN over TCP can be severely affected by the choice of Internet Service Providers (ISPs): some ISPs use odd MTU sizes or packet fragmenting rules, resulting in extremely poor performance of OpenVPN-over-TCP compared to nonencrypted TCP traffic.

The encryption protocol

It has been said that OpenVPN implements TLS over UDP. This is more or less true, but the way OpenVPN uses TLS is different from the way a web browser uses it. Thus, when OpenVPN is run over TCP (using port 443 is a common method to duck firewalls), the traffic is distinguishable from normal TLS traffic. A firewall that uses Deep Packet Inspection (DPI) can easily filter out OpenVPN traffic.

The main difference between OpenVPN-TLS and browser-TLS is the way packets are signed. OpenVPN offers features to protect against DoS attacks by signing the control channel packets using a special static key (--tls-auth ta.key 0|1). Data channel packets, which are sent over the same UDP or TCP connection, are signed completely differently and are very readily distinguished from HTTPS traffic. The OpenVPN website ( depicts how packets are encrypted for UDP transport, which is illustrated below.

The same mechanism is used for TCP transport (

This is also the main reason why port-sharing, where OpenVPN and a secure web server share the same IP address and port number, can actually work.

The control and data channels

OpenVPN uses two virtual channels to communicate between the client and server:

  • A TLS control channel to exchange configuration information and cipher material between the client and server. This channel is used mostly when the VPN connection is started, as well as for exchanging new encryption keying material. This keying material is renewed after a certain period (based on the --reneg-sec, --reneg-bytes, or --reneg-pkts options).

  • A data channel over which the encrypted payload is exchanged.

The exception to this is the older pre-shared key point-to-point mode, in which only the data channel is used.

Encryption and authentication (signing) for the control channel and the data channel are determined differently. The control channel is initiated using a TLS-style protocol, similar to how a secure website connection is initiated. During control channel initialization, the encryption cipher and hashing algorithm are negotiated between the client and server.

Encryption and authentication algorithms for the data channel are not negotiable, but they are set in both the client and server configuration files for OpenVPN. The current default settings are Blowfish as the encryption cipher and SHA1 as the hashing algorithm. The ability to also negotiate cipher and hashing algorithms for the data channel are high on the wish list of the development team, but this requires an extensive change to the code.

Ciphers and hashing algorithms

OpenVPN supports a wide range of encryption ciphers and hashing algorithms. The ciphers are used to encrypt the payload, while the HMAC function makes use of a digest or hashing algorithm to authenticate incoming packets. As OpenVPN uses a control channel and a data channel, there are two sets of ciphers and hashing algorithms that can be configured.

The control channel cipher and hashing algorithms are normally negotiated at startup. The list of available combinations of ciphers and hashing algorithms can be displayed using the following command:

$ openvpn --show-tls

The available TLS Ciphers listed in order of preference:


This output was retrieved on a CentOS 6 host using the OpenSSL 1.0.1e library.

The available combinations depend largely on the exact version of the SSL library used. You can specify a list of tls-ciphers in the OpenVPN configuration file in a manner that is very similar to configuring the Apache mod_ssl module:


List all ciphers on a single line; the preceding output was modified for readability.

For the data channel, the encryption cipher and hashing algorithm are controlled using the --cipher and --auth options. If the cipher and authentication algorithm are not specified, then the default values of bf-cbc and sha1 are used, respectively.

To retrieve the list of available encryption ciphers, use the following command:

$ openvpn --show-ciphers

The following ciphers and cipher modes are available for use with OpenVPN. Each cipher shown here may be used as a parameter to the --cipher option. The default key size is shown regardless of, whether or not it can be changed with the --keysize directive. Using a CBC mode is recommended. In a static key mode, only a CBC mode is allowed:

BF-CBC 128 bit default key (variable)
BF-CFB 128 bit default key (variable) (TLS client/server mode)
BF-OFB 128 bit default key (variable) (TLS client/server mode)
AES-128-CBC 128 bit default key (fixed)
AES-128-OFB 128 bit default key (fixed) (TLS client/server mode)
AES-128-CFB 128 bit default key (fixed) (TLS client/server mode)
AES-192-CBC 192 bit default key (fixed)
AES-192-OFB 192 bit default key (fixed) (TLS client/server mode)
AES-192-CFB 192 bit default key (fixed) (TLS client/server mode)
AES-256-CBC 256 bit default key (fixed)
AES-256-OFB 256 bit default key (fixed) (TLS client/server mode)
AES-256-CFB 256 bit default key (fixed) (TLS client/server mode)

In this output, only the most commonly-used ciphers are shown. The list of available ciphers again depends on the exact version of the underlying crypto library. However, in most cases, the Blowfish (BF-*) and AES (AES-*) ciphers should be available.

Similarly, for the authentication (HMAC-signing) algorithms, we use the following command to list all the available options:

$ openvpn --show-digests

The following message digests are available for use with OpenVPN. A message digest is used in conjunction with the HMAC function to authenticate received packets. You can specify a message digest as a parameter to the --auth option:

SHA 160 bit digest size
SHA1 160 bit digest size
ecdsa-with-SHA1 160 bit digest size
SHA256 256 bit digest size
SHA384 384 bit digest size
SHA512 512 bit digest size
SHA224 224 bit digest size

In this output, only the most commonly-used digests or hashing algorithms are shown. The list of available digests depends on the exact version of the underlying crypto library. In most cases, the SHA-1 and SHA-2 family of hashing algorithms should be available.

OpenSSL versus PolarSSL

Starting with OpenVPN 2.3, support for a new SSL library has been added. The PolarSSL library ( can be compiled in instead of the default OpenSSL library. The main reason for adding a second library was to ensure the independence of the underlying encryption libraries and to ensure that no copyright issues would arise, as the OpenSSL copyright license is different from the one that OpenVPN uses.