There are many VPN products available on the market, both commercial and open source. Almost all of these VPN products can be separated into the following four categories:
PPTP-protocol based VPNs
IPSec-protocol based VPNs
SSL-based VPNs
OpenVPN
Some people argue that OpenVPN is also an SSL-based VPN, as it uses an SSL or TLS-like protocol to establish a secure connection. However, we have created a separate category for OpenVPN, as it is different from almost every other SSL-based VPN solution.
We will now go into more detail about each of the four types of VPNs:
One of the oldest VPN protocols is the Point-to-Point Tunneling Protocol (PPTP) developed by Microsoft and Ascend in 1999. It is officially registered as RFC2637 (see https://www.ietf.org/rfc/rfc2637.txt for the full standard). The PPTP client has been included in Windows ever since 1995 and is still included in most operating systems.
Nowadays, the PPTP protocol is considered fundamentally insecure, as the strength of the security of the connection is directly related to the strength of the authentication mechanism chosen (for example, the password). Thus, an insecure password leads to an insecure VPN connection. Most PPTP setups use the MS-CHAPv2 protocol for encrypting passwords, and it is this protocol which is fundamentally broken. The security of the PPTP protocol, including the Microsoft MS-CHAPv2 extensions, has been discussed in the article available at https://www.schneier.com/paper-pptpv2.html.
It is also possible to use X.509 certificates for securing a PPTP connection, which does lead to a fairly secure connection. However, not all PPTP clients support EAP-TLS, which is needed to allow the use of X.509 certificates.
PPTP uses two channels, a control channel for setting up the connection and another channel for data transport. The control channel is initiated over TCP port 1723. The data channel uses the General Routing Encapsulation (GRE) protocol, which is IP protocol 47. For comparison, "regular" TCP/IP traffic is done using IP protocol 6 (TCP) and 17 (UDP).
PPTP clients are available on almost all operating systems, ranging from Windows to Linux and Unix derivatives to iOS and Android devices.
The IPSec standard is the official IEEE/IETF standard for IP security. It is officially registered as RFC2411 (see https://www.ietf.org/rfc/rfc2411.txt for the full standard). IPSec is also built into the IPv6 standard.
IPSec operates at layer 2 and 3 of the OSI model of the network stack. It introduces the concept of security policies, which makes it extremely flexible and powerful, but also notoriously hard to configure and troubleshoot. Security policies allow an administrator to encrypt traffic between two endpoints based on many parameters, such as the source and destination IP address, as well as the source and destination TCP or UDP ports.
IPSec can be configured to use pre-shared keys or X.509 certificates to secure the VPN connection. Additionally, it uses either X.509 certificates, one-time passwords, or username/password protocols to authenticate the VPN connection.
There are two modes of operation in IPSec: tunneling mode and transport mode. Transport mode is used most often in combination with the Level 2 Tunneling Protocol (L2TP). This L2TP protocol performs the user authentication as described in the preceding section. The IPSec clients built into most operating systems usually perform IPSec+L2TP, although it is also possible to set up an IPSec-only connection. The IPSec VPN client built into Microsoft Windows uses IPSec+L2TP by default, but it is possible to disable or bypass it. However, this involves cryptic commands and security policy changes.
Like PPTP, IPSec also uses two channels: a control channel for setting up the connection and one for data transport. The control channel is initiated over UDP port 500 or 4500. The data channel uses the Encapsulated Security Payload (ESP) protocol, which is IP protocol 50. For comparison, "regular" TCP/IP traffic is done using IP protocol 6 (TCP) and 17 (UDP). The integrity of IPSec packets is ensured using Hash-based Message Authentication Code (HMAC), which is the same method that OpenVPN uses.
One of the main disadvantages of IPSec is that many vendors have implemented extensions to the standard, which makes it hard (if not impossible) to connect two IPSec endpoints from different vendors.
IPSec software is included in almost all operating systems, as well as firewall, router, and switch firmware.
The most commonly used VPNs nowadays are SSL-based VPNs, which are based on the SSL/TLS protocol. SSL-based VPNs are often called client-less VPNs or web-based VPNs, although there are some vendors that provide separate client software, such as Cisco AnyConnect and Microsoft SSTP. Most SSL-based VPNs use the same network protocol as is used for secure website (HTTPS), while OpenVPN uses a custom format for encrypting and signing data traffic. This is the main reason why OpenVPN is listed as a separate VPN category.
There is no well-defined standard for SSL-based VPNs, but most use the SSL/TLS protocol to set up and secure the connection. The connection is secured in most cases by using X.509 certificates, with one-time password or username/password protocols for authenticating the connection. SSL-based VPNs are very similar to the connections used to secure websites (HTTPS) and the same protocol and channel (TCP and port 443) is often used.
Even though SSL-based VPNs are often called web-based or client-less, there are quite a few vendors that use a browser plugin or ActiveX control to "enhance" the VPN connection. This makes the VPN noninteroperable with unsupported browsers or operating systems.
OpenVPN is often called an SSL-based VPN, as it uses the SSL/TLS protocol to secure the connection. However, OpenVPN also uses HMAC in combination with a digest (or hashing) algorithm for ensuring the integrity of the packets delivered. It can be configured to use pre-shared keys as well as X.509 certificates. These features are not typically offered by other SSL-based VPNs.
Furthermore, OpenVPN uses a virtual network adapter (a tun or tap device) as an interface between the user-level OpenVPN software and the operating system. In general, any operating system that has support for a tun/tap device can run OpenVPN. This currently includes Linux, Free/Open/NetBSD, Solaris, AIX, Windows, and Mac OS, as well as iOS/Android devices. For all these platforms, client software needs to be installed, which sets OpenVPN apart from client-less or web-based VPNs.
The OpenVPN protocol is not defined in an RFC standard, but the protocol is publicly available because OpenVPN is a piece of open source software. The fact that it is open source actually makes OpenVPN more secure than closed-source VPNs, as the code is continually inspected by different people. Also, there is very little chance of secret backdoors being built into OpenVPN.
OpenVPN has the notion of a control channel and a data channel, both of which are encrypted and secured differently. However, all traffic passes over a single UDP or TCP connection. The control channel is encrypted and secured using SSL/TLS, the data channel is encrypted using a custom encryption protocol.
The default protocol and port for OpenVPN is UDP and port 1194. Before IANA granted OpenVPN an official port assignment, older clients (2.0-beta16 and older) defaulted to port 5000.