VPN protocols explained: how do they work?

Imagine that you want to send an expensive tea set to your grandmother abroad. If you also want it to reach her in one piece, you’ll have to wrap it in bubble wrap, add a “fragile” sticker on the box, and choose a reputable delivery service. But if you’re sending a baseball bat for your cousin, your preparation will be different.

The point is, there are many ways to send a package, and the same is also true for data exchanges over the Internet. VPN protocols determine how your data is sent over a VPN network. Let’s dig into these VPN protocols and how they work.

What is a VPN protocol?

A VPN protocol is a set of rules on how the data will be packaged and sent over a private network. They establish VPN tunnels to safely exchange data.

If you want to transfer data to a server, you could just connect your device and the server using a cable. But what if the server is on another continent? VPN allows you to connect to a remote server as if you’re connected with a wire, making a separate and inaccessible network to outsiders.

To make this sort of connection possible, we need VPN protocols (sometimes also called tunneling protocols) that make two things possible:

• Establish a safe tunnel for your data to reach the remote server. So, your connection remains separate from the public internet, making your data difficult to intercept.
• Bypass firewall restrictions. E.g. you can “wrap” BitTorrent traffic in the ubiquitous TCP or UDP protocols, bypassing firewall blocks on torrent traffic.

This can be very useful not only for bypassing firewalls. For example, you cannot run protocols on networks that don’t support them. This poses a problem if a website that you’re frequently using isn’t secure, i.e., uses HTTP. You cannot force the website owner to update it to HTTPS. However, you can establish an encrypted connection to a remote server and access the website from it. So, you’re creating a safe passage for your data to reach the VPN server when otherwise it wouldn’t be possible.

How do VPN protocols work?

Among the things communication protocols define are parameters like data packet size, error correction types, authentication techniques, address format, and much more. It’s quite likely that you will have heard some of the more popular communication protocols, such as TCP/IP, HTTP, BitTorrent, or SMTP.

There are a variety of tunneling protocols. Essentially, they’re used for the same function – to allow data exchange between two networks. For that, they’re using some method of authentication to ensure there’s a genuine VPN server on the other end. Then, there’s some encryption method to make the exchanged data impossible to inspect from the outside.

Although they work similarly, each VPN protocol has particular traits that make them better at some specific tasks and worse at others. No protocol is best for everything, it all depends on the context, and there are many points to consider when choosing them.

Common VPN protocols

Internet Key Exchange version 2 (IKEv2)

IKEv2 is very common on VPN mobile apps. It’s a compound tunneling protocol, which uses Encapsulating Security Payload (ESP) to secure the packet transmissions with Mobility and Multihoming Protocol for endpoint tunnels. The latter means that when the connection with a VPN server is interrupted, you can reconnect without resetting the connection. If you live in a remote location, this is a lifesaver. IKEv2 will automatically reconnect after disconnecting and continue as if nothing happened. Plus, it works great on mobile devices because of widespread support and the fact that you can seemlessly switch between wifi and cellular data.

The protocol also supports advanced cipher functions. You can use it with 3DES and AES for encryption, with the latter being the safest method. By default, IKEv2 uses UDP port 500, so it provides good compatibility and should bypass most firewalls.

OpenVPN

Probably the most popular VPN protocol. OpenVPN uses TLS with SSL/TLS for private key exchange. It relies on the OpenSSL crypto library – an index of secure cryptography algorithms that make your tunnel safer. OpenVPN is similar to SSTP (to be discussed below) – both use the 443 TCP port (reserved for HTTPS traffic), which means you can bypass many firewalls.

OpenVPN offers lots of room for customization. For starters, it uses either the User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) – standard network protocols, but both somewhat different.

In UDP mode, OpenVPN is much faster because you’re sacrificing additional data verification checks to save time. It usually requires multiple data verifications in TCP mode, slowing the exchanges down, hence the lower speeds. That way, you can switch between the usage situation and always have the best option at hand.

Layer 2 Tunneling Protocol over Internet Protocol Security (L2TP/IPSec)

By default, L2TP doesn’t have any encryption, so it’s added via IPSec, and it uses a 256bit AES variant. L2TP creates the tunnel and handles authentication.

L2TP itself is a combination of two protocols that proceeded it: Layer 2 Forwarding Protocol and Point to Point Tunneling Protocol. This VPN protocol is considered secure, but there’s a caveat. L2TP/IPSec was co-developed with NSA, and there are suspicions that there may be backdoors to L2TP/IPSec traffic.

WireGuard

The most recent major tunneling protocol, WireGuard offers by far the best connection speeds, while maintaining security. This protocol solves problems that arise from IPSec and OpenVPN implementations, which tend to be quite complicated and more prone to misconfigurations. Wireguard has very few lines of code and is therefore very easy to implement or patch.

WireGuard uses publicly available cryptography packages like Poly1305 for data authentication and ChaCha20 for encryption. It also has a built-in hashing ensured by the BLAKE2s function – a masterpiece of modern cryptography applied for network safety.

By default, if no special port is selected, Wireguard uses 51820 UDP, which is still a standard port, but less used than some others. This means a lower chance of interference with other devices using it.

WireGuard offers tools for developers to add their extensions and scripts, making it a lot safer than it already is.

For more information, read our extended guide about Wireguard protocol.

Secure Socket Tunneling Protocol (SSTP)

This protocol is an advancement of PPTP and L2TP protocols. What it really does is that it sends PPTP or L2TP traffic through the SSL 3.0 channel. This addition allows data integrity checks and encryption measures. Plus, it uses the standard SSL and TCP port 443, so you can bypass most firewalls. It also means that the exchanged data can be unsupervised and work around closed ports and other restrictions put in place by your network administrator.

The SSTP protocol uses 2048-bit certificates for authentication and the 256-bit SSL cipher for encryption. It’s one of the most secure VPN protocols, even though it’s not very recent. The only caveat is that it was originally developed by Microsoft, which makes some distrust its privacy. However, as of now, there is no proof that there are backdoors that could reveal your traffic.

PPTP or Point to Point Tunneling Protocol

One of the first tunneling protocols, PPTP is gravely outdated. This protocol encrypts data in transit with the MPPE cipher, which is very vulnerable to attacks using modern computers. Sent data packets could potentially be intercepted and modified, which is the main concern. The protocol has no methods to verify whether legitimate sources sent the data.

Proprietary VPN protocols

Some VPN service providers have created their own tunneling protocols. Some of these offer better speeds, security, or an improved ability to bypass firewalls

The main problem with proprietary protocols is that we don’t necessarily know what exactly is under the hood. However, in most cases, these are modified versions of familiar protocols, such as OpenVPN or WireGuard.

Catapult Hydra

Found only in the Hotspot Shield VPN service, Catapult Hydra is based on TLS 1.2 and uses RSA certificates with 2048-bit keys for authentication with ephemeral keys, purging them after each session. Aside from that, not much is known about Catapult Hydra, however, Hotspot Shield has often appeared among the fastest VPN services, probably in some part due to the protocol.

NordLynx

NordVPN’s NordLynx is actually modified Wireguard. The default implementation of Wireguard uses the same IP address for each connected user, which means the tunnel is private, but it displays an obvious pattern that could expose you. So, the developers added a fix, which they call “double NAT” (Network Address Translation) to randomize your IP address without keeping your data logs.

As all implementations of WireGuard, NordLynx is very fast.

KeepSolid Wise

Only available in VPN Unlimited, the KeepSolid Wise is a modified version of OpenVPN. Depending on the configuration, it uses TCP 443 and UDP 443 ports, which should be enough to bypass most firewalls. KeepSolid claims the protocol improves on OpenVPN performance. However, it seems that when running in TCP mode, the protocol sometimes suffers from TCP meltdown. This is a known issue of OpenVPN tunneling protocol when there isn’t sufficient headroom in the untunneled network. It may result in performance drops, slowing down your connection a lot.

Lightway

This is a proprietary VPN protocol from ExpressVPN. Lightway uses the wolfSSL cryptography library that meets the FIPS 140-2 standard. For authentication, it depends on TLS to secure the tunnel. The protocol is very fast when switching networks. It makes it one of the best options for a mobile phone or when your internet connectivity suffers from frequent interrupts. With all that said, Lightway is still very new, so it’s hard to draw conclusions about it.

What is the downside of using a proprietary VPN protocol?

One of the biggest advantages of some common tunneling protocols like OpenVPN or WireGuard is that they are open-source. This gives the best transparency as anyone who’s willing can inspect its source code to verify it’s safe.

So, the biggest downside of proprietary VPN protocol is that they likely won’t be open-source to protect the developers’ work. Some users might have concerns about this loss of transparency.

VPN protocol comparison

If you’ve been wondering whether IKEv2 is more secure than OpenVPN, you can see everything in the table. Although it’s still in development, WireGuard seems to solve most issues that are plaguing older protocols, especially when it comes to security and efficiency.

Choosing the Best VPN protocols for different use cases

Although it would seem tempting to call Wireguard the best tunneling protocol, the reality may not be as clear cut. Also, not all your devices will support all the existing protocols (relevant if you’re using the VPN without a VPN app). In such cases, it’s good to know which protocols you should use in which scenarios.

Streaming

If you think about streaming media, you’re likely to need speed over privacy. VPNs are often used to avoid geo-blocking, and in such a situation, you’re probably less afraid about the police knocking down your door because you’ve watched a TV series on Hulu. For the best performance, you should probably use Wireguard, if it’s available. However, IKEv2, L2TP/IPSec, or even OpenVPN in UDP mode are all decent choices.

Downloads

If you want to download data from P2P networks, you’ll have to combine speed with privacy in equal parts. Downloaded torrents put your IP out in the open due to the transparency of BitTorrent. It makes you an easy target for copyright holders as well as hackers. So, you should be using the protocol versions that are secure and speedy like Wireguard, or OpenVPN in UDP mode.

Gaming

If you need a VPN for gaming, your focus should be a slow ping as possible. The best option to reduce is to pick a fast tunneling protocol like IKEv2 or WireGuard and connect to a nearby location. The closer it is, the less distance the signal has to travel back and forth, adding less latency.

Privacy

Suppose you’re a whistleblower or someone working in a very restrictive country. In that case, the privacy and security of the connection should be your top goals. For this reason, you should pick only the safest possible protocols: Wireguard and OpenVPN. You also want to seek out a VPN that has some sort of “stealth mode” to bypass firewalls.

Thinking of trying out a VPN service? Read one of our VPN guides or reviews

FAQ