IPsec vs. OpenVPN — what are the differences?

Quick answer: IPsec is a suite of secure network protocols that are based on open IETF standards and OpenVPN is an open-source protocol that is used by an application with the same name. In other words, IPSec is a framework but OpenVPN is an end-user solution. It doesn’t follow that OpenVPN is worse than IPsec. If OpenVPN as a solution fulfills your requirements it could be the best choice for you. 

The strength of IPsec lies in three words: open IETF standards. By meeting these standards, two different implementations from different vendors can successfully operate together in the most secure way without having a single common line of code. IPsec is usually supported by the majority of network devices and operating systems out-of-the-box and you don’t need to install any additional packages (here we do not look at licensing and export control in commercial products). In the case of OpenVPN, we install OpenVPN packages on our systems. Thus, as opposed to IPsec the same code is used on both sides of the secure connection. 

What is a VPN? 

VPN protocols often employ encryption techniques that secure the connection and protect it from eavesdropping or data manipulation when transferred over untrusted parts of the network (i.e. the Internet). One of the advantages of VPNs is their ability to connect separated and often geographically dispersed networks as if (from the point of view of the end user) there were no extra routers in between.

This allows communication without the need for reconfiguring third-party networking devices. A VPN is like a tunnel between two devices or locations. For example, it can allow remote access to a company's resources, it can provide a secure communications channel between company offices, or even allow interconnectivity with parts of private networks belonging to other companies. 

This article focuses on encrypted types of VPN protocols. 

OpenVPN – a definition

OpenVPN is an example of a virtual private network (VPN) protocol that is used by an application with the same name. The OpenVPN application exists now in two versions: the free and open-source OpenVPN Community Edition and its commercial version, OpenVPN Access Server with additional features. In this article we consider the Community Edition.

IPsec – a definition 

"IPsec" stands for Internet Protocol Security, (please note that starting from the third generation, the standardized abbreviation is IPsec not IPSec). IPsec is a framework of connected protocols that ensures data confidentiality. It protects data by encrypting packets before their transmission over a network. IPSec also cares about data integrity—it checks that the transmission does not influence the packets and does not change them. This VPN protocol provides data authentication at the IP layer, and can protect more than one data flow. IPsec also has an anti-replay feature that identifies and rejects replayed packets.
IPsec is most commonly used for site-to-site connectivity, such as allowing communication between multiple offices belonging to the same company.

Are you wondering how IPsec works? Check out our previous article to get the details. 

Main characteristics of IPsec and OpenVPN 

IPsec       is a suite of secure network protocols that has been developed by the Internet Engineering Task Force       (IETF) since 1992. IPsec is based on three main protocols:

• Authentication Header (AH) that provides data integrity and data origin authentication but doesn’t provide confidentiality (and for this reason, it is almost never used now).
• Encapsulating Security Payload (ESP) that provides confidentiality, data integrity, and data origin authentication.
• Internet Security Association and Key Management Protocol (ISAKMP) that provides a framework for authentication and key exchange.

In general, we use ISAKMP + ESP. ISAKMP uses the UDP protocol and port 500. ESP uses its own IP protocol with the number 50. But if we have to traverse NAT then ESP is encapsulated in the UDP protocol and uses port 4500. So, without NAT we use UDP/500 for ISAKMP and IP protocol 50 for ESP, with NAT we use UDP/500 + UDP/4500 respectively. 

OpenVPN       (the solution and protocol) was originally developed in 2001. Compared with IPsec it offers much more flexibility because it can work over the TCP or UDP protocols using any ports and even can work via an HTTP proxy.

It is worth mentioning that flexibility doesn’t mean invisibility: even if you configure your OpenVPN connection using TCP/443 any equipment or software solution with basic DPI functionality could distinguish OpenVPN traffic from normal HTTPS and block it if necessary. But using OpenVPN in such a configuration can still help traverse traditional stateful firewalls or other systems that use packet filters or access control lists.

Differences between OpenVPN and IPsec 

This section delves into the cardinal disparities between these two widely-used VPN protocols, each offering unique advantages. Understanding these differences will facilitate better-informed decisions when it comes to securing your network infrastructure.

Installation process

OpenVPN requires extra software that the operating system (OS) usually does not have installed by default. On the other hand, IPsec is directly supported by many modern operating systems, such as Windows, Ubuntu, macOS, Android, iOS/iPadOS and can be used out of the box.

Security level

With regard to data encryption, IPsec and OpenVPN can be considered similar. IPsec defines several encryption algorithms but the most often used one is AES with a 256-bit key. OpenVPN uses the OpenSSL library for encryption and in theory supports any encryption algorithm provided by OpenSSL. However in practice AES with a 256-bit key is usually used. This is unsurprising because all modern CPUs are optimized for AES encryption with a 256-bit key. 

The main difference is in the authentication and key exchange procedures (authentication with a pre-shared key is straightforward and works the same way). IPsec uses the open standard Internet Key Exchange (IKE) protocol and OpenVPN uses its own custom protocol that is based on SSL/TLS. Maybe this is the best illustration of the difference between IPsec and OpenVPN: IKE is a fairly complex protocol (there are two versions – IKEv1 and IKEv2 – and both are still used; the protocol is described in several RFC documents), but the custom TLS-based protocol of OpenVPN is much simpler and does the job. 

Performance

With regard to performance there is no big difference between IPsec and OpenVPN. The time needed to establish a connection and key exchange may differ but that really doesn’t matter because the main job is encryption/decryption. Doing this job on the same system using the same encryption algorithm will demonstrate similar results.

However IPsec will always run a little bit faster due to being part of the kernel, unlike OpenVPN that runs in the user space.

Compatibility with devices and operating systems

IPsec is supported on almost all network devices such as routers and firewalls. Also it is supported on some access points. IPsec is implemented natively in the majority of server, desktop and mobile operating systems.

An OpenVPN package is available for the main operating systems (Windows, Linux, MacOS, Solaris, BSD-based systems) and is integrated in some small routers’ firmware, such as OpenWRT and DD-WRT. It is also integrated in the open-source software router VyOS and open-source software firewall pfSense. It is not supported on the majority of network devices.

Firewall port requirements 

OpenVPN uses a UDP or TCP port, allowing for flexible configuration choices. On the other hand, IPsec uses predefined communication channels, UDP 500 and UDP 4500, to establish the encrypted tunnel and ESP for the transmission of encrypted data. 

The advantages and disadvantages of both solutions are shown below. In the next section, we will explore them in more depth.

Use cases

There are two main scenarios for using a VPN: site-to-site (sometimes referred as S2S, or even L2L - LAN-to-LAN) and client-to-site (sometimes referred as remote-access or C2S, or even C2L - Client-to-LAN). For a site-to-site VPN, IPsec is usually used. For client-to-site, IPsec is often used but the main network vendors have their own proprietary solutions in the form of end-user applications (Palo Alto GlobalProtect, Cisco AnyConnect, Fortinet FortiClient, etc.) that can use IPsec or SSL.

Such applications have two main functions: establish a VPN connection and get control over end-user workstations. OpenVPN can be used here in a similar way but only provides the first function - establishing a VPN connection.

Management and maintenance

Site-to-site VPN is usually configured by network specialists and for them IPsec configuration is not a complex task, regardless of the network equipment or operating systems used (however, sometimes they may come across tricky combinations, such as Linux + Cisco ASA).

For client-to-site VPN, configuration of the client's side can be a complex task because usually it is done by users without any specialized networking knowledge. In this case, for the average user, configuration of OpenVPN will be easier than, for example, L2TP with IPsec. OpenVPN configuration on the server’s side, due to its complexity, will be similar to IPsec but again it will be done by network specialists.

Pros and cons of IPsec and OpenVPN 

Benefits 

Everything above, taken together, makes IPsec strong protection for a network. IPsec implementation in the firewall or router does not require changing any software on the user or server systems. IPsec is also a part of the user system—this is a real benefit when working with remote employees with no permissions to install software on their computers.

When it comes to OpenVPN, with its 256-bit encryption keys and high-end ciphers, information takeover is difficult. The ability to use any port on TCP or UDP means a connection through OpenVPN will appear as a regular HTTPS and pass through the firewall without any problem (unless it is explicitly blocked by DPI-capable equipment). 

Disadvantages 

While IPsec is a well-established and commonly used protocol, the encrypted tunnel setup requires the appropriate configuration of multiple parameters, which may seem quite overwhelming for a new user. If the VPN is not working as expected, troubleshooting usually involves browsing through a large number of logs with complex terminology.

As in the case of IPSec, OpenVPN's configuration can be demanding and complex. A lot of manual settings and a wide variety of possible options require expert knowledge. An incorrect setup can do more harm than good. As mentioned earlier, OpenVPN is not preinstalled on any OS, so it requires configuration of third-party software. 

IPsec or OpenVPN – which one to choose?

The answer is as usual – it depends. First you need to check if OpenVPN is supported by both sides of the secure connection you need to establish. IPsec is supported by almost all routers and firewalls as well as by major operating systems. The OpenVPN package is available for the main operating systems (Windows, Linux, MacOS, Solaris, BSD-based systems), it is integrated in some small routers firmware such as OpenWRT and DD-WRT and it is also integrated in open-source software router VyOS and open-source software firewall pfSense.

In cases where OpenVPN is available as a package, you need to decide how to support this solution. Here we have considered the open-source, free Community Edition of OpenVPN that is supported by the community. IPsec in commercial operating systems is supported by OS vendor in contradiction to OpenVPN as third-party software.

In some cases, OpenVPN can be considered as a viable simple alternative to commercial proprietary client-to-site remote access solutions such as Palo Alto GlobalProtect, Cisco AnyConnect, etc. Of course, OpenVPN doesn’t have such advanced features as health check or split tunneling based on applications but similar to the above products, its client application can be easily installed by a user without specialized technical knowledge unlike manual IPsec configuration. 

WireGuard – an alternative? 

If neither OpenVPN or IPsec wins you over, there is another option—WireGuard. This solution is well known for its easy implementation, safety, cross-platforming, and high performance      . It is a great choice when you want to quickly set up a secure connection without the need for complex configuration. You may consider WireGuard as a modern alternative to OpenVPN with better performance and a simpler configuration process. 

Conclusion 

When determining which VPN solution is right for your needs, it's essential to consider the specific use cases. IPsec shines in scenarios like site-to-site connections between network devices, securing transmissions between company branches, and selective communication with partner companies' networks. On the other hand, OpenVPN excels in providing a client-to-site connection, offering remote access for company users. If ease of use and flexibility in authentication are paramount, WireGuard may be the answer.

This article doesn't define a single "correct" choice but instead places these solutions within a broader context, highlighting their strengths and weaknesses. By sharing this knowledge, we aim to provide the information needed to make decisions tailored to unique requirements. Your network's security is in your hands – choose wisely.

Whether you're seeking the robustness of IPsec, the versatility of OpenVPN, or the efficiency of WireGuard, our expert insights will help you navigate a path to a safer and more secure network.

Original post date 02/23/2022, update date 09/22/2023.