IPSec — what is it and how does it work?

The Internet has expanded more and faster than anyone expected. The rate of development has been so rapid that the Internet Protocol standards (IPv4) cannot meet the current expectations and requirements, especially regarding security and supporting the growing number of IP addresses. As the network expanded, the amount of sensitive data grew. Information, like financial transactions or internal company communications, needs to be secure. IPSec was introduced as one of the solutions protecting this kind of data by encryption and authentication between endpoints.

In this article, you can familiarize yourself with this solution—how it works, the core protocols, and more about the pros and cons of IPSec.

What is IP Security—definition 

The history of IPSec       began in 1993 when IPSec VPN was used to connect the east and west coasts of the United States for the first time. IPSec secures higher-level protocols and resolves the earlier addressing problems of IPv4. Put simply, IPSec makes communication secure. The “IPSec'' abbreviation comes from security, so IPSec stands for IP Security. 

This framework ensures data confidentiality (packets are encrypted before their transmission over a network) and data integrity (checking if packets were unchanged during transmission) at the IP layer. IPSec has a feature called antireplay that identifies and rejects replayed packets. What is also worth mentioning is that IPSec can protect more than one data flow.

How does IPSec work? Step by step

The graphic below is a visualization of how IPSec works. The process can be divided into five main steps. The picture alone cannot fully convey the process’s complexity. For that reason, each step is fully described to give the best possible explanation. 

Step 1: Interesting traffic 

Many enterprise firewalls may be able to establish IPSec tunnels on demand. The firewall monitors the traffic flowing through its interfaces and checks if that traffic matches the configured policy. If a match occurs it is assumed that the traffic should go through the secure channel. However, nowadays this step looks slightly different—the modern IPSec sets up ready tunnels (which can be “IDLE”).

In many cases, two types of policy statements can be distinguished: 

• Policy-based VPN—encrypts and encapsulates traffic flow according to a defined policy.
• Route-based VPN—applies routed tunnel interfaces as the network endpoints, and passing traffic is put into the VPN. Generating/transiting interesting traffic initializes the next phase. 

Step 2: Negotiation

Here, IPSec is used to negotiate the necessary policies for setting up a secure circuit. IKE stands for Internet Key Exchange and its main function is to authenticate IP Security peers and build a secure channel between them, making the IKE exchanges possible. Two modes are possible in this phase: 

• main mode—when the negotiation phase ends, the IKE SA (security association) is set up, and the IPSec circuit is defined. 
• aggressive mode—there is no negotiation in this mode, and the IKE SA is simply accepted. This mode is faster but less secure (and is currently rarely used). 

Step 3: IPSec circuit 

In this phase, the IPSec tunnel is set up. Steps taken during the IKE phase two include negotiating the IPSec (SA) parameters that IKE SAs protect, and establishing and regularly renegotiating IPSec SAs. 

Step 4: Data transfer 

Having completed IKE phase two, IPSec peers can now transfer data between them using encryption from IPSec SAs. 

Step 5: Tunnel termination 

Deletion or timing out can cause termination of IPSec SAs. When this happens the keys are discarded, and IKE phase two (and sometimes also one) is performed again to generate new SAs and keys. However, the new SAs can be set up before the old ones expire to keep the flow continuous. 

How does IPSec work in practice? If you are interested in details, read the case study about connecting different VPC/VNets to multiple clouds with IPSec as one of the core technologies used.

IP Security core protocols 

The main aim of the core protocols is to ensure security when encoding information. In IPSec, there are two to mention—the AH and ESP protocols. The Authentication Header (AH) protocol is responsible for data authentication but currently, it is rarely used. The AH aims to confirm that the delivered packet stayed integrated during the transfer. ESP (Encapsulating Security Payload protocol) encrypts packets—even if hackers intercept the packet, they do not have access to the packet content. 

IPSec, like any solution, has its advantages and disadvantages; there is nothing surprising about that. Still, it is good to know where the limitations lie and how to profit from the solution’s strengths. 

Benefits and disadvantages of IPSec 

IPSec is a respected solution because of features like:

• The most prominent IPSpec advantage is the fact that it works well between different devices (Windows, Cisco, Juniper, and many more); it is an open solution with a long list of compatible solutions. 
• IPSec is the only VPN protocol that is standardized in RFC. That means IPSec has the greatest chance of working well with different endpoints/implementations.
• A server or firewall (that determines IPSec flow) with IPSec implemented becomes strong protection for the network.
• IPSec is transparent to applications because it is above the transport layer. 
• Software on user or server systems does not need to be changed when IPSec is implemented in the firewall/router. 
• IPSec can provide security even for individual workers/users—IPSec enables remote access to a company’s sources for remote workers.
• Another advantage for business use, IPSec is already installed in the user system. This is a real advantage when workers lack permission to install any software on their computers.

As we all know, there is no perfect solution. IP Security also has its disadvantages. Below you can find the most crucial ones. That does not mean that IPSec is not worth your attention; only that it’s necessary to keep these points in mind while developing a project:

• IPSec configuration, because of its complexity, can be more demanding compared to other VPN protocols; this VPN requires more extensive knowledge. 
• IPSec can require additional firewall configuration—if the tunnel ends do not support NAT-T (Network Address Translation-Traversal), the L3 ESP protocol has to be run in the firewall—please keep in mind that not every firewall can meet these requirements.

Read the differences between IPSec vs OpenVPN.

Conclusion 

The IPSec topic is broad and this article merely highlights the most significant information you should know up front: key protocols, components, the advantages and limitations of IPSec. However, this solution has many more interesting aspects which will be described in future articles—stay tuned.