Internet Protocol Security (IPsec) is a secure network protocol suite that authenticates and encrypts the packets of data to provide secure encrypted communication between two computers/networks. IPsec is usually used to set up VPNs, and it works by encrypting IP packets, along with authenticating the source where the packets come from.
Read more about encryption here.
In fact, IPsec describes the framework for providing security at the IP layer, as well as the suite of protocols designed to provide that security, through authentication and encryption of IP network packets.
IPsec can protect network traffic with the following features:
- Confidentiality: by encrypting data.
- Integrity: By calculating a hash value, the sender and receiver will be able to check if changes have been made to the packet.
- Authentication: the sender and receiver will authenticate each other.
- Anti-replay: even if a packet is encrypted and authenticated, an attacker could try to capture these packets and send them again. By using sequence numbers, IPsec will not transmit any duplicate packets. (Source: networklessons.com )
Uses of IPsec
- Encrypting application layer data
- Providing security for routers sending routing data across the public internet
- Providing authentication without encryption
- Protecting network data by setting up circuits using IPsec tunneling
What Is an IPsec VPN?
A virtual private network (VPN) is an encrypted connection between two or more computers. VPN connections take place over public networks, but the data exchanged over the VPN is still private because it is encrypted. Many VPNs use the IPsec protocol suite to establish and run these encrypted connections. In fact IPsec is mostly used to form a secure tunnel in VPNs, however, not all VPNs use IPsec.
How Does IPsec Work?
IPsec includes protocols for establishing mutual authentication between agents at the beginning of a session and negotiation of cryptographic keys to use during the session. IPsec can protect data flows between a pair of hosts (host-to-host), between a pair of security gateways (network-to-network), or between a security gateway and a host (network-to-host).
Included in IPsec are protocols that define the cryptographic algorithms used to encrypt, decrypt and authenticate packets, as well as the protocols needed for secure key exchange and key management.
In fact, IPsec it is made up of three main elements:
- Authentication Header (AH): This protocol ensures that data packets are from a trusted source and that the data has not been tampered with, like a tamper-proof seal on a consumer product. The anti-replay protection, protects against unauthorized transmission of packets. It does not protect data’s confidentiality.
- Encapsulating Security Payload (ESP): It provides data integrity, encryption, authentication and anti-replay. It also provides authentication for payload.
- Security Associations (SAs): SA refers to a number of protocols used for negotiating encryption keys and algorithms. One of the most common SA protocols is Internet Key Exchange (IKE).
The Internet Security Association and Key Management Protocol (ISAKMP) defines how Security Associations (SAs) are set up and used to define direct connections between two hosts that are using IPsec. ISAKMP provides a framework for authentication and key exchange, with actual authenticated keying material provided either by manual configuration with pre-shared keys, Internet Key Exchange (IKE and IKEv2), Kerberized Internet Negotiation of Keys (KINK), or IPSECKEY DNS records.
The Internet Key Exchange (IKE) protocol and IKEv2 are used to manage the cryptographic keys used by hosts for IPsec.
IPsec works on five steps:
Step 1- Define interesting traffic: This step occurs when a host recognizes that a packet should be transmitted using IPsec. This may be done by checking the IP address of the source or destination against policy configurations. This is done when the system sending the packet apply an appropriate encryption. The incoming packets are also checked by the host that they are encrypted properly or not.
Step 2- IKE Phase 1: This step also called is IKE Phase 1and it allows to negotiate the policy sets they use for the secured circuit, authenticate themselves to each other, and initiate a secure channel between the two hosts. IKE Phase 1 sets up an initial secure channel between hosts using IPsec; that secure channel is then used to securely negotiate the way the IPsec circuit will encrypt and/or authenticate data sent across the IPsec circuit.
There are two options during IKE Phase 1:
- Main mode: This mode provides greater security because it sets up a secure tunnel for exchanging session algorithms and keys
- Aggressive mode: This mode allows some of the session configuration data to be passed as plain-text but enables hosts to establish an IPsec circuit more quickly.
Step 3 – IKE Phase 2: This step in setting up an IPsec circuit is the IKE Phase 2, which itself is conducted over the secure channel setup in IKE Phase 1. It requires the two hosts to negotiate and initiate the security association for the IPsec circuit carrying actual network data. In the second phase, the two hosts negotiate the type of cryptographic algorithms to use on the session, as well as agreeing on secret keying material to be used with those algorithms.
Step 4 – Data transfer: This step is the actual exchange of data across the newly created IPsec encrypted tunnel. These packets are encrypted and decrypted by the hosts using IPsec SAs.
Step 5 – IPSec tunnel termination: This step is termination of the IPsec tunnel, usually when the communication between the hosts is complete, when the session times out or when a previously specified number of bytes has been passed through the IPsec tunnel. In other words, when the communication between the hosts is completed or the session times out then the IPsec tunnel is terminated by discarding the keys by both the hosts.
IPsec can use both ESP and AH in either tunnel or transport mode.
- Tunnel Mode: This mode is used between two dedicated routers, with each router acting as one end of a virtual “tunnel” through a public network. In this mode the entire data packet is either encrypted or authenticated (or both). At each end of the tunnel, the routers decrypt the IP headers to deliver the packets to their destinations.
- Transport Mode: In this mode, the payload of each packet is encrypted, but the original IP header is not. Intermediary routers are thus able to view the final destination of each packet. There are also some other changes, depending on whether ESP or AH is being used.
Tunnel mode is more common because many VPNs rely on this mode.