A Virtual Private Network (VPN) is often described as a secure tunnel through an untrusted network like the internet. While that metaphor works, it oversimplifies what’s really happening behind the scenes. How do devices in different locations trust each other over a public network? How is the data kept private, tamper-proof, and verifiable from end to end?

The answer lies in IPSec (Internet Protocol Security). In this post, we’ll start with the basics of VPNs, then dive deep into IPSec: its technologies, security functions, protocols, modes, and the connection procedure that makes private communication over a public network possible.

What is a VPN?

A Virtual Private Network (VPN) is a secure communication overlay that protects and controls traffic between two or more endpoints over an untrusted network.

it builds an encrypted tunnel that ensures:

  • Confidentiality: no one can read the data in transit
  • Integrity: the data hasn’t been tampered with
  • Authentication: the endpoints are who they claim to be
  • Access Control: only authorized parties can send/receive traffic through the tunnel

VPN Types:

  • Site-to-Site VPN: Secure communication between fixed networks, VPN gateways are preconfigured with information to establish a secure tunnel, and internal hosts have no knowledge that a VPN is being used.
  • Remote-Access VPN: Enables individual users to securely connect to a network from remote locations. The VPN connection is typically initiated on demand by the user through VPN client software or a web browser (for SSL VPNs).

While the VPN concept describes secure tunneling, IPSec defines the mechanics of how it’s done. IPSec isn’t a single protocol; it’s a framework of protocols and algorithms that collectively provide Confidentiality, Integrity, Authentication and Secure Key Exchange.

IPSec Security Functions

  1. Confidentiality: is achieved by encrypting the data. The degree of confidentiality depends on the encryption algorithm and the length of the key used in the encryption algorithm, such as: AES, DES or SEAL.
  2. Integrity: means that the data that is received is exactly the same data that was sent. Hashing algorithms verify that data hasn’t been tampered with:
  • MD5(128-bit): Basic integrity check, now less recommended.
  • SHA (160-bit or higher): Preferred for stronger protection.
  1. Authentication: Ensures endpoints are legitimate before secure communication begins:
    • Pre-shared Keys (PSK): A simple password value is entered into each peer manually.
    • RSA: Stronger authentication using digital certificates to authenticate peers.
  2. Diffie-Hellman Key Exchange: DH provides a way for two peers to establish a shared secret key that only they know, even though they are communicating over an insecure channel.

IPSec Protocols

IPSec defines two primary protocols that determine how data packets are secured: Authentication Header (AH) and Encapsulation Security Protocol (ESP).

AH (Authentication Header)

Provides data integrity and authentication but does not encrypt the payload. It is suitable in environments where data confidentiality is not a requirement, but authentication and integrity are. How it works?

  • The sender generates a hash value for the entire packet (except mutable fields like the IP Time-to-Live field) using a shared secret key.
  • This hash value is inserted into a new AH header and added to the packet.
  • The receiving device performs the same hash calculation and compares it to the value in the AH header.
  • If the two values match, the packet is verified as authentic and unaltered.

ESP (Encapsulation Security Protocol)

Provides confidentiality (encryption), integrity, authentication, and anti-replay protection. ESP is the preferred protocol for most VPN deployments, it works by:

  • Encrypting the payload using symmetric encryption (AES, 3DES, or SEAL).
  • It then calculates a hash over the encrypted data to ensure integrity and authentication.
  • An ESP header is added to carry encryption and authentication parameters.
  • The receiver decrypts the payload, validates the hash, and verifies the packet’s authenticity.
  • ESP can include a sequence number for each packet to prevent attackers from capturing and replaying packets to disrupt communication.

IPSec protocols (AH or ESP) can operate in two modes, depending on how packets are encapsulated and the type of connection being secured.

  • Transport Mode: In Transport Mode, only the payload of the IP packet is encrypted and authenticated. The original IP header remains intact and visible. It is less resource-intensive than Tunnel Mode, but it offers a lower level of protection since key details about communication remain exposed.
  • Tunnel Mode: Tunnel Mode provides maximum security by encrypting the entire original IP packet, including both its header and payload. Once encrypted, this packet is encapsulated within a new packet that has a fresh IP header used solely for routing through the public network. This approach ensures that no part of the original packet can be inspected during transit, protecting not only the data but also the identities of the communicating devices. This mode is the standard choice for site-to-site VPNs, where security is a top priority and VPN gateways at each endpoint handle the encapsulation and encryption.

IPSec Connection Procedure

The process of establishing an IPSec VPN tunnel is handled by the Internet Key Exchange (IKE) protocol, which operates in two distinct phases. Phase 1 lays the groundwork for secure communication, while Phase 2 defines how actual data traffic will be protected.

Phase 1: Establishing a Secure Channel

Phase 1 is where two VPN peers first agree on how to communicate securely. At this stage, the devices negotiate a common set of cryptographic policies known as ISAKMP (Internet Security Association and Key Management Protocol) policies. These policies define which encryption algorithms, hashing methods, authentication techniques, and key exchange mechanisms will be used. Once both sides agree on a compatible policy, they authenticate each other. Authentication can be done through pre-shared keys (PSK), where both peers use the same secret key, or through digital certificates (RSA), which provide a more scalable and secure method in large networks.

Finally, Phase 1 establishes a secure channel for further negotiation by performing a Diffie-Hellman key exchange. This ensures that both sides derive a shared secret key without transmitting it over the network in cleartext. The key outcome of Phase 1 is the creation of an encrypted control channel, known as the ISAKMP Security Association (SA). This channel is used exclusively for exchanging additional keying information and policies, not for sending application data.

Phase 2: Negotiating IPSec Security Associations

Once the initial secure channel is in place, Phase 2 begins. This phase uses the established ISAKMP channel to negotiate the specific IPSec parameters that will be applied to actual data traffic flowing through the VPN. the peers agree on the IPSec protocol to use (ESP or AH), whether encryption, authentication, or both will be applied, and the lifetime of the session keys. Unlike Phase 1, which sets up a single secure channel, Phase 2 creates one or more Security Associations (SAs)—unidirectional agreements that define how traffic will be encrypted and authenticated in each direction of communication.

The result of Phase 2 is a fully operational IPSec tunnel where all traffic between the peers is encrypted, authenticated, and integrity-verified according to the negotiated policies. Once these steps are complete, data can flow securely across the network until the session expires or is manually terminated.

Conclusion

IPSec is far more than just a protocol; it is a framework that provides confidentiality, integrity, authentication, and secure key exchange for VPNs. By leveraging AH or ESP, operating in transport or tunnel mode, and using a robust two-phase IKE negotiation process, IPSec ensures that sensitive data remains private and tamper-proof across untrusted networks.

References