IP Encapsulating Security Payload ESP

While the AH solves the problem of IP packet integrity and authentication, it does not address the issue of confidentiality. In many cases, you need to make sure that nobody is gaining unauthorized access to the payload of your IP packets as they travel over an untrusted network. The IP Encapsulated Security Payload (ESP) specification allows for a security gateway to completely encapsulate a private outgoing packet before it travels via the public Internet. This can be done in one of two ways: ESP transport mode and ESP tunnel mode.

ESP Transport Mode

You can use ESP transport mode to protect the payload of IP packets exchanged between two security gateways that are implementing IPsec security. When ESP transport mode is used, the security gateway inserts an ESP header between the original IP and TCP headers (see Figure 12.4). This header contains a pointer to the security association (SA) that was negotiated between the two security gateways, as well as to the pointers to the next ESP packet that the endpoint should expect.

Figure 12.4 IPsec ESP transport mode packet format

After the TCP header and TCP payload, the security gateway inserts an ESP trailer that contains a variable number of padding characters to make sure the packet to be protected is of a fixed length.

Finally, the security gateway appends an ESP authentication value consisting of a hash value calculated using the entire ESP packet (minus the authentication value) as the input. This works similarly to the AH protocol and ensures the integrity of the packet.

Note that ESP encrypts the contents of the original TCP header and payload and ensures the authentication of the entire ESP packet.

ESP Tunnel Mode

Note that ESP transport mode protects the TCP header and TCP payload of the original packet, but it does not protect the original IP header. If you need to ensure the confidentiality, integrity, and authentication of the entire original IP packet, you should opt for ESP tunnel mode. ESP tunnel mode effectively encapsulates the entire IP packet and inserts it into a brand new packet whose source IP address is typically the externally reachable address of the security gateway that is doing the IPsec encapsulation.

The ESP tunnel mode packet format, illustrated in Figure 12.5, is similar to the format used for ESP transport mode, except for the fact that the original IP header is also authenticated and encrypted.

Figure 12.5 IPsec ESP tunnel mode packet format

Another advantage of ESP tunnel mode is that packets originating from an IPsec-protected network need not use legal Internet addresses because those addresses will be effectively "hidden" while the packet is in transit over the Internet.

While the ESP specification (defined in RFC2406) allows for new encryption algorithms to be used, it also mandates that IPsec-conformant implementations include support for the DES encryption algorithm, as well as for the MD5 and SHA-1 authentication algorithms mentioned earlier in this chapter in the section "IP Authentication Headers." Since the strength of the DES algorithm has been brought into question recently, most IPsec software vendors typically support stronger encryption algorithms such as 3DES, IDEA, and Blowfish.

As is usually the case, the Linux open-source community has stepped up to develop a robust implementation of the IPsec standard. The FreeS/WAN Linux IPsec package is the topic of discussion in the next section of this chapter.

Continue reading here: Installing FreeSWAN

Was this article helpful?