Windows Server 2003 : Protecting Network Communications with Internet Protocol Security - IPSec Basics (part 1) - Security Advantages of IPSec

8/22/2012 5:22:54 PM
Before you jump right in and start using IPSec, it's important that you learn about the protocol itself, both what it is capable of and how it works. Since it is possible to shut down all communications on a network by using IPSec improperly, taking the time up front to learn the basics can save you many hours of troubleshooting and embarrassment. It can also ensure that you are aware of the many features of IPSec and are able to fully utilize the many advantages of this complex protocol.

1. Security Advantages of IPSec

IPSec was originally developed for IPv6, and then back-ported to IPv4. IPSec has many advantages, including the following:

  • It can be used to block, permit, and secure communications.

  • It provides in-depth defense by layering multiple security technologies (such as authentication, confidentiality, and integrity).

  • It can be used to tunnel communications across networks or to secure them on the LAN.

  • It provides flexibility by offering a variety of security algorithms to choose from.

1.1. Blocking, permitting, and securing communications

You implement IPSec by writing IPSec Policies, a collection of elements that define how IPSec is used on a specific computer. To secure communications using IPSec, a policy must be present on both IPSec peers (the two computers involved in the communication) and each policy must be compatible with the other.

IPSec or Personal Firewall?

Since IPSec and a personal firewall (either Microsoft's built in firewall or any other product) can be used to permit or prevent communications between a computer and another device on the network, when should you use one or the other? While the answer is not always clear, here are some guidelines:

  • First, and most obvious, a personal firewall is not equipped to provide secure communications between computers. The purpose of a personal firewall is to block or allow communications between the computer it's installed on and any other device. For example, a personal firewall cannot encrypt communications between the computer it is installed on and any other computer. (A full-featured firewall may provide secure remote communications using VPN services.) When you need to secure communications between computers on a LAN, IPSec should be your methodology of choice.

  • Second, IPSec was not designed to act as a personal firewall. It can only permit or block communications based on protocol type (selection and port number) or device identification (IP address). These operations can provide rudimentary firewall protection similar to that of access control lists (ACLs) on a router. Modern firewalls provide stateful filtering and application layer filtering, neither of which can be done by IPSec. IPSec, however, is a very effective blocking mechanism and may often be your choice when you want to establish simple block/permit scenarios, when a personal firewall is not available, or when it's necessary to create complex protection for communications of a specific type.

  • Third, IPSec can be scripted for ease of implementation. In a Windows Server 2003 or Windows 2000 domain, IPSec policies can be configured in Group Policy and rapidly deployed to thousands of computers. This makes it ideal when simple permitting and blocking must be quickly deployed and centrally managed without purchasing third-party products. While configuration of some personal firewalls can also be centrally managed, to provide centralized management in a mixed Windows environment (Windows 2000, Windows XP, and Windows Server 2003) would require additional expense. IPSec also can block ingress (incoming) and egress (outgoing) communications, while some firewalls (including the free Microsoft firewall) only block incoming communications.

  • Finally, IPSec can be scripted and applied to systems during startup to protect communications before all services are initialized. Many personal firewalls do not do this. (Boot protection via IPSec is enabled by default when the Windows Firewall installed by Windows XP SP2 is installed.)

type of communication) begins, the connection is negotiated using the information contained in the policy. Figure 1 illustrates the location of policies when you want to secure a communication.

IPSec RFCs and Drafts

The IPSec protocol is defined in many Internet Engineering Task Force (IETF) Requests for Comments (RFCs) and drafts. You should examine this information. You can read it at Following are a few of the important RFCs:

  • RFC 2409, The Internet Key Exchange (IKE)

  • RFC 2402, IP Authentication Header

  • RFC 2406, IP Encapsulating Security Payload (ESP)

  • RFC 2401, Security Architecture for the Internet Protocol

  • RFC 3948, UDP Encapsulation of IPSec ESP Packets

  • RFC 3947, Negotiation of NAT-Traversal in the IKE

Figure 1. Two IPSec policies are required to secure communications between computers

In addition to securing communications between computers, you can block or permit communications to and from a single computer. A properly crafted IPSec policy can mimic a port filtering router. Just as you might use the router's ACLs to block port 21 traffic from computer A while permitting port 80 traffic, or permit port 1434 traffic from a specific host on network B to a specific host on network A, you can block and permit traffic at the host level using IPSec. Figures 2 and 3 illustrate this point. Figure 2 shows a simple scenario in which a router controls communications between network A and network B. Figure 3 mimics this setup using an IPSec policy on a single computer.

1.2. Defense in depth

To many people, encryption is synonymous with communications protection, and IPSec can be used to encrypt the data being passed between two computers. However, encryption only provides confidentiality . Captured, encrypted data is protected

Figure 2. Using a port filtering router

Figure 3. Using IPSec to filter host communications

from those who do not have the ability to decrypt it. There are, however, other requirements for protecting communications, and IPSec can also provide them. The following will help you understand how IPSec provides confidentiality and some of these other requirements:


IPSec uses secret key encryption to protect data. Knowledge of the secret key is only shared between IPSec peers. Since an attacker does not have the key, the attacker cannot decrypt the message. While encryption provides strong protection, its success requires that the secret key is itself kept secret, that the encryption algorithm and its implementation be exemplary so that no weakness in its design or implementation allow an attack to beat it, and that the key size be large enough to prevent brute-force attacks. Brute-force attacks are those that attempt to use every possible key combination. If the key size is small, it may be possible to deduce the key in this manner. IPSec provides the implementer with the ability to select from among the most current encryption algorithms. The Windows implementation provides choices between Data Encryption Standard (DES) and Triple DES (3DES). DES is the former United States government encryption standard for data. Triple DES is a more secure version of that standard. In general, you should always choose the more secure encryption algorithm. However, you must also consider the issues of speed and compatibility. Triple DES will increase processing time, albeit slightly on today's systems. Processing time will probably not be a factor in encryption strength selection. However, compatibility may be. Both systems must be able to use the same algorithm and some IPSec -capable devices may not provide the option.


Communications might be intercepted and different, encrypted data substituted for the original data payload, or the original encrypted data might be tampered with. IPSec provides integrity by requiring an Integrity Check Value (ICV), a cryptographic hash-based message authentication code (HMAC), of the data in each packet. A hash is a one-way, mathematical summary of some piece of data. A cryptographic hash uses a secret key in its calculation. When the IPSec peer receives a communication, it can verify that the data received has not been tampered with. It does so by creating its own ICV and matching it with the one received with the communication. Windows IPSec can be configured to use either HMAC Message Digest algorithm 5 (MD5) or HMAC Secure Hash Algorithm-1 (SHA-1) for integrity. SHA-1 is considered to be a more secure algorithm. However, while processing speed should not be a factor here, you will also want to consider compatibility.

Data origin authentication

If a normal hash algorithm were used, it could verify the integrity of the message, but it could not validate who sent the message. However, because a cryptographic hash is used, and only the IPSec peer shares the knowledge of the secret key, the origin of the data can be authenticated. Attackers cannot calculate a correct ICV because they do not have the key and, therefore, can't get at the data.


An attacker might attempt to replace one end of the communications stream by pretending to be one of the IPSec peer computers, or by inserting himself between both hosts to intercept and then pass along data—the classic "man-in-the-middle" attack. To combat this, IPSec requires mutual peer authentication. Windows IPSec policies can use Kerberos, certificates, or a shared secret for authentication.

Key management

Without sound key management , encryption keys may be exposed to attackers. Eventually, an encryption key can be broken, given enough time and computing resources. IPSec's secret keys (which are used to encrypt data) are never transported across the network. Instead, both peers calculate the keys using the Internet Key Exchange (IKE) algorithm. IPSec uses a combination of a master key and session keys, and the policy can require that the keys change frequently.

Replay resistance

If IPSec traffic is captured by an attacker, can it be used at a later time in a replay attack? Several features of IPSec prevent this. First, each communication is assigned a Security Association (SA) and is given a timestamp and sequence number. SAs are a set of information that identify a secured communication and enable its session key lookup. Since many SAs may exist for any specific host, each SA is uniquely identified by a Security Parameters Index (SPI). The SA consists of the SPI, the session key, and the security protocol. Once a communication is started, the SA, the SA sequence number, and the timestamp can uniquely identify each packet. IPSec communications can be configured to establish new SAs periodically and to time out an SA if its lifetime is exceeded. If the attacker waits too long to attempt a replay attack, the attack will fail because the SA no longer exists. If the SPI of the SA of the replay packet matches a current SA SPI but its timestamp is out of sequence, the packet will be dropped. If the SPI, timestamp, and sequence number match a combination that has already been received, the packet will be dropped.

1.3. IPSec tunnels

IPSec is often used to protect communications between computers on the LAN. It can, however, also be used in tunnel mode to carry data from LAN to LAN across a wide area network. When an IPSec tunnel is used, data is encapsulated within a new header and routed between two endpoints. The tunnel creates a virtual network. One endpoint sits on one network (as an example, perhaps corporate headquarters), and the other sits on another (for example, a branch office network). The data may be routed across the Internet or any other network(s).  The tunnel does not make the data any more secure. IPSec tunnels are not often recommended because there are more effective tunneling protocols available. In fact, Windows VPNs that use IPSec use the layer 2 tunneling protocol instead of the IPSec tunnel.

1.4. Flexibility

IPSec is flexible because of the following characteristics:

  • You can choose from a selection of authentication, encryption, and integrity algorithms to best suit your environment. For example, you can select algorithms that meet constraints supplied by a requirement to support IPsec between devices supporting different algorithms, or perhaps where communications must be protected and legal constraints do not allow some algorithms.

  • You can select key management options such as the frequency of changing keys or the strength of the keys. This means, once again, that you may be able to use IPSec between more types of devices. Selecting very frequent key change or longer keys may reduce performance. If this is a concern, the option to balance security against performance is available.

  • You can choose between transport or tunnel mode. In tunnel mode, data is encapsulated and may be transported between two networks over a third. In transport mode, no tunnel is required, which is suitable for securing LAN communications.

  • You can choose to use encryption with Encapsulating Security Payload (ESP), or to use superior integrity protection with an Authentication Header (AH). ESP and AH are the two IPSec sub-protocols. To select specific security features of IPSec (such as integrity or encryption), you must first select AH and/or ESP. Both protocols offer several of the same security benefits, but differ in how they do so. ESP offers an additional benefit—encryption. You should be aware of these differences in order to select the right protocol for your requirements. More information on how the protocols differ follows.

  •  Windows Vista : Communicating with Windows Mail - Handling Incoming Messages (part 2) - Customizing the Message Columns, Setting Read Options
  •  Windows Vista : Communicating with Windows Mail - Handling Incoming Messages (part 1) - Processing Messages
  •  Windows Vista : Communicating with Windows Mail - Setting Up Mail Accounts
  •  Ultra-X P.H.D PCI2 - Solve PC Problems Easily (Part 2)
  •  Ultra-X P.H.D PCI2 - Solve PC Problems Easily (Part 1)
  •  Confessions Of An Internet Troll (Part 2)
  •  Confessions Of An Internet Troll (Part 1)
  •  Windows Vista or Windows Server 2008 : Architecture of Group Policy - Domain Controller Selection During GPO Management
  •  Windows Vista or Windows Server 2008 : Architecture of Group Policy - Group Policy Dependencies
  •  Retina MacBook Pro
  •  Suitcase Fusion 4
  •  Canon PIXMA MX895
  •  Samsung Series 5 Ultra
  •  ASUS U32U : Upscale Version Of A Netbook
  •  The Ubuntu Server Project (Part 6)
  •  The Ubuntu Server Project (Part 5)
  •  The Ubuntu Server Project (Part 4)
  •  The Terminal : Command line interface for the Mac, Unix foundation
  •  Sharp Aquos LC32LE340M : Just Enough
  •  Quicksilver : Giving your Mac a boost of power
    Top 10
    Windows Vista : Windows PowerShell (part 2) - PowerShell Variables, PowerShell Scripts, Automate Scripts with the Task Scheduler
    Windows Vista : Windows PowerShell (part 1) - CmdLets and Aliases, Pipelines
    Windows 7 : How to Troubleshoot Bluetooth Problems, Troubleshooting Tools
    Iphone Application : Building Rotatable and Resizable User Interfaces - Swapping Views on Rotation
    Iphone Application : Building Rotatable and Resizable User Interfaces - Reframing Controls on Rotation
    Open GL : Advanced Geometry Management - Synchronizing When OpenGL Begins to Draw
    Open GL : Advanced Geometry Management - Clipping and Determining What Should Be Drawn
    Downloading and Installing 3CX Phone System (part 3) - Checking the status of 3CX
    Downloading and Installing 3CX Phone System (part 2) - Starting the install
    Downloading and Installing 3CX Phone System (part 1)
    Most View
    Programming .NET Framework 3.5 : Placing Text
    Accessing XML Data in Silverlight
    Oracle Coherence 3.5 : Accessing the data grid (part 1) - Coherence console
    Installing SharePoint 2010 Using PowerShell
    Create a TCP/IP Client and Server
    Mobile Application Security : The Apple iPhone - Local Data Storage: Files, Permissions, and Encryption
    Buying Guide: CPU Cooling Equipment (Part 1)
    Onyx Calypso 9.7 Tablet
    100 Windows Speed-Up Tips (Part 2) : Streamline your PC, Set up SMART
    Complete Guide for Your PC & How to Buy It (Part 1) - Choose PC System
    Toshiba MQ01ABD100 1TB Hard Drive
    Nikon Coolpix L25
    Using System Center Operations Manager to Simplify Management of SharePoint 2010
    Dell Inspiron 14R 5420 Review (Part 2)
    Kingston HyperX 3K SSD : SSD For The Budget Conscious
    Windows Tips & Tricks (Part 4)
    A Case For Quality (Part 4) - Maroo – Seth Aaron Series: Dotty, ColcaSac – Juan Valdez iPad Sleeve
    LEPA G1000 - Surpassing 80 Plus Gold Level
    Windows 8's Unexpected Features (Part 1)
    Separating BPM and SOA Processes : Disputes on the Model Stack