Before you jump right in and
start using IPSec, it's important that you learn about the protocol
itself, both what it is capable of and how it works. Since it is
possible to shut down all communications on a network by using IPSec
improperly, taking the time up front to learn the basics can save you
many hours of troubleshooting and embarrassment. It can also ensure that
you are aware of the many features of IPSec and are able to fully
utilize the many advantages
of this complex protocol.
1. Security Advantages of IPSec
IPSec was originally developed for IPv6, and then back-ported to IPv4. IPSec has many advantages, including the following:
It can be used to block, permit, and secure communications.
provides in-depth defense by layering multiple security technologies
(such as authentication, confidentiality, and integrity).
It can be used to tunnel communications across networks or to secure them on the LAN.
It provides flexibility by offering a variety of security algorithms to choose from.
1.1. Blocking, permitting, and securing communications
You implement IPSec by writing IPSec Policies,
a collection of elements that define how IPSec is used on a specific
computer. To secure communications using IPSec, a policy must be present on both IPSec peers
(the two computers involved in the communication) and each policy must
be compatible with the other.
Since IPSec and
a personal firewall (either Microsoft's built in firewall or any other
product) can be used to permit or prevent communications between a
computer and another device on the network, when should you use one or
the other? While the answer is not always clear, here are some
First, and most
obvious, a personal firewall is not equipped to provide secure
communications between computers. The purpose of a personal firewall is
to block or allow communications between the computer it's installed on
and any other device. For example, a personal firewall cannot encrypt
communications between the computer it is installed on and any other
computer. (A full-featured firewall may provide secure remote
communications using VPN services.) When you need to secure
communications between computers on a LAN, IPSec should be your
methodology of choice.
IPSec was not designed to act as a personal firewall. It can only
permit or block communications based on protocol type (selection and
port number) or device identification (IP address). These operations can
provide rudimentary firewall protection similar to that of access control lists
(ACLs) on a router. Modern firewalls provide stateful filtering and
application layer filtering, neither of which can be done by IPSec.
IPSec, however, is a very effective blocking mechanism and may often be
your choice when you want to establish simple block/permit scenarios,
when a personal firewall is not available, or when it's necessary to
create complex protection for communications of a specific type.
IPSec can be scripted for ease of implementation. In a Windows Server
2003 or Windows 2000 domain, IPSec policies can be configured in Group
Policy and rapidly deployed to thousands of computers. This makes it
ideal when simple permitting and blocking must be quickly deployed and
centrally managed without purchasing third-party products. While
configuration of some personal firewalls can also be centrally managed,
to provide centralized management in a mixed Windows environment
(Windows 2000, Windows XP, and Windows Server 2003) would require
additional expense. IPSec also can block ingress (incoming) and egress
(outgoing) communications, while some firewalls (including the free
Microsoft firewall) only block incoming communications.
IPSec can be scripted and applied to systems during startup to protect
communications before all services are initialized. Many personal
firewalls do not do this. (Boot protection via IPSec is enabled by
default when the Windows Firewall installed by Windows XP SP2 is
type of communication) begins, the connection is negotiated using the information contained in the policy. Figure 1 illustrates the location of policies when you want to secure a communication.
The IPSec protocol is
defined in many Internet Engineering Task Force (IETF) Requests for
Comments (RFCs) and drafts. You should examine this information. You can
read it at http://www.ietf.org/rfc.html. Following are a few of the important RFCs:
RFC 2409, The Internet Key Exchange (IKE)
RFC 2402, IP Authentication Header
RFC 2406, IP Encapsulating Security Payload (ESP)
RFC 2401, Security Architecture for the Internet Protocol
RFC 3948, UDP Encapsulation of IPSec ESP Packets
RFC 3947, Negotiation of NAT-Traversal in the IKE
Figure 1. Two IPSec policies are required to secure communications between computers
In addition to
securing communications between computers, you can block or permit
communications to and from a single computer. A properly crafted IPSec
policy can mimic a port filtering router. Just as you might use the
router's ACLs to block port 21 traffic from computer A while permitting
port 80 traffic, or permit port 1434 traffic from a specific host on
network B to a specific host on network A, you can block and permit
traffic at the host level using IPSec. Figures 2 and 3 illustrate this point. Figure 2 shows a simple scenario in which a router controls communications between network A and network B. Figure 3 mimics this setup using an IPSec policy on a single computer.
1.2. Defense in depth
To many people,
encryption is synonymous with communications protection, and IPSec can
be used to encrypt the data being passed between two computers. However,
encryption only provides confidentiality
. Captured, encrypted data is protected
Figure 2. Using a port filtering router
Figure 3. Using IPSec to filter host communications
from those who do not
have the ability to decrypt it. There are, however, other requirements
for protecting communications, and IPSec can also provide them. The
following will help you understand how IPSec provides confidentiality
and some of these other requirements:
secret key encryption to protect data. Knowledge of the secret key is
only shared between IPSec peers. Since an attacker does not have the
key, the attacker cannot decrypt the message. While encryption provides
strong protection, its success requires that the secret key is itself
kept secret, that the encryption algorithm and its implementation be
exemplary so that no weakness in its design or implementation allow an
attack to beat it, and that the key size be large enough to prevent
brute-force attacks. Brute-force attacks
are those that attempt to use every possible key combination. If the
key size is small, it may be possible to deduce the key in this manner.
IPSec provides the implementer with the ability to select from among the
most current encryption algorithms. The Windows implementation provides
choices between Data Encryption Standard (DES) and Triple DES (3DES).
DES is the former United States government encryption standard for data.
Triple DES is a more secure version of that standard. In general, you
should always choose the more secure encryption algorithm. However, you
must also consider the issues of speed and compatibility. Triple DES
will increase processing time, albeit slightly on today's systems.
Processing time will probably not be a factor in encryption strength
selection. However, compatibility may be. Both systems must be able to
use the same algorithm and some IPSec
-capable devices may not provide the option.
might be intercepted and different, encrypted data substituted for the
original data payload, or the original encrypted data might be tampered
with. IPSec provides integrity
by requiring an Integrity Check Value (ICV), a cryptographic hash-based message authentication
code (HMAC), of the data in each packet. A hash is a one-way, mathematical summary of some piece of data. A cryptographic hash
uses a secret key in its calculation. When the IPSec peer receives a
communication, it can verify that the data received has not been
tampered with. It does so by creating its own ICV and matching it with
the one received with the communication. Windows IPSec can be configured
to use either HMAC Message Digest algorithm 5 (MD5) or HMAC Secure Hash
Algorithm-1 (SHA-1) for integrity. SHA-1 is considered to be a more
secure algorithm. However, while processing speed should not be a factor
here, you will also want to consider compatibility.
Data origin authentication
If a normal
hash algorithm were used, it could verify the integrity of the message,
but it could not validate who sent the message. However, because a
cryptographic hash is used, and only the IPSec peer shares the knowledge
of the secret key, the origin of the data can be authenticated.
Attackers cannot calculate a correct ICV because they do not have the
key and, therefore, can't get at the data.
An attacker might
attempt to replace one end of the communications stream by pretending to
be one of the IPSec peer computers, or by inserting himself between
both hosts to intercept and then pass along data—the classic
"man-in-the-middle" attack. To combat this, IPSec requires mutual peer
authentication. Windows IPSec policies can use Kerberos, certificates,
or a shared secret for authentication.
Without sound key management
, encryption keys may be exposed to attackers. Eventually, an encryption
key can be broken, given enough time and computing resources. IPSec's
secret keys (which are used to encrypt data) are never transported
across the network. Instead, both peers calculate the keys using the
Internet Key Exchange (IKE) algorithm. IPSec uses a combination of a
master key and session keys, and the policy can require that the keys
If IPSec traffic
is captured by an attacker, can it be used at a later time in a replay
attack? Several features of IPSec prevent this. First, each
communication is assigned a Security Association
(SA) and is given a timestamp and sequence number. SAs are a set of
information that identify a secured communication and enable its session
key lookup. Since many SAs may exist for any specific host, each SA is
uniquely identified by a Security Parameters Index
(SPI). The SA consists of the SPI, the session key, and the security
protocol. Once a communication is started, the SA, the SA sequence
number, and the timestamp can uniquely identify each packet. IPSec
communications can be configured to establish new SAs periodically and
to time out an SA if its lifetime is exceeded. If the attacker waits too
long to attempt a replay attack, the attack will fail because the SA no
longer exists. If the SPI of the SA of the replay packet matches a
current SA SPI but its timestamp is out of sequence, the packet will be
dropped. If the SPI, timestamp, and sequence number match a combination
that has already been received, the packet will be dropped.
1.3. IPSec tunnels
IPSec is often used to
protect communications between computers on the LAN. It can, however,
also be used in tunnel mode to carry data from LAN to LAN across a wide
area network. When an IPSec tunnel is used, data is encapsulated within a
new header and routed between two endpoints. The tunnel creates a
virtual network. One endpoint sits on one network (as an example,
perhaps corporate headquarters), and the other sits on another (for
example, a branch office network). The data may be routed across the
Internet or any other network(s).
The tunnel does not make the data any more secure. IPSec tunnels are
not often recommended because there are more effective tunneling
protocols available. In fact, Windows VPNs that use IPSec use the layer 2
tunneling protocol instead of the IPSec tunnel.
IPSec is flexible because of the following characteristics:
You can choose
from a selection of authentication, encryption, and integrity
algorithms to best suit your environment. For example, you can select
algorithms that meet constraints supplied by a requirement to support
IPsec between devices supporting different algorithms, or perhaps where
communications must be protected and legal constraints do not allow some
can select key management options such as the frequency of changing
keys or the strength of the keys. This means, once again, that you may
be able to use IPSec between more types of devices. Selecting very
frequent key change or longer keys may reduce performance. If this is a
concern, the option to balance security against performance is
You can choose between transport or tunnel mode. In tunnel mode, data is encapsulated and may be transported between two networks over a third. In transport mode, no tunnel is required, which is suitable for securing LAN communications.
can choose to use encryption with Encapsulating Security Payload (ESP),
or to use superior integrity protection with an Authentication Header
(AH). ESP and AH are the two IPSec sub-protocols. To select specific
security features of IPSec (such as integrity or encryption), you must
first select AH and/or ESP. Both protocols offer several of the same
security benefits, but differ in how they do so. ESP offers an
additional benefit—encryption. You should be aware of these differences
in order to select the right protocol for your requirements. More
information on how the protocols differ follows.