I’ve been meaning to write something about Internet protocol security (IPSec) ever since I heard Steve Lamb talk about it a few months back but Owen Cutajar blogged about Steve Friedl’s Illustrated Guide to IPSec a few days back which gives a much better description than I ever will! Steve’s site has a whole load of useful technical tips, but as his URL might give away, he comes at things from a UNIX perspective.

For Windows users who are interested in implementing IPSec, I recommend that you read both Steve Lamb’s blog and Steve Friedl’s Illustrated Guide to IPSec, but what follows is a brief description of some high-level concepts which might help to put it all into context.

Although it sounds complex, symmetric key cryptography is a very basic method of encrypting messages (e.g. DES or AES/Rijndael) using a shared secret. The plain text input is encrypted to produce cipher text which is transmitted to the intended recipient, who can then decrypt it to produce plain text output. An example of such a mechanism is the Caesar shift, whereby characters are shifted by a known number of places (the shared secret), so that for example if the shared secret is 3, A becomes D, B becomes E, and so on. Symmetric key cryptography is simple, and fast, but relies on some form of mechanism for exchanging keys (shared secrets).

Public key cryptography is an asymmetric encryption mechanism, whereby knowledge of the encryption key doesn’t provide the methods to decrypt the message. The recipient of the message generates a pair of keys (using a certificate authority) and publishes the public key in a directory so that anyone can send them encrypted messages that only they can read. This pair of keys is actually a single key split mathematically using a one-way algorithm (i.e. one which current mathematics does not allow to be reversed). When sending a message, it is encrypted with the recipient’s public key and they can decrypt it (using their private key). Unfortunately even this method has its weaknesses as it is slow, subject to what is known as a “known ciphertext” attack and requires the public key to be trusted (i.e. to be from a known certificate authority).

The real-world answer is often a hybrid encryption process whereby a symmetric session key is encrypted using the recipient’s public key and then, once this key has been decrypted by the recipient (using their private key), they can read messages encrypted using the session key. The session key is transmitted with the encrypted message as a digital envelope. Once the message exchange is complete (whether that is literally the transfer of a message, or a communication session) the session key is disregarded (i.e. its life is finite – dictated by the length of the session).

IPSec is used to authenticate and/or encrypt TCP/IP communications, securing either specific ports or all IP traffic and is obligatory for IPv6.

In an Active Directory environment, IPSec is generally configured via group policy and both the client and the server must be configured. No reply is issued to rejected packets – they are simply dropped. Installing a certificate authority (CA) is a simple process (although because a lot of the configuration is wizard-based, it can be difficult to appreciate exactly what has been done). Windows Server 2003 Certificate Services allows a hierarchy of CAs to be implemented (generally with the root CA kept offline once the hierarchy is established) as well as adhering to public key standards from RSA, Entrust and Verisign (licensed by Microsoft to avoid any per-certificate cost issues). Once a certificate has been issued the client no longer needs to communicate with the CA. Of course, internal CAs are only suitable for internal use of IPSec (a trusted CA needs to be used for securing traffic across the Internet).

One of the advantages of IPSec is that, because it works at the network layer, it can be used to provide secure data transfer without affecting applications; however the downside is that architects (or administrators) should carefully consider the impact that encrypting all traffic would cause as some security software (e.g. intrusion detection systems) will no longer function.