IPsec is a name that's been given to a suite of security protocols used to secure IP traffic between computers.
When two computers (peers) use IPsec to communicate, they create two kinds of
security associations. In the first, called
main mode or
phase one, the peers mutually authenticate themselves to each other, thus establishing trust between the computers. In the second, called
quick mode or
phase two, two peers negotiate the particulars of the security association, including how they digitally sign and encrypt traffic between them.
A computer can have only one IPsec policy assigned at a time. The policy can have any number of rules, each of which has a filter list and a filter action. Filter lists contain one or more filters that specify: source and destination addresses, source and destination port numbers, and protocol types. Filter actions specify the behaviors of the rule: whether to permit traffic, block traffic, or negotiate the pair of IPsec security associations. Actions that specify negotiating security can have many options, including encryption suites, per-packet authentication methods, how often to generate new keys, how to respond to incoming insecure requests, and whether to communicate with computers that don't support IPsec.
Each rule in an IPsec policy combines:
- One filter list with one filter action
- The security association's mode (transport or tunnel)
- One of three phase-one authentication methods
Traffic that matches a particular filter list is processed according to the settings in the linked filter action.
The three phase-one authentication methods are preshared keys, digital certificates, and Kerberos.
Filter actions that negotiate security can choose one or both of two different phase two security methods (protocols): Authentication Header (AH) and Encapsulating Security Payload (ESP). They can be run individually or on top of one another. In addition, they both support two distinct modes of operation: transport and tunnel. AH provides only authentication and integrity protection; it doesn't encrypt the channel. ESP provides full CIA on the channel, but costs a bit more in terms of bandwidth. Ferguson and Schneier recommend ESP in tunnel mode as the only option. Keith Brown further suggests that you use ESP in transport mode for securing communications within your organization. L2TP+IPsec VPNs use tranport mode. In Windows IPsec, tunnel mode is supported only for site-to-site VPNs on RRAS gateways and not for any kind of client-to-client or client-to-server communications.
Transport mode and tunnel mode are the two kinds of phase one security association modes.
Key exchange is described by a couple of layered standards: Internet Security Association Key Management Protocol (ISAKMP), and Internet Key Exchange (IKE). A Security Association (SA) is identified by a triplet that consists of a peer's IP address, a protocol identifier (AH or ESP), and an index to a set of parameters (such as what encryption and hash algorithms should be used to protect packets). One of the things that the key exchange protocols do is help establish an SA between two machines.
IPsec is thus very useful for protecting communications between machines, but it doesn't help a server implement any form of user authentication or authorization.
In Windows, you enable IPsec via security policy. Either you can edit the local security policy of individual machines or you can use domain group policy to configure IPsec in a consistent way on a whole group of machines.
To configure IPsec related settings, it's recommended to use the ESP protocol using 3DES encryption with HMAC-SHA1 integrity protection. Another option you should enable is called "Session key perfect forward secrecy (PFS)."
References:
