Setting up IPsec bypass

Mark Minasi explains how to set up IPsec bypass, a technology that requires both your clients and their machines to be authenticated prior to accessing your network. This chapter is an excerpt from Minasi's book, "Mastering Windows Server 2003 Upgrade Edition for SP1 and R2."

SP1 and R2

Windows Firewall Basics
By Mark Minasi

Mark Minasi is a best-selling author, commentator and all-around alpha geek. He is best known for his books in the Mastering Windows series. The following excerpt is from chapter eight of Minasi's newest book, Mastering Windows Server 2003 Upgrade Edition for SP1 and R2, entitled "Windows Firewall Basics." Read the entire chapter here.

Check out the first half of this chapter excerpt, Automatic exceptions: IPsec bypass, from Mark Minasi's book, Mastering Windows Server 2003 Upgrade Edition for SP1 and R2.

How to set up IPsec bypass

That's the overview. As always, the specifics can get a bit uglier. Let's look at the steps in some more detail.

You need Active Directory

IPsec bypass needs AD because configuring IPsec bypass on a given server requires that you specify the name of a group (I'm simplifying when I say "group," because in actuality it's a list of groups and machines, but I'll fill in the details on that in a bit; for now, it's just easier to say "group") containing the client systems that you want to be able to bypass the firewall in talking to the server. Local machine groups, however, can't contain machine accounts, just user accounts.

IPsec bypass also requires AD because IPsec bypass requires an authenticated IPsec relationship between the client and the server, and IPsec bypass is extra picky about that authentication in that it insists on Kerberos authentication. Kerberos authentication on IPsec relationships in the Microsoft world only works between members of the same Active Directory forest.

A minimum client-server configuration that can exploit IPsec bypass, then, must have at least one client and at least one server who are members of the same domain. (I found it easiest to do my initial IPsec bypass experiments by testing IPsec bypass on a member server, which required a three-machine setup rather than playing with IPsec bypass on a domain controller, but either arrangement works.)

For the sake of example, suppose you've got a slightly different setup than the one we've been using so far in this chapter. Let's suppose that you've got three systems in a test domain called bigfirm. com:

  • A domain controller name at It's also a DNS server and hosts the zone for the AD. as well as the two other systems point to this DNS server for their DNS needs. It needn't have Windows firewall enabled for this example, although if you did want it firewalled, then you'd have to make the changes to WF discussed in the previous section, "Windows Firewall and Domain Controllers."

  • A member server called at It hosts a secure website that you want only someone sitting at, your next system, to be able to access. (If you want to test this out, then you don't need anything fancier than to put a file in c:inetpubwwwroot on that contains the words "Welcome to the secure website!") It is a domain member and, obviously, has IIS running. When setting IPsec bypass up on any server, I find it easier testing-wise to keep WF disabled until I've tested my IPsec policies, as you'll see as you go along. Recall that points to dc1.bigfirm. com for DNS.

  • An XP client called, the privileged system who will be the only one to be able to view's website. It is also a domain member and sits at It really doesn't matter whether its firewall is up or down. points to for DNS.

Create a group of trusted client machines in AD

If you're going to use IPsec bypass to restrict which machines can access a given server, than you must create a domain group and make those machines members of that group. I've used global groups in my experience with IPsec bypass, but I'd imagine that domain local groups would work fine, provided that the clients and the server were all in the same group.

You might run into one problem, however, when adding a machine to that group. When you want to add a member to a group, of course, then you open up the group in Active Directory Users and Computers, where you're presented with a property page with tabs labeled General, Members, Member Of, and Managed By. If you click the Members tab, then you add a member by clicking the Add button, which offers you the Select Users, Contacts, or Computers dialog box that you've no doubt seen many times. In the Enter the Object Names to Select field, you type the name of the machine, and click OK or Check Names, and that's where the fun starts.

If you've never added a machine to a group before, then you may not notice that in the Select Users, Contacts, or Computers dialog is a field called Select This Object Type, and it says by default Users or Other Objects. Apparently, "other objects" don't include computer accounts. Next to the field is a button labeled Object Types which, if clicked, will show you a list of the kinds of things that the dialog box will accept, and by default you'll see check boxes next to Other Objects and Users…but not next to Contacts and, significantly for us in this case, Computers. Just check the box next to Computers to return to Select Users, Contacts, and Computers, and now it'll accept your machine names.

If you intend to use IPsec bypass to protect more than one server, you'll probably want to create a separate group for each server.

WARNING: And remember that whenever a user or machine becomes a member of a group, then that user or machine doesn't actually get the benefits of being a member of that group until the user or machine logs off, then on. That means that if you were, for example, to create a group that you decided to call "trusted" and put a machine named xpclient in that group, then xpclient would not look to the rest of the network like a member of trusted until you rebooted xpclient. I figured that a simple netdom reset command would do the trick, but as far as I can see, adding a machine to a group or deleting a machine from a group doesn't really take effect until you reboot the machine.

If you're following along in my example then at this point you should start up Active Directory Users and Computers and create a new global group in called trusted, and add xpclient to it. Then reboot xpclient to make the membership change actually happen.

Create and test IPsec policies linking the client machines and servers

Next, examine how the client systems communicate with the server, and create IPsec policies on both the clients and the server to allow them to communicate either with signed (integrity in the lingo of the Windows IPsec interface, or AH in IPsec RFC-speak) or encrypted (integrity and encryption in the Windows UI, ESP in RFC-speak) communications. If all you're interested in is the IPsec bypass, than either signing or encrypting will do fine because IPsec bypass is interested only in seeing the client machine authenticate to the server machine, and signing and encrypting are the only two of the four IPsec filter actions that require authentication. (The other two IPsec filter actions are permit and block.) If your main interest in IPsec is as a vehicle to IPsec bypass, then remember that signing takes a lot less CPU power than encrypting and therefore signing might be preferable.

Let's quickly review IPsec. (It's covered in more detail in Chapter 6 of Mastering Windows Server 2003.) Recall that IPsec policies are composed of one or more IPsec rules, and that IPsec rules incorporate two main mandatory parts and a sometimes-mandatory one:

    IP filter list. This tells the IPsec policy agent running on every 2000, XP, and 2003 system when to wake up and do something. More specifically, IP filter lists look like "use this rule when you see traffic coming from X to Y from port A to port B." X and Y can be very general, as in "from any IP address to any IP address"; a bit more specific, as in "from any IP address to my IP address"; very specific, as in "from to"; or anything in between. The source and destination ports can be either "any port" or a particular port. IP filter lists also specify whether a communication is TCP, UDP, or another protocol. So, for example, if one of your clients were at and the server were at and you only needed the client to be able to access the web service on, then you could build an IP filter list along the lines of "this filter kicks in when there's traffic from, any port, to on port 80, protocol TCP." Alternatively, someone just liking IPsec's ability to, say, digitally sign traffic might create an IP filter list that says "any data coming from any IP address on the Internet to my IP address." Such a filter would always kick in for any IP traffic. Filter lists can also be "mirrored" to save time in creating them. The idea behind mirroring is that if you want, for example, your data to be encrypted when A talks to B, then you probably also want encryption when B replies to A. An IPsec rule can incorporate as many IP filter lists as you like.

    Filter action. This says, "Now that the criteria for the IP filter list's criteria have been met, what do we do?" There are four possible actions: permit the traffic to pass untouched, block the traffic from arriving, sign the traffic, or encrypt it. Thus, you could use IPsec as a means to block some arbitrary port 107 by first creating an IP filter list that says, "Employ this rule whenever there is data coming from any IP address, any port to my IP address, TCP port 107," and the resultant action would be, "Block the traffic." By writing many of these rules, one could create a software firewall on any IPsec-aware system. If signing or encrypting, then there are many different cryptographic algorithms, and both sides of the conversation would have to agree on an algorithm.

    Authentication. If the filter action is either signing or encrypting, then both sides of the conversation must authenticate to each other. Windows IPsec supports three ways of authenticating. First, both sides might know the same text password, like "open sesame" or the like. Second, both sides might offer certificates to prove their identity. (Note again that when IPsec says "authentication," it wants one machine to authenticate itself to another machine—there are no user accounts involved here.) Finally, Windows IPsec supports using Active Directory's Kerberos protocol to authenticate between the two parties to the conversation. As mentioned before, IPsec bypass requires Kerberos authentication; shared secrets or certificates will not work.

Putting it together, then, to allow a client at to always sign any communications with the web server on the system at, then you'd build two IPsec policies: one for the client and one for the server.

The server policy would look like:

  • IP filter list: "This filter activates when data originates from anywhere on the Internet on any port to my IP address, TCP port 80. This filter is mirrored, so whatever action we take on receiving this incoming data we should also do to my replies."

  • Filter action: "Digitally sign all traffic, using SHA-1." (SHA is Secure Hashing Algorithm, and it's commonly used for digital signatures.)

  • Authentication: "Let's use Kerberos."

You create IPsec policies on a computer either by working in the GUI interface of the local Group Policy Editor, or by creating a domain-based group policy object containing an IPsec policy, or from the command line with either ipseccmd on XP or netsh ipsec on 2003. (And please forgive me for not taking you through the process click by click, but it's not the focus of this chapter and besides, if I did, then I'd have to reproduce about 25 pages from Chapter 6 of the earlier book.)

The client would also need an IPsec policy. It might look like:

  • IP filter list: "This filter activates when data originates from my IP address on any port to on TCP port 80. It is mirrored." Thus, the client has a fairly specific filter that only kicks in when doing web communication with a particular system.

  • Filter action: "Digitally sign all traffic, using SHA-1."

  • Authentication: "Let's use Kerberos."

Once you've got these policies in place, I highly recommend testing them. I'd fire up a copy of IE on the client and point it at the server to ensure that the IPsec relationship ("security association" is the RFC-ese for "relationship") can be established. While I'm at it, here are a few notes on testing and troubleshooting an IPsec security association.

Check out the first half of this chapter excerpt, Automatic exceptions: IPsec bypass, from Mark Minasi's book, "Mastering Windows Server 2003 Upgrade Edition for SP1 and R2."

Mark Minasi is a best-selling author, commentator and all-around alpha geek. Mark is best known for his books in the Mastering Windows series. What separates him from others is that he knows how to explain technical things to normal humans, and make them laugh while doing it. Mark's firm, MR&D, is based in Pungo, a town in Virginia's Tidewater area that is distinguished by having one -- and only one -- traffic light.
Copyright 2005 TechTarget

Read more on Antivirus, firewall and IDS products