Domain Isolation Rules
Isolation rules are the rules that restrict connections based on various criteria, such as domain membership. Building a rule to allow inbound connections only from domain members is actually quite simple with this wizard. Make sure Isolation is selected, as shown in Figure 4, and click Next to get the window in Figure 5.
In Figure 5, you get to define whether you want to require or request authentication, and on which types of connections. If you are building a domain isolation rule for clients, you probably want to require authentication on inbound connections, as clients should not be servers-but only request it on outbound connections as not all servers may support IPsec. With prior versions of Windows, it was paramount to also have a large number of exceptions that did not require authentication. The reason was that the fall-back to clear when a client requested authentication and the server did not support it was longer than the reset interval on most connections. The result was that even if a computer was set up to request but not require authentication, all outbound connections configured this way failed. This was fixed in Windows Vista by changing the fall-back to clear time-out value when a computer requests authentication. A computer will now fall back to clear text much faster, making it possible to request encryption in scenarios where the request may be ignored. Previously such requests would almost invariable result in failed connections. This change was also back-ported to down-level platforms. See Microsoft Knowledge Base article 914841 for more details.
The next step is to decide which type of authentication to use. Figure 6 shows the screen for this. The default selection, somewhat surprisingly, is "Default."
On this screen you may notice that the option for pre-shared keys has been removed. Pre-shared key IPsec authentication is still supported, under the Advanced option, but is highly discouraged. Not only are pre-shared keys an absolutely nightmare to manage, but they are also stored in clear-text in the registry, making them notoriously insecure. The default option uses whichever authentication options are configured via Group Policy for the firewall itself. The defaults can be configured in the firewall properties, as shown in Figure 7.
Paradoxically, the dialog box in Figure 7 that you use to configure the default authentication options also has a default option. In other words, the default is to use default, which defaults to default. Fortunately, the default is "Computer (using Kerberos v5)," which is probably what you want anyway, if you haven't become so confused by now you have given up entirely. (By default, it is actually not as confusing as the defaults may seem.) Leave it alone and you get computer authentication using Kerberos v5.
The only thing left after Figure 7 is to configure which profiles you want this rule used in. You may not want it in all profiles. If it is a domain isolation rule, for instance, and it uses Kerberos as the authentication protocol, it would be quite meaningless to allow it to be used when the computer is in the public profile.
It is also important that you create the appropriate exceptions using directional rules to allow the IPsec secured traffic to make it through. In other words, if you have a connection security rule that allows some traffic, but there is no inbound rule that permits it, the traffic is blocked because the inbound firewall will block it before the IPsec filter even comes into play. However, if you have a directional rule that allows the traffic if it is secure, the connection security rule will take effect and allow traffic that meets its requirements.
For this reason, we recommend that you create an inbound rule in the firewall that allows all traffic from domain members if it is secure. This creates a blanket IPsec bypass that permits all traffic that matches the conditions of one or more connection security rules. If no connection security rules permit the connection, it will be blocked at the IPsec layer.
Server Isolation Rules
Server Isolation rules are also much simpler to build in Windows Vista than in earlier versions of Windows. First, go back to the screen shown in Figure 4 and select a server-to-server rule. Next, select which computers are in the two endpoints. This is shown in Figure 8.
As you can see in Figure 8, you can configure server isolation rules for ranges of IP addresses. However, you cannot configure them for hostnames. In other words, server-to-server isolation rules can be difficult to build for clients where you typically use DHCP. However, if you simply want to permit inbound connections from one or a few systems, such as a management station, the problem can be solved by ensuring that the management station(s) have predefined IP addresses. In that case, you would simply specify one of the endpoints as "Any IP address." As far as IPsec is concerned, there is no concept of "my IP address." IPsec does not consider the direction traffic was sent in the same way a firewall does. Therefore, either endpoint 1 or endpoint 2 can be "Any IP address." It makes no difference either way.
Once you have selected the endpoints, the rest of the rule configuration proceeds exactly like a domain isolation rule, with selecting the connection requirements authentication, which profiles the rule applies in, and so forth. Oddly enough, the Domain member authentication options do not appear in the authentication dialog box for server-to-server rules. You have to select Advanced and then Customize … to select Computer (Kerberos v5) to use Kerberos for authentication. Stranger still, the Preshared key option is shown in this dialog box. There may be some solid reasoning behind this dialog box design, but it could also be that Microsoft did not complete the UI design for connection security rules.
