ISA To TMG – Site To Site IPSec VPN

I recently encountered an issue creating a site to site IPSec VPN tunnel between Microsoft ISA Server 2006 and Microsoft Forefront TMG 2010. The IPSec tunnel itself would establish correctly, however I found I could only contact resources at either end of tunnel if I initiated the connection from either the ISA or TMG servers themselves. If I attempted to send a PING command from a workstation on the ISA internal network, to a workstation on the TMG internal network it would simply timeout. CIFS, FTP and web traffic all suffered the same fate.

After creating the topology in a test virtual environment, I was able to replicate the exact same issue I was experiencing in the production environment. After a few days of performing traces, I contacted Microsoft Partner Support to ask if they had experienced this problem before. They informed me they had and it was addressed in post SP1 hotfix KB 980674. Even though my TMG toplogy did not contain an NLB, and the KB article specifically states the hotfix is for NLB topologies, it resolved my issue never the less. To ensure your ISA to TMG IPSec site to site VPN works correctly ensure the following actions are performed:

1. Ensure the name of the IPSec VPN is the same on both the ISA and TMG servers. For example, in the site to site VPN wizard on your TMG server, if you name the tunnel “IPSec VPN” this name must also be used when you set the tunnel up on the ISA server.

2. Change the IPSec Phase 1 and Phase 2 security settings on your TMG server to match those on your ISA server. By default, TMG uses a higher level of IPSec encryption than ISA. You can obtain the IPSec security settings by reviewing them in the properties of the site to site VPN tunnel as shown in the below screenshot.

3. Ensure the network address ranges you assign in the site to site tunnel wizard are correct. For example, if you are setting up the VPN connection on your TMG server and the ISA servers internal network address range is 172.16.10.0 – 172.16.10.255 you must specify the network in the tunnel wizard as 172.16.0.0 – 172.16.255.255, otherwise traffic will not pass over the tunnel correctly.

4. Ensure your ISA and TMG servers are fully service packed and all hotfixes are applied, in particular hotfix KB 980674 on your TMG server, even if you do not have an NLB configuration.

Hopefully this help will solve any cross platform IPSec VPN issues you are experiencing.

ISA Server 2006 – Missing RRAS Ports

After recently rebooting a Microsoft ISA 2006 server, I found myself in the situation of being unable to establish VPN connectivity to it from a remote location. After performing several troubleshooting methods, I identified that there were no available PPTP RAS ports listed in the Routing and Remote Access (RRAS) console.

This explained why VPN connections could not be established, however I could see that a pool of ten ports had been configured but for some reason were not available. Some research on this issue led me to identify this was the result of a recent Windows Update. If you have applied KB 956570 to your ISA Server and then rebooted it, you may find yourself in the same situation. To resolve this issue, perform the following actions on the affected server:

1. Click Start and select Run

2. In the Run dialog box enter the following registry command to reset the servers reserved port threshold. Please note, enter this command without the quotation marks:

“reg add HKLM\SYSTEM\CurrentControlSet\Services\Fweng\Parameters /v ReservedPortThreshold /t REG_DWORD /d 1250 /f”

3. To uninstall the update, please run the following script which has been provided by Microsoft for this purpose. I have collated this script, which can be downloaded here for your convenience.

4. Create a new folder on the C: drive of your ISA Server named KBFix and copy the downloaded script (KB956570.vbs) to this folder location.

5. Open the command prompt and browse to the new folder you have just created. This can be achieved by typing cd c:\kbfix and then pressing return on your keyboard.

6. Once in the correct directory type cscript KB956570.vbs and then press return on your keyboard. This will now start the process of removing the update.

7. Once the removal process is complete, reboot your ISA Server and VPN connectivity will be restored.

For reference, here is the official Microsoft TechNet KB article on the problem: http://support.microsoft.com/kb/956570