I recently encountered an issue creating a site to site IPSec VPN tunnel between Microsoft ISA Server 2006 and Microsoft Forefront TMG 2010. The IPSec tunnel itself would establish correctly, however I found I could only contact resources at either end of tunnel if I initiated the connection from either the ISA or TMG servers themselves. If I attempted to send a PING command from a workstation on the ISA internal network, to a workstation on the TMG internal network it would simply timeout. CIFS, FTP and web traffic all suffered the same fate.
After creating the topology in a test virtual environment, I was able to replicate the exact same issue I was experiencing in the production environment. After a few days of performing traces, I contacted Microsoft Partner Support to ask if they had experienced this problem before. They informed me they had and it was addressed in post SP1 hotfix KB 980674. Even though my TMG toplogy did not contain an NLB, and the KB article specifically states the hotfix is for NLB topologies, it resolved my issue never the less. To ensure your ISA to TMG IPSec site to site VPN works correctly ensure the following actions are performed:
1. Ensure the name of the IPSec VPN is the same on both the ISA and TMG servers. For example, in the site to site VPN wizard on your TMG server, if you name the tunnel “IPSec VPN” this name must also be used when you set the tunnel up on the ISA server.
2. Change the IPSec Phase 1 and Phase 2 security settings on your TMG server to match those on your ISA server. By default, TMG uses a higher level of IPSec encryption than ISA. You can obtain the IPSec security settings by reviewing them in the properties of the site to site VPN tunnel as shown in the below screenshot.
3. Ensure the network address ranges you assign in the site to site tunnel wizard are correct. For example, if you are setting up the VPN connection on your TMG server and the ISA servers internal network address range is 172.16.10.0 – 172.16.10.255 you must specify the network in the tunnel wizard as 172.16.0.0 – 172.16.255.255, otherwise traffic will not pass over the tunnel correctly.
4. Ensure your ISA and TMG servers are fully service packed and all hotfixes are applied, in particular hotfix KB 980674 on your TMG server, even if you do not have an NLB configuration.
Hopefully this help will solve any cross platform IPSec VPN issues you are experiencing.
