IPSec issues with Vista Service Pack1


New Member

I am facing a strange issue. I am using a Vista Service Pack1 installed PC as one end of an IPSec connection. The other end is a Linux node.

Am using "advfirewall consec" to configure rules. Rule properties are:
transport mode, DHgroup2 , AES128 bit mode, 3600 secs lifetime, action=requireinrequireout of main mode.
Quickmode parameters are: 3600secs life time, 5 mins idle timeout, AES128, SHA1.

According to my application the linux node has to initate the ISAKMP negotation. If this happens then the IPSec negotiation is complete and I checked the SA's formed by using: "netsh firewall dynamic" show mmsas, show qmsas. Both the commands displayed one entry each for the other side. This is the successful case. In this case there are 3 pairs of main mode packets and two pairs of quick mode packets exchanged.

Failure case is: I have a TCP application running on the Vista PC which also tries to connect to the Linux box periodically to read statistics from that box. In the failure case both the Linux end and the Vista end start ISAKMP mani mode negotiation respectively ( 3 sets of exchanges happen successfully) and as a result upon execution of the command "netsh ipsec dynamic show mmsas" I see two main SA's with different cookie pairs established between the Vista and the Linux box. On the other side - Linux end we see only one main mode SA. The IPSec connection is sestablished successfuly with One main mode pair and onse quick mode. There exists an additional main mode security associations. Here starts the problem. Based upon captured packets in Vista PC I observed that the PC now initiates ISAKMP quick mode packets towards the Linux node ( I assume this quick mode packets it sends if for the second MMSA that it has created - correct me If I am wrong). These quick mod packets are not answered by the Linux side. It has only one MM SA and one QM SA established. After 6 retransmissions of the Quick mode packets by the Vista PC, the SA is now broken. I see a ISAKMP informational packet originated by the Vista PC. I have enabled IKE logging and have checked the retransmission count, timer values are set preoperly etc. After these packets when I see the command "show mmsas" and show qmsas" I observe that one main mode packet is left over without its corresponding quick mode. The IPSec negotiation does not succeed here on. Upon clearing up all SA's and restarting the negotiation every thing works fine. The TCP application that we run in the back ground tires to connect to the box and read logs from the box. This in needed in our case.

I have two questions in this regard:
1) Is it possible to have two main mode SA's between two ends of an IPSec connection with the same parameters ? can't the Vista PC detect if there already exists a MMSA and not create a new MMSA.
2) If the answer for 1) is yes (there are prossible two MMSA's even with same configuration), how do we handle the error situation that I have explained above.

Someone, please answer my question or redirect me - whom to approach for such issues.


My Computer