[Openswan Users] Wndows 2008 server IPsec client and openswan - malformed payload

[Gupta, Deepak (Deepak)]

Hello,

We are trying to setup a tunnel (host to host, no NAT-T needed) between a Windows 2008 server and a RHEL 5.3 box running openswan 2.6.14.  For authenctication we are using x509 certs.

And seeing the following error in /var/log/secure:

got payload 0x100(ISAKMP_NEXT_HASH) needed: 0x100 opt: 0x0
Oct  7 17:43:20 STATION_B pluto[11790]: "ag-10.120.51.12" #2: byte 2 of ISAKMP Hash Payload must be zero, but is not
Oct  7 17:43:20 STATION_B pluto[11790]: "ag-10.120.51.12" #2: malformed payload in packet
Oct  7 17:43:20 STATION_B pluto[11790]: | payload malformed after IV
Oct  7 17:43:20 STATION_B pluto[11790]: |   a2 8f 0d 4b  62 18 77 e5
Oct  7 17:43:20 STATION_B pluto[11790]: "ag-10.120.51.12" #2: sending notification PAYLOAD_MALFORMED to 10.120.51.12:500

I have attached the secure file and also a txt version of the wireshark capture of the exchange.

My initial take is that this is some issue on Windows end during phase1, or perhaps with the certificates, I can't tell for sure.  I don't have access to the Windows end, just the Linux end.

Any pointers would be appreciated.

Thanks,

-Deepak
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: gba_ipsec_DES_02.txt
Url: http://lists.openswan.org/pipermail/users/attachments/20101007/c80d49a5/attachment-0002.txt 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: secure_07oct_DES_02.txt
Url: http://lists.openswan.org/pipermail/users/attachments/20101007/c80d49a5/attachment-0003.txt