[Openswan Users] next payload type of ISAKMP Hash Payload has an unknown value

[Brett Curtis]

Hello,

Versions: openswan-2.4.7 / 2.6.18-gentoo-r3
x86_64 arch

I am trying to switch my VPN over to x509 certs from PSK this weekend  
while down time is allowed.

I have followed Nate's Guide and everything as far as creating the  
certs went well. My openswan loads all the correct certs and starts  
up fine.
I imported my .p12 into windows both by hand and with the  
certimport.exe tool. This seemed to work fine in both cases.

However when I try to connect I get the windows 786 lt2p error and  
this in my openswan logs.

Jan 13 14:49:30 defender64 pluto[6562]: packet from  
74.65.156.181:500: ignoring Vendor ID payload [MS NT5 ISAKMPOAKLEY  
00000004]
Jan 13 14:49:30 defender64 pluto[6562]: packet from  
74.65.156.181:500: ignoring Vendor ID payload [FRAGMENTATION]
Jan 13 14:49:30 defender64 pluto[6562]: packet from  
74.65.156.181:500: received Vendor ID payload [draft-ietf-ipsec-nat-t- 
ike-02_n] method set to=106
Jan 13 14:49:30 defender64 pluto[6562]: packet from  
74.65.156.181:500: ignoring Vendor ID payload [Vid-Initial-Contact]
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: responding to Main Mode from unknown peer  
74.65.156.181
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: transition from state STATE_MAIN_R0 to state  
STATE_MAIN_R1
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: STATE_MAIN_R1: sent MR1, expecting MI2
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: NAT-Traversal: Result using draft-ietf-ipsec-nat-t- 
ike-02/03: peer is NATed
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: transition from state STATE_MAIN_R1 to state  
STATE_MAIN_R2
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: STATE_MAIN_R2: sent MR2, expecting MI3
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: next payload type of ISAKMP Hash Payload has an  
unknown value: 51
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: malformed payload in packet
Jan 13 14:49:30 defender64 pluto[6562]: | payload malformed after IV
Jan 13 14:49:30 defender64 pluto[6562]: |   e7 12 22 63  76 fe 09 0c   
0e 2a b9 ec  7b 5e 1b 52
Jan 13 14:49:30 defender64 pluto[6562]: |   9a c7 1f 66
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: sending notification PAYLOAD_MALFORMED to  
74.65.156.181:500
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: next payload type of ISAKMP Hash Payload has an  
unknown value: 39
Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: malformed payload in packet
Jan 13 14:50:40 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181 #1: max number of retransmissions (2) reached  
STATE_MAIN_R2
Jan 13 14:50:40 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]  
74.65.156.181: deleting connection "roadwarrior-osx-xp" instance with  
peer 74.65.156.181 {isakmp=#0/ipsec=#0}

I have searched these errors for a few hours now with no luck.

Thanks for any help.

-Brett