[Openswan Users] next payload type of ISAKMP Hash Payload has an unknown value

[Paul Wouters]

On Mon, 5 Feb 2007, Brett Curtis wrote:

> Sorry to reply to myself  (I switched email addresses for this list) anyways
> I am still having the issue as described below. It is not mtu related as I
> am now testing on subnets inside my LAN.
>
> What should I look into?

Do you have a > 1024bit key in your certificate? that was cause IKE
fragmentation that won't work with Openswan.

Create/check the OAKLEY.LOG on windows to see what it is doing, and if
that is what you think it hsould do.

Paul

> >Versions: openswan-2.4.7 / 2.6.18-gentoo-r3
> >x86_64 arch
>
> >I have followed Nate's Guide and everything as far as creating the
> >certs went well. My openswan loads all the correct certs and starts
> >up fine.
> >I imported my .p12 into windows both by hand and with the
> >certimport.exe tool. This seemed to work fine in both cases.
>
> >However when I try to connect I get the windows 786 lt2p error and
> >this in my openswan logs.
>
> >Jan 13 14:49:30 defender64 pluto[6562]: packet from
> >74.65.156.181:500: ignoring Vendor ID payload [MS NT5 ISAKMPOAKLEY
> >00000004]
> >Jan 13 14:49:30 defender64 pluto[6562]: packet from
> >74.65.156.181:500: ignoring Vendor ID payload [FRAGMENTATION]
> >Jan 13 14:49:30 defender64 pluto[6562]: packet from
> >74.65.156.181:500: received Vendor ID payload [draft-ietf-ipsec-nat-t-
> >ike-02_n] method set to=106
> >Jan 13 14:49:30 defender64 pluto[6562]: packet from
> >74.65.156.181:500: ignoring Vendor ID payload [Vid-Initial-Contact]
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: responding to Main Mode from unknown peer
> >74.65.156.181
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: transition from state STATE_MAIN_R0 to state
> >STATE_MAIN_R1
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: STATE_MAIN_R1: sent MR1, expecting MI2
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: NAT-Traversal: Result using draft-ietf-ipsec-nat-t-
> >ike-02/03: peer is NATed
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: transition from state STATE_MAIN_R1 to state
> >STATE_MAIN_R2
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: STATE_MAIN_R2: sent MR2, expecting MI3
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: next payload type of ISAKMP Hash Payload has an
> >unknown value: 51
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: malformed payload in packet
> >Jan 13 14:49:30 defender64 pluto[6562]: | payload malformed after IV
> >Jan 13 14:49:30 defender64 pluto[6562]: |   e7 12 22 63  76 fe 09 0c
> >0e 2a b9 ec  7b 5e 1b 52
> >Jan 13 14:49:30 defender64 pluto[6562]: |   9a c7 1f 66
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: sending notification PAYLOAD_MALFORMED to
> >74.65.156.181:500
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: next payload type of ISAKMP Hash Payload has an
> >unknown value: 39
> >Jan 13 14:49:30 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: malformed payload in packet
> >Jan 13 14:50:40 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181 #1: max number of retransmissions (2) reached
> >STATE_MAIN_R2
> >Jan 13 14:50:40 defender64 pluto[6562]: "roadwarrior-osx-xp"[1]
> >74.65.156.181: deleting connection "roadwarrior-osx-xp" instance with
> >peer 74.65.156.181 {isakmp=#0/ipsec=#0}
> >
> >I have searched these errors for a few hours now with no luck.
> >
> >Thanks for any help.
>

-- 
Building and integrating Virtual Private Networks with Openswan:
http://www.amazon.com/gp/product/1904811256/104-3099591-2946327?n=283155