[Openswan Users] Openswan<--->vigor malformed payload

[Peter McGill]

Abdul-Wahid,

Set pfs=no in ipsec.conf, in ipsec.secrets should have : PSK "secret".
Verify that PSK's are the same on both sides, reenter if neccessary.
Make sure the Vigor is using Perfect Forward Secrecy (PFS).
(pfs=no, makes pfs optional, it will still use it if Vigor asks for it,
but this way if Vigor doesn't the connection will still work.)
Make sure that the Vigor isn't using Diffie-Hellman (DH) Group 1 (768 
bit), but instead DH Group 2 (1024 bit) or 5 (1536 bit). 1 is 
obsolete/insecure and openswan will not work with it.
I'm assuming this is your only roadwarrior (dynamic address) client
connection. If using psk's, then all roadwarriors must use the same one.

Peter

Abdul-Wahid Paterson wrote:
> Hi,
>  
> I have setup my Vigor 2600 for VPN tunnel to openswan but I am receiving 
> malformed payload messages. The ISAKMP AS is established ok but when 
> starting IPSec SA it gets the malformed packets. What could be the 
> possible cause of this.
>  
> Conf...
>  
> 
> conn kscdubai
>         authby=secret
>         left=196.202.140.66 <http://196.202.140.66>
>         leftsubnet=10.1.0.0/17 <http://10.1.0.0/17>
>         leftnexthop=196.202.140.65 <http://196.202.140.65>
>         leftsourceip=10.1.0.1 <http://10.1.0.1>
>         right=%any
>         rightsubnet=10.0.97.0/24 <http://10.0.97.0/24>
>         keyingtries=0
>         keyexchange=ike
>         esp=aes128-sha1,aes128-md5,3des-sha1,3des-md5
>         ike=aes256-sha1,aes128-sha1,aes128-md5,3des-sha1,3des-md5
>         compress=no
>         pfs=yes
>         auto=add
>  
> Sep  2 15:36:46 ksckhf0101 pluto[15566]: "kscdubai"[1] 86.98.26.69 
> <http://86.98.26.69> #43: STATE_MAIN_R3: sent MR3, ISAKMP SA established 
> {auth=OAKLEY_PRESHARED_KEY cipher=oakley_3des_cbc_192 prf=oakley_sha 
> group=modp1024}
> Sep  2 15:36:47 ksckhf0101 pluto[15566]: "kscdubai"[1] 86.98.26.69 
> <http://86.98.26.69> #44: responding to Quick Mode {msgid:4e9d3b76}
> Sep  2 15:36:47 ksckhf0101 pluto[15566]: "kscdubai"[1] 86.98.26.69 
> <http://86.98.26.69> #44: ERROR: netlink XFRM_MSG_NEWPOLICY response for 
> flow tun.10000 at 196.202.140.66 <mailto:tun.10000 at 196.202.140.66> included 
> errno 17: File exists
> Sep  2 15:36:47 ksckhf0101 pluto[15566]: "kscdubai"[1] 86.98.26.69 
> <http://86.98.26.69> #44: transition from state STATE_QUICK_R0 to state 
> STATE_QUICK_R1
> Sep  2 15:36:47 ksckhf0101 pluto[15566]: "kscdubai"[1] 86.98.26.69 
> <http://86.98.26.69> #44: STATE_QUICK_R1: sent QR1, inbound IPsec SA 
> installed, expecting QI2
> Sep  2 15:36:50 ksckhf0101 pluto[15566]: "kscdubai"[1] 86.98.26.69 
> <http://86.98.26.69> #44: next payload type of ISAKMP Hash Payload has 
> an unknown value: 206
> Sep  2 15:36:50 ksckhf0101 pluto[15566]: "kscdubai"[1] 86.98.26.69 
> <http://86.98.26.69> #44: malformed payload in packet
> Sep  2 15:36:50 ksckhf0101 pluto[15566]: "kscdubai"[1] 86.98.26.69 
> <http://86.98.26.69> #44: sending notification PAYLOAD_MALFORMED to 
> 86.98.26.69:500 <http://86.98.26.69:500>
>  
> Any pointers would be great.
> 
> AW
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Users at openswan.org
> http://lists.openswan.org/mailman/listinfo/users
> Building and Integrating Virtual Private Networks with Openswan: 
> http://www.amazon.com/gp/product/1904811256/104-3099591-2946327?n=283155