[Openswan Users] openswan with sonicwall, payload malformed

[paul pantages]

Hello,

I am trying to get an ipsec vpn client connected to a SonicWall
appliance as per the instructions at:
http://wiki.openswan.org/index.php/Openswan/SonicWall

In particular, this page refs a tech note from SonicWall
http://www.sonicwall.com/downloads/SonicOS_Enhanced_to_Openswan_Using_GroupVPN_with_XAUTH.pdf

I am able to make it through phase I (reaching STATE_MAIN_I4) but get
the old "Hash Payload has an unknown value: xx" / PAYLOAD_MALFORMED
error. (below)

I am not prompted for uid/password,
so I am assuming that the preamble to the Xauth is what is failing.

I also saw the notes at
http://wiki.openswan.org/index.php/Openswan/QuickProblemSolvingGuide
and have switched to

    Basic Constraints: CA:TRUE
    keyUsage = cRLSign, keyCertSign, digitalSignature, *keyEncipherment

etc. in openssl.cnf as directed, but this did not change the behaviour
at all.

Any ideas would be much appreciated.

Thank you, PdP

8<-------------------------------------------------------------------------------------------------------------
Client Side:

> uname -a
Linux rigel 2.6.20-1.2962.fc6 #1 SMP Tue Jun 19 18:50:05 EDT 2007 x86_64
x86_64 x86_64 GNU/Linux

> ipsec version
Linux Openswan U2.4.5/K2.6.20-1.2962.fc6 (netkey)
See `ipsec --copyright' for copyright information.

Sonicwall:

PRO 1260 Enhanced
SonicOS Enhanced 3.2.3.0-6e
SonicROM 3.1.0.4

8<-------------------------------------------------------------------------------------------------------------

[root at rigel pdp]# ipsec verify
Checking your system to see if IPsec got installed and started correctly:
Version check and ipsec on-path                                 [OK]
Linux Openswan U2.4.5/K2.6.20-1.2962.fc6 (netkey)
Checking for IPsec support in kernel                            [OK]
NETKEY detected, testing for disabled ICMP send_redirects       [OK]
NETKEY detected, testing for disabled ICMP accept_redirects     [OK]
Checking for RSA private key (/etc/ipsec.secrets)               [OK]
Checking that pluto is running                                  [OK]
Checking for 'ip' command                                       [OK]
Checking for 'iptables' command                                 [OK]
Opportunistic Encryption Support                                [DISABLED]
[root at rigel pdp]#

8<-------------------------------------------------------------------------------------------------------------

conn myclient
      left=172.16.1.35
      leftsubnet=172.16.1.35/32
      leftxauthclient=yes
      right=sonicwall.wan.addr
      rightsubnet=172.30.0.0/24	 # corporate private network
      rightxauthserver=yes
      keyingtries=1
      pfs=no
      auto=add
      auth=esp
      esp=3des-sha1
      ike=3des-sha1
      xauth=yes
      authby=secret
      aggrmode=no
      rightid=@sonicwall.unique.firewall.identifier

8<-------------------------------------------------------------------------------------------------------------

[root at rigel cisco-vpnclient]# ipsec whack --name myclient --xauthuser
user --xauthpass pass --initiate

002 "myclient" #1: initiating Main Mode
104 "myclient" #1: STATE_MAIN_I1: initiate
003 "myclient" #1: ignoring unknown Vendor ID payload [5b362bc820f60003]
003 "myclient" #1: received Vendor ID payload [RFC 3947] method set to=110
002 "myclient" #1: enabling possible NAT-traversal with method 3
002 "myclient" #1: transition from state STATE_MAIN_I1 to state
STATE_MAIN_I2
106 "myclient" #1: STATE_MAIN_I2: sent MI2, expecting MR2
003 "myclient" #1: ignoring unknown Vendor ID payload [404bf439522ca3f6]
003 "myclient" #1: received Vendor ID payload [XAUTH]
003 "myclient" #1: received Vendor ID payload [Dead Peer Detection]
002 "myclient" #1: I did not send a certificate because I do not have one.
003 "myclient" #1: NAT-Traversal: Result using 3: i am NATed
002 "myclient" #1: transition from state STATE_MAIN_I2 to state
STATE_MAIN_I3
108 "myclient" #1: STATE_MAIN_I3: sent MI3, expecting MR3
003 "myclient" #1: Mode Config message is unacceptable because it is for
an incomplete ISAKMP SA (state=STATE_MAIN_I3)
010 "myclient" #1: STATE_MAIN_I3: retransmission; will wait 20s for response
002 "myclient" #1: Main mode peer ID is ID_FQDN:
'@sonicwall.unique.firewall.identifier'
002 "myclient" #1: transition from state STATE_MAIN_I3 to state
STATE_MAIN_I4
004 "myclient" #1: STATE_MAIN_I4: ISAKMP SA established
{auth=OAKLEY_PRESHARED_KEY cipher=oakley_3des_cbc_192 prf=oakley_sha
group=modp1024}
003 "myclient" #1: next payload type of ISAKMP Hash Payload has an
unknown value: 120
003 "myclient" #1: malformed payload in packet
002 "myclient" #1: sending notification PAYLOAD_MALFORMED to
sonicwall.wan.addr:4500
003 "myclient" #1: next payload type of ISAKMP Hash Payload has an
unknown value: 120
003 "myclient" #1: malformed payload in packet
002 "myclient" #1: sending notification PAYLOAD_MALFORMED to
sonicwall.wan.addr:4500

8<-------------------------------------------------------------------------------------------------------------