[Openswan Users] PAYLOAD_MALFORMED to lsipsectool

[Mihajlo Cvetanović]

I'm trying to establish a road warrior connection from win200 to FC5, 
with certificates and lsipsectool. Openswan keeps complaining about some 
malformed payloads. What is wrong here? I've put root certificate and 
certificate with CN=mac22 into Windows repository via lsipsectools, but 
they both appear in "Root Certificates" list. Desired network 
configuration is:

80.80.80.0/24...[80.80.80.53/24(eth0),10.0.0.3/8(eth1)]===10.0.0.22/8(win2000)

Ethereal capture file is also attached.

/etc/ipsec.conf
=================================
version    2.0    # conforms to second version of ipsec.conf specification

config setup
    interfaces="ipsec0=eth1"

conn mihajlo-sale
    left=10.0.0.3
    leftcert=/etc/ipsec.d/certs/westCert.pem
    rightrsasigkey=%cert
    right=%any
    rightid="C=SR, ST=srbija, O=netset, CN=mac22"
    auto=add

#Disable Opportunistic Encryption
include /etc/ipsec.d/examples/no_oe.conf
=================================

/var/log/secure
=================================
Jun 15 16:31:34 localhost ipsec__plutorun: Starting Pluto subsystem...
Jun 15 16:31:34 localhost pluto[3525]: Starting Pluto (Openswan Version 
2.4.5 X.509-1.5.4 PLUTO_SENDS_VENDORID PLUTO_USES_KEYRR; Vendor ID 
OEnMCu\177xOp at c)
Jun 15 16:31:34 localhost pluto[3525]: Setting NAT-Traversal port-4500 
floating to off
Jun 15 16:31:34 localhost pluto[3525]:    port floating activation 
criteria nat_t=0/port_fload=1
Jun 15 16:31:34 localhost pluto[3525]:   including NAT-Traversal patch 
(Version 0.6c) [disabled]
Jun 15 16:31:34 localhost pluto[3525]: ike_alg_register_enc(): 
Activating OAKLEY_AES_CBC: Ok (ret=0)
Jun 15 16:31:34 localhost pluto[3525]: starting up 1 cryptographic helpers
Jun 15 16:31:34 localhost pluto[3525]: started helper pid=3526 (fd:6)
Jun 15 16:31:34 localhost pluto[3525]: Using KLIPS IPsec interface code 
on 2.6.15
Jun 15 16:31:34 localhost pluto[3525]: Changing to directory 
'/etc/ipsec.d/cacerts'
Jun 15 16:31:34 localhost pluto[3525]:   loaded CA cert file 
'caCert.pem' (1127 bytes)
Jun 15 16:31:34 localhost pluto[3525]: Changing to directory 
'/etc/ipsec.d/aacerts'
Jun 15 16:31:34 localhost pluto[3525]: Changing to directory 
'/etc/ipsec.d/ocspcerts'
Jun 15 16:31:34 localhost pluto[3525]: Changing to directory 
'/etc/ipsec.d/crls'
Jun 15 16:31:34 localhost pluto[3525]:   Warning: empty directory
Jun 15 16:31:34 localhost pluto[3525]:   loaded host cert file 
'/etc/ipsec.d/certs/westCert.pem' (960 bytes)
Jun 15 16:31:34 localhost pluto[3525]: added connection description 
"mihajlo-sale"
Jun 15 16:31:34 localhost pluto[3525]: listening for IKE messages
Jun 15 16:31:34 localhost pluto[3525]: adding interface ipsec0/eth1 
10.0.0.3:500
Jun 15 16:31:34 localhost pluto[3525]: loading secrets from 
"/etc/ipsec.secrets"
Jun 15 16:31:34 localhost pluto[3525]:   loaded private key file 
'/etc/ipsec.d/private/west.key' (963 bytes)
Jun 15 16:31:41 localhost pluto[3525]: packet from 10.0.0.22:500: 
ignoring Vendor ID payload [MS NT5 ISAKMPOAKLEY 00000002]
Jun 15 16:31:41 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
responding to Main Mode from unknown peer 10.0.0.22
Jun 15 16:31:41 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
transition from state STATE_MAIN_R0 to state STATE_MAIN_R1
Jun 15 16:31:41 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
STATE_MAIN_R1: sent MR1, expecting MI2
Jun 15 16:31:41 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
transition from state STATE_MAIN_R1 to state STATE_MAIN_R2
Jun 15 16:31:41 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
STATE_MAIN_R2: sent MR2, expecting MI3
Jun 15 16:31:41 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
next payload type of ISAKMP Hash Payload has an unknown value: 116
Jun 15 16:31:41 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
malformed payload in packet
Jun 15 16:31:41 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
sending notification PAYLOAD_MALFORMED to 10.0.0.22:500
Jun 15 16:31:52 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
byte 2 of ISAKMP Hash Payload must be zero, but is not
Jun 15 16:31:52 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
malformed payload in packet
Jun 15 16:31:52 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
sending notification PAYLOAD_MALFORMED to 10.0.0.22:500
Jun 15 16:32:11 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
ignoring informational payload, type INVALID_COOKIE
Jun 15 16:32:11 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
received and ignored informational message
Jun 15 16:32:51 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22 #1: 
max number of retransmissions (2) reached STATE_MAIN_R2
Jun 15 16:32:51 localhost pluto[3525]: "mihajlo-sale"[1] 10.0.0.22: 
deleting connection "mihajlo-sale" instance with peer 10.0.0.22 
{isakmp=#0/ipsec=#0}
Jun 15 16:36:44 localhost pluto[3525]: shutting down
Jun 15 16:36:44 localhost pluto[3525]: forgetting secrets
Jun 15 16:36:44 localhost pluto[3525]: "mihajlo-sale": deleting connection
Jun 15 16:36:44 localhost pluto[3525]: shutting down interface 
ipsec0/eth1 10.0.0.3:500
=================================

Log file of lsipsectool (from Win2000)
=================================
16:33:01: Starting Tunnel

16:33:01: IKE Encryption: 3des
IKE Integrity: md5
Remote Gateway Address: 10.0.0.3
Remote Monitor Address: 80.80.80.53
Remote Network: 80.80.80.0/255.255.255.0
Local Address: 10.0.0.22
Local Network: 10.0.0.0/255.0.0.0

16:33:02: 15 Consecutive Unsuccessfull ECHO REQUEST [ Waiting 5 Secs ]...

16:33:07: 30 Consecutive Unsuccessfull ECHO REQUEST [ Waiting 5 Secs ]...

16:33:10: Stoping Tunnel
=================================


-------------- next part --------------
A non-text attachment was scrubbed...
Name: rw_lsipsectool.cap
Type: application/octet-stream
Size: 1566 bytes
Desc: not available
Url : http://lists.openswan.org/pipermail/users/attachments/20060615/3e7f6774/rw_lsipsectool.obj