[Openswan Users] PAYLOAD_MALFORMED error with cisco PIX

[Tharanga]

Hi List,

iam new to the openswan project. iam connecting to  a  cisco pix .

conn tunnelipsec
        type= tunnel
        authby= secret
        #aggrmode=yes
        left= 203.94.xx.xx
        leftnexthop= 203.94.xx.xx
        #leftsubnet= 147.120.0.0/24
        right= 194.247.yy.yy
        #rightnexthop= 203.97.9.161
        rightsubnet= 194.247.yy.yy/24
        esp= 3des-sha1-1536
        ike= 3des-sha1-1536
        keyexchange= ike
        pfs= no
        auto= start

iam using 3DES, sha1, and Diffie-helman group 5 with a shared key.. when i
try to connect it says tunnel is up. (status comand).
but logs says
Jun  3 11:33:33 SMS-GW pluto[7441]: "tunnelipsec" #9: max number of
retransmissions (2) reached STATE_MAIN_I3.  Possible authentication failure:
no acceptable response to our first encrypted message
Jun  3 11:33:33 SMS-GW pluto[7441]: |    responder cookie:
Jun  3 11:33:33 SMS-GW pluto[7441]: |    responder cookie:
Jun  3 11:33:33 SMS-GW pluto[7441]: |    responder cookie:


un  3 11:44:43 SMS-GW pluto[8715]: "tunnelipsec" #1: malformed payload in
packet
Jun  3 11:44:43 SMS-GW pluto[8715]: | payload malformed after IV
Jun  3 11:44:43 SMS-GW pluto[8715]: |   7f 5a 91 4c  ea 27 ac 58
Jun  3 11:44:43 SMS-GW pluto[8715]: "tunnelipsec" #1: sending notification
PAYLOAD_MALFORMED to


and iam getting  PAYLOAD_MALFORMED error in the logs.

ipsec.secretes looks lks this

y.y.yy.yy  x.x.xx xx: PSK "mysharedkey"


iam using fedora core 8 2.6.23.1-42.fc8  x86_64.
strongswan version - Linux Openswan 2.4.12 (klips)

Checking your system to see if IPsec got installed and started correctly:
Version check and ipsec on-path                                 [OK]
Linux Openswan 2.4.12 (klips)
Checking for IPsec support in kernel                            [OK]
KLIPS detected, checking for NAT Traversal support              [FAILED]
Checking for RSA private key (/etc/ipsec.secrets)               [OK]
Checking that pluto is running                                  [OK]
Two or more interfaces found, checking IP forwarding            [OK]
Checking NAT and MASQUERADEing                                  [N/A]
Checking for 'ip' command                                       [OK]
Checking for 'iptables' command                                 [OK]
Opportunistic Encryption Support                                [DISABLED]


where i went wrong ?

many thanks,
Tharanga