[Openswan Users] PAYLOAD_MALFORMED error with cisco PIX

[Peter McGill]

Tharanga,

Check that all your settings match the other end, including...
Aggressive Mode: Off (aggrmode=no, note: turning this on decreases security.)
Perfect Forward Secrecy: Off (pfs=no, note: turning this on increases security.)
Phase1 & Phase2: 3DES SHA1 Diffie Helman Group 5
	(ike=3des-sha1-modp1535 and esp=3des-sha1, note esp does not show the
	dh group, instead it inherits the dh group from phase1/ike.)
No NAT-T, each IPSec endpoint has a public internet IP address.
Key in ipsec.secrets is specified by ip addresses of endpoints, not hostname/ids.
(this is a limitation of psk keys.)
left and right are the public internet ip addresses of the two ipsec computers.
leftsubnet and rightsubnet are private lan networks behind the ipsec computers.
Are you sure you intended to comment out the leftsubnet line?
(leftsubnet defaults to left, if not specified.)
Also aggrmode defaults to no, if not specified.


Peter McGill
IT Systems Analyst
Gra Ham Energy Limited 

> -----Original Message-----
> From: users-bounces at openswan.org 
> [mailto:users-bounces at openswan.org] On Behalf Of Tharanga
> Sent: June 5, 2008 4:18 AM
> To: users at openswan.org
> Subject: [Openswan Users] how to enable Diffie helman group 5
> 
> Hi All,
> 
> i need to enable 3DES,SHA1 with Diffie-helman group 5 witha 
> preshared key.
> authby=psk also not working ?
> 
> i tried.
> 
> authby=secret
> esp=3des-sha1-modp1536
> ike=3des-sha1-modp1536
> 
> but this was not recognized?  can i change  keyexchange= 
> ikev2 ? it gave an
> error ?
> 
> pls help.

> -----Original Message-----
> From: users-bounces at openswan.org 
> [mailto:users-bounces at openswan.org] On Behalf Of Tharanga
> Sent: June 4, 2008 10:08 PM
> To: users at openswan.org
> Subject: [Openswan Users] PAYLOAD_MALFORMED error with cisco PIX
> 
> Hi List,
> 
> iam new to the openswan project. iam connecting to  a  cisco pix .
> 
> conn tunnelipsec
>         type= tunnel
>         authby= secret
>         #aggrmode=yes
>         left= 203.94.xx.xx
>         leftnexthop= 203.94.xx.xx
>         #leftsubnet= 147.120.0.0/24
>         right= 194.247.yy.yy
>         #rightnexthop= 203.97.9.161
>         rightsubnet= 194.247.yy.yy/24
>         esp= 3des-sha1-1536
>         ike= 3des-sha1-1536
>         keyexchange= ike
>         pfs= no
>         auto= start
> 
> iam using 3DES, sha1, and Diffie-helman group 5 with a shared 
> key.. when i
> try to connect it says tunnel is up. (status comand).
> but logs says
> Jun  3 11:33:33 SMS-GW pluto[7441]: "tunnelipsec" #9: max number of
> retransmissions (2) reached STATE_MAIN_I3.  Possible 
> authentication failure:
> no acceptable response to our first encrypted message
> Jun  3 11:33:33 SMS-GW pluto[7441]: |    responder cookie:
> Jun  3 11:33:33 SMS-GW pluto[7441]: |    responder cookie:
> Jun  3 11:33:33 SMS-GW pluto[7441]: |    responder cookie:
> 
> 
> un  3 11:44:43 SMS-GW pluto[8715]: "tunnelipsec" #1: 
> malformed payload in
> packet
> Jun  3 11:44:43 SMS-GW pluto[8715]: | payload malformed after IV
> Jun  3 11:44:43 SMS-GW pluto[8715]: |   7f 5a 91 4c  ea 27 ac 58
> Jun  3 11:44:43 SMS-GW pluto[8715]: "tunnelipsec" #1: sending 
> notification
> PAYLOAD_MALFORMED to
> 
> 
> and iam getting  PAYLOAD_MALFORMED error in the logs.
> 
> ipsec.secretes looks lks this
> 
> y.y.yy.yy  x.x.xx xx: PSK "mysharedkey"
> 
> 
> iam using fedora core 8 2.6.23.1-42.fc8  x86_64.
> strongswan version - Linux Openswan 2.4.12 (klips)
> 
> Checking your system to see if IPsec got installed and 
> started correctly:
> Version check and ipsec on-path                                 [OK]
> Linux Openswan 2.4.12 (klips)
> Checking for IPsec support in kernel                            [OK]
> KLIPS detected, checking for NAT Traversal support            
>   [FAILED]
> Checking for RSA private key (/etc/ipsec.secrets)               [OK]
> Checking that pluto is running                                  [OK]
> Two or more interfaces found, checking IP forwarding            [OK]
> Checking NAT and MASQUERADEing                                  [N/A]
> Checking for 'ip' command                                       [OK]
> Checking for 'iptables' command                                 [OK]
> Opportunistic Encryption Support                              
>   [DISABLED]
> 
> 
> where i went wrong ?
> 
> many thanks,
> Tharanga
> 
> 
> _______________________________________________
> Users at openswan.org
> http://lists.openswan.org/mailman/listinfo/users
> Building and Integrating Virtual Private Networks with Openswan: 
> http://www.amazon.com/gp/product/1904811256/104-3099591-294632
> 7?n=283155