[Openswan Users] PAYLOAD_MALFORMED error with cisco PIX

[Peter McGill]

Tharanga,

1535 was a typo, it most definately is 1536.
Consider the binary, 2^10 (1024) + 2^9 (512) = 1536.
My key point for the esp line, was that you don't use -modp* for it.
Only the ike line takes the dh group, the esp line doesn't.
Phase2 then uses the same dh group as was in phase1/ike.

Peter McGill

Tharanga wrote:
> Hi Peter,
> 
> Thxs for the mail. and  i have verified all the settings with the cisco end.
> i have changed my esp, and ike settings as follows.
> 
> esp=3des-sha1-modp1535
>  ike=3des-sha1-modp1535
> 
> now its not starting it says
> 
> un  6 07:07:29 SMS-GW ipsec__plutorun: 034 esp string error: modp group not
> found, enc_alg="3des", auth_alg="sha1", modp="modp1535"
> Jun  6 07:07:29 SMS-GW ipsec__plutorun: ...could not add conn "tunnelipsec"
> Jun  6 07:07:29 SMS-GW ipsec__plutorun: 021 no connection named
> "tunnelipsec"
> Jun  6 07:07:29 SMS-GW ipsec__plutorun: ...could not route conn
> "tunnelipsec"
> Jun  6 07:07:29 SMS-GW ipsec__plutorun: 021 no connection named
> "tunnelipsec"
> Jun  6 07:07:29 SMS-GW ipsec__plutorun: ...could not start conn
> "tunnelipsec"
> 
> sould it be modp1535 or 1536 ?
> please let me know.
> 
> many thanks,
> Tharanga
> 
> 
> 
> 
> Tharanga,
> 
> Check that all your settings match the other end, including...
> Aggressive Mode: Off (aggrmode=no, note: turning this on decreases
> security.)
> Perfect Forward Secrecy: Off (pfs=no, note: turning this on increases
> security.)
> Phase1 & Phase2: 3DES SHA1 Diffie Helman Group 5
> (ike=3des-sha1-modp1535 and esp=3des-sha1, note esp does not show the
> dh group, instead it inherits the dh group from phase1/ike.)
> No NAT-T, each IPSec endpoint has a public internet IP address.
> Key in ipsec.secrets is specified by ip addresses of endpoints, not
> hostname/ids.
> (this is a limitation of psk keys.)
> left and right are the public internet ip addresses of the two ipsec
> computers.
> leftsubnet and rightsubnet are private lan networks behind the ipsec
> computers.
> Are you sure you intended to comment out the leftsubnet line?
> (leftsubnet defaults to left, if not specified.)
> Also aggrmode defaults to no, if not specified.
> 
> 
> Peter McGill
> IT Systems Analyst
> Gra Ham Energy Limited
> 
>