[Openswan Users] Openswan 2.3.1 vs XP SP2 , malformed payload in packet

[Elias Valea Peri]

Hello everybody

We've a VPN infrastructure running ok using Openswan from 2 years ago,
interconecting near to 100 Linux gateways and networks.
Recently we've tried to mount a IPSec/L2TP server to allow company users
(XP_SP2 and Pocket PCs) to connect to our network (thanks to Jacco de
Leeuw and Nate

Carlson impressive work!).
All the clients behind a client-side NAT are capable to connect to the VPN
and runnig ok, but surprising!! a few clients that are directly connected
to the

Internet and own a public IP (i.e using a modem) not.
Looking at the openswan logs the problem is always the same during the SA
establishment:

May  2 15:33:54 vpnsrv pluto[27852]: packet from <client-ip>:500: ignoring
Vendor ID payload [MS NT5 ISAKMPOAKLEY 00000004]
May  2 15:33:54 vpnsrv pluto[27852]: packet from <client-ip>:500: ignoring
Vendor ID payload [FRAGMENTATION]
May  2 15:33:54 vpnsrv pluto[27852]: packet from <client-ip>:500: received
Vendor ID payload [draft-ietf-ipsec-nat-t-ike-02_n] method set to=106
May  2 15:33:54 vpnsrv pluto[27852]: packet from <client-ip>:500: ignoring
Vendor ID payload [Vid-Initial-Contact]
May  2 15:33:54 vpnsrv pluto[27852]: "roadwarrior-l2tp"[2] <client-ip> #2:
responding to Main Mode from unknown peer <client-ip>
May  2 15:33:54 vpnsrv pluto[27852]: "roadwarrior-l2tp"[2] <client-ip> #2:
transition from state STATE_MAIN_R0 to state STATE_MAIN_R1
May  2 15:33:55 vpnsrv pluto[27852]: "roadwarrior-l2tp"[2] <client-ip> #2:
NAT-Traversal: Result using draft-ietf-ipsec-nat-t-ike-02/03: both are
NATed
May  2 15:33:55 vpnsrv pluto[27852]: "roadwarrior-l2tp"[2] <client-ip> #2:
transition from state STATE_MAIN_R1 to state STATE_MAIN_R2
May  2 15:33:56 vpnsrv pluto[27852]: "roadwarrior-l2tp"[2] <client-ip> #2:
next payload type of ISAKMP Hash Payload has an unknown value: 105
May  2 15:33:56 vpnsrv pluto[27852]: "roadwarrior-l2tp"[2] <client-ip> #2:
malformed payload in packet
May  2 15:33:56 vpnsrv pluto[27852]: "roadwarrior-l2tp"[2] <client-ip> #2:
sending notification PAYLOAD_MALFORMED to <client-ip>:500
May  2 15:33:56 vpnsrv pluto[27852]: "roadwarrior-l2tp"[2] <client-ip> #2:
failed to build notification for spisize=0
May  2 15:33:56 vpnsrv pluto[27852]: "roadwarrior-l2tp"[2] <client-ip> #2:
next payload type of ISAKMP Hash Payload has an unknown value: 128
May  2 15:33:56 vpnsrv pluto[27852]: "roadwarrior-l2tp"[2] <client-ip> #2:
malformed payload in packet

'malformed payload in packet','failed to build notification', 'incorrect
Payload Hash...'
It seems that there is something between the clients and the server that
'modifies' the packets, or that some packet fragments doesnt arrives to
the

server...
We've tried :
- Using forceencaps to force NAT-T works (you can see it in the log 'both
are NATed')
- Apply the recently addded (experimental) NAT-OA patch
- Allow Transport-mode NAT and switch to Transport
- investigate about IPSec-passthrough devices between server and clients,
but there're many ISPs involved and is difficult to detect...
... but the problem remains

A few words about the infrastructure:
- Clients :
XP SP2 fully patched, or Ms Pocket PC 2003 (4.20.0)
- Server :
owns a public IP (not natted, but firewalled -> ESP/AH, UDP500 and UDP4500
allowed)
kernel 2.6.10
openswan 2.3.1
l2tpd 0.69 (brings the clients a virtual-IP inside a private-DMZ)
x.509 authentication
Excerpt from ipsec.conf :

version 2.0

config setup
        interfaces=%defaultroute
        nat_traversal=yes
        klipsdebug=none
        plutodebug=none
        virtual_private=%v4:10.0.0.0/8,%v4:172.16.0.0/12,%v4:192.168.0.0/16
        uniqueids=yes

conn %default
        keyingtries=3
        compress=yes
        disablearrivalcheck=no
        authby=rsasig
        leftrsasigkey=%cert
        rightrsasigkey=%cert
        forceencaps=yes

conn exclude-lo
        authby=never
        left=127.0.0.1
        leftsubnet=127.0.0.0/8
        right=127.0.0.2
        rightsubnet=127.0.0.0/8
        type=passthrough
        auto=route

#Disable Opportunistic Encryption
include /etc/ipsec.d/examples/no_oe.conf

conn roadwarrior-l2tp
        left=<Server-public-ip>
        leftcert=/etc/ipsec.d/certs/ServerCert.pem
        leftid="/C=xx/ST=XXX/O=XXXX/OU=XX/CN=x.x.x/emailAddress=x at x"
        leftprotoport=17/1701
        right=%any
        rightid="/C=yy/ST=YYY/O=YYYY/OU=YY/CN=yyyyyyy/emailAddress=y at y"
        rightprotoport=17/1701
        rightsubnet=vhost:%no,%priv
        pfs=no
        auto=add


... any proposal will be very thankful
... thanks in advance