[Openswan Users] X509 problem, PAYLOAD_MALFORMED

Hideo GOTO gotoh at eis.co.jp
Mon Feb 25 05:13:22 EST 2008 

Hi everyone,

I am trying to switch from PSK to X509, but cannot achieve to get correct
result after one week of googling and trials. I would be happy if anyone
could give me any advise.

With PSK, it (L2TP/IPsec) works perfect. When I try X509 based L2TP/IPSec
connection to an Openswan 2.4.9 server from a WindowsXP(SP2) client, it
aborts immediateley, and log shows "PAYLOAD_MALFORMED". On the client side,
certificates seems to be well installed by MMC to local computer account.
PSK setting is confirmed to be disabled.
I wonder if certificates for X509/IPSec requires special parameters such as
key usage, extendedKeyUsage. I permitted my self to include here-below
Openssl text out put of Openswan side certificate. The other side is the
same excepting CN. 

I am sure I am making some stupid mistake, that I cannot find myself. Thanks
in advance for any advise.

Hideo GOTO,  gotoh at eis.co.jp

System info:
    OS: Fedora8 (kernel 2.6.23 w/o patches (=NETKEY))
    Openswan: 2.4.9
    L2TPD: xl2tpd 1.1.12

    Host is configured as 3 segment firewall using IPTABLES (all blocking
rules are disabled during tests.). 
    Test is done by simulation inside a private network with following IP
configuration:
	eth0: 192.168.xx   (external net)
	eth1: 172.16.xx     (private net )
	eth2: 10.xxxx	   (dmz net : disabled during tests.)

   All I/F has MTU of 1500


*** OPENSWAN SIDE CERTIFICATE ***
[root at ns01 certs]# openssl x509 -text -noout -in cl20080225145659.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 12 (0xc)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=JP, ST=Tokyo, L=Shinagawa-ku, O=Scholar Consult Co.,Ltd.,
OU=CA, CN=privateCA/emailAddress=postmaster at scholar.co.jp
        Validity
            Not Before: Feb 25 05:57:00 2008 GMT
            Not After : Feb 24 05:57:01 2011 GMT
        Subject: C=JP, ST=Tokyo, O=Scholar Consult Co.,Ltd., OU=security,
CN=eistestSwan-2008/02/25-14:56/emailAddress=postmaster at scholar.co.jp
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (1024 bit)
                Modulus (1024 bit):
                    00:ce:76:ca:19:a8:81:60:91:7f:a4:df:24:1a:84:
                    eb:1f:d7:82:ce:f4:9e:e8:23:a7:aa:5d:ec:c0:32:
                    2d:b2:84:00:3b:4d:e3:1d:33:31:df:32:d1:4d:82:
                    b2:7d:01:37:ea:48:f3:5e:60:d7:63:fa:ae:54:5f:
                    fd:17:1e:3e:a3:7d:37:16:52:08:64:1b:a8:4f:f9:
                    68:19:84:f6:50:14:2b:7f:67:9f:d6:f4:87:3a:97:
                    66:e7:43:f1:5a:e4:ba:d5:41:24:35:25:a4:24:af:
                    25:2e:22:7e:53:8c:79:71:a4:02:ef:fe:88:75:da:
                    2e:d5:bf:25:82:36:c7:75:fb
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Cert Type:
                SSL Client, S/MIME, Object Signing
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                CD:7B:34:BE:90:91:DC:60:36:1E:58:BB:83:00:1A:BE:B4:06:E1:61
            X509v3 Authority Key Identifier:
 
keyid:E0:F4:94:BB:3C:4C:D7:54:50:71:6F:42:18:21:72:BD:79:21:67:B3
                DirName:/C=JP/ST=Tokyo/L=Shinagawa-ku/O=Scholar Consult
Co.,Ltd./OU=CA/CN=privateCA/emailAddress=postmaster at scholar.co.jp
                serial:B2:87:33:5A:86:D2:4C:DC

            X509v3 Extended Key Usage:
                TLS Web Client Authentication
    Signature Algorithm: sha1WithRSAEncryption
        64:0c:f4:84:ed:d1:e3:d9:f1:46:08:e3:49:7d:3e:59:f5:bb:
        9b:35:a9:45:0d:c8:ac:9f:71:29:77:8e:70:51:aa:fe:88:d3:
        a9:f5:51:77:df:35:ef:a5:c5:67:8d:61:52:bf:6a:40:32:1b:
        b5:d2:5e:0c:6a:90:f0:a6:36:5c:88:1f:24:86:23:30:c1:2b:
        e5:fb:d2:04:09:4d:08:f8:82:26:aa:04:57:03:02:ea:b3:62:
        2f:fe:97:6a:f4:69:e6:db:ff:dd:df:4e:e7:15:aa:15:f1:1a:
        9f:1d:a5:9b:5e:ff:fe:38:91:26:4b:17:92:34:cb:e9:cb:72:
        71:cd



*** LAST PART OF LOG (/var/log/secure : plutodebug=control) ***

Feb 25 18:36:49 ns01 pluto[16987]: | inserting event EVENT_RETRANSMIT,
timeout in 10 secon
ds for #2
Feb 25 18:36:49 ns01 pluto[16987]: "l2tp-X.509"[2] 192.168.14.121 #2:
STATE_MAIN_R1: sent
MR1, expecting MI2
Feb 25 18:36:49 ns01 pluto[16987]: | modecfg pull: noquirk policy:push
not-client
Feb 25 18:36:49 ns01 pluto[16987]: | phase 1 is done, looking for phase 1 to
unpend
Feb 25 18:36:49 ns01 pluto[16987]: | next event EVENT_RETRANSMIT in 10
seconds for #2
Feb 25 18:36:50 ns01 pluto[16987]: |
Feb 25 18:36:50 ns01 pluto[16987]: | *received 360 bytes from
192.168.14.121:500 on eth0 (
port=500)
Feb 25 18:36:50 ns01 pluto[16987]: |  processing packet with exchange
type=ISAKMP_XCHG_IDP
ROT (2)
Feb 25 18:36:50 ns01 pluto[16987]: | ICOOKIE:  7c 14 9f 08  56 ab a9 25
Feb 25 18:36:50 ns01 pluto[16987]: | RCOOKIE:  63 c4 90 17  79 34 4e 49
Feb 25 18:36:50 ns01 pluto[16987]: | peer:  c0 a8 0e 79
Feb 25 18:36:50 ns01 pluto[16987]: | state hash entry 7
Feb 25 18:36:50 ns01 pluto[16987]: | peer and cookies match on #2, provided
msgid 00000000
 vs 00000000
Feb 25 18:36:50 ns01 pluto[16987]: | state object #2 found, in STATE_MAIN_R1
Feb 25 18:36:50 ns01 pluto[16987]: | processing connection l2tp-X.509[2]
192.168.14.121
Feb 25 18:36:50 ns01 pluto[16987]: "l2tp-X.509"[2] 192.168.14.121 #2:
NAT-Traversal: Resul
t using draft-ietf-ipsec-nat-t-ike-02/03: no NAT detected
Feb 25 18:36:50 ns01 pluto[16987]: | inserting event EVENT_NAT_T_KEEPALIVE,
timeout in 20
seconds
Feb 25 18:36:50 ns01 pluto[16987]: | helper -1 doing build_kenonce op id: 0
Feb 25 18:36:50 ns01 pluto[16987]: | processing connection l2tp-X.509[2]
192.168.14.121
Feb 25 18:36:50 ns01 pluto[16987]: | started looking for secret for C=JP,
ST=Tokyo, O=Scho
lar Consult Co.,Ltd., OU=security, CN=eistestSwan-2008/02/25-14:56,
E=postmaster at scholar.c
o.jp->192.168.14.121 of kind PPK_PSK
Feb 25 18:36:50 ns01 pluto[16987]: | instantiating him to 0.0.0.0
Feb 25 18:36:50 ns01 pluto[16987]: | actually looking for secret for C=JP,
ST=Tokyo, O=Sch
olar Consult Co.,Ltd., OU=security, CN=eistestSwan-2008/02/25-14:56,
E=postmaster at scholar.
co.jp->0.0.0.0 of kind PPK_PSK
Feb 25 18:36:50 ns01 pluto[16987]: | concluding with best_match=0 best=(nil)
(lineno=-1)
Feb 25 18:36:50 ns01 pluto[16987]: | complete state transition with STF_OK
Feb 25 18:36:50 ns01 pluto[16987]: "l2tp-X.509"[2] 192.168.14.121 #2:
transition from stat
e STATE_MAIN_R1 to state STATE_MAIN_R2
Feb 25 18:36:50 ns01 pluto[16987]: | sending reply packet to
192.168.14.121:500 (from port
=500)
Feb 25 18:36:50 ns01 pluto[16987]: | sending 528 bytes for STATE_MAIN_R1
through eth0:500
to 192.168.14.121:500:
Feb 25 18:36:50 ns01 pluto[16987]: | inserting event EVENT_RETRANSMIT,
timeout in 10 secon
ds for #2
Feb 25 18:36:50 ns01 pluto[16987]: "l2tp-X.509"[2] 192.168.14.121 #2:
STATE_MAIN_R2: sent
MR2, expecting MI3
Feb 25 18:36:50 ns01 pluto[16987]: | modecfg pull: noquirk policy:push
not-client
Feb 25 18:36:50 ns01 pluto[16987]: | phase 1 is done, looking for phase 1 to
unpend
Feb 25 18:36:50 ns01 pluto[16987]: | complete state transition with
STF_INLINE
Feb 25 18:36:50 ns01 pluto[16987]: | next event EVENT_RETRANSMIT in 10
seconds for #2
Feb 25 18:36:50 ns01 pluto[16987]: |
Feb 25 18:36:50 ns01 pluto[16987]: | *received 84 bytes from
192.168.14.121:500 on eth0 (p
ort=500)
:%
Feb 25 18:36:50 ns01 pluto[16987]: | peer and cookies match on #2, provided
msgid 00000000
 vs 00000000/00000000
Feb 25 18:36:50 ns01 pluto[16987]: | p15 state object #2 found, in
STATE_MAIN_R2
Feb 25 18:36:50 ns01 pluto[16987]: | processing connection l2tp-X.509[2]
192.168.14.121
Feb 25 18:36:50 ns01 pluto[16987]: "l2tp-X.509"[2] 192.168.14.121 #2: next
payload type of
 ISAKMP Hash Payload has an unknown value: 180
Feb 25 18:36:50 ns01 pluto[16987]: "l2tp-X.509"[2] 192.168.14.121 #2:
malformed payload in
 packet
Feb 25 18:36:50 ns01 pluto[16987]: | payload malformed after IV
Feb 25 18:36:50 ns01 pluto[16987]: |   dd e7 06 6c  a1 bd 6b c5  ac 0d 04 30
c2 39 f0 f9
Feb 25 18:36:50 ns01 pluto[16987]: |   08 27 e0 36
Feb 25 18:36:50 ns01 pluto[16987]: "l2tp-X.509"[2] 192.168.14.121 #2:
sending notification
 PAYLOAD_MALFORMED to 192.168.14.121:500
Feb 25 18:36:50 ns01 pluto[16987]: | sending 40 bytes for notification
packet through eth0
:500 to 192.168.14.121:500:
Feb 25 18:36:50 ns01 pluto[16987]: | next event EVENT_RETRANSMIT in 10
seconds for #2
Feb 25 18:36:50 ns01 pluto[16987]: |
Feb 25 18:36:50 ns01 pluto[16987]: | *received 84 bytes from
192.168.14.121:500 on eth0 (p
ort=500)
Feb 25 18:36:50 ns01 pluto[16987]: |  processing packet with exchange
type=ISAKMP_XCHG_INF
O (5)
Feb 25 18:36:50 ns01 pluto[16987]: | ICOOKIE:  7c 14 9f 08  56 ab a9 25
Feb 25 18:36:50 ns01 pluto[16987]: | RCOOKIE:  63 c4 90 17  79 34 4e 49
Feb 25 18:36:50 ns01 pluto[16987]: | peer:  c0 a8 0e 79
Feb 25 18:36:50 ns01 pluto[16987]: | state hash entry 7
Feb 25 18:36:50 ns01 pluto[16987]: | peer and cookies match on #2, provided
msgid 00000000
 vs 00000000/00000000
Feb 25 18:36:50 ns01 pluto[16987]: | p15 state object #2 found, in
STATE_MAIN_R2
Feb 25 18:36:50 ns01 pluto[16987]: | processing connection l2tp-X.509[2]
192.168.14.121
Feb 25 18:36:50 ns01 pluto[16987]: "l2tp-X.509"[2] 192.168.14.121 #2: next
payload type of
 ISAKMP Hash Payload has an unknown value: 198
Feb 25 18:36:50 ns01 pluto[16987]: "l2tp-X.509"[2] 192.168.14.121 #2:
malformed payload in
 packet
Feb 25 18:36:50 ns01 pluto[16987]: | next event EVENT_RETRANSMIT in 10
seconds for #2


*** RESULT OF  "ipsec auto --listall ***
000
000 List of Public Keys:
000
000 Feb 25 18:16:20 2008, 1024 RSA Key AwEAAc52y, until Feb 24 14:57:01 2011
ok
000        ID_DER_ASN1_DN 'C=JP, ST=Tokyo, O=Scholar Consult Co.,Ltd.,
OU=security, CN=eistestSwan-2008/02/25-14:56, E=postmaster at scholar.co.jp'
000        Issuer 'C=JP, ST=Tokyo, L=Shinagawa-ku, O=Scholar Consult
Co.,Ltd., OU=CA, CN=privateCA, E=postmaster at scholar.co.jp'
000
000 List of X.509 End Certificates:
000
000 Feb 25 18:16:20 2008, count: 1
000        subject: 'C=JP, ST=Tokyo, O=Scholar Consult Co.,Ltd.,
OU=security, CN=eistestSwan-2008/02/25-14:56, E=postmaster at scholar.co.jp'
000        issuer:  'C=JP, ST=Tokyo, L=Shinagawa-ku, O=Scholar Consult
Co.,Ltd., OU=CA, CN=privateCA, E=postmaster at scholar.co.jp'
000        serial:   0c
000        pubkey:   1024 RSA Key AwEAAc52y, has private key
000        validity: not before Feb 25 14:57:00 2008 ok
000                  not after  Feb 24 14:57:01 2011 ok
000        subjkey:
cd:7b:34:be:90:91:dc:60:36:1e:58:bb:83:00:1a:be:b4:06:e1:61
000        authkey:
e0:f4:94:bb:3c:4c:d7:54:50:71:6f:42:18:21:72:bd:79:21:67:b3
000        aserial:  00:b2:87:33:5a:86:d2:4c:dc
000
000 List of X.509 CA Certificates:
000
000 Feb 25 18:16:20 2008, count: 1
000        subject: 'C=JP, ST=Tokyo, L=Shinagawa-ku, O=Scholar Consult
Co.,Ltd., OU=CA, CN=privateCA, E=postmaster at scholar.co.jp'
000        issuer:  'C=JP, ST=Tokyo, L=Shinagawa-ku, O=Scholar Consult
Co.,Ltd., OU=CA, CN=privateCA, E=postmaster at scholar.co.jp'
000        serial:   00:b2:87:33:5a:86:d2:4c:dc
000        pubkey:   1024 RSA Key AwEAAaDBH
000        validity: not before Feb 25 01:31:12 2008 ok
000                  not after  Feb 22 01:31:12 2018 ok
000        subjkey:
e0:f4:94:bb:3c:4c:d7:54:50:71:6f:42:18:21:72:bd:79:21:67:b3
000        authkey:
e0:f4:94:bb:3c:4c:d7:54:50:71:6f:42:18:21:72:bd:79:21:67:b3
000        aserial:  00:b2:87:33:5a:86:d2:4c:dc
000
000 List of X.509 CRLs:
000
000 Feb 25 18:16:20 2008, revoked certs: 0
000        issuer:  'C=JP, ST=Tokyo, L=Shinagawa-ku, O=Scholar Consult
Co.,Ltd., OU=CA, CN=privateCA, E=postmaster at scholar.co.jp'
000        updates:  this Feb 25 16:23:37 2008
000                  next Mar 26 16:23:37 2008 ok

More information about the Users mailing list