[Openswan Users] Windows 8.1 L2TP/IPsec with certificate not working

[Daniel Minder]

Dear all,

I have openswan 2.6.39 running on a public server (no NAT on server side).

On a Windows machine I have a VPN configured with L2TP/IPsec and 
certifacte based authentification. The certificate is self-signed and 
imported to the certificate store of the computer.

This setup works perfectly on Windows Vista. However, it does not work 
on Windows 8.1 any more.

I switched on plutodebug and compared the two session establishments. 
 From Vista, I see messages as defined in RFC2409 (IKEv1) section 5.1:
         Initiator                          Responder
        -----------                        -----------
         HDR, SA                     -->
                                     <--    HDR, SA
         HDR, KE, Ni                 -->
                                     <--    HDR, KE, Nr
         HDR*, IDii, [ CERT, ] SIG_I -->
For Windows 8.1, the first four messages are similar, but the 5th is 
different. Instead of exchange type ISAKMP_XCHG_IDPROT I get this one:

| *received 92 bytes from xx.xx.xx.xx:500 on eth2 (port=500)
|   a3 aa 87 ba  dc 53 77 55  bf af cf f6  02 a7 88 3d
|   08 10 05 01  9c 83 6b 16  00 00 00 5c  3a 59 56 8c
|   1c 39 4a ae  18 31 2d 40  87 20 69 a2  67 0a 0c 9b
|   80 d0 f4 89  70 fb c4 5a  2f ff bd 0a  63 ac 06 43
|   fe 62 ef f3  86 91 42 65  10 a6 6b 5d  1d b9 03 5e
|   b8 77 e9 84  76 ba 3d 5f  ca a0 46 7c
| **parse ISAKMP Message:
|    initiator cookie:
|   a3 aa 87 ba  dc 53 77 55
|    responder cookie:
|   bf af cf f6  02 a7 88 3d
|    next payload type: ISAKMP_NEXT_HASH
|    ISAKMP version: ISAKMP Version 1.0 (rfc2407)
|    exchange type: ISAKMP_XCHG_INFO
|    flags: ISAKMP_FLAG_ENCRYPTION
|    message ID:  9c 83 6b 16
|    length: 92
|  processing version=1.0 packet with exchange type=ISAKMP_XCHG_INFO (5)
| ICOOKIE:  a3 aa 87 ba  dc 53 77 55
| RCOOKIE:  bf af cf f6  02 a7 88 3d
| state hash entry 8
| peer and cookies match on #5, provided msgid 00000000 vs 00000000/00000000
| p15 state object #5 found, in STATE_MAIN_R2
| processing connection l2tp-X.509[3] xx.xx.xx.xx
| last Phase 1 IV:
| current Phase 1 IV:  f8 c1 47 d0  e5 e9 a4 e4  b2 4d cf 60  eb c5 f1 b2
| current Phase 1 IV:  31 d6 f1 f0
| computed Phase 2 IV:
|   23 38 ca b8  0d a0 a1 70  52 3e 83 25  eb 83 ab 3d
|   b8 bd 98 0f
| received encrypted packet from xx.xx.xx.xx:500
| decrypting 64 bytes using algorithm OAKLEY_AES_CBC
| decrypted:
|   db 62 b5 6c  a9 b3 8a da  35 50 eb 23  71 76 38 36
|   0a 99 cb 61  b1 62 2e 56  00 00 00 1c  00 00 00 01
|   01 10 00 1c  a3 aa 87 ba  dc 53 77 55  bf af cf f6
|   02 a7 88 3d  00 00 00 00  00 00 00 00  00 00 00 00
| next IV:  1d b9 03 5e  b8 77 e9 84  76 ba 3d 5f  ca a0 46 7c
| got payload 0x100(ISAKMP_NEXT_HASH) needed: 0x100 opt: 0x0
"l2tp-X.509"[3] xx.xx.xx.xx #5: next payload type of ISAKMP Hash Payload 
has an unknown value: 219


Does Windows 8.1 really send me a broken packet? I guess it's not the 
decryption since the last 12 bytes are all 0 which really seems like 
padding bytes for the encryption to work. I guess the ISAKMP_XCHG_INFO 
exchange type also requires the general payload header, i.e. the first 
byte should indicate the next payload type - which is 219 here.


Has anyone successfully set up VPN with certs on Win 8.1? Any help is 
appreciated!


Best regards,
Daniel