[Openswan dev] next payload type of ISAKMP Hash Payload has an unknown value XX error

[Paul Wouters]

While trying to create an ipsec tunnel on XP SP2, which seems to want something
special in my certificates, because it is rejecting it, I found amongst other
things in my logs on the Openswan end:

next payload type of ISAKMP Hash Payload has an unknown value XX

Corresponding this with the oakley.log I see (sorry in dutch):

10-18: 12:41:20:252:7a0 Receive: (get) SA = 0x00109598 from 82.161.125.16.500
10-18: 12:41:20:252:7a0 ISAKMP Header: (V1.0), len = 396
10-18: 12:41:20:252:7a0   I-COOKIE 3e704a932133235a
10-18: 12:41:20:252:7a0   R-COOKIE 702e9f026ad1976e
10-18: 12:41:20:252:7a0   exchange: Oakley Main Mode
10-18: 12:41:20:252:7a0   flags: 0
10-18: 12:41:20:252:7a0   next payload: KE
10-18: 12:41:20:252:7a0   message ID: 00000000
10-18: 12:41:20:252:7a0 processing payload KE
10-18: 12:41:20:292:7a0 processing payload NONCE
10-18: 12:41:20:292:7a0 processing payload CRP
10-18: 12:41:20:292:7a0 processing payload NATDISC
10-18: 12:41:20:292:7a0 Processing NatHash
10-18: 12:41:20:292:7a0 Nat hash 0be820fc559b9495a65a04f0be4272f5
10-18: 12:41:20:292:7a0 04327d23
10-18: 12:41:20:292:7a0 SA StateMask2 f
10-18: 12:41:20:292:7a0 processing payload NATDISC
10-18: 12:41:20:292:7a0 Processing NatHash
10-18: 12:41:20:292:7a0 Nat hash ad42fbdf8397401ecc92869716af8891
10-18: 12:41:20:292:7a0 1056b7e1
10-18: 12:41:20:292:7a0 SA StateMask2 8f
10-18: 12:41:20:292:7a0 ClearFragList
10-18: 12:41:20:292:7a0 constructing ISAKMP Header
10-18: 12:41:20:292:7a0 constructing ID
10-18: 12:41:20:292:7a0 Received no valid CRPs.  Using all configured
10-18: 12:41:20:292:7a0 Looking for IPSec only cert
10-18: 12:41:20:402:7a0 failed to get chain 80092004
10-18: 12:41:20:402:7a0 Looking for any cert
10-18: 12:41:20:402:7a0 failed to get chain 80092004
10-18: 12:41:20:402:7a0 ProcessFailure: sa:00109598 centry:00000000 status:35ee
10-18: 12:41:20:402:7a0 isadb_set_status sa:00109598 centry:00000000 status 35ee 
10-18: 12:41:20:452:7a0 Sleuteluitwisselingsmodus (hoofdmodus)
10-18: 12:41:20:452:7a0 Bron-IP-adres 193.110.157.23  Bron-IP-adresmasker 255.255.255.255  Bestemmings-IP-adres 82.161.125.16  Bestemmings-IP-adresmaskerer 255.255.255.255  Protocol 0  Bronpoort 0  Bestemmingspoort 0  Lokaal IKE-adres 193.110.157.23  IKE-peeradres 82.161.125.16
10-18: 12:41:20:452:7a0 Identiteit op basis van certificaat    Peer-IP-adres: 82.161.125.16
10-18: 12:41:20:452:7a0 Mijzelf
10-18: 12:41:20:452:7a0 IKE kan geen binding maken met geldig computercertificaat
10-18: 12:41:20:452:7a0 0x80092004 0x0
10-18: 12:41:20:452:7a0 ProcessFailure: sa:00109598 centry:00000000 status:35ee
10-18: 12:41:20:452:7a0 constructing ISAKMP Header
10-18: 12:41:20:452:7a0 constructing HASH (null)
10-18: 12:41:20:452:7a0 constructing NOTIFY 28
10-18: 12:41:20:452:7a0 constructing HASH (Notify/Delete)
10-18: 12:41:20:452:7a0
10-18: 12:41:20:452:7a0 Sending: SA = 0x00109598 to 82.161.125.16:Type 1.500
10-18: 12:41:20:452:7a0 ISAKMP Header: (V1.0), len = 84
10-18: 12:41:20:452:7a0   I-COOKIE 3e704a932133235a
10-18: 12:41:20:452:7a0   R-COOKIE 702e9f026ad1976e
10-18: 12:41:20:452:7a0   exchange: ISAKMP Informational Exchange
10-18: 12:41:20:452:7a0   flags: 1 ( encrypted )
10-18: 12:41:20:452:7a0   next payload: HASH
10-18: 12:41:20:452:7a0   message ID: 97649456
10-18: 12:41:20:452:7a0 Ports S:f401 D:f401
10-18: 12:41:20:482:7a0

First, the problem seems to be that my certificate is invalid for some reason. I will
debug that seperately, but Windows at least wants to construct and send a Notify/Delete
message. I think the problem is at:

10-18: 12:41:20:452:7a0 constructing ISAKMP Header
10-18: 12:41:20:452:7a0 constructing HASH (null)
10-18: 12:41:20:452:7a0 constructing NOTIFY 28
10-18: 12:41:20:452:7a0 constructing HASH (Notify/Delete)
10-18: 12:41:20:452:7a0
10-18: 12:41:20:452:7a0 Sending: SA = 0x00109598 to 82.161.125.16:Type 1.500

I'm rather sceptical of the hash on (null). So I guess the Openswan error is
an actual error saying Windows send us a corrupted ISAKMP message.

I am seeing something similar with the same certificates on a Windows 2000 machine,
so I am not sure this is an error in the XP SP2. It's likely an error as a result
of something I'm doing wrong with my current set of certificates, though I don't
know yet what that might be.

Has anyone gotten an X509 connection working using between Openswan-2 and XP SP2 ?
Google suggests more people have problems after upgrading to SP2.

And yes, the certificate is in the right place in the store according to the MMC,
and has private key and valid certificate path.

Paul