[Openswan Users] PSK works, certificates not

[Arno Lehmann]

Hello,

this is my first post to the list, so please be patient if I ask a FAQ 
(I did look through the archives, and used google, but couldn't find 
answers to my problem that solved them...)

Ok, my setup:

Quite basic at the moment:
I have an internal network, 192.168.0.0/24, where I run a server that 
will become a VPN gateway. This is "balrog" at 192.168.0.22.

A test client "phoenix" is at .88. This machine runs MS Windows Vista 
business.

I set up the server with Linux Openswan U2.4.6/K2.6.18.8-0.3-default 
(netkey) (as distributed by SuSE's OpenSuse 10.2). My first tests used 
PSK authentication and worked fine, mostly following jacco's manual.

Then I went on to try certificates, which will be a requirement later. 
(And, most probably, I'll have to fight with NAT and having the server 
listening to different networks, and all this sort of fun - later)

I created a connection in ipsec.conf like this:

config setup
         interfaces="ipsec0=eth2 ipsec1=eth1 ipsec2=eth0"
         klipsdebug=
         plutodebug=
         manualstart=
         syslog=
         plutowait=yes
         nhelpers=0
         # 
virtual_private=%v4:10.0.0.0/8,%v4:172.16.0.0/12,%v4:192.168.0.0/16,%v
4:!192.168.0.0/24,%v4:!192.168.37.0/24
         # nat_traversal=yes

conn intern-cert
         authby=rsasig
         rightrsasigkey=%cert
         leftcert=ITS-VPN.pem
         left=192.168.0.22
         leftprotoport=17/1701
         right=%any
         rightprotoport=17/1701
         rightsubnet=vhost:%no,%priv
         rightca=%same
         auto=add

(the defaults are unchanged from the PSK setup, and the connection 
itself is also similar to the working PSK one.

I created a (sub) CA (I'm using tinyCA for other certificate handling 
already) and created two certificates, one for the VPN server, and one 
for the windows client.

I packaged the windows one as a pkcs12 file and installed it on that 
machine.

I copied the server-related files to the vpn gateway 
(/etc/ipsec.d/certs, .../cacerts, .../private - certificate, CA 
certificate, private key for certificate, respectivels) and restarted 
the ipsec subsystem.

When I keep the gateway certificate protected by a password, and have 
a line like ": RSA <keyfile> "password" in ipsec.secrets, I get these 
messages:

> Jul 17 01:18:05 balrog pluto[13311]: loading secrets from "/etc/ipsec.secrets"
> Jul 17 01:18:05 balrog pluto[13311]:   could not open private key file '/etc/ipsec.d/private/ITS-VPN.pem'
> Jul 17 01:18:05 balrog pluto[13311]: "/etc/ipsec.secrets" line 14: error loading RSA private key file
> Jul 17 01:18:05 balrog ipsec__plutorun: 003 "/etc/ipsec.secrets" line 14: error loading RSA private key file

If I unlock the key file (and comment out the line in ipsec.secrets), 
I get no messages in the log.

When I start the connection from the windows client, I quickly get a 
message that the connection could not be established, with a result 
code of 810, which would indicate a certificate problem, I think.

On the server side, in the log, I find lines like

> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: transition from state STATE_MAIN_R0 to state STATE_MAIN_R1
> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: STATE_MAIN_R1: sent MR1, expecting MI2
> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: transition from state STATE_MAIN_R1 to state STATE_MAIN_R2
> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: STATE_MAIN_R2: sent MR2, expecting MI3
> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: next payload type of ISAKMP Hash Payload has an unknown value: 51
> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: malformed payload in packet
> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: sending notification PAYLOAD_MALFORMED to 192.168.0.88:500

The malformed payload message is, as far as I understand after much 
reading, more or less only an artifact of an unencrypted message sent 
because encryption could not be established.

After much trial and error, and not getting anywhere, I tried the 
CA.sh script to set up the CA and create certificates, but without any 
success - the problem persists. I even deleted the whole CA for the 
VPN and recreated it to make sure I did not accidentially use the 
wrong files or anything... (I admit this is close to the usual 
windows-related advice to reinstall, but I *am* getting desperate ;-)

By now, I think that windows, for some reason, does not like my 
certificates.

Unfortunately, I don't know what I'm doing wrong :-(

Any hints from some of you, perhaps?

-- 
Arno Lehmann
IT-Service Lehmann
www.its-lehmann.de