[Openswan Users] PSK works, certificates not

Tue Jul 17 03:55:22 EDT 2007

>Arno Lehmann <al at its-lehmann.de> wrote:

>Hello,

>this is my first post to the list, so please be patient if I ask a FAQ 
(I did look through the archives, and used google, but couldn't find 
answers to my problem that solved them...)

>Ok, my setup:

>Quite basic at the moment:
>I have an internal network, 192.168.0.0/24, where I run a server that 
will become a VPN gateway. This is "balrog" at 192.168.0.22.

>A test client "phoenix" is at .88. This machine runs MS Windows Vista 
business.

>I set up the server with Linux Openswan U2.4.6/K2.6.18.8-0.3-default 
(netkey) (as distributed by SuSE's OpenSuse 10.2). My first tests used 
PSK authentication and worked fine, mostly following jacco's manual.

>Then I went on to try certificates, which will be a requirement later. 
(And, most probably, I'll have to fight with NAT and having the server 
listening to different networks, and all this sort of fun - later)

>I created a connection in ipsec.conf like this:

>config setup
         >interfaces="ipsec0=eth2 ipsec1=eth1 ipsec2=eth0"
         >klipsdebug=
         >plutodebug=
         >manualstart=
         >syslog=
         >plutowait=yes
         >nhelpers=0
         ># 

Just disable klipsdebug, plutodebug, mannualstart, syslog or gives options yes or no but not leave blanck.

>virtual_private=%v4:10.0.0.0/8,%v4:172.16.0.0/12,%v4:192.168.0.0/16,%v
>4:!192.168.0.0/24,%v4:!192.168.37.0/24
         ># nat_traversal=yes

>conn intern-cert
         >authby=rsasig
         >rightrsasigkey=%cert
         >leftcert=ITS-VPN.pem
         >left=192.168.0.22
         >leftprotoport=17/1701
         >right=%any
         >rightprotoport=17/1701
         >rightsubnet=vhost:%no,%priv
>rightca=%same
         >auto=add

Here you have to specify 
leftrsasigkey=%cert
leftsubnet=192.168.0.0/24

>(the defaults are unchanged from the PSK setup, and the connection 
>itself is also similar to the working PSK one.

>I created a (sub) CA (I'm using tinyCA for other certificate handling 
already) and created two certificates, one for the VPN server, and one 
for the windows client.

>I packaged the windows one as a pkcs12 file and installed it on that 
machine.

>I copied the server-related files to the vpn gateway 
(/etc/ipsec.d/certs, .../cacerts, .../private - certificate, CA 
certificate, private key for certificate, respectivels) and restarted 
the ipsec subsystem.

>When I keep the gateway certificate protected by a password, and have 
a line like ": RSA  "password" in ipsec.secrets, I get these 
messages:

you have to specify this thing in ipsec.secrets file
: RSA ITS-VPN.key "password"
This ITS-VPN.key is key file which is copied to private folder. This solve that key not found problem.

>> Jul 17 01:18:05 balrog pluto[13311]: loading secrets from "/etc/ipsec.secrets"
>> Jul 17 01:18:05 balrog pluto[13311]:   could not open private key file '/etc/ipsec.d/private/ITS-VPN.pem'
>> Jul 17 01:18:05 balrog pluto[13311]: "/etc/ipsec.secrets" line 14: error loading RSA private key file
>> Jul 17 01:18:05 balrog ipsec__plutorun: 003 "/etc/ipsec.secrets" line 14: error loading RSA private key file

>If I unlock the key file (and comment out the line in ipsec.secrets), I get no messages in the log.

>When I start the connection from the windows client, I quickly get a 
message that the connection could not be established, with a result 
code of 810, which would indicate a certificate problem, I think.

>On the server side, in the log, I find lines like

>> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: transition from state STATE_MAIN_R0 to state STATE_MAIN_R1
>> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: STATE_MAIN_R1: sent MR1, expecting MI2
>> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: transition from state STATE_MAIN_R1 to state STATE_MAIN_R2
>> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: STATE_MAIN_R2: sent MR2, expecting MI3
>> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: next payload type of ISAKMP Hash Payload has an unknown value: 51
>> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: malformed payload in packet
>> Jul 17 01:23:03 balrog pluto[14159]: "intern-cert"[1] 192.168.0.88 #1: sending notification PAYLOAD_MALFORMED to 192.168.0.88:500

>The malformed payload message is, as far as I understand after much 
reading, more or less only an artifact of an unencrypted message sent 
because encryption could not be established.

>After much trial and error, and not getting anywhere, I tried the 
CA.sh script to set up the CA and create certificates, but without any 
success - the problem persists. I even deleted the whole CA for the 
VPN and recreated it to make sure I did not accidentially use the 
wrong files or anything... (I admit this is close to the usual 
windows-related advice to reinstall, but I *am* getting desperate ;-)

Just to make certificate refer http://www.natecarlson.com/linux/ipsec-x509.php guide. Also make crl.pem file. then follow http://www.jacco2.dds.nl/networking/vista-openswan.html to go ahead.

>By now, I think that windows, for some reason, does not like my 
certificates.

>Unfortunately, I don't know what I'm doing wrong :-(

>Any hints from some of you, perhaps?

>-- 
>Arno Lehmann
>IT-Service Lehmann
>www.its-lehmann.de

Regards,

Dhaval

---------------------------------
Pinpoint customers who are looking for what you sell. 

---------------------------------
Luggage? GPS? Comic books? 
Check out fitting  gifts for grads at Yahoo! Search.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openswan.org/pipermail/users/attachments/20070717/292213bf/attachment.html 