[Openswan Users] Linux (debian lenny) client to Checkpoint Firewall NGx R65 using certificates - secureclient ok, openswan ko - PAYLOAD_MALFORMED - SOLVED!!!!

[Luca Arzeni]

Hi there,
since I've been struggling for few monts about this issue, and now I
was able to solve it, I think it's a good thing (TM) to spread the
word about the solution.

Client: Debian Linux Openswan U2.4.12/K , kernel 2.6.26-2, using klips
Server: Checkpoint Firewall NGx R65

The connection works flawlessy under windows with checkpoint secure
client using certificates (fw admin assigned me a p12 certificate); it
also worked under Red Hat 7.3 with an old kernel module released from
checkpoint.

Solution: the PAYLOAD_MALFORMED issue was caused by the nat-traversal option.
I had to
1) disable the nat traversal in my config, and
2) setup, on our firewall a couple of rules to pass traffico on port
500 and 4500 to the linux box where openswan runs.

A second issue comes from the server ID, which, for checkpoint, must
be the public IP (it refuses to send me the certificate signature), so
I needed to setup it with rightid.

The setup works also with netkey (tested), but I needed klips to set the mtu.

I also tested with openswan 2.6.25, 2.6.26, 2.6.28 under debian
squeeze (2.6.32) and, with few different settings, all thing worked
fine.

Thanks to all people that helped.

Regards, Luca



On Fri, May 14, 2010 at 7:38 PM, Luca Arzeni <l.arzeni at gmail.com> wrote:
> Thanks Ondrej,
> I was not meaning that openswan is broken.
> I've also read that checkpoint does some strange hybrid mode
> authentication and so on.
>
> I'm simply saying that it could be more useful if I could add a feature...
>
> Bye, Luca
>
>
>
> On Thu, May 13, 2010 at 10:55 PM, Ondrej Valousek <webserv at s3group.cz> wrote:
>> Luca,
>>
>> It does not prove anything as you are still using CP-provided client right?
>> CP-client will always understand CP firewall. I think it might still use
>> SecurID with secrificates. I do not know.
>> Once you are able to connect with non-CP client, I will say yes, there could
>> be something wrong with openswan, but now....
>>
>> Cheers,
>>
>> Ondrej
>>
>> On 13.05.2010 18:16, Luca Arzeni wrote:
>>
>> Alas,
>> administrator said that all people is now using certificates, and no
>> one is using securID, so I'm the (un)lucky guy.
>>
>> I goggled around a little about ISAKMP_NEXT_N and found that
>>
>> ISAKMP_NEXT_N is an always-welcome payload_type (Notification)
>> ISAKMP_NEXT_D is an always-welcome payload_type (Delete)
>>
>> Now, I recall that I've read that, after a successfull connection, CP
>> sends some packets to see if connection is properly established. If I
>> could ignore them, and go ahead, probably the connection would
>> succeed...
>> What do you think about this? Do I need to patch openswan to reach this
>> goal?
>> Thanks, Luca
>>
>>
>> On Thu, May 13, 2010 at 3:04 PM, Ondrej Valousek <webserv at s3group.cz> wrote:
>>
>>
>> My wild guess is that the your Checkpoint only accepts SecurID clients and
>> not authentication using certificates.
>>
>>
>> Yes, that's probably it. At main mode, your CP responds with ISAKMP_NEXT_N
>> (which I do not know what it is) whereas it should respond with
>> ISAKMP_NEXT_KE (which is most likely Key Exchange request -Paul to
>> clarify...)
>>
>> O.
>>
>>
>>
>>
>