[Openswan dev] [Openswan Users] pluto segfaults when using SHA2 256 hash

[Paul Wouters]

On Thu, 2 Feb 2012, Abhinav Bhagwat wrote:

> Thanks Paul. That works. However, I see another issue. If I connect two linux boxes it works fine.
> Simiarly if I connect two windows boxes, it works fine. However, if I try to connect to a windows 2K8 box
> to a linux box, it does not work. Phase 1 and phase 2 SAs are both successfully established. But, when I
> telnet to windows box, the ESP packet reaches the windows box but there is not reply back. If I replace
> sha256 with sha1, it all works fine.

That is probably due to the SHA2 256 Linux bug. In all kernels up to
2.6.32 or so, the SHA256 was truncated. For newer kernels, it requires
a different call via kernel_netlink to use the fixed up version of the
XFM code that fixed the truncation.

I started work on fixing that, but it did not yet quite work as
expected.

A quick and dirty hack could be to change the kernel truncation and
recompile the kernel. That would be in linux-2.6.xx/net/crypto/xfrm_algo.c
around the section:

{
         .name = "hmac(sha256)",
         .compat = "sha256",

         .uinfo = {
                 .auth = {
                         .icv_truncbits = 96,
                         .icv_fullbits = 256,
                 }
         },

While the draft had 96, the final RFC has fullbits/2, so you should make
it 128. For sha256 it would be "256".

I hope to get back on track to fix that so we can specify:

 	phase2alg=aes128-sha2_256-128

and

 	phase2alg=aes128-sha2_256-96

But those changes are more invasive then I had time for a few weeks ago.

I'll forward another message to the list with details that did not make
it to the dev archives.

Paul