[Openswan dev] [Openswan Users] pluto segfaults when using SHA2 256 hash

[Paul Wouters]

On Mon, 6 Feb 2012, Abhinav Bhagwat wrote:

> Thanks Paul. That works! After recompiling the kernel with icv_tuncbits=128
> I was able to successfully create a transport tunnel between linux and
> windows.

Great! Thanks for the feedback. I will add a check in "ipsec verify".

Paul

> ____________________________________________________________________________
> From: Paul Wouters <paul at nohats.ca>
> Sent: Friday, February 3, 2012 10:48 PM
> Subject: Re: [Openswan Users] pluto segfaults when using SHA2 256 hash
> 
> On Thu, 2 Feb 2012, Abhinav Bhagwat wrote:
> 
> > Thanks Paul. That works. However, I see another issue. If I connect two
> linux boxes it works fine.
> > Simiarly if I connect two windows boxes, it works fine. However, if I try
> to connect to a windows 2K8 box
> > to a linux box, it does not work. Phase 1 and phase 2 SAs are both
> successfully established. But, when I
> > telnet to windows box, the ESP packet reaches the windows box but there is
> not reply back. If I replace
> > sha256 with sha1, it all works fine.
> 
> That is probably due to the SHA2 256 Linux bug. In all kernels up to
> 2.6.32 or so, the SHA256 was truncated. For newer kernels, it requires
> a different call via kernel_netlink to use the fixed up version of the
> XFM code that fixed the truncation.
> 
> I started work on fixing that, but it did not yet quite work as
> expected.
> 
> A quick and dirty hack could be to change the kernel truncation and
> recompile the kernel. That would be in linux-2.6.xx/net/crypto/xfrm_algo.c
> around the section:
> 
> {
>         .name = "hmac(sha256)",
>         .compat = "sha256",
> 
>         .uinfo = {
>                 .auth = {
>                         .icv_truncbits = 96,
>                         .icv_fullbits = 256,
>                 }
>         },
> 
> While the draft had 96, the final RFC has fullbits/2, so you should make
> it 128. For sha256 it would be "256".
> 
> I hope to get back on track to fix that so we can specify:
> 
>     phase2alg=aes128-sha2_256-128
> 
> and
> 
>     phase2alg=aes128-sha2_256-96
> 
> But those changes are more invasive then I had time for a few weeks ago.
> 
> I'll forward another message to the list with details that did not make
> it to the dev archives.
> 
> Paul
> 
> 
> 
>