[Openswan dev] [Openswan Users] pluto segfaults when using SHA2 256 hash

[Paul Wouters]

On Thu, 2 Feb 2012, Abhinav Bhagwat wrote:

Here is the additional information,

Paul


> ---------- Forwarded message ----------
> Date: Fri, 2 Dec 2011 09:41:34
> From: Paul Wouters <paul at xelerance.com>
> Subject: ESP with SHA2 trunction problem for XFRM with rhel5/6

> 
> I'm working on resolving the badly truncated SHA2 auth algorithms for ESP.
> This is to fix interop with other devices like Checkpoint.
> 
> The bug is related to the draft and final version of RFC-4868. In the draft
> the SHA2_256 was truncated at 96 bits, but in the final version this became
> nnn/2, that is 128bits for SHA2_256 and 256bits for SHA2_512. The Linux
> XFRM/NETKEY interface did not fix the truncation. Instead they cloned the
> XFRMA_ALG_AUTH rta_type to XFRMA_ALG_AUTH_TRUNC, and cloned the xfrm_algo
> struct to xfrm_algo_auth. The latter one has an additional field called
> alg_trunc_len. I pulled the newer xfrm code into programs/pluto/linux26/
> 
> I know this fix does NOT appear in the rhel 5 or 6 kernels. It does
> appear in 2.6.38. Avesh's updated xfrm code did not have all the required
> code, I assume he updated it based on the rhel6 kernel. I pulled in newer
> xfrm headers to gain XFRMA_ALG_AUTH_TRUNC support.
> 
> Additionally, openswan did not pass the truncated hash length properly
> from the esp= option all the way to netlink_add_sa(). I'm working on
> fixing that.

[...]