[Openswan dev] assertion failed with AES and no keylen

[Mathieu Lafon]

There is a failed assertion in kernel.c when the other peer want to use
AES but has not sent the KEY_LENGTH attribute :

#3: responding to Quick Mode
| kernel_alg_esp_enc_keylen():alg_id=12, keylen=16
| keymat_len=32 key_len=32 authkeylen=16
#3: ASSERTION FAILED at kernel.c:2301: st->st_esp.keymat_len == key_len
    + ei->authkeylen

This has been detected during interoperability tests with Bewan LanBooster
6104xg.

In that case, AES-128 should be used (see kernel_alg.c for comments on 
that)
but we want enough keying material for a 256b key.

Proposed patch (Openswan-1) :

diff -u -r1.3 kernel.c
--- openswan/pluto/kernel.c     4 May 2004 12:51:24 -0000       1.3
+++ openswan/pluto/kernel.c     14 May 2004 09:54:14 -0000
@@ -2280,8 +2280,12 @@
                        (int)key_len, (int)ei->enckeylen);
                        goto fail;
                }
-       } else {
+       } else { 
+#ifndef NO_KERNEL_ALG
+               key_len = 
kernel_alg_esp_enc_keylen(st->st_esp.attrs.transid);
+#else
                key_len = ei->enckeylen;
+#endif
        }
        /* Grrrrr.... f*cking 7 bits jurassic algos  */


-- 
Mathieu Lafon - Arkoon Network Security
-------------- next part --------------
A non-text attachment was scrubbed...
Name: patch-aes-nokeylen.diff
Type: application/octet-stream
Size: 428 bytes
Desc: not available
Url : http://lists.openswan.org/pipermail/dev/attachments/20040514/04117f35/patch-aes-nokeylen.obj