[Openswan Users] FATAL ERROR: unable to malloc... after kernel update

[Nels Lindquist]

I'm running OpenSWAN 2.6.41 on CentOS 6 (x86_64 arch).  Recently we
rebooted our VPN endpoint after some updates (including a new kernel)
and now IPSEC (particularly pluto) fails to start.

Here's the startup logs from /var/log/secure:

Aug 16 21:17:55 yeggate ipsec__plutorun: Starting Pluto subsystem...
Aug 16 21:17:55 yeggate pluto[42107]: nss directory plutomain: /etc/ipsec.d
Aug 16 21:17:55 yeggate pluto[42107]: NSS Initialized
Aug 16 21:17:55 yeggate pluto[42107]: Non-fips mode set in
/proc/sys/crypto/fips_enabled
Aug 16 21:17:55 yeggate pluto[42107]: Starting Pluto (Openswan Version
2.6.41; Vendor ID OSWsxljF at TSY) pid:42107
Aug 16 21:17:55 yeggate pluto[42107]: Non-fips mode set in
/proc/sys/crypto/fips_enabled
Aug 16 21:17:55 yeggate pluto[42107]: LEAK_DETECTIVE support [disabled]
Aug 16 21:17:55 yeggate pluto[42107]: OCF support for IKE [disabled]
Aug 16 21:17:55 yeggate pluto[42107]: SAref support [disabled]: Protocol
not available
Aug 16 21:17:55 yeggate pluto[42107]: SAbind support [disabled]:
Protocol not available
Aug 16 21:17:55 yeggate pluto[42107]: NSS support [enabled]
Aug 16 21:17:55 yeggate pluto[42107]: HAVE_STATSD notification support
not compiled in
Aug 16 21:17:55 yeggate pluto[42107]: Setting NAT-Traversal port-4500
floating to on
Aug 16 21:17:55 yeggate pluto[42107]:    port floating activation
criteria nat_t=1/port_float=1
Aug 16 21:17:55 yeggate pluto[42107]:    NAT-Traversal support  [enabled]
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_register_enc(): Activating
OAKLEY_AES_CBC: Ok (ret=0)
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_register_hash():
Activating OAKLEY_SHA2_512: Ok (ret=0)
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_register_hash():
Activating OAKLEY_SHA2_256: Ok (ret=0)
Aug 16 21:17:55 yeggate pluto[42107]: starting up 7 cryptographic helpers
Aug 16 21:17:55 yeggate pluto[42107]: started helper (thread)
pid=140265451960064 (fd:7)
Aug 16 21:17:55 yeggate pluto[42107]: started helper (thread)
pid=140265310385920 (fd:9)
Aug 16 21:17:55 yeggate pluto[42107]: started helper (thread)
pid=140265441470208 (fd:11)
Aug 16 21:17:55 yeggate pluto[42107]: started helper (thread)
pid=140265430980352 (fd:13)
Aug 16 21:17:55 yeggate pluto[42107]: started helper (thread)
pid=140265420490496 (fd:15)
Aug 16 21:17:55 yeggate pluto[42107]: started helper (thread)
pid=140265410000640 (fd:17)
Aug 16 21:17:55 yeggate pluto[42107]: started helper (thread)
pid=140265399510784 (fd:19)
Aug 16 21:17:55 yeggate pluto[42107]: Using Linux XFRM/NETKEY IPsec
interface code on 2.6.32-279.19.1.el6.x86_64
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_register_enc(): Activating
aes_ccm_8: Ok (ret=0)
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_add(): ERROR: algo_type
'0', algo_id '0', Algorithm type already exists
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_register_enc(): Activating
aes_ccm_12: FAILED (ret=-17)
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_add(): ERROR: algo_type
'0', algo_id '0', Algorithm type already exists
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_register_enc(): Activating
aes_ccm_16: FAILED (ret=-17)
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_add(): ERROR: algo_type
'0', algo_id '0', Algorithm type already exists
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_register_enc(): Activating
aes_gcm_8: FAILED (ret=-17)
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_add(): ERROR: algo_type
'0', algo_id '0', Algorithm type already exists
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_register_enc(): Activating
aes_gcm_12: FAILED (ret=-17)
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_add(): ERROR: algo_type
'0', algo_id '0', Algorithm type already exists
Aug 16 21:17:55 yeggate pluto[42107]: ike_alg_register_enc(): Activating
aes_gcm_16: FAILED (ret=-17)
Aug 16 21:17:55 yeggate pluto[42107]: FATAL ERROR: unable to malloc
9223372036854775807 bytes for CA cert

I tried rebuilding OpenSWAN against the new kernel headers, but it
didn't make any difference.


-- 
Nels Lindquist
<nlindq at maei.ca>