[Openswan dev] Possible Memory Leak in Openswan 2.4.5

[Jim Barber]

Sorry, sent this to Paul instead of to the list.

--------

Paul Wouters wrote:
> On Mon, 31 Jul 2006, Jim Barber wrote:
> 
>> What would happen is that Phase 1 of the IPSec tunnel did the shared key
>> exchange and successfully completed.
>> The phase 2 part failed with an error about INVALID_ID_INFORMATION due to the
>> mismatched 10.128.0.0/22 vs 10.128.0.0/16.
> 
> Can you check and see if this problem is similar to:
> http://bugs.xelerance.com/view.php?id=645

I've checked out that bug and I don't have any of the error messages reported in it.
The only messages I got are like the ones I put into my original post.

>> With a mis-configuration like this, after a while the pluto process starts to
>> take up a huge amount of memory.
> 
>> In the logs you can see messages such as 'starting keying attempt $X of an
>> unlimited number' where $X is a number.
> 
> Could you reomcpile openswan and edit programs/pluto/Makefile and enable
> -DLEAK_DETECTIVE
> 
> then run it until you see a lot of memory being eaten, and nicely restart
> openswan (/etc/init.d/openswan restart). This should log a lot of memory
> debugging to syslog, which I would be interested in seeing.

Given this is on a server which is supposed to be production, I don't really have the luxury of building the code and playing around.
The bad tunnel was one I set up initially for testing and then I implemented other production tunnels.
Upon making some tweaks to the production tunnels for the new netmask value I failed to tweak the test one which is what exposed the problem.
Time has rolled on, and that test tunnel is now a production one as well so I can't really play with it.

In theory this bug should hopefully be easy enough to reproduce by setting up a bad tunnel that will complete Phase 1 correctly, but fail Phase 2 and leave it to retry for a long time.

If you have trouble reproducing it, what I may be able to do is set up two virtual PCs on my desktop computer with a minimal debian install on each and create a badly configured tunnel between them and see what happens.
Then if they fail, have a go at recompiling the debian package with the LEAK_DETECTIVE flag, install it on the virtual PCs and try again.

Jim.