[Openswan Users] OpenSWAN / Native 2.6 IPSec: MTU / PMTUD woes

[Jan Koop]

Hi list,

I just wanted to share my partially horrifying experience with Native 
2.6 Kernel IPSec, OpenSWAN and path MTU discovery (PTMUD) with the list.

Problem description: Large packets near LAN MTU (1500) get dropped when 
passing tunnel (with the DF flag set), TCP sessions hang (e.g. ssh-login 
OK, 'ls' works, but cat'ting a large file makes the terminal hang), the 
effect on UDP connections is unknown to me.

I tracked the problem down to some kind of MTU issue, usually with KLIPS 
under kernel 2.4 I used parameters like overridemtu to control such 
issues when necessary, but that only applies to KLIPS with its ipsecN 
interfaces. Under kernel 2.6 native ipsec pmtud with ipsec is reported 
to be broken in the posts i dug out.
I found a partial solution, only for TCP, namely MSS clamping with 
netfilter. This did work, but a particular UDP application still wasn't 
working. On the search for a solution I came across a way to achieve a 
"overridemtu"-like behavior. This didn't solve the problem with the 
UDP-dependant application though :( , but can be used as an alternative 
to MSS clamping.
I just modified the _updown script to set the mtu to 1412 (hopefully 
enough room for all overheads to come) on the routes it sets up.

diff:
------------------------------------------------------------------------------------------------------- 
/usr/lib/ipsec/_updown      2005-01-06 13:55:08.000000000 +0100
+++ /root/bin/ipsec_updown.mtufix       2005-02-07 09:40:57.309394632 +0100
@@ -316,6 +316,9 @@
            parms3="$parms3 src ${PLUTO_MY_SOURCEIP%/*}"
        fi

+       # fix MTU issues
+       parms3="$parms3 mtu 1412"
+
        case "$PLUTO_PEER_CLIENT_NET/$PLUTO_PEER_CLIENT_MASK" in
        "0.0.0.0/0.0.0.0")
                # opportunistic encryption work around
----------------------------------------------------------------------------------------------------

Maybe this wouldn't be bad to be controllable via ipsec.conf in some way 
like "if overridemtu is set, then if KLIPS, then set MTU on ipsecN 
interface, else set MTU on host/net route"

Well, this does work now for TCP. If I understand correctly, this is a 
static solution, while clamping the MSS to the pmtu would be more dynamic.

I will have to look in other places for a fix to the UDP problem - if 
anyone has any idea why apple remote desktop shows a mouse pointer, but 
no graphics after the connection succeeded - please share it with me ;). 
It used to work with a 2.4 / KLIPS-based gateway, but when i switched 
over to 2.6 native IPSec the woes began.

Bye,
Jan