[Openswan dev] "auth failed on incoming packet" when using NAT-T

[Markus Hanauska]

No reply on user list, maybe someone on dev list can help.


Hi there!

I have a little Problem with NAT-T, using OpensSWAN 2.2.0, KLIPS on 
Linux Kernel 2.4.28 (with NAT-T patch, that I had to partly patch in by 
hand; but I read here that the bug is already known and fixed), the 
distribution is Fedora Core 1. OpenSWAN plays the server here, on the 
other side a patched version of Racoon (Kame project) with NAT-T 
support plays the client. It's a host to network connection.

Without NAT-T I can make a tunnel and ping the private interface of my 
Linux gateway. Everthing works like expected. Packages arive and leave 
the interface as ESP. The NAT device in my test setup uses IPSec Pass 
Through, but that only works for one connection from behind that NAT 
device.

For a second connection NAT-T is necessary. But the second I enable 
NAT-T, nothing works anymore.

Pluto makes the connection as before, detecting NAT-T draft-2, 
switching to port 4500 during phase 1 (so does Racoon) and they both do 
phase 1 and phase 2 like a charm. The tunnel is established, both sides 
are sure about having a working tunnel. But all traffic is thrown away 
by KLIPS with the message:

"auth failed on incoming packet"

The client side sends out the ESP packages, wrapped in UDP packages, on 
port 4500. I know pretty well how  the client side works, after all 
I've written the wrapper all by myself.

Okay, you may say that the problem is home-made, my wrapper might be 
wrong. Could be, but then why does it work with SonicWALL, Netscreen 
and Zyxel devices? And it works with old NAT-T (the one on port 500) 
and with new NAT-T (the one on port 4500) - e.g. the small SonicWALLs 
use the old one (RFC draft 0/1), the big ones use the new one (RFC 
draft 2); whereby the real new NAT-T starts with draft 3 (where the 
payload numbers have been altered).

I've read all the RFCs and I do exactly what is written there. Add an 
UDP header in-between, correct the IP header (length, checksum, next 
protocol) and add a non-ESP marker (draft 2 and above) or a non-IKE 
marker (draft 0-1) where necessary. If this code were wrong, it 
wouldn't work with hardware solutions of other vendors.

The next thing you may guess is, maybe the ESP implementation is 
broken. If this was the case, why does it work perfectly the second I 
disable NAT-T on the OpenSWAN side? After all the ESP implementation 
doesn't know that I wrap the packages (the packages are caught and 
altered between protocol and network layer, transparent for both 
layers), for the ESP layer, everything is like without NAT-T.

What else can I tell you? Both sides show that they use 3des and MD5 on 
the tunnel and authentication during IKE was pre-shared key.

On the client side, I have setkey and can thus easy access to the SPI 
or the key used to encrypt the packages, and so on. I don't know how to 
get such output on the server (OpenSWAN) side.

Maybe you can help me with that, give me some hints on how to get more 
info useful for debugging the problem. Or maybe you have an idea what 
might cause the problem. Any advice will be greatly appreciated!

Thanks in advance.

-- 
Best Regards,
         Markus Hanauska