[Openswan Users] Connecting to OS X Server for L2TP with NAT-T

[Kevin Locke]

Hello All,

I'm attempting to connect to a Mac OS X 10.7.5 server for L2TP from
Openswan 2.6.38 on Debian.  Both systems are behind a NAT.  I have had
success connecting with both Windows and OS X clients, but no luck
with Openswan.

Configuration and log information are as follows (for server
vpn.example.com with address 69.X.X.X and client 174.Y.Y.Y):

-8<-- ipsec.conf ------------------------------------------------------
version	2.0

config setup
	nat_traversal=yes
	oe=off
	protostack=netkey

conn myconnection
	authby=secret
	type=transport
	left=%defaultroute
	leftprotoport=17/1701
	right=vpn.example.com
	rightid=10.0.0.10
	rightprotoport=17/1701
	auto=add
-8<--------------------------------------------------------------------

Note that I set rightid to the server's private IP address to get past
the error "we require peer to have ID '69.X.X.X', but peer
declares '10.0.0.10'".

Also note that I've also tried adding "pfs=no", "rekey=yes",
"keyingtries=3", and "forceencaps=yes" in various combinations with
the same result.

-8<-- Client log ------------------------------------------------------
002 "myconnection" #1: initiating Main Mode
104 "myconnection" #1: STATE_MAIN_I1: initiate
003 "myconnection" #1: received Vendor ID payload [RFC 3947] method set to=115 
003 "myconnection" #1: received Vendor ID payload [Dead Peer Detection]
002 "myconnection" #1: enabling possible NAT-traversal with method RFC 3947 (NAT-Traversal)
002 "myconnection" #1: transition from state STATE_MAIN_I1 to state STATE_MAIN_I2
106 "myconnection" #1: STATE_MAIN_I2: sent MI2, expecting MR2
003 "myconnection" #1: NAT-Traversal: Result using draft-ietf-ipsec-nat-t-ike (MacOS X): both are NATed
002 "myconnection" #1: transition from state STATE_MAIN_I2 to state STATE_MAIN_I3
108 "myconnection" #1: STATE_MAIN_I3: sent MI3, expecting MR3
002 "myconnection" #1: Main mode peer ID is ID_IPV4_ADDR: '10.0.0.10'
002 "myconnection" #1: transition from state STATE_MAIN_I3 to state STATE_MAIN_I4
004 "myconnection" #1: STATE_MAIN_I4: ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY cipher=aes_128 prf=oakley_sha group=modp1024}
002 "myconnection" #2: initiating Quick Mode PSK+ENCRYPT+PFS+UP+IKEv2ALLOW+SAREFTRACK {using isakmp#1 msgid:4e01d877 proposal=defaults pfsgroup=OAKLEY_GROUP_MODP1024}
117 "myconnection" #2: STATE_QUICK_I1: initiate
003 "myconnection" #2: length of ISAKMP Notification Payload is smaller than minimum
003 "myconnection" #2: malformed payload in packet
002 "myconnection" #2: sending notification PAYLOAD_MALFORMED to 69.X.X.X:4500
003 "myconnection" #2: length of ISAKMP Notification Payload is smaller than minimum
003 "myconnection" #2: malformed payload in packet
002 "myconnection" #2: sending notification PAYLOAD_MALFORMED to 69.X.X.X:4500
003 "myconnection" #2: length of ISAKMP Notification Payload is smaller than minimum
003 "myconnection" #2: malformed payload in packet
002 "myconnection" #2: sending notification PAYLOAD_MALFORMED to 69.X.X.X:4500
-8<--------------------------------------------------------------------

-8<-- Server log ------------------------------------------------------
IKE Packet: receive success. (Information message).
Connecting.
IPSec Phase1 started (Initiated by peer).
IKE Packet: receive success. (Responder, Main-Mode message 1).
IKE Packet: transmit success. (Responder, Main-Mode message 2).
IKE Packet: receive success. (Responder, Main-Mode message 3).
IKE Packet: transmit success. (Responder, Main-Mode message 4).
IKEv1 Phase1 AUTH: success. (Responder, Main-Mode Message 5).
IKE Packet: receive success. (Responder, Main-Mode message 5).
IKEv1 Phase1 Responder: success. (Responder, Main-Mode).
IKE Packet: transmit success. (Responder, Main-Mode message 6).
IPSec Phase1 established (Initiated by peer).
IPSec Phase2 started (Initiated by peer).
IKE Packet: receive success. (Responder, Quick-Mode message 1).
IKE Packet: transmit success. (Responder, Quick-Mode message 2).
IKE Packet: transmit success. (Phase2 Retransmit).
--- last message repeated 2 times ---
Received retransmitted packet from 174.Y.Y.Y[4500].
IKE Packet: transmit success. (Phase2 Retransmit).
--- last message repeated 5 times ---
-8<--------------------------------------------------------------------

Similar errors with Mac OS X have been discussed a few times that I
could find:

* It is mentioned on Jacco de Leeuw's extremely useful website at
  http://www.jacco2.dds.nl/networking/linux-l2tp.html#NAT-T as a
  known incompatibility with Mac OS X server 10.4.3 and below.
* It was discussed in a thread from 2005
  https://lists.openswan.org/pipermail/dev/2005-September/001066.html
  relating to https://www.openswan.org/issues/462 and updated in 2006
  https://lists.openswan.org/pipermail/dev/2006-February/001288.html

But my reading of the above suggests that the issue was resolved long
ago (both by better RFC 3947 support in OS X 10.4.5 and inclusion of
patches to work around the issues) and I couldn't find any current
open issues that seemed to match my problem.

I'd appreciate any suggestions that you might have for how to solve
or further diagnose the issue.

Thanks!

Kevin