[Openswan Users] Openswan cannot make connection with DrayTek in Aggressive Mode (packet rejected: should have been encrypted)

[Steve Leung]

Hi,


I'm facing the problem with Openswan IPsec U2.6.35/K2.6.28.10 while
connecting to DrayTek Vigor2910 using Aggressive Mode.

===
Here are the log messages:

Sep 9 13:33:02 openswan openswan: "IPSEC1/1x1"[1] 10.0.0.1 #18:
transition from state STATE_AGGR_R0 to state STATE_AGGR_R1
Sep 9 13:33:02 openswan openswan: | sending reply packet to
10.0.0.1:500 (from port 500)
Sep 9 13:33:02 openswan openswan: | sending 356 bytes for
STATE_AGGR_R0 through eth2:500 to 10.0.0.1:500 (using #18)
Sep 9 13:33:02 openswan openswan: | inserting event EVENT_RETRANSMIT,
timeout in 10 seconds for #18
Sep 9 13:33:02 openswan openswan: "IPSEC1/1x1"[1] 10.0.0.1 #18:
STATE_AGGR_R1: sent AR1, expecting AI2
Sep 9 13:33:02 openswan openswan: | modecfg pull: noquirk policy:push not-client
Sep 9 13:33:02 openswan openswan: | phase 1 is done, looking for phase
2 to unpend
Sep 9 13:33:02 openswan openswan: | complete state transition with STF_INLINE
Sep 9 13:33:02 openswan openswan: | complete state transition with STF_INLINE
Sep 9 13:33:02 openswan openswan: | * processed 0 messages from
cryptographic helpers
Sep 9 13:33:02 openswan openswan: | next event EVENT_RETRANSMIT in 4
seconds for #14
Sep 9 13:33:02 openswan openswan: | next event EVENT_RETRANSMIT in 4
seconds for #14
Sep 9 13:33:03 openswan openswan: |
Sep 9 13:33:03 openswan openswan: | *received 88 bytes from
10.0.0.1:500 on eth2 (port=500)
Sep 9 13:33:03 openswan openswan: | processing version=1.0 packet with
exchange type=ISAKMP_XCHG_AGGR (4)
Sep 9 13:33:03 openswan openswan: | ICOOKIE: c1 ae 25 e1 c0 b0 28 e5
Sep 9 13:33:03 openswan openswan: | RCOOKIE: bb 29 54 ad 53 46 6f cf
Sep 9 13:33:03 openswan openswan: | state hash entry 24
Sep 9 13:33:03 openswan openswan: | v1 peer and cookies match on #18,
provided msgid 00000000 vs 00000000
Sep 9 13:33:03 openswan openswan: | v1 state object #18 found, in STATE_AGGR_R1
Sep 9 13:33:03 openswan openswan: | processing connection IPSEC1/1x11 10.0.0.1
Sep 9 13:33:03 openswan openswan: "IPSEC1/1x1"[1] 10.0.0.1 #18: packet
rejected: should have been encrypted
Sep 9 13:33:03 openswan openswan: "IPSEC1/1x1"[1] 10.0.0.1 #18:
sending notification INVALID_FLAGS to 10.0.0.1:500
Sep 9 13:33:03 openswan openswan: | sending 40 bytes for notification
packet through eth2:500 to 10.0.0.1:500 (using #18)
Sep 9 13:33:03 openswan openswan: | * processed 0 messages from
cryptographic helpers
Sep 9 13:33:03 openswan openswan: | next event EVENT_RETRANSMIT in 3
seconds for #14
Sep 9 13:33:03 openswan openswan: | next event EVENT_RETRANSMIT in 3
seconds for #14
Sep 9 13:33:03 openswan openswan: |
Sep 9 13:33:03 openswan openswan: | *received 148 bytes from
10.0.0.1:500 on eth2 (port=500)
Sep 9 13:33:03 openswan openswan: | processing version=1.0 packet with
exchange type=ISAKMP_XCHG_QUICK (32)
Sep 9 13:33:03 openswan openswan: | ICOOKIE: c1 ae 25 e1 c0 b0 28 e5
Sep 9 13:33:03 openswan openswan: | RCOOKIE: bb 29 54 ad 53 46 6f cf
Sep 9 13:33:03 openswan openswan: | state hash entry 24
Sep 9 13:33:03 openswan openswan: | v1 peer and cookies match on #18,
provided msgid 02ebe05b vs 00000000
Sep 9 13:33:03 openswan openswan: | v1 state object not found
Sep 9 13:33:03 openswan openswan: | ICOOKIE: c1 ae 25 e1 c0 b0 28 e5
Sep 9 13:33:03 openswan openswan: | RCOOKIE: bb 29 54 ad 53 46 6f cf
Sep 9 13:33:03 openswan openswan: | state hash entry 24
Sep 9 13:33:03 openswan openswan: | v1 peer and cookies match on #18,
provided msgid 00000000 vs 00000000
Sep 9 13:33:03 openswan openswan: | v1 state object #18 found, in STATE_AGGR_R1
Sep 9 13:33:03 openswan openswan: | processing connection IPSEC1/1x11 10.0.0.1
Sep 9 13:33:03 openswan openswan: "IPSEC1/1x1"[1] 10.0.0.1 #18: Quick
Mode message is unacceptable because it is for an incomplete ISAKMP SA
Sep 9 13:33:03 openswan openswan: | payload malformed after IV
Sep 9 13:33:03 openswan openswan: | 26 58 de 9f ef 69 d6 cd 98 e1 a8
fa 19 c8 fc 26
Sep 9 13:33:03 openswan openswan: "IPSEC1/1x1"[1] 10.0.0.1 #18:
sending notification PAYLOAD_MALFORMED to 10.0.0.1:500
Sep 9 13:33:03 openswan openswan: | sending 40 bytes for notification
packet through eth2:500 to 10.0.0.1:500 (using #18)
Sep 9 13:33:03 openswan openswan: | * processed 0 messages from
cryptographic helpers
Sep 9 13:33:03 openswan openswan: | next event EVENT_RETRANSMIT in 3
seconds for #14

===
and my ipsec.conf:

version 2.0
config setup
protostack=netkey
nat_traversal=on
virtual_private=%v4:10.0.0.0/8,%v4:192.168.0.0/16,%v4:172.16.0.0/12
oe=no
uniqueids=yes
nhelpers=0
plutodebug=control

conn IPSEC1
type=tunnel
authby=secret
left=10.0.0.2
leftsubnets={10.7.1.1/16}
leftid=ipsec at abc
right=%any
rightsubnets={192.168.1.1/24}
rightid=ipsec at draytek
ike=3des-md5;modp1024
phase2alg=3des-md5
pfs=no
forceencaps=off
ikev2=no
aggrmode=on
salifetime=28800s
ikelifetime=3600s
dpddelay=10
dpdtimeout=30
dpdaction=restart
rekey=yes
keyingtries=%forever
auto=start

===

Also I found this should be something similar with BUG#1218
(https://bugs.openswan.org/issues/1218).

As I see the error "packet rejected: should have been encrypted", I
think it's a problem from DrayTek at first, so to make sure the
DrayTek's IPsec Aggressive mode is working well, I have created
another IPsec profile on DrayTek to connect with Juniper SSG140, and
everything goes well, so I think it's not the problem on DrayTek's
side.

Anyone know if there is anything wrong with my openswan config?

Thank you very much :)



Best regards,
Steve