From owner-freebsd-net@freebsd.org Mon Dec 26 22:53:02 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D1F1C92362 for ; Mon, 26 Dec 2016 22:53:02 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:1008:bcb:1:1:0:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CB0231CEE for ; Mon, 26 Dec 2016 22:53:01 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e02:5301:d0be:dcb:6e13:1bd3] (p2003008C2E025301D0BE0DCB6E131BD3.dip0.t-ipconnect.de [IPv6:2003:8c:2e02:5301:d0be:dcb:6e13:1bd3]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3tnZ5b51J7zSbp for ; Mon, 26 Dec 2016 23:52:51 +0100 (CET) From: Michael Grimm Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\)) Subject: IPSec tunnel, VNET jail and routing issue Message-Id: Date: Mon, 26 Dec 2016 23:52:50 +0100 To: freebsd-net@freebsd.org X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3259) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Dec 2016 22:53:02 -0000 Hi -- I am referring to the following (simplified) setup: [hostA /ix0 / 2001:dead::1 / 1.2.3.4] <=3D=3D=3D=3D=3D IPsec tunnel = =3D=3D=3D=3D=3D> [hostB / ix0 / 2001:beef::10 / 10.20.30.40] | | = =20 | | [jail1 / bridge0 / fd00:a::1 / 10.1.1.1] [jail1 = / bridge0 / fd00:b::2 / 10.2.2.2] All my jails are VNET jails, that use the bridge0 (epair) device. Thus, = all IPv4 and IPv6 addresses of my local networks an A and B are bound to = the bridge0 interface! But, the IPsec tunnel (via racoon) is anchored at = public IPv4 addresses on ix0 at both hosts. Task: route all local traffic from hostA to hostB via the tunnel. Working: IPv6 traffic is running fine, meaning, that I can reach every = jail from every host. That has been working for years. Issue: I recently wanted to extend my setup to local IPv4 addresses of = my jails, and failed miserably. Configuration (shown for hostA, only): setkey.conf # hostA hostB = hostA hostB=20 spdadd fd00:a::/64 fd00:b::/64 any -P out ipsec = esp/tunnel/1.2.3.4-10.20.30.40/require; spdadd fd00:a::/64 2001:beef::/56 any -P out ipsec = esp/tunnel/1.2.3.4-10.20.30.40/require; spdadd 2001:dead::/56 fd00:b::/64 any -P out ipsec = esp/tunnel/1.2.3.4-10.20.30.40/require; # hostB hostA = hostB hostA=20 spdadd fd00:b::/64 fd00:a::/64 any -P in ipsec = esp/tunnel/10.20.30.40-1.2.3.4/require; spdadd fd00:b::/64 2001:dead::/56 any -P in ipsec = esp/tunnel/10.20.30.40-1.2.3.4/require; spdadd 2001:beef::/56 fd00:a::/64 any -P in ipsec = esp/tunnel/10.20.30.40-1.2.3.4/require; # hostA hostB = hostA hostB=20 spdadd 10.1.1.0/24 10.2.2.0/24 any -P out ipsec = esp/tunnel/1.2.3.4-10.20.30.40/require; spdadd 10.1.1.0/24 10.20.30.40 any -P out ipsec = esp/tunnel/1.2.3.4-10.20.30.40/require; spdadd 1.2.3.4 10.2.2.0/24 any -P out ipsec = esp/tunnel/1.2.3.4-10.20.30.40/require; # hostB hostA = hostB hostA=20 spdadd 10.2.2.0/24 10.1.1.0/24 any -P in ipsec = esp/tunnel/10.20.30.40-1.2.3.4/require; spdadd 10.2.2.0/24 1.2.3.4 any -P in ipsec = esp/tunnel/10.20.30.40-1.2.3.4/require; spdadd 10.20.30.40 10.1.1.0/24 any -P in ipsec = esp/tunnel/10.20.30.40-1.2.3.4/require; There is no specific routing regarding the tunnel defined. All = should be done by this spdadd's. Achieved sofar: #) I can reach each jail at the other site from the host. #) Allowing arpproxy_all=3D"YES" will satisfy ARP (MACs from = opposite VNET jails will become assigned).=20 I do not know if that is needed, but now ping from jails to = the opposite jails will at least start to send ICMP packages. Unsolved issue: I cannot reach opposite jails from another host's jail; e.g.: = ping 10.20.30.40 in jail1@hostA will not work. Observations so far: #) tcpdump shows for "ping 10.2.2.2 in jail1@hostA" ICMP traffic = at the bridge0 at hostA: IP 10.1.1.1 > 10.2.2.2: ICMP echo request, id 20099, seq = 0, length 64=20 and at bridge0 at hostB: IP 10.1.1.1 > 10.2.2.2: ICMP echo request, id 15233, seq = 6, length 64=20 IP 10.2.2.2 > 10.1.1.1: ICMP echo reply, id 15233, seq = 6, length 64=20 =09 Hmm: hostA doesn't get an echo replay, although hostB did send = one.=20 #) tcpdump shows for "ping 10.2.2.2 at hostA" *no* ICMP traffic = at hostA@bridge0 or hostA@ix0 but ICMP traffic at hostB@bridge0: IP 1.2.3.4 > 10.2.2.2: ICMP echo request, id 60543, seq = 0, length 64=20 IP 10.2.2.2 > 1.2.3.4: ICMP echo reply, id 60543, seq 0, = length 64=20 =09 Hmm: it's working. #) It looks to me as if the tunnel does not recognise "spdadd = 10.1.1.0/24 10.2.2.0/24" and vice versa settings because those IPs are = bound to the bridge. #) Whenever an IP bound to ix0 is involved (host to jail) the = corresponding spdadd parts are recognised. #) adding static routes like "add route 10.2.2.0/24 1.2.3.4" and = alike do not solve my issue.=20 Questions: #) Is this an issue with IPsec/racoon? #) Is this a routing issue? #) Why does IPv6 address space work (identical configuration = regarding jails, firewalling, routing, et al.) #) Any other idea? Sorry for this lengthy post, and any feedback is highly welcome, Michael