From owner-freebsd-ipfw@freebsd.org Fri Nov 20 23:12:22 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0F0AA33416 for ; Fri, 20 Nov 2015 23:12:22 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6FA2A1837; Fri, 20 Nov 2015 23:12:21 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 7E7BB2C3241; Sat, 21 Nov 2015 09:08:14 +1000 (AEST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id kBtjACVFzQUU; Sat, 21 Nov 2015 09:08:14 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 43CC12C32B8; Sat, 21 Nov 2015 09:08:14 +1000 (AEST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id PMSJdcvxF3_K; Sat, 21 Nov 2015 09:08:14 +1000 (AEST) Received: from [172.20.10.3] (unknown [1.132.70.186]) by mail.7sq.com.au (Postfix) with ESMTPSA id 620ED2C3241; Sat, 21 Nov 2015 09:08:13 +1000 (AEST) Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: Kernel NAT issues From: Nathan Aherne In-Reply-To: <20151119032200.T27669@sola.nimnet.asn.au> Date: Sat, 21 Nov 2015 09:12:12 +1000 Cc: Julian Elischer , freebsd-ipfw@freebsd.org Message-Id: <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> <20151119032200.T27669@sola.nimnet.asn.au> To: Ian Smith X-Mailer: Apple Mail (2.2104) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2015 23:12:23 -0000 I am not exactly sure how to draw the setup so it doesn=E2=80=99t = confuse the situation. The setup is extremely simple (I am not running = vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface) network = or with public IPs. The jails with private IPs are the HTTP app jails. = The Host runs a HTTP Proxy (nginx) and forwards traffic to each HTTP App = jail based on the URL it receives. The jails with public IPs are things = like database jails which cannot be proxied by the Host. I can happily communicate with any jail from my laptop (externally) but = when I want one jail to communicate with another jail (for example an = App Jail communicating with the database jail) the traffic shows as = backwards (destination:port -> source:port) in the IPFW logs (tshark = shows the traffic correctly source:port -> destination:port). The jail = to jail traffic tries to go over the lo1 interface (backwards) and is = blocked. Below is some IPFW logs of an App jail (10.0.0.25) = communicating with the database jail (aaa.bbb.ccc.ddd) IPFW logs. The lines labelled UNKNOWN is the check-state rule = (everything is labelled UNKNOWN even if it is KNOWN traffic) Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 = 10.0.0.25:42957 out via lo1 tshark output (loopback and wan interface capture for port 5432) Capturing on 'Loopback' and 'bce0' 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957=E2=86=925432 = [SYN] Seq=3D0 Win=3D65535 Len=3D0 MSS=3D16344 WS=3D64 SACK_PERM=3D1 = TSval=3D142885525 TSecr=3D0 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142888539 TSecr=3D0 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142891767 TSecr=3D0 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142894976 TSecr=3D0 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142898180 TSecr=3D0 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142901389 TSecr=3D0 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP = Retransmission] 42957=E2=86=925432 [SYN] Seq=3D0 Win=3D65535 Len=3D0 = MSS=3D16344 WS=3D64 SACK_PERM=3D1 TSval=3D142907602 TSecr=3D0 > If so, what sort of routing is setup on both host and jails? Routing is what would be added by default (whatever the host system adds = when adding an IP), there is no custom routing. I have wondered if I = need to modify the routing table to get this to work.=20 Below is the output of netstat -rn www.xxx.yy.zzz is the gateway address eee.fff.gg.hhh is the database jail public IP aaa.bbb.cc.ddd is the public IP for NAT lll.mmm.nn.ooo is the Hosts public IP Routing tables Internet: Destination Gateway Flags Netif Expire default www.xxx.yy.zzz UGS bce0 10.0.0.1 link#6 UH lo1 10.0.0.2 link#6 UH lo1 10.0.0.3 link#6 UH lo1 10.0.0.4 link#6 UH lo1 10.0.0.5 link#6 UH lo1 10.0.0.6 link#6 UH lo1 10.0.0.7 link#6 UH lo1 10.0.0.8 link#6 UH lo1 10.0.0.9 link#6 UH lo1 10.0.0.10 link#6 UH lo1 10.0.0.11 link#6 UH lo1 10.0.0.12 link#6 UH lo1 10.0.0.13 link#6 UH lo1 10.0.0.14 link#6 UH lo1 10.0.0.15 link#6 UH lo1 10.0.0.16 link#6 UH lo1 10.0.0.17 link#6 UH lo1 10.0.0.18 link#6 UH lo1 10.0.0.19 link#6 UH lo1 10.0.0.20 link#6 UH lo1 10.0.0.21 link#6 UH lo1 10.0.0.22 link#6 UH lo1 10.0.0.23 link#6 UH lo1 10.0.0.24 link#6 UH lo1 10.0.0.25 link#6 UH lo1 10.0.0.26 link#6 UH lo1 www.xxx.yy.zzz/25 link#1 U bce0 eee.fff.gg.hhh link#1 UHS lo0 eee.fff.gg.hhh/32 link#1 U bce0 aaa.bbb.cc.ddd link#1 UHS lo0 aaa.bbb.cc.ddd/32 link#1 U bce0 lll.mmm.nn.ooo link#1 UHS lo0 127.0.0.1 link#5 UH lo0 Internet6: Destination Gateway Flags = Netif Expire ::/96 ::1 UGRS = lo0 ::1 link#5 UH = lo0 ::ffff:0.0.0.0/96 ::1 UGRS = lo0 fe80::/10 ::1 UGRS = lo0 fe80::%lo0/64 link#5 U = lo0 fe80::1%lo0 link#5 UHS = lo0 ff01::%lo0/32 ::1 U = lo0 ff02::/16 ::1 UGRS = lo0 ff02::%lo0/32 ::1 U = lo0 > Anything like ? > = http://kb.juniper.net/InfoCenter/index?page=3Dcontent&id=3DKB24639&actp=3D= search = Yes just like that. Regards, Nathan > On 19 Nov 2015, at 2:46 am, Ian Smith wrote: >=20 > On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: >> On 11/18/15 8:40 AM, Nathan Aherne wrote: >>> For some reason hairpin (loopback nat or nat reflection) does not = seem to >>> be working, which is why I chose IPFW in the first place. >=20 >> it would be good to see a diagram of what this actually means. >=20 > Anything like ? > = http://kb.juniper.net/InfoCenter/index?page=3Dcontent&id=3DKB24639&actp=3D= search >=20 > Was this so one jail can only access service/s provided by other = jail/s,=20 > both/all with internal NAT'd addresses, by using only the public = address=20 > and port of the 'router', which IIRC this is a single system with = jails? >=20 > If so, what sort of routing is setup on both host and jails? >=20 > (blindfolded, no idea where I've pinned the donkey's tail :) >=20 > cheers, Ian