Date: Sat, 21 Nov 2015 09:12:12 +1000 From: Nathan Aherne <nathan@reddog.com.au> To: Ian Smith <smithi@nimnet.asn.au> Cc: Julian Elischer <julian@freebsd.org>, freebsd-ipfw@freebsd.org Subject: Re: Kernel NAT issues Message-ID: <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> In-Reply-To: <20151119032200.T27669@sola.nimnet.asn.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <C1C25100-FBD4-42F4-94F7-965B270D927F@reddog.com.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> <20151119032200.T27669@sola.nimnet.asn.au>
next in thread | previous in thread | raw e-mail | index | archive | help
I am not exactly sure how to draw the setup so it doesn’t confuse the situation. The setup is extremely simple (I am not running vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface) network or with public IPs. The jails with private IPs are the HTTP app jails. The Host runs a HTTP Proxy (nginx) and forwards traffic to each HTTP App jail based on the URL it receives. The jails with public IPs are things like database jails which cannot be proxied by the Host. I can happily communicate with any jail from my laptop (externally) but when I want one jail to communicate with another jail (for example an App Jail communicating with the database jail) the traffic shows as backwards (destination:port -> source:port) in the IPFW logs (tshark shows the traffic correctly source:port -> destination:port). The jail to jail traffic tries to go over the lo1 interface (backwards) and is blocked. Below is some IPFW logs of an App jail (10.0.0.25) communicating with the database jail (aaa.bbb.ccc.ddd) IPFW logs. The lines labelled UNKNOWN is the check-state rule (everything is labelled UNKNOWN even if it is KNOWN traffic) Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 tshark output (loopback and wan interface capture for port 5432) Capturing on 'Loopback' and 'bce0' 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 TSecr=0 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142888539 TSecr=0 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142891767 TSecr=0 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142894976 TSecr=0 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142898180 TSecr=0 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142901389 TSecr=0 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142907602 TSecr=0 > If so, what sort of routing is setup on both host and jails? Routing is what would be added by default (whatever the host system adds when adding an IP), there is no custom routing. I have wondered if I need to modify the routing table to get this to work. Below is the output of netstat -rn www.xxx.yy.zzz is the gateway address eee.fff.gg.hhh is the database jail public IP aaa.bbb.cc.ddd is the public IP for NAT lll.mmm.nn.ooo is the Hosts public IP Routing tables Internet: Destination Gateway Flags Netif Expire default www.xxx.yy.zzz UGS bce0 10.0.0.1 link#6 UH lo1 10.0.0.2 link#6 UH lo1 10.0.0.3 link#6 UH lo1 10.0.0.4 link#6 UH lo1 10.0.0.5 link#6 UH lo1 10.0.0.6 link#6 UH lo1 10.0.0.7 link#6 UH lo1 10.0.0.8 link#6 UH lo1 10.0.0.9 link#6 UH lo1 10.0.0.10 link#6 UH lo1 10.0.0.11 link#6 UH lo1 10.0.0.12 link#6 UH lo1 10.0.0.13 link#6 UH lo1 10.0.0.14 link#6 UH lo1 10.0.0.15 link#6 UH lo1 10.0.0.16 link#6 UH lo1 10.0.0.17 link#6 UH lo1 10.0.0.18 link#6 UH lo1 10.0.0.19 link#6 UH lo1 10.0.0.20 link#6 UH lo1 10.0.0.21 link#6 UH lo1 10.0.0.22 link#6 UH lo1 10.0.0.23 link#6 UH lo1 10.0.0.24 link#6 UH lo1 10.0.0.25 link#6 UH lo1 10.0.0.26 link#6 UH lo1 www.xxx.yy.zzz/25 link#1 U bce0 eee.fff.gg.hhh link#1 UHS lo0 eee.fff.gg.hhh/32 link#1 U bce0 aaa.bbb.cc.ddd link#1 UHS lo0 aaa.bbb.cc.ddd/32 link#1 U bce0 lll.mmm.nn.ooo link#1 UHS lo0 127.0.0.1 link#5 UH lo0 Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 ::1 link#5 UH lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 fe80::/10 ::1 UGRS lo0 fe80::%lo0/64 link#5 U lo0 fe80::1%lo0 link#5 UHS lo0 ff01::%lo0/32 ::1 U lo0 ff02::/16 ::1 UGRS lo0 ff02::%lo0/32 ::1 U lo0 > Anything like ? > http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search <http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search>; Yes just like that. Regards, Nathan > On 19 Nov 2015, at 2:46 am, Ian Smith <smithi@nimnet.asn.au> wrote: > > On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: >> On 11/18/15 8:40 AM, Nathan Aherne wrote: >>> For some reason hairpin (loopback nat or nat reflection) does not seem to >>> be working, which is why I chose IPFW in the first place. > >> it would be good to see a diagram of what this actually means. > > Anything like ? > http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search > > Was this so one jail can only access service/s provided by other jail/s, > both/all with internal NAT'd addresses, by using only the public address > and port of the 'router', which IIRC this is a single system with jails? > > If so, what sort of routing is setup on both host and jails? > > (blindfolded, no idea where I've pinned the donkey's tail :) > > cheers, Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9D81BDD4-200C-40AB-AB24-B1112881E43A>