Date: Sat, 21 Nov 2015 22:08:26 +0800 From: Julian Elischer <julian@freebsd.org> To: Nathan Aherne <nathan@reddog.com.au> Cc: "freebsd-ipfw@freebsd.org" <Freebsd-ipfw@freebsd.org> Subject: Re: Kernel NAT issues Message-ID: <56507ADA.8030202@freebsd.org> In-Reply-To: <3BF360A8-35E6-4043-8AFF-87D983F29C66@reddog.com.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <C1C25100-FBD4-42F4-94F7-965B270D927F@reddog.com.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> <564C8879.8070307@freebsd.org> <20151119032200.T27669@sola.nimnet.asn.au> <9D81BDD4-200C-40AB-AB24-B1112881E43A@reddog.com.au> <3BF360A8-35E6-4043-8AFF-87D983F29C66@reddog.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
I note, no answers yet.. I plan on spending some time tomorrow trying to understand this issue. just letting you know.. On 21/11/2015 10:06 AM, Nathan Aherne wrote: > I had a bit of a think about how to describe what I am trying to > achieve. > > I am treating each jail likes its own little "virtual machine”. The > jail provides certain services, using things like nginx or nodejs, > php-fpm, mysql or postgresql. The jails can control connections to > themselves by configuring the firewall ports that are opened on the > IP their IP (10.0.0.0/16 or a public IP). I know the jails have no > firewall of their own, the firewall is configured from the host. > > I want each jail or “virtual machine” to be able to communicate with > one another and the wider internet. When a jail does a DNS query for > another App jail, it may get a public IP on its own Host (or it may > get another host) and it has no issues being able to communicate > with another jail on the same host. > > At the moment all of the above is working perfectly except for jail > to jail communication on the same host (when the communication is > not directly between 10.0.0.0/16 IP addresses). > > Regards, > > Nathan > >> On 21 Nov 2015, at 9:12 am, Nathan Aherne <nathan@reddog.com.au >> <mailto:nathan@reddog.com.au>> wrote: >> >> I am not exactly sure how to draw the setup so it doesn’t confuse >> the situation. The setup is extremely simple (I am not running >> vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface) >> network or with public IPs. The jails with private IPs are the HTTP >> app jails. The Host runs a HTTP Proxy (nginx) and forwards traffic >> to each HTTP App jail based on the URL it receives. The jails with >> public IPs are things like database jails which cannot be proxied >> by the Host. >> >> I can happily communicate with any jail from my laptop (externally) >> but when I want one jail to communicate with another jail (for >> example an App Jail communicating with the database jail) the >> traffic shows as backwards (destination:port -> source:port) in the >> IPFW logs (tshark shows the traffic correctly source:port -> >> destination:port). The jail to jail traffic tries to go over the >> lo1 interface (backwards) and is blocked. Below is some IPFW logs >> of an App jail (10.0.0.25) communicating with the database jail >> (aaa.bbb.ccc.ddd) >> >> IPFW logs. The lines labelled UNKNOWN is the check-state rule >> (everything is labelled UNKNOWN even if it is KNOWN traffic) >> >> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP >> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1 >> >> tshark output (loopback and wan interface capture for port 5432) >> >> Capturing on 'Loopback' and 'bce0' >> 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN] >> Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525 >> TSecr=0 >> 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142888539 TSecr=0 >> 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142891767 TSecr=0 >> 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142894976 TSecr=0 >> 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142898180 TSecr=0 >> 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142901389 TSecr=0 >> 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP >> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344 >> WS=64 SACK_PERM=1 TSval=142907602 TSecr=0 >> >> >>> If so, what sort of routing is setup on both host and jails? >> >> Routing is what would be added by default (whatever the host system >> adds when adding an IP), there is no custom routing. I have >> wondered if I need to modify the routing table to get this to work. >> >> Below is the output of netstat -rn >> >> www.xxx.yy <http://www.xxx.yy/>.zzz is the gateway address >> eee.fff.gg.hhh is the database jail public IP >> aaa.bbb.cc.ddd is the public IP for NAT >> lll.mmm.nn.ooo is the Hosts public IP >> >> >> Routing tables >> >> Internet: >> Destination Gateway Flags Netif Expire >> default www.xxx.yy <http://www.xxx.yy/>.zzz UGS bce0 >> 10.0.0.1 link#6 UH lo1 >> 10.0.0.2 link#6 UH lo1 >> 10.0.0.3 link#6 UH lo1 >> 10.0.0.4 link#6 UH lo1 >> 10.0.0.5 link#6 UH lo1 >> 10.0.0.6 link#6 UH lo1 >> 10.0.0.7 link#6 UH lo1 >> 10.0.0.8 link#6 UH lo1 >> 10.0.0.9 link#6 UH lo1 >> 10.0.0.10 link#6 UH lo1 >> 10.0.0.11 link#6 UH lo1 >> 10.0.0.12 link#6 UH lo1 >> 10.0.0.13 link#6 UH lo1 >> 10.0.0.14 link#6 UH lo1 >> 10.0.0.15 link#6 UH lo1 >> 10.0.0.16 link#6 UH lo1 >> 10.0.0.17 link#6 UH lo1 >> 10.0.0.18 link#6 UH lo1 >> 10.0.0.19 link#6 UH lo1 >> 10.0.0.20 link#6 UH lo1 >> 10.0.0.21 link#6 UH lo1 >> 10.0.0.22 link#6 UH lo1 >> 10.0.0.23 link#6 UH lo1 >> 10.0.0.24 link#6 UH lo1 >> 10.0.0.25 link#6 UH lo1 >> 10.0.0.26 link#6 UH lo1 >> www.xxx.yy.zzz/25 <http://www.xxx.yy.zzz/25>; link#1 U >> bce0 >> eee.fff.gg.hhh link#1 UHS lo0 >> eee.fff.gg.hhh/32 link#1 U bce0 >> aaa.bbb.cc <http://aaa.bbb.cc/>.ddd link#1 UHS lo0 >> aaa.bbb.cc.ddd/32 link#1 U bce0 >> lll.mmm.nn.ooo link#1 UHS lo0 >> 127.0.0.1 link#5 UH lo0 >> >> Internet6: >> Destination Gateway Flags >> Netif Expire >> ::/96 ::1 UGRS lo0 >> ::1 link#5 UH lo0 >> ::ffff:0.0.0.0/96 ::1 UGRS >> lo0 >> fe80::/10 ::1 UGRS lo0 >> fe80::%lo0/64 link#5 U lo0 >> fe80::1%lo0 link#5 UHS lo0 >> ff01::%lo0/32 ::1 U lo0 >> ff02::/16 ::1 UGRS lo0 >> ff02::%lo0/32 ::1 U lo0 >> >>> Anything like ? >>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search >> >> Yes just like that. >> >> Regards, >> >> Nathan >> >>> On 19 Nov 2015, at 2:46 am, Ian Smith <smithi@nimnet.asn.au >>> <mailto:smithi@nimnet.asn.au>> wrote: >>> >>> On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote: >>>> On 11/18/15 8:40 AM, Nathan Aherne wrote: >>>>> For some reason hairpin (loopback nat or nat reflection) does >>>>> not seem to >>>>> be working, which is why I chose IPFW in the first place. >>> >>>> it would be good to see a diagram of what this actually means. >>> >>> Anything like ? >>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search >>> >>> Was this so one jail can only access service/s provided by other >>> jail/s, >>> both/all with internal NAT'd addresses, by using only the public >>> address >>> and port of the 'router', which IIRC this is a single system with >>> jails? >>> >>> If so, what sort of routing is setup on both host and jails? >>> >>> (blindfolded, no idea where I've pinned the donkey's tail :) >>> >>> cheers, Ian >> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?56507ADA.8030202>