Date: Wed, 18 Nov 2015 10:40:14 +1000 From: Nathan Aherne <nathan@reddog.com.au> To: Ian Smith <smithi@nimnet.asn.au> Cc: freebsd-ipfw@freebsd.org Subject: Re: Kernel NAT issues Message-ID: <32DEEFB3-E41F-40CD-8E1A-520FB261C572@reddog.com.au> In-Reply-To: <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <C1C25100-FBD4-42F4-94F7-965B270D927F@reddog.com.au> <20151014232026.S15983@sola.nimnet.asn.au> <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Everyone, I think I have worked this out and have a working Stateful IPFW NAT (its = worked fine for a few weeks) rule set. Hopefully this saves someone else = a few weeks of their lives. For some reason hairpin (loopback nat or nat reflection) does not seem = to be working, which is why I chose IPFW in the first place. I am actually getting some really weird log results from IPFW (for inter = jail communication ). IPFW logs show traffic flowing in the opposite = direction to what tshark shows. Tshark shows the traffic in the = direction I would expect. IPFW is blocking the traffic because it is = seeing it as going the wrong way. Anyone have an idea how I can solve = this issue, I would super appreciate any pointers! To clarify inter jail communication - not using the local jail IPs but = DNS - they all resolve to the same public IP. I cannot use split DNS. = ##########################################################################= ##########################################################################= ### #!/bin/sh # Configuration wif=3D"bce0" # WAN interface jif=3D"lo1" # Jail interface jsn=3D"10.0.0.0/16" # Jail subnet plip=3D"10.0.0.1" # Proxy IP hwip=3D=93aaa.bbb.ccc.ddd" # Host WAN IP nwip=3D=93www.xxx.yyy.zzz" # NAT WAN IP # Script variables cmd=3D"ipfw -q add" ks=3D"keep-state" sks=3D"setup keep-state" # Flush all rules ipfw -q -f flush # Enable NAT ipfw nat 1 config ip $nwip log # Allow all loopback traffic $cmd 5 allow ip from any to any via lo0 # Block any traffic not for Host $cmd 6 deny ip from any to not me in via $wif $cmd 100 nat 1 log ip from any to $nwip recv $wif $cmd 101 check-state log # Host $cmd 110 allow icmp from any to $hwip in via $wif $ks $cmd 111 allow tcp from any to $hwip 65222 in via $wif $sks $cmd 112 allow icmp from $hwip to any out via $wif $ks $cmd 113 allow tcp from $hwip to any 53, 80, 443, 22, 65222 out via $wif = $sks $cmd 114 allow udp from $hwip to any 53, 123 out via $wif $ks # Incoming NAT $cmd 120 skipto 65510 log tcp from any to $jsn recv $wif $sks $cmd 121 skipto 65510 log udp from any to $jsn recv $wif $ks # Outgoing NAT $cmd 122 skipto 65510 log tcp from $jsn to not $jsn xmit $wif $sks $cmd 123 skipto 65510 log udp from $jsn to not $jsn xmit $wif $ks # JAILS RULES 200-65000 # Block any other traffic $cmd 65501 deny log ip from any to any # Outgoing NAT $cmd 65510 nat 1 log ip from $jsn to any xmit $wif $cmd 65511 allow log ip from $nwip to any xmit $wif # Block any other traffic #$cmd 65519 deny log ip from any to any # Incoming NAT $cmd 65520 allow log ip from any to $jsn recv $wif # Block any other traffic $cmd 65522 deny log ip from any to any = ##########################################################################= ##########################################################################= ### Regards, Nathan > On 21 Oct 2015, at 9:02 am, Nathan Aherne <nathan@reddog.com.au> = wrote: >=20 > Hi Ian, >=20 > Thank you very much for your response! Sorry about the late response, = I have been offline for a few days. >=20 > I think I may have worked this issue out. I am bringing up a bunch of = Jails today to test my firewall rules in the hopes that I have corrected = my problem. I will reply back either way. >=20 > Regards, >=20 > Nathan >=20 >> On 15 Oct 2015, at 12:51 am, Ian Smith <smithi@nimnet.asn.au> wrote: >>=20 >> On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote: >>> Hi Ian, >>>=20 >>> Thank you for your response. >>>=20 >>> I didn=FF=FFt post my ruleset because I should be able to fix the = issue=20 >>> myself but I see now that my request to explain =FF=FFhow NAT = works=FF=FF was=20 >>> incorrect. >>>=20 >>> I have now included my ruleset below (as well as my initial email). >>=20 >> Hi Nathan, >>=20 >> I was really hoping someone who knows more about stateful rule = handling=20 >> (and jail networking) might have a go at this. Oh well I'll try, but=20= >> I'm a lousy mindreader, and really don't know which of the below=20 >> constitutes 'hairpin NAT'. Perhaps showing your 'netstat -finet -an'=20= >> and 'netstat -finet -rn' may shed light on routing? And 'ifconfig'? >>=20 >>> # Enable NAT >>> ipfw nat 1 config ip $jip same_ports log >>=20 >> I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that=20 >> WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP = routed=20 >> to you, and so traffic to it won't be subject to NAT .. correct? But=20= >> the WWW... address and all 10.0/16 addresses are jails, not any = separate=20 >> boxes you gateway for, right? Just the one external interface, = right? >>=20 >>> 00005 allow ip from any to any via lo0 >>> 00006 deny ip from any to not me in via bce0 >>> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0 >>> 00101 check-state >>=20 >> Ok, inbound from WAN is nat'd and existing stateful flows followed by=20= >> executing the rule that originally kept state. Where this is a = skipto,=20 >> skipto will be performed. But where it's a nat rule, I've no idea ..=20= >> see below, but you really don't want to add keep-state (again) there. >>=20 >>> 00110 allow icmp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ = recv bce0 keep-state >>=20 >> Hmm. I'd limit this to perhaps icmptypes 0,3,8,11 - though a = stateless=20 >> rule would make more sense especially for inbound ICMP. But moving = on .. >>=20 >>> 00111 allow tcp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ = dst-port 65222 recv bce0 setup keep-state >>=20 >> Ok, but showting why plain text works better than HTML on lists :) >>=20 >>> 00112 allow icmp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any = xmit bce0 keep-state >>> 00113 allow tcp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any = dst-port 53,80,443,22,65222 xmit bce0 setup keep-state >>> 00114 allow udp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any = dst-port 53,123 xmit bce0 keep-state >>=20 >> Smells ok. >>=20 >>> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup = keep-state >>> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 = keep-state >>=20 >> Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's=20= >> got a problem .. see below. These two are inbound traffic (recv) and = as=20 >> is, skipping to 65501 will fall through two outbound rules to be = denied. >>=20 >> Either allow them here directly, or likely better, skipto a separate >> target that then allows (or denies) them, if that's what you = intended? >>=20 >>> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 setup keep-state >>> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit = bce0 keep-state >>=20 >> Ok, this traffic does needs to be NAT'd on the way out. >>=20 >>> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup = keep-state >>> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out = setup keep-state >>> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state >>=20 >> Not clear why these tcp ports are open inbound and outbound? = Presumably=20 >> this is jail-to-jail traffic? Perhaps not relevant to your problem. >>=20 >>> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup = keep-state >>> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out = setup keep-state >>> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state >>> 65500 deny log ip from any to any >>=20 >> Ok. >>=20 >>> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 = keep-state >>=20 >> This the target for outbound traffix, xmit bce0, so nat is = appropriate. =20 >> Does jail-to-jail traffic travels via lo1? Or what? >>=20 >> This won't do anything to inbound traffic, but that really shouldn't = get=20 >> here except returns as the result of check-state - not from 120 & = 121. >>=20 >> But keep-state is not ok, state was already set on the skipto. I = don't=20 >> know how this extra keep-state might behave - does anyone have an = idea? >>=20 >> Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic. >>=20 >>> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state >>=20 >> So, only remaining traffic is outbound from the host itself, and = traffic=20 >> that is to 10.0/16, but not from AAA... is to be dropped, correct? >>=20 >> I'm not sure whether 'allow ip .. keep-state' covers tcp, udp, icmp=20= >> states .. myself, I'd go for separate rules for each eg tcp, udp, .. = and=20 >> I'd do it somewhere else than as a fall through from outbound nat = rule,=20 >> it's confusing here, to me anyway .. unless I've missed the reason? >>=20 >>> 65534 deny log ip from any to any >>> 65535 deny ip from any to any >>=20 >>=20 >> Ok, now for your demo of the problem from the later mail, which I've=20= >> reformated to quote properly, so: >>=20 >>> To further illustrate my issue, this is a small log output. >>>=20 >>> I am running host google.com <http://google.com/>; in the jail, which=20= >>> has the IP 10.0.0.1. The UNKNOWN line is logging on the check-state=20= >>> rule. >>=20 >> I see you don't have logging on 101 above now. Probably best. >>=20 >>> I would expect the first piece of traffic out would be UNKNOWN=20 >>> (does not have an entry in the state table) but it seems the=20 >>> returning traffic is also showing as UNKNOWN (the second 101). >>=20 >> I've never logged a check-state, but UNKNOWN may not mean that .. >>=20 >>> You can see that the traffic is returning on the same port it went=20= >>> out on, so its obviously the returning traffic. I am not sure why=20 >>> state is not being kept? >>=20 >> Well perhaps it is .. the return packet is from 8.8.8.8 to 10.0.0.1, = so=20 >> it's been correctly NAT'd on the way in. Get rid of that keep-state = on=20 >> the nat rule at 65501 and see if not creating double entries in the=20= >> state table helps. And change the skipto target on 120 & 121 to only=20= >> pass outbound traffic to outbound NAT rule/s. >>=20 >> Once you've done outbound NAT, probably best just to 'allow [log] = all'? >>=20 >>> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 >>> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP = 10.0.0.1:57446 8.8.8.8:53 out via bce0 >>> Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 = 8.8.8.8:53 out via bce0 >>> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >>> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >>> Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 = 10.0.0.1:57446 in via bce0 >>=20 >> That said, I can see why this return packet would be denied even if = it=20 >> were in the nat table: it would execute 'skipto 65501', which nat = rule=20 >> does not apply, as it's not outbound, and rule 65502 does not apply, = as=20 >> it's neither from AAA... nor outbound, so it's then denied by 65534. >>=20 >> Hope this helps. Please cc me on any response to the list. >>=20 >> It would be great if someone else might care to lend an oar here; I'm=20= >> paddling out of my depth. >>=20 >> cheers, Ian >>=20 >> [..] >=20 > _______________________________________________ > freebsd-ipfw@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw > To unsubscribe, send any mail to = "freebsd-ipfw-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?32DEEFB3-E41F-40CD-8E1A-520FB261C572>