Date: Wed, 21 Oct 2015 09:02:33 +1000 From: Nathan Aherne <nathan@reddog.com.au> To: Ian Smith <smithi@nimnet.asn.au> Cc: freebsd-ipfw@freebsd.org Subject: Re: Kernel NAT issues Message-ID: <9908EC22-344F-4D0B-8930-7D2C70B084A1@reddog.com.au> In-Reply-To: <20151014232026.S15983@sola.nimnet.asn.au> References: <94B91F98-DE01-4A10-8AB5-4193FE11AF3F@reddog.com.au> <20151013142301.B67283@sola.nimnet.asn.au> <C1C25100-FBD4-42F4-94F7-965B270D927F@reddog.com.au> <20151014232026.S15983@sola.nimnet.asn.au>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Ian, Thank you very much for your response! Sorry about the late response, I have been offline for a few days. I think I may have worked this issue out. I am bringing up a bunch of Jails today to test my firewall rules in the hopes that I have corrected my problem. I will reply back either way. Regards, Nathan > On 15 Oct 2015, at 12:51 am, Ian Smith <smithi@nimnet.asn.au> wrote: > > On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote: >> Hi Ian, >> >> Thank you for your response. >> >> I didn˙˙t post my ruleset because I should be able to fix the issue >> myself but I see now that my request to explain ˙˙how NAT works˙˙ was >> incorrect. >> >> I have now included my ruleset below (as well as my initial email). > > Hi Nathan, > > I was really hoping someone who knows more about stateful rule handling > (and jail networking) might have a go at this. Oh well I'll try, but > I'm a lousy mindreader, and really don't know which of the below > constitutes 'hairpin NAT'. Perhaps showing your 'netstat -finet -an' > and 'netstat -finet -rn' may shed light on routing? And 'ifconfig'? > >> # Enable NAT >> ipfw nat 1 config ip $jip same_ports log > > I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that > WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP routed > to you, and so traffic to it won't be subject to NAT .. correct? But > the WWW... address and all 10.0/16 addresses are jails, not any separate > boxes you gateway for, right? Just the one external interface, right? > >> 00005 allow ip from any to any via lo0 >> 00006 deny ip from any to not me in via bce0 >> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0 >> 00101 check-state > > Ok, inbound from WAN is nat'd and existing stateful flows followed by > executing the rule that originally kept state. Where this is a skipto, > skipto will be performed. But where it's a nat rule, I've no idea .. > see below, but you really don't want to add keep-state (again) there. > >> 00110 allow icmp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ recv bce0 keep-state > > Hmm. I'd limit this to perhaps icmptypes 0,3,8,11 - though a stateless > rule would make more sense especially for inbound ICMP. But moving on .. > >> 00111 allow tcp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ dst-port 65222 recv bce0 setup keep-state > > Ok, but showting why plain text works better than HTML on lists :) > >> 00112 allow icmp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any xmit bce0 keep-state >> 00113 allow tcp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 53,80,443,22,65222 xmit bce0 setup keep-state >> 00114 allow udp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 53,123 xmit bce0 keep-state > > Smells ok. > >> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state >> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state > > Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's > got a problem .. see below. These two are inbound traffic (recv) and as > is, skipping to 65501 will fall through two outbound rules to be denied. > > Either allow them here directly, or likely better, skipto a separate > target that then allows (or denies) them, if that's what you intended? > >> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 setup keep-state >> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state > > Ok, this traffic does needs to be NAT'd on the way out. > >> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup keep-state >> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup keep-state >> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state > > Not clear why these tcp ports are open inbound and outbound? Presumably > this is jail-to-jail traffic? Perhaps not relevant to your problem. > >> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup keep-state >> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup keep-state >> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state >> 65500 deny log ip from any to any > > Ok. > >> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state > > This the target for outbound traffix, xmit bce0, so nat is appropriate. > Does jail-to-jail traffic travels via lo1? Or what? > > This won't do anything to inbound traffic, but that really shouldn't get > here except returns as the result of check-state - not from 120 & 121. > > But keep-state is not ok, state was already set on the skipto. I don't > know how this extra keep-state might behave - does anyone have an idea? > > Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic. > >> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state > > So, only remaining traffic is outbound from the host itself, and traffic > that is to 10.0/16, but not from AAA... is to be dropped, correct? > > I'm not sure whether 'allow ip .. keep-state' covers tcp, udp, icmp > states .. myself, I'd go for separate rules for each eg tcp, udp, .. and > I'd do it somewhere else than as a fall through from outbound nat rule, > it's confusing here, to me anyway .. unless I've missed the reason? > >> 65534 deny log ip from any to any >> 65535 deny ip from any to any > > > Ok, now for your demo of the problem from the later mail, which I've > reformated to quote properly, so: > >> To further illustrate my issue, this is a small log output. >> >> I am running host google.com <http://google.com/>; in the jail, which >> has the IP 10.0.0.1. The UNKNOWN line is logging on the check-state >> rule. > > I see you don't have logging on 101 above now. Probably best. > >> I would expect the first piece of traffic out would be UNKNOWN >> (does not have an entry in the state table) but it seems the >> returning traffic is also showing as UNKNOWN (the second 101). > > I've never logged a check-state, but UNKNOWN may not mean that .. > >> You can see that the traffic is returning on the same port it went >> out on, so its obviously the returning traffic. I am not sure why >> state is not being kept? > > Well perhaps it is .. the return packet is from 8.8.8.8 to 10.0.0.1, so > it's been correctly NAT'd on the way in. Get rid of that keep-state on > the nat rule at 65501 and see if not creating double entries in the > state table helps. And change the skipto target on 120 & 121 to only > pass outbound traffic to outbound NAT rule/s. > > Once you've done outbound NAT, probably best just to 'allow [log] all'? > >> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0 >> Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0 > > That said, I can see why this return packet would be denied even if it > were in the nat table: it would execute 'skipto 65501', which nat rule > does not apply, as it's not outbound, and rule 65502 does not apply, as > it's neither from AAA... nor outbound, so it's then denied by 65534. > > Hope this helps. Please cc me on any response to the list. > > It would be great if someone else might care to lend an oar here; I'm > paddling out of my depth. > > cheers, Ian > > [..]
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9908EC22-344F-4D0B-8930-7D2C70B084A1>