Date: Thu, 24 Jul 2003 15:53:17 -0700 From: Luigi Rizzo <luigi@FreeBSD.org> To: Sean Chittenden <seanc@FreeBSD.org> Cc: ipfw@FreeBSD.org Subject: Re: Dynamic rules not being matched after divert... Message-ID: <20030724155317.A35706@xorpc.icir.org> In-Reply-To: <20030724203657.GA415@perrin.int.nxad.com>; from seanc@FreeBSD.org on Thu, Jul 24, 2003 at 01:36:57PM -0700 References: <20030724203657.GA415@perrin.int.nxad.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jul 24, 2003 at 01:36:57PM -0700, Sean Chittenden wrote: > I'm setting up an ipfw2+natd gateway and am pretty convinced there's a > bug in the way that ipfw2 promotes dynamic rules to being fully > established. Thanks for the detailed report, but I am afraid to say that ipfw2 works correctly, it's your ruleset which is buggy. What happens is that if you run the session from the laptop, you have the following packets: #1 in sis0 SYN laptop->host.example.com this matches rule 01200, creates a stateful entry and is accepted. The IP layer decides it needs to be forwarded and the firewall is invoked again by ip_output() with packet #1. This matches rule 00300, and is reinjected at 00400 as #2: #2 in-sis0 out sis1 SYN fw->host.example.com this matches rule 01800, creates a stateful entry and is accepted. The SYN/ACK coming back is #3: #3 in sis1 SYN-ACK host.example.com->fw matches rule 200, gets diverted and reinjected at 300 as #4: #4 in sis1 SYN-ACK host.example.com->laptop which matches dynamic rule 01200 and is accepted. >From now on, forward packets (laptop->host.example.com) will match rule 00500 (dynamic rule 01200) in the way in, go to ip_output(), match rule 00300, get translated into (fw->host.example.com), and match rule 00500 (dynamic rule 01800). Return packets (host.example.com->fw) match rule 00200, get translated into (host.example.com->laptop) and match rule 00500 (dynamic rule 01200) on the way in, and then again rule 00500 (dynamic rule 01200) on the way out. Note that RETURN PACKETS NEVER MATCH DYNAMIC RULE 01800! The counts you see only refer to forward packets. The situation is different for locally generated packets because they are not translated, so return packets will match. In order to fix your ruleset I would suggest to put the keep-state into a 'skipto' action so that even after a check-state you will have a chance for further processing of the packet. But it is extremely tricky to get it right. cheers luigi ----- (your rules numbered, for reference) ---- # Basic anti-spoofing 00100 set 1 deny log all from any to 0.0.0.0/8,... via ${oif} # Allow connections from the internal network to be NAT'ed 00200 set 1 divert natd all from any to me in recv ${oif} 00300 set 1 divert natd all from not me to any recv ${iif} out xmit ${oif} # Finish the last of the basic anti-spoofing now that packets have been NAT'ed 00400 set 1 deny log all from 0.0.0.0/8,... to any via ${oif} # Allow all connections that have dynamic rules built for them, # but deny established connections that don't have a dynamic rule. 00500 set 1 check-state 00600 set 1 deny log tcp from any to any established 00700 set 1 deny log ip from any to any in frag 00800 set 1 deny log ip from any to any not verrevpath in # Allow all localhost connections 00900 set 1 allow tcp from me to any out via lo0 setup keep-state 01000 set 1 deny log tcp from me to any out via lo0 01100 set 1 allow ip from me to any out via lo0 keep-state # Allow all connections from the internal network 01200 set 1 allow tcp from 192.168.1.0/24 to any in via ${iif} setup keep-state 01300 set 1 deny log tcp from 192.168.1.0/24 to any 01400 set 1 allow ip from 192.168.1.0/24 to any in via ${iif} keep-state 01500 set 1 allow ip from any to any in via ${iif} keep-state # Allow all connections from my network card that I initiate 01600 set 1 allow tcp from me to any out xmit any setup keep-state 01700 set 1 deny log tcp from me to any 01800 set 1 allow ip from me to any out xmit any keep-state # Enable ICMP to work since it is not a stateful protocol 01900 set 1 allow icmp from any to any icmptypes 0,3,8,11,12,13,14 # This sends a RESET to all ident packets. 02000 set 1 reset log tcp from any to me 113 in recv any # Deny all the rest. 02100 set 1 deny log ip from any to any ##### END SCRIPT > Here's the rule set that I'm using: > > ##### BEGIN SCRIPT > fwcmd="/sbin/ipfw" > > # Inside interface > iif="sis0" > > # Outside interface > oif="sis1" > > $fwcmd -f flush > > # Basic anti-spoofing > $fwcmd add set 1 deny log all from any to 0.0.0.0/8,10.0.0.0/8,169.254.0.0/16,192.0.2.0/24,172.16.0.0/12,192.168.0.0/16,224.0.0.0/4,240.0.0.0/4 via ${oif} > > # Allow connections from the internal network to be NAT'ed > $fwcmd add set 1 divert natd all from any to me in recv ${oif} > $fwcmd add set 1 divert natd all from not me to any recv ${iif} out xmit ${oif} > > # Finish the last of the basic anti-spoofing now that packets have been NAT'ed > $fwcmd add set 1 deny log all from 0.0.0.0/8,10.0.0.0/8,169.254.0.0/16,192.0.2.0/24,172.16.0.0/12,192.168.0.0/16,224.0.0.0/4,240.0.0.0/4 to any via ${oif} > > # Allow all connections that have dynamic rules built for them, > # but deny established connections that don't have a dynamic rule. > $fwcmd add set 1 check-state > $fwcmd add set 1 deny log tcp from any to any established > $fwcmd add set 1 deny log ip from any to any in frag > $fwcmd add set 1 deny log ip from any to any not verrevpath in > > # Allow all localhost connections > $fwcmd add set 1 allow tcp from me to any out via lo0 setup keep-state > $fwcmd add set 1 deny log tcp from me to any out via lo0 > $fwcmd add set 1 allow ip from me to any out via lo0 keep-state > > # Allow all connections from the internal network > $fwcmd add set 1 allow tcp from 192.168.1.0/24 to any in via ${iif} setup keep-state > $fwcmd add set 1 deny log tcp from 192.168.1.0/24 to any > $fwcmd add set 1 allow ip from 192.168.1.0/24 to any in via ${iif} keep-state > $fwcmd add set 1 allow ip from any to any in via ${iif} keep-state > > # Allow all connections from my network card that I initiate > $fwcmd add set 1 allow tcp from me to any out xmit any setup keep-state > $fwcmd add set 1 deny log tcp from me to any > $fwcmd add set 1 allow ip from me to any out xmit any keep-state > > # Enable ICMP to work since it is not a stateful protocol > $fwcmd add set 1 allow icmp from any to any icmptypes 0,3,8,11,12,13,14 > > # This sends a RESET to all ident packets. > $fwcmd add set 1 reset log tcp from any to me 113 in recv any > > # Deny all the rest. > $fwcmd add set 1 deny log ip from any to any > ##### END SCRIPT > > Packets can travel across the firewall correctly, however the dynamic > rule for TCP connections made from the Intranet to the Internet, > aren't correct. For example: > > laptop% ssh host.example.com 'sleep 30 && exit' & > # sudo ipfw -d show > [snip] > 01800 14 2929 (13s) STATE tcp 64.x.y.z 49678 <-> 69.u.v.w 22 > > The src address has been translated correctly and I can use SSH, > however, because the connection hasn't been promoted to a fully > established state, if I stop typing and let the connection idle for > more than 20s, the dynamic rule cleans up and I have to re-establish a > connection. If I hop onto the gateway directly and issue the same ssh > command, things work correctly: > > fw% ssh host.example.com 'sleep 30 && exit' & > # sudo ipfw -d show > [snip] > 01800 30 4605 (297s) STATE tcp 64.x.y.z 49506 <-> 69.u.v.w 22 > > From what I can tell, the check-state rule is failing to match rules > that have been divert'ed/nat'ed, which seems exceedingly broken given > the only thing that the check-state rule needs to do is match the src > ip, src port, dst ip, and dst port. In case someone asks, yes, the > connection is setup correctly according to tcpdump: > > # tcpdump -n -i sis1 tcp > tcpdump: listening on sis1 > 13:27:43.955298 64.x.y.z.49680 > 69.u.v.w.22: S 311390851:311390851(0) win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 4601787 0> (DF) > 13:27:43.995120 69.u.v.w.22 > 64.x.y.z.49680: S 73338311:73338311(0) ack 311390852 win 32768 <mss 1460> (DF) > 13:27:43.997839 64.x.y.z.49680 > 69.u.v.w.22: . ack 1 win 65535 (DF) > 13:27:44.043933 69.u.v.w.22 > 64.x.y.z.49680: P 1:41(40) ack 1 win 33580 (DF) > 13:27:44.047096 64.x.y.z.49680 > 69.u.v.w.22: P 1:42(41) ack 41 win 65535 (DF) > 13:27:44.106271 69.u.v.w.22 > 64.x.y.z.49680: P 41:577(536) ack 42 win 33580 (DF) > 13:27:44.111365 64.x.y.z.49680 > 69.u.v.w.22: P 42:586(544) ack 577 win 65164 (DF) > 13:27:44.288589 69.u.v.w.22 > 64.x.y.z.49680: . ack 586 win 33580 (DF) > 13:27:44.291236 64.x.y.z.49680 > 69.u.v.w.22: P 586:610(24) ack 577 win 65535 (DF) > 13:27:44.346961 69.u.v.w.22 > 64.x.y.z.49680: P 577:857(280) ack 610 win 33580 (DF) > 13:27:44.404742 64.x.y.z.49680 > 69.u.v.w.22: P 610:882(272) ack 857 win 65535 (DF) > 13:27:44.519718 69.u.v.w.22 > 64.x.y.z.49680: P 857:1641(784) ack 882 win 33580 (DF) > 13:27:44.606056 64.x.y.z.49680 > 69.u.v.w.22: P 882:898(16) ack 1641 win 65535 (DF) > 13:27:44.737744 69.u.v.w.22 > 64.x.y.z.49680: . ack 898 win 33580 (DF) > [snip] > 13:27:45.120273 64.x.y.z.49680 > 69.u.v.w.22: P 2178:2290(112) ack 2329 win 65535 (DF) > 13:27:45.120804 64.x.y.z.49680 > 69.u.v.w.22: P 2290:2354(64) ack 2329 win 65535 (DF) [tos 0x10] > 13:27:45.178022 69.u.v.w.22 > 64.x.y.z.49680: . ack 2354 win 33516 (DF) > 13:27:45.181483 69.u.v.w.22 > 64.x.y.z.49680: P 2329:2377(48) ack 2354 win 33580 (DF) > 13:27:45.278207 64.x.y.z.49680 > 69.u.v.w.22: . ack 2377 win 65535 (DF) [tos 0x10] > 13:27:47.651804 64.x.y.z.49680 > 69.u.v.w.22: F 2354:2354(0) ack 2377 win 65535 (DF) [tos 0x10] > 13:27:47.691808 69.u.v.w.22 > 64.x.y.z.49680: . ack 2355 win 33580 (DF) > 13:27:47.694033 69.u.v.w.22 > 64.x.y.z.49680: F 2377:2377(0) ack 2355 win 33580 (DF) > 13:27:47.696684 64.x.y.z.49680 > 69.u.v.w.22: . ack 2378 win 65534 [tos 0x10] > 13:27:47.728191 64.x.y.z.49680 > 69.u.v.w.22: . ack 2377 win 0 > 13:27:47.729124 64.x.y.z.49680 > 69.u.v.w.22: . ack 2378 win 65534 [tos 0x10] > 13:27:47.768700 69.u.v.w.22 > 64.x.y.z.49680: R 73340688:73340688(0) win 0 > 13:27:47.775331 69.u.v.w.22 > 64.x.y.z.49680: R 73340689:73340689(0) win 0 > > > I've stuck 'count log // test' rules in and have verified that > everything is correct coming across the wire/kernel, however ipfw2's > just not detecting that the connection has been fully established. If > anyone would like, please feel free to use the above rules and test > them out, they should be quite good and a fair bit more secure than > what's suggested currently (the above works for surfing the web/pop3, > but is infuriating if you use ssh), though they're not quite working > right yet. > > For a while I thought it had to do with net.inet.ip.fw.one_pass, but > toggling that has no bearing on the outcome at all. > Comments/suggestions? -sc > > -- > Sean Chittenden
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030724155317.A35706>