From owner-freebsd-ipfw@freebsd.org Thu Aug 20 15:35:06 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8AF809BFFFB for ; Thu, 20 Aug 2015 15:35:06 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8DA9416D8 for ; Thu, 20 Aug 2015 15:35:04 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id t7KFZ1ic063337; Fri, 21 Aug 2015 01:35:01 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 21 Aug 2015 01:35:00 +1000 (EST) From: Ian Smith To: andreas scherrer cc: freebsd-ipfw@freebsd.org Subject: Re: ipfw's "via" rule option/match pattern Message-ID: <20150821013137.E8515@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Aug 2015 15:35:06 -0000 Argh, fluffed freebsd-ipfw@ address, checked everything else :( ---------- Forwarded message ---------- Date: Fri, 21 Aug 2015 01:14:58 +1000 (EST) From: Ian Smith To: andreas scherrer Cc: freebsd-questions@freebsd.org, freeb-ipfw@freebsd.org Subject: Re: ipfw's "via" rule option/match pattern In freebsd-questions Digest, Vol 585, Issue 3, Message: 9 On Wed, 19 Aug 2015 00:41:35 +0200 andreas scherrer wrote: > Dear all > > I am struggling to understand ifpw's "via" rule option. I am on FreeBSD > 10.1-release-p16 > > I have read ipfw(8) [1] many times by now but I still find it confusing: > > "The via keyword causes the interface to always be checked. If recv or > xmit is used instead of via, then only the receive or transmit interface > (respectively) is checked." > > What does "the interface is checked" mean in this context? And what is a > common use case for the "via" rule option as opposed to recv/xmit? "in > via" seems to be the same as "in recv" anyway (because "in xmit" is > invalid) if I read "via" as "recv or xmit"... The interface is checked to match the one (or ones) specified. in recv is indeed the same as in via, because as received from ip(6)_input, only the receive interface is known .. the transmit interface is only known on outgoing packets after routing has chosen it, before ip(6)_output (or ether_demux and ether_output_frame respectively if matching at layer 2 - see ipfw(8) PACKET FLOW) > I have also read (and I believe understood) [2], [3] and [4]. But it > somehow contradicts what I see on my system. Or at least I think so. Yes, I believe you're right .. and [2] and [4] were written by me. I had a look at the code tonight (duh!) and suspect I've been at least partially wrong all the years I've been posting about this :( > Mainly they all state that something like > > ... out via ifX > > matches packets that are *going through* a box (read: are forwarded) if > they came in on interface ifX and are leaving the box via ifY. These packets need not be being forwarded .. they might originate on this box, and then they have no recieve interface, ie never came 'in'. > That would be because the packet's receive interface is checked when it > leaves the box and because it entered the box on ifX the rule matches > (even if the packet leaves the box on ifY). > > [4]: "Again, 'out via $wan_if' is ambiguous, and includes packets > _received_ on $wan_if and now being transmitted to the inside" > > When I run a quick test, sending one ICMP echo request from > 192.168.32.10 to 192.168.38.17 (two devices communicating via the box > that has the "count" rules listed below configured), I get the following > result: A good set of tests for all combinations. Something else I saw recently made me doubt that my own understanding of this was correct, and your tests seem to confirm that I've been misadvising people for, oh, the best part of 10 years .. here's the code, which I've checked hasn't functionally changed at all since 2012, and little from 2002 with Luigi's first ip_fw2.c (tabs lost): case O_RECV: match = iface_match(m->m_pkthdr.rcvif, (ipfw_insn_if *)cmd, chain, &tablearg); break; case O_XMIT: match = iface_match(oif, (ipfw_insn_if *)cmd, chain, &tablearg); break; case O_VIA: match = iface_match(oif ? oif : m->m_pkthdr.rcvif, (ipfw_insn_if *)cmd, chain, &tablearg); break; iface_match() (qv) does the test vs iface name or IP address, returning 1 on a match, but begins by returning 0 if the passed interface is NULL. In the case of O_VIA, if the outside iface is specified then that iface (only) is tested; the rcvif is only checked if there's NO out iface. This directly contradicts what I've been telling myself and others for years :( I guess what's amazing is that nobody who'd know better ever pulled me up on such statements, increasing confidence in wrongness :) > ----- > When 192.168.38.17 does not answer the ping: > 00350 2 168 count icmp from 192.168.32.10 to 192.168.38.17 recv re0.32 > 00350 0 0 count icmp from 192.168.38.17 to 192.168.32.10 recv re0.38 > 00351 1 84 count icmp from 192.168.32.10 to 192.168.38.17 in recv re0.32 > 00351 0 0 count icmp from 192.168.38.17 to 192.168.32.10 in recv re0.38 > 00352 1 84 count icmp from 192.168.32.10 to 192.168.38.17 out recv re0.32 > 00352 0 0 count icmp from 192.168.38.17 to 192.168.32.10 out recv re0.38 > 00355 1 84 count icmp from 192.168.32.10 to 192.168.38.17 via re0.32 > 00355 0 0 count icmp from 192.168.38.17 to 192.168.32.10 via re0.38 > 00356 1 84 count icmp from 192.168.32.10 to 192.168.38.17 in via re0.32 > 00356 0 0 count icmp from 192.168.38.17 to 192.168.32.10 in via re0.38 > 00357 0 0 count icmp from 192.168.32.10 to 192.168.38.17 out via re0.32 > 00357 0 0 count icmp from 192.168.38.17 to 192.168.32.10 out via re0.38 > 00358 1 84 count icmp from 192.168.32.10 to 192.168.38.17 out recv re0.32 xmit re0.38 > 00358 0 0 count icmp from 192.168.38.17 to 192.168.32.10 out recv re0.38 xmit re0.32 > > When 192.168.38.17 does answer the ping: > 00350 2 168 count icmp from 192.168.32.10 to 192.168.38.17 recv re0.32 > 00350 2 168 count icmp from 192.168.38.17 to 192.168.32.10 recv re0.38 > 00351 1 84 count icmp from 192.168.32.10 to 192.168.38.17 in recv re0.32 > 00351 1 84 count icmp from 192.168.38.17 to 192.168.32.10 in recv re0.38 > 00352 1 84 count icmp from 192.168.32.10 to 192.168.38.17 out recv re0.32 > 00352 1 84 count icmp from 192.168.38.17 to 192.168.32.10 out recv re0.38 > 00355 1 84 count icmp from 192.168.32.10 to 192.168.38.17 via re0.32 > 00355 1 84 count icmp from 192.168.38.17 to 192.168.32.10 via re0.38 > 00356 1 84 count icmp from 192.168.32.10 to 192.168.38.17 in via re0.32 > 00356 1 84 count icmp from 192.168.38.17 to 192.168.32.10 in via re0.38 > 00357 0 0 count icmp from 192.168.32.10 to 192.168.38.17 out via re0.32 > 00357 0 0 count icmp from 192.168.38.17 to 192.168.32.10 out via re0.38 > 00358 1 84 count icmp from 192.168.32.10 to 192.168.38.17 out recv re0.32 xmit re0.38 > 00358 1 84 count icmp from 192.168.38.17 to 192.168.32.10 out recv re0.38 xmit re0.32 > ----- > > According to the statement in [4] I would expect rule 357 to match... Yes; [4] is clearly wrong in this respect. 'out via' does NOT check the receive interface if the transmit interface is known. > Can anyone shed some light on that topic for me? I have a feeling that > my understanding that "via" is "recv or xmit" is not correct. > > Thanks in advance > andreas > > [1] https://www.freebsd.org/cgi/man.cgi?query=ipfw(8) > [2] > https://lists.freebsd.org/pipermail/freebsd-questions/2005-July/094739.html > [3] > https://groups.google.com/forum/#!topic/comp.unix.bsd.freebsd.misc/AkDSKlUmVok > [4] > https://lists.freebsd.org/pipermail/freebsd-questions/2011-June/231271.html Posting this also to freebsd-ipfw@ for comments, clarifications, or plain old getting beaten up. Perhaps ipfw(8) could benefit other users by clarifying in particular this 'out via' behaviour. Please cc me on responses in -questions@, I'm subscribed to the digest. cheers, Ian From owner-freebsd-ipfw@freebsd.org Sat Aug 22 03:28:45 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10FC19BF1F1 for ; Sat, 22 Aug 2015 03:28:45 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) by mx1.freebsd.org (Postfix) with ESMTP id 6AEC6125E for ; Sat, 22 Aug 2015 03:28:43 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id E4A722C18CA for ; Sat, 22 Aug 2015 13:19:15 +1000 (EST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id et6KSIv2C2_q for ; Sat, 22 Aug 2015 13:19:15 +1000 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id AF9582C18DF for ; Sat, 22 Aug 2015 13:19:15 +1000 (EST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id NHw6QZjkXxhi for ; Sat, 22 Aug 2015 13:19:15 +1000 (EST) Received: from [192.168.1.8] (220-244-73-44.tpgi.com.au [220.244.73.44]) by mail.7sq.com.au (Postfix) with ESMTPSA id 6C47A2C18CA for ; Sat, 22 Aug 2015 13:19:15 +1000 (EST) From: Nathan Aherne Subject: IPFW and in kernel nat Message-Id: Date: Sat, 22 Aug 2015 13:19:37 +1000 To: freebsd-ipfw@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) X-Mailer: Apple Mail (2.2102) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Aug 2015 03:28:45 -0000 Hi Everyone, First time message to this list, so I am sorry if I do something against = the rules. I have posted this questions on the FreeBSD forums in two different = places but have not had a single response in several days, so thought I = might get more success here. I have spent many days solely reading about = IPFW and in kernel nat in the hopes that I would be able to get what I = want to achieve working without having to ask for help. I have found it = extremely hard to find what I would think is a regular use use case for = IPFW and nat. There are examples but there are just so many that are bad = or hard to follow, including the handbook and IPFW that its extremely = easy to get confused, as I have. Because of this, I will be posting my = eventual solution back to the forums in the hopes that it stops someone = from wasting days. I have a single wan interface (bce0) that has two public IPs attached to = it. I would like to use one of these IPs for the host (wanip1) and jails = and another for a jail (wanip2) that requires a public IP. I have a = cloned lo0 interface, lo1 with a subnet of 10.1.0.0/24 which all the = jails reside on. I would like to forward ports 80,443 coming in on = wanip1 to the jail proxy, which then forwards the traffic off to the = correct jail. I would like for the host server to be able to have its = own firewall and also receive traffic on port 65222. I have a couple of = other jails that require some other ports as you will see in my = ipfw.rules script. I would also like each jail to be able to connect to = the internet. I would like the jails to be able to have their own firewalls, which I = currently allow by including a firewall script from within each jail. I have net.inet.ip.fw.one_pass=3D0 set in /etc/sysctl.conf Below is my ipfw.rules script. I would super appreciate it if someone = could show me where I have gone wrong. = **************************************************************************= **************************************************************************= ************************* #!/bin/sh ###################################################### # Configuration wif=3D"bce0" # WAN interface wip=3D"119.111.111.111" # WAN IP j1if=3D"lo1" # Jails Interface j1net=3D"10.1.0.0/24" # Jails Network skip=3D"skipto 30000" jcmd=3D"ipfw -q add 10000=E2=80=9D # Jails rules are inserted here ###################################################### # IPFW variables cmd=3D"ipfw -q add" ks=3D"keep-state" sks=3D"setup keep-state" ###################################################### ipfw -q -f flush # Flush all rules ###################################################### # NAT on Jail1 WAN IP ipfw nat 1 config ip $wip same_ports unreg_only reset \ redirect_port tcp 10.1.0.1:80 80 \ redirect_port tcp 10.1.0.1:443 443 \ redirect_port tcp 10.1.0.2:65432 65432 \ redirect_port tcp 10.1.0.3:65444 65444 \ redirect_port tcp 10.1.0.3:65333 65333 # Jail1 Network - allow all traffic $cmd 10 allow ip from any to any via $j1if ###################################################### # Allow all traffic on Loopback $cmd 999 allow ip from any to any via lo0 # NAT Rule for incoming packets on WAN IP $cmd 1000 nat 1 ip4 from any to any in via $wif # Check stage table $cmd 2000 check-state ###################################################### # HOST ONLY # Ping $cmd 2100 allow icmp from $wip to any out $ks $cmd 2101 allow icmp from any to $wip in $ks # DNS $cmd 2102 allow tcp from $wip to any 53 out $sks $cmd 2103 allow udp from $wip to any 53 out $ks # Ports $cmd 2104 allow tcp from $wip to any 80 out $ks $cmd 2105 allow tcp from $wip to any 433 out $ks # SSH $cmd 2106 allow tcp from $wip to any 22 out $ks $cmd 2107 allow tcp from $wip to any 65222 out $ks $cmd 2108 allow tcp from any to $wip 65222 in $ks # OpenNTP $cmd 2109 allow udp from $wip to any 123 out $ks ###################################################### # Jails =20 # Out $cmd 3004 $skip ip4 from any to any out xmit $wif $ks # In $cmd 3000 $skip tcp from any to any dst-port 80 in via $wif $sks $cmd 3001 $skip tcp from any to any dst-port 443 in via $wif $sks $cmd 3002 $skip tcp from any to any dst-port 65444 in via $wif $sks $cmd 3003 $skip tcp from any to any dst-port 65432 in via $wif $sks $cmd 3004 $skip tcp from any to any dst-port 65333 in via $wif $sks ###################################################### # jail.example.com . /usr/jails/jail.example.com/ipfw.rules $cmd 30000 nat 1 ip from $jnet to any out ###################################################### # Deny Remainder and Log $cmd deny log all from any to any ###################################################### = **************************************************************************= **************************************************************************= ************************* Regards, Nathan From owner-freebsd-ipfw@freebsd.org Sat Aug 22 04:19:52 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AEE6E9BF99C for ; Sat, 22 Aug 2015 04:19:52 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from frv189.fwdcdn.com (frv189.fwdcdn.com [212.42.77.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4D7EEB61 for ; Sat, 22 Aug 2015 04:19:51 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from [10.10.1.28] (helo=frv196.fwdcdn.com) by frv189.fwdcdn.com with esmtp ID 1ZT028-0007eM-TX for freebsd-ipfw@freebsd.org; Sat, 22 Aug 2015 07:03:56 +0300 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Type:MIME-Version:References:In-Reply-To:Message-Id:Cc:To:Subject:From:Date; bh=CV2dDiK4m2MORq9k3dHvXxUn99MBnT9ynnswjlVrUzU=; b=Sa1HykcpnPNPPuXr+qBpOKwjOW3AHzmaECMuhNjc5dOYZ7ocSQdURuK8KpPEvGbQklXlD8T15xRhOfuJaxKdC0TJ2c0IV7DOxxGq9poRGEpcT8zJ3AFsjVHBym2REx1i/lgkgtf17DsHsq0eGw5XykLnziShSv+3z9rJYpFC7+4=; Received: from [10.10.10.34] (helo=frv34.fwdcdn.com) by frv196.fwdcdn.com with smtp ID 1ZT020-000OpY-L6 for freebsd-ipfw@freebsd.org; Sat, 22 Aug 2015 07:03:48 +0300 Date: Sat, 22 Aug 2015 07:03:48 +0300 From: wishmaster Subject: Re: IPFW and in kernel nat To: Nathan Aherne Cc: freebsd-ipfw@freebsd.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1440215094.231702740.aw1gnxg5@frv34.fwdcdn.com> In-Reply-To: References: X-Reply-Action: reply Received: from artemrts@ukr.net by frv34.fwdcdn.com; Sat, 22 Aug 2015 07:03:48 +0300 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Aug 2015 04:19:52 -0000 Hi,  --- Original message --- From: "Nathan Aherne" Date: 22 August 2015, 06:28:51 Hi Everyone, First time message to this list, so I am sorry if I do something against the rules. I have posted this questions on the FreeBSD forums in two different places but have not had a single response in several days, so thought I might get more success here. I have spent many days solely reading about IPFW and in kernel nat in the hopes that I would be able to get what I want to achieve working without having to ask for help. I have found it extremely hard to find what I would think is a regular use use case for IPFW and nat. There are examples but there are just so many that are bad or hard to follow, including the handbook and IPFW that its extremely easy to get confused, as I have. Because of this, I will be posting my eventual solution back to the forums in the hopes that it stops someone from wasting days. I have a single wan interface (bce0) that has two public IPs attached to it. I would like to use one of these IPs for the host (wanip1) and jails and another for a jail (wanip2) that requires a public IP. I have a cloned lo0 interface, lo1 with a subnet of 10.1.0.0/24 which all the jails reside on. I would like to forward ports 80,443 coming in on wanip1 to the jail proxy, which then forwards the traffic off to the correct jail. I would like for the host server to be able to have its own firewall and also receive traffic on port 65222. I have a couple of other jails that require some other ports as you will see in my ipfw.rules script. I would also like each jail to be able to connect to the internet. I would like the jails to be able to have their own firewalls, which I currently allow by including a firewall script from within each jail. I have net.inet.ip.fw.one_pass=0 set in /etc/sysctl.conf Below is my ipfw.rules script. I would super appreciate it if someone could show me where I have gone wrong. ***************************************************************************************************************************************************************************** #!/bin/sh ###################################################### # Configuration wif="bce0" # WAN interface wip="119.111.111.111" # WAN IP j1if="lo1" # Jails Interface j1net="10.1.0.0/24" # Jails Network skip="skipto 30000" jcmd="ipfw -q add 10000” # Jails rules are inserted here ###################################################### # IPFW variables cmd="ipfw -q add" ks="keep-state" sks="setup keep-state" ###################################################### ipfw -q -f flush # Flush all rules ###################################################### # NAT on Jail1 WAN IP ipfw nat 1 config ip $wip same_ports unreg_only reset \ redirect_port tcp 10.1.0.1:80 80 \ redirect_port tcp 10.1.0.1:443 443 \ redirect_port tcp 10.1.0.2:65432 65432 \ redirect_port tcp 10.1.0.3:65444 65444 \ redirect_port tcp 10.1.0.3:65333 65333 # Jail1 Network - allow all traffic $cmd 10 allow ip from any to any via $j1if ###################################################### # Allow all traffic on Loopback $cmd 999 allow ip from any to any via lo0 # NAT Rule for incoming packets on WAN IP $cmd 1000 nat 1 ip4 from any to any in via $wif # Check stage table $cmd 2000 check-state ###################################################### # HOST ONLY # Ping $cmd 2100 allow icmp from $wip to any out $ks $cmd 2101 allow icmp from any to $wip in $ks # DNS $cmd 2102 allow tcp from $wip to any 53 out $sks $cmd 2103 allow udp from $wip to any 53 out $ks # Ports $cmd 2104 allow tcp from $wip to any 80 out $ks $cmd 2105 allow tcp from $wip to any 433 out $ks # SSH $cmd 2106 allow tcp from $wip to any 22 out $ks $cmd 2107 allow tcp from $wip to any 65222 out $ks $cmd 2108 allow tcp from any to $wip 65222 in $ks # OpenNTP $cmd 2109 allow udp from $wip to any 123 out $ks ###################################################### # Jails # Out $cmd 3004 $skip ip4 from any to any out xmit $wif $ks # In $cmd 3000 $skip tcp from any to any dst-port 80 in via $wif $sks $cmd 3001 $skip tcp from any to any dst-port 443 in via $wif $sks $cmd 3002 $skip tcp from any to any dst-port 65444 in via $wif $sks $cmd 3003 $skip tcp from any to any dst-port 65432 in via $wif $sks $cmd 3004 $skip tcp from any to any dst-port 65333 in via $wif $sks ###################################################### # jail.example.com . /usr/jails/jail.example.com/ipfw.rules $cmd 30000 nat 1 ip from $jnet to any out ###################################################### # Deny Remainder and Log $cmd deny log all from any to any ##################################################### As for me, ipfw is quite complex to configure in not trivial configurations where there are many interfaces. So my advise, you should use per-interface ACL and VIMAGE for jails for better traffic control. Small example below from the real machine. # tables with interfaces, in ipfw -fq table 10 flush ipfw table 10 add nfe0 3000 ipfw table 10 add ue0 3500 ipfw table 10 add lo0 4000 ipfw table 10 add sk0 5000 ipfw table 10 add epair1a 6000 ipfw table 10 add epair2a 6000 ipfw table 10 add epair3a 6000 ipfw table 10 add epair4a 6000 ipfw table 10 add epair999a 6000 ipfw table 10 add tun1 6100 # out ipfw -fq table 11 flush ipfw table 11 add nfe0 13000 ipfw table 11 add ue0 13500 ipfw table 11 add lo0 14000 ipfw table 11 add sk0 15000 ipfw table 11 add epair1a 16000 ipfw table 11 add epair2a 16000 ipfw table 11 add epair3a 16000 ipfw table 11 add epair4a 16000 ipfw table 11 add epair999a 16000 ipfw table 11 add tun1 16100 #nat rules ..... $cmd 100 skipto tablearg log all from any to any in recv "table(10)" $cmd 110 skipto tablearg log all from any to any out xmit "table(11)" #rules for interfaces # rl0 $cmd 1000 allow log ip4 from any to any $cmd 1099 deny log all from any to any $cmd 11000 allow log ip4 from any to any $cmd 11099 deny log all from any to any # epair[0-9]a in  (from jail) $cmd 6000 nat 2 log ip4 from "table(12)" 80,81,443,5555 to me $cmd 6001 allow log ip4 from "table(12)" 80,81,443,5555 to $nonroute 1025-65535 $cmd 6003 fwd 127.0.0.1,25 log ip4 from any to me 25 $cmd 6010 check-state log $cmd 6020 allow log ip4 from 192.168.254.10 to not $nonroute $ks $cmd 6030 allow log ip4 from 192.168.254.2,192.168.254.254 to not $nonroute $ks    # allows requests to wild world from basejail & j1 only! $cmd 6099 deny log all from any to any # epair[0-9]a out    (to jail) $cmd 16000 nat 2 log ip4 from $nonroute 1025-65535 to "table(12)" 80,81,443,5555 $cmd 16010 allow log ip4 from me to "table(12)" $ks   # for www redirect ext_if -> jail_if and for $lan -> www jail $cmd 16090 check-state log $cmd 16099 deny log all from any to any ### IN ext_if sk0 $cmd 5000 call 30000 log all from any to any $cmd 5010 nat 1 log ip4 from any to me in $cmd 5011 call 25000 log all from any to any $cmd 5020 check-state log $cmd 5100 allow log tcp from any to me 10001 setup limit src-addr 5 $cmd 5122 allow log tcp from any to me 25 setup limit src-addr 100 $cmd 5127 allow log tcp from any to me 2112 setup limit src-addr 10 $cmd 5128 allow log tcp from any to me 49152-65535 limit src-addr 10 $cmd 5150 allow log udp from any to me 11944 limit src-addr 5 $cmd 5152 allow log ip4 from any to me 67,68 $cmd 5200 allow log tcp from any to "table(12)" 80,81,443,5555 setup limit src-addr 20 $cmd 5210 allow log tcp from any to 192.168.10.2 55551 setup limit src-addr 100                         # my torrent $cmd 5211 allow log udp from any to 192.168.10.2 55551 limit src-addr 100                       # my torrent $cmd 5215 allow log udp from any to 192.168.254.10 11945 limit src-addr 2 $cmd 5310 allow log icmp from any to any icmptypes 0,3,4,8,11 $cmd 5999 deny log all from any to any ... and so on ..... I massively use stateful feature and call subrequests. Hope this helps. Cheers, Vitaliy From owner-freebsd-ipfw@freebsd.org Sat Aug 22 04:25:11 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1795D9BFB05 for ; Sat, 22 Aug 2015 04:25:11 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) by mx1.freebsd.org (Postfix) with ESMTP id 72326F04 for ; Sat, 22 Aug 2015 04:25:09 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 9CEF12C186E for ; Sat, 22 Aug 2015 14:24:45 +1000 (EST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id 6C0Wum1jvm_l for ; Sat, 22 Aug 2015 14:24:45 +1000 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 630562C18CA for ; Sat, 22 Aug 2015 14:24:45 +1000 (EST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 1W2APkF94XLF for ; Sat, 22 Aug 2015 14:24:45 +1000 (EST) Received: from [172.20.10.2] (unknown [1.128.35.164]) by mail.7sq.com.au (Postfix) with ESMTPSA id E87582C186E for ; Sat, 22 Aug 2015 14:24:44 +1000 (EST) From: Nathan Aherne Message-Id: Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: IPFW and in kernel nat Date: Sat, 22 Aug 2015 14:25:07 +1000 In-Reply-To: <1440215094.231702740.aw1gnxg5@frv34.fwdcdn.com> Cc: freebsd-ipfw@freebsd.org References: <1440215094.231702740.aw1gnxg5@frv34.fwdcdn.com> X-Mailer: Apple Mail (2.2102) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Aug 2015 04:25:11 -0000 Hi Vitaliy, Thanks for sending this though, its much appreciated. I will take = onboard your recommendation about using vimage and have a bit more of a = read about it, in concert with the rules you have posted. At the moment, = I would like to get my server working without vimage with the view to = implementing it later. Regards, Nathan > On 22 Aug 2015, at 2:03 pm, wishmaster wrote: >=20 > Hi, =20 >=20 > --- Original message --- > From: "Nathan Aherne" > Date: 22 August 2015, 06:28:51 >=20 > Hi Everyone, >=20 > First time message to this list, so I am sorry if I do something = against the rules. >=20 > I have posted this questions on the FreeBSD forums in two different = places but have not had a single response in several days, so thought I = might get more success here. I have spent many days solely reading about = IPFW and in kernel nat in the hopes that I would be able to get what I = want to achieve working without having to ask for help. I have found it = extremely hard to find what I would think is a regular use use case for = IPFW and nat. There are examples but there are just so many that are bad = or hard to follow, including the handbook and IPFW that its extremely = easy to get confused, as I have. Because of this, I will be posting my = eventual solution back to the forums in the hopes that it stops someone = from wasting days. >=20 > I have a single wan interface (bce0) that has two public IPs attached = to it. I would like to use one of these IPs for the host (wanip1) and = jails and another for a jail (wanip2) that requires a public IP. I have = a cloned lo0 interface, lo1 with a subnet of 10.1.0.0/24 which all the = jails reside on. I would like to forward ports 80,443 coming in on = wanip1 to the jail proxy, which then forwards the traffic off to the = correct jail. I would like for the host server to be able to have its = own firewall and also receive traffic on port 65222. I have a couple of = other jails that require some other ports as you will see in my = ipfw.rules script. I would also like each jail to be able to connect to = the internet. >=20 > I would like the jails to be able to have their own firewalls, which I = currently allow by including a firewall script from within each jail. >=20 > I have net.inet.ip.fw.one_pass=3D0 set in /etc/sysctl.conf >=20 > Below is my ipfw.rules script. I would super appreciate it if someone = could show me where I have gone wrong. >=20 > = **************************************************************************= **************************************************************************= ************************* > #!/bin/sh >=20 > ###################################################### > # Configuration > wif=3D"bce0" # WAN interface > wip=3D"119.111.111.111" # WAN IP >=20 > j1if=3D"lo1" # Jails Interface > j1net=3D"10.1.0.0/24" # Jails Network > skip=3D"skipto 30000" > jcmd=3D"ipfw -q add 10000=E2=80=9D # Jails rules are inserted here > ###################################################### > # IPFW variables > cmd=3D"ipfw -q add" > ks=3D"keep-state" > sks=3D"setup keep-state" > ###################################################### > ipfw -q -f flush # Flush all rules > ###################################################### > # NAT on Jail1 WAN IP > ipfw nat 1 config ip $wip same_ports unreg_only reset \ > redirect_port tcp 10.1.0.1:80 80 \ > redirect_port tcp 10.1.0.1:443 443 \ > redirect_port tcp 10.1.0.2:65432 65432 \ > redirect_port tcp 10.1.0.3:65444 65444 \ > redirect_port tcp 10.1.0.3:65333 65333 >=20 > # Jail1 Network - allow all traffic > $cmd 10 allow ip from any to any via $j1if > ###################################################### > # Allow all traffic on Loopback > $cmd 999 allow ip from any to any via lo0 >=20 > # NAT Rule for incoming packets on WAN IP > $cmd 1000 nat 1 ip4 from any to any in via $wif >=20 > # Check stage table > $cmd 2000 check-state > ###################################################### > # HOST ONLY > # Ping > $cmd 2100 allow icmp from $wip to any out $ks > $cmd 2101 allow icmp from any to $wip in $ks > # DNS > $cmd 2102 allow tcp from $wip to any 53 out $sks > $cmd 2103 allow udp from $wip to any 53 out $ks > # Ports > $cmd 2104 allow tcp from $wip to any 80 out $ks > $cmd 2105 allow tcp from $wip to any 433 out $ks > # SSH > $cmd 2106 allow tcp from $wip to any 22 out $ks > $cmd 2107 allow tcp from $wip to any 65222 out $ks > $cmd 2108 allow tcp from any to $wip 65222 in $ks > # OpenNTP > $cmd 2109 allow udp from $wip to any 123 out $ks > ###################################################### > # Jails =20 > # Out > $cmd 3004 $skip ip4 from any to any out xmit $wif $ks > # In > $cmd 3000 $skip tcp from any to any dst-port 80 in via $wif $sks > $cmd 3001 $skip tcp from any to any dst-port 443 in via $wif $sks > $cmd 3002 $skip tcp from any to any dst-port 65444 in via $wif $sks > $cmd 3003 $skip tcp from any to any dst-port 65432 in via $wif $sks > $cmd 3004 $skip tcp from any to any dst-port 65333 in via $wif $sks > ###################################################### >=20 > # jail.example.com > . /usr/jails/jail.example.com/ipfw.rules >=20 > $cmd 30000 nat 1 ip from $jnet to any out >=20 > ###################################################### > # Deny Remainder and Log > $cmd deny log all from any to any > ##################################################### > As for me, ipfw is quite complex to configure in not trivial = configurations where there are many interfaces. So my advise, you should = use per-interface ACL and VIMAGE for jails for better traffic control. = Small example below from the real machine. >=20 > # tables with interfaces, in > ipfw -fq table 10 flush > ipfw table 10 add nfe0 3000 > ipfw table 10 add ue0 3500 > ipfw table 10 add lo0 4000 > ipfw table 10 add sk0 5000 > ipfw table 10 add epair1a 6000 > ipfw table 10 add epair2a 6000 > ipfw table 10 add epair3a 6000 > ipfw table 10 add epair4a 6000 > ipfw table 10 add epair999a 6000 > ipfw table 10 add tun1 6100 >=20 > # out > ipfw -fq table 11 flush > ipfw table 11 add nfe0 13000 > ipfw table 11 add ue0 13500 > ipfw table 11 add lo0 14000 > ipfw table 11 add sk0 15000 > ipfw table 11 add epair1a 16000 > ipfw table 11 add epair2a 16000 > ipfw table 11 add epair3a 16000 > ipfw table 11 add epair4a 16000 > ipfw table 11 add epair999a 16000 > ipfw table 11 add tun1 16100 >=20 > #nat rules ..... >=20 > $cmd 100 skipto tablearg log all from any to any in recv "table(10)" > $cmd 110 skipto tablearg log all from any to any out xmit "table(11)" >=20 > #rules for interfaces >=20 > # rl0 > $cmd 1000 allow log ip4 from any to any > $cmd 1099 deny log all from any to any >=20 > $cmd 11000 allow log ip4 from any to any > $cmd 11099 deny log all from any to any >=20 > # epair[0-9]a in (from jail) > $cmd 6000 nat 2 log ip4 from "table(12)" 80,81,443,5555 to me > $cmd 6001 allow log ip4 from "table(12)" 80,81,443,5555 to $nonroute = 1025-65535 > $cmd 6003 fwd 127.0.0.1,25 log ip4 from any to me 25 > $cmd 6010 check-state log >=20 > $cmd 6020 allow log ip4 from 192.168.254.10 to not $nonroute $ks > $cmd 6030 allow log ip4 from 192.168.254.2,192.168.254.254 to not = $nonroute $ks # allows requests to wild world from basejail & j1 = only! >=20 > $cmd 6099 deny log all from any to any >=20 > # epair[0-9]a out (to jail) > $cmd 16000 nat 2 log ip4 from $nonroute 1025-65535 to "table(12)" = 80,81,443,5555 > $cmd 16010 allow log ip4 from me to "table(12)" $ks # for www = redirect ext_if -> jail_if and for $lan -> www jail >=20 > $cmd 16090 check-state log >=20 > $cmd 16099 deny log all from any to any >=20 > ### IN ext_if sk0 > $cmd 5000 call 30000 log all from any to any >=20 > $cmd 5010 nat 1 log ip4 from any to me in > $cmd 5011 call 25000 log all from any to any >=20 > $cmd 5020 check-state log >=20 > $cmd 5100 allow log tcp from any to me 10001 setup limit src-addr 5 > $cmd 5122 allow log tcp from any to me 25 setup limit src-addr 100 > $cmd 5127 allow log tcp from any to me 2112 setup limit src-addr 10 > $cmd 5128 allow log tcp from any to me 49152-65535 limit src-addr 10 > $cmd 5150 allow log udp from any to me 11944 limit src-addr 5 > $cmd 5152 allow log ip4 from any to me 67,68 > $cmd 5200 allow log tcp from any to "table(12)" 80,81,443,5555 setup = limit src-addr 20 > $cmd 5210 allow log tcp from any to 192.168.10.2 55551 setup limit = src-addr 100 # my torrent > $cmd 5211 allow log udp from any to 192.168.10.2 55551 limit src-addr = 100 # my torrent > $cmd 5215 allow log udp from any to 192.168.254.10 11945 limit = src-addr 2 >=20 > $cmd 5310 allow log icmp from any to any icmptypes 0,3,4,8,11 >=20 > $cmd 5999 deny log all from any to any >=20 > ... > and so on > ..... >=20 >=20 > I massively use stateful feature and call subrequests. Hope this = helps. >=20 > Cheers, > Vitaliy >=20 >=20 >=20