From owner-freebsd-ipfw@freebsd.org Sun Aug 30 02:59:24 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0AA69C5FF7 for ; Sun, 30 Aug 2015 02:59:24 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from mail.7sq.com.au (mail.7sq.com.au [119.148.74.199]) by mx1.freebsd.org (Postfix) with ESMTP id E02C2E43 for ; Sun, 30 Aug 2015 02:59:22 +0000 (UTC) (envelope-from nathan@reddog.com.au) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 382EF2C0008 for ; Sun, 30 Aug 2015 12:48:45 +1000 (EST) Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id iGz_6V4fURI1 for ; Sun, 30 Aug 2015 12:48:45 +1000 (EST) Received: from localhost (localhost [127.0.0.1]) by mail.7sq.com.au (Postfix) with ESMTP id 04D542C18EC for ; Sun, 30 Aug 2015 12:48:45 +1000 (EST) X-Virus-Scanned: amavisd-new at mail.7sq.com.au Received: from mail.7sq.com.au ([127.0.0.1]) by localhost (mail.7sq.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 6iuCnOqiS-Xb for ; Sun, 30 Aug 2015 12:48:44 +1000 (EST) Received: from [172.20.10.2] (unknown [1.129.28.125]) by mail.7sq.com.au (Postfix) with ESMTPSA id 247142C0008 for ; Sun, 30 Aug 2015 12:48:44 +1000 (EST) From: Nathan Aherne Message-Id: Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: IPFW and in kernel nat Date: Sun, 30 Aug 2015 12:49:32 +1000 References: To: freebsd-ipfw@freebsd.org In-Reply-To: X-Mailer: Apple Mail (2.2102) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 30 Aug 2015 02:59:24 -0000 I have done some further work on this and I have been able to get kernel = nat working in ipfw but some functionality I am after didn=E2=80=99t = work. As I said in the below email I have a couple of jails with public ips, = those jails are unable to communicate with the jails on the private = network behind nat. The traffic seems to go via the loopback interface, = which is then blocked because its not going through nat. I have also found that while jails can communicate with each other = (hairpin nat is working) some of the jails cannot. The jails that cannot = communicate with each other are the ones which have ports redirected to = them. For instance, I have a jail on 10.1.0.9 and it is unable to = communicate with the jail on 10.1.0.2 via port 65432. The traffic seems = to be going over the loopback interface and not via nat. Any traffic = between jails where there is no redirected ports (via nat) work fine. I = have no custom routes, only what the system sets up when adding ips via = /etc/rc.conf. It seems this may be a bug with ipfw and kernel nat? Regards, Nathan Aherne nathan@reddog.com.au +61 402 555 080 > On 22 Aug 2015, at 1:19 pm, Nathan Aherne = wrote: >=20 > Hi Everyone, >=20 > First time message to this list, so I am sorry if I do something = against the rules. >=20 > I have posted this questions on the FreeBSD forums in two different = places but have not had a single response in several days, so thought I = might get more success here. I have spent many days solely reading about = IPFW and in kernel nat in the hopes that I would be able to get what I = want to achieve working without having to ask for help. I have found it = extremely hard to find what I would think is a regular use use case for = IPFW and nat. There are examples but there are just so many that are bad = or hard to follow, including the handbook and IPFW that its extremely = easy to get confused, as I have. Because of this, I will be posting my = eventual solution back to the forums in the hopes that it stops someone = from wasting days. >=20 > I have a single wan interface (bce0) that has two public IPs attached = to it. I would like to use one of these IPs for the host (wanip1) and = jails and another for a jail (wanip2) that requires a public IP. I have = a cloned lo0 interface, lo1 with a subnet of 10.1.0.0/24 which all the = jails reside on. I would like to forward ports 80,443 coming in on = wanip1 to the jail proxy, which then forwards the traffic off to the = correct jail. I would like for the host server to be able to have its = own firewall and also receive traffic on port 65222. I have a couple of = other jails that require some other ports as you will see in my = ipfw.rules script. I would also like each jail to be able to connect to = the internet. >=20 > I would like the jails to be able to have their own firewalls, which I = currently allow by including a firewall script from within each jail. >=20 > I have net.inet.ip.fw.one_pass=3D0 set in /etc/sysctl.conf >=20 > Below is my ipfw.rules script. I would super appreciate it if someone = could show me where I have gone wrong. >=20 > = **************************************************************************= **************************************************************************= ************************* > #!/bin/sh >=20 > ###################################################### > # Configuration > wif=3D"bce0" # WAN interface > wip=3D"119.111.111.111" # WAN IP >=20 > j1if=3D"lo1" # Jails Interface > j1net=3D"10.1.0.0/24" # Jails Network > skip=3D"skipto 30000" > jcmd=3D"ipfw -q add 10000=E2=80=9D # Jails rules are inserted here > ###################################################### > # IPFW variables > cmd=3D"ipfw -q add" > ks=3D"keep-state" > sks=3D"setup keep-state" > ###################################################### > ipfw -q -f flush # Flush all rules > ###################################################### > # NAT on Jail1 WAN IP > ipfw nat 1 config ip $wip same_ports unreg_only reset \ > redirect_port tcp 10.1.0.1:80 80 \ > redirect_port tcp 10.1.0.1:443 443 \ > redirect_port tcp 10.1.0.2:65432 65432 \ > redirect_port tcp 10.1.0.3:65444 65444 \ > redirect_port tcp 10.1.0.3:65333 65333 >=20 > # Jail1 Network - allow all traffic > $cmd 10 allow ip from any to any via $j1if > ###################################################### > # Allow all traffic on Loopback > $cmd 999 allow ip from any to any via lo0 >=20 > # NAT Rule for incoming packets on WAN IP > $cmd 1000 nat 1 ip4 from any to any in via $wif >=20 > # Check stage table > $cmd 2000 check-state > ###################################################### > # HOST ONLY > # Ping > $cmd 2100 allow icmp from $wip to any out $ks > $cmd 2101 allow icmp from any to $wip in $ks > # DNS > $cmd 2102 allow tcp from $wip to any 53 out $sks > $cmd 2103 allow udp from $wip to any 53 out $ks > # Ports > $cmd 2104 allow tcp from $wip to any 80 out $ks > $cmd 2105 allow tcp from $wip to any 433 out $ks > # SSH > $cmd 2106 allow tcp from $wip to any 22 out $ks > $cmd 2107 allow tcp from $wip to any 65222 out $ks > $cmd 2108 allow tcp from any to $wip 65222 in $ks > # OpenNTP > $cmd 2109 allow udp from $wip to any 123 out $ks > ###################################################### > # Jails =20 > # Out > $cmd 3004 $skip ip4 from any to any out xmit $wif $ks > # In > $cmd 3000 $skip tcp from any to any dst-port 80 in via $wif $sks > $cmd 3001 $skip tcp from any to any dst-port 443 in via $wif $sks > $cmd 3002 $skip tcp from any to any dst-port 65444 in via $wif $sks > $cmd 3003 $skip tcp from any to any dst-port 65432 in via $wif $sks > $cmd 3004 $skip tcp from any to any dst-port 65333 in via $wif $sks > ###################################################### >=20 > # jail.example.com > . /usr/jails/jail.example.com/ipfw.rules = >=20 > $cmd 30000 nat 1 ip from $jnet to any out >=20 > ###################################################### > # Deny Remainder and Log > $cmd deny log all from any to any > ###################################################### >=20 > = **************************************************************************= **************************************************************************= ************************* >=20 > Regards, >=20 > Nathan >=20