Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 22 Aug 2015 14:25:07 +1000
From:      Nathan Aherne <nathan@reddog.com.au>
Cc:        freebsd-ipfw@freebsd.org
Subject:   Re: IPFW and in kernel nat
Message-ID:  <C84335C7-3F27-42BC-A682-D1CB13321DC2@reddog.com.au>
In-Reply-To: <1440215094.231702740.aw1gnxg5@frv34.fwdcdn.com>
References:  <C513D7E5-D890-4230-A4EE-B94180444396@reddog.com.au> <1440215094.231702740.aw1gnxg5@frv34.fwdcdn.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Hi Vitaliy,

Thanks for sending this though, its much appreciated. I will take =
onboard your recommendation about using vimage and have a bit more of a =
read about it, in concert with the rules you have posted. At the moment, =
I would like to get my server working without vimage with the view to =
implementing it later.

Regards,

Nathan

> On 22 Aug 2015, at 2:03 pm, wishmaster <artemrts@ukr.net> wrote:
>=20
> Hi, =20
>=20
> --- Original message ---
> From: "Nathan Aherne" <nathan@reddog.com.au>
> Date: 22 August 2015, 06:28:51
>=20
> Hi Everyone,
>=20
> First time message to this list, so I am sorry if I do something =
against the rules.
>=20
> I have posted this questions on the FreeBSD forums in two different =
places but have not had a single response in several days, so thought I =
might get more success here. I have spent many days solely reading about =
IPFW and in kernel nat in the hopes that I would be able to get what I =
want to achieve working without having to ask for help. I have found it =
extremely hard to find what I would think is a regular use use case for =
IPFW and nat. There are examples but there are just so many that are bad =
or hard to follow, including the handbook and IPFW that its extremely =
easy to get confused, as I have. Because of this, I will be posting my =
eventual solution back to the forums in the hopes that it stops someone =
from wasting days.
>=20
> I have a single wan interface (bce0) that has two public IPs attached =
to it. I would like to use one of these IPs for the host (wanip1) and =
jails and another for a jail (wanip2) that requires a public IP. I have =
a cloned lo0 interface, lo1 with a subnet of 10.1.0.0/24 which all the =
jails reside on. I would like to forward ports 80,443 coming in on =
wanip1 to the jail proxy, which then forwards the traffic off to the =
correct jail. I would like for the host server to be able to have its =
own firewall and also receive traffic on port 65222. I have a couple of =
other jails that require some other ports as you will see in my =
ipfw.rules script. I would also like each jail to be able to connect to =
the internet.
>=20
> I would like the jails to be able to have their own firewalls, which I =
currently allow by including a firewall script from within each jail.
>=20
> I have net.inet.ip.fw.one_pass=3D0 set in /etc/sysctl.conf
>=20
> Below is my ipfw.rules script. I would super appreciate it if someone =
could show me where I have gone wrong.
>=20
> =
**************************************************************************=
**************************************************************************=
*************************
> #!/bin/sh
>=20
> ######################################################
> # Configuration
> wif=3D"bce0" # WAN interface
> wip=3D"119.111.111.111" # WAN IP
>=20
> j1if=3D"lo1" # Jails Interface
> j1net=3D"10.1.0.0/24" # Jails Network
> skip=3D"skipto 30000"
> jcmd=3D"ipfw -q add 10000=E2=80=9D # Jails rules are inserted here
> ######################################################
> # IPFW variables
> cmd=3D"ipfw -q add"
> ks=3D"keep-state"
> sks=3D"setup keep-state"
> ######################################################
> ipfw -q -f flush # Flush all rules
> ######################################################
> # NAT on Jail1 WAN IP
> ipfw nat 1 config ip $wip same_ports unreg_only reset \
>                         redirect_port tcp 10.1.0.1:80 80 \
>                         redirect_port tcp 10.1.0.1:443 443 \
>                         redirect_port tcp 10.1.0.2:65432 65432 \
>                         redirect_port tcp 10.1.0.3:65444 65444 \
>                         redirect_port tcp 10.1.0.3:65333 65333
>=20
> # Jail1 Network - allow all traffic
> $cmd 10 allow ip from any to any via $j1if
> ######################################################
> # Allow all traffic on Loopback
> $cmd 999 allow ip from any to any via lo0
>=20
> # NAT Rule for incoming packets on WAN IP
> $cmd 1000 nat 1 ip4 from any to any in via $wif
>=20
> # Check stage table
> $cmd 2000 check-state
> ######################################################
> # HOST ONLY
> # Ping
> $cmd 2100 allow icmp from $wip to any out $ks
> $cmd 2101 allow icmp from any to $wip in $ks
> # DNS
> $cmd 2102 allow tcp from $wip to any 53 out $sks
> $cmd 2103 allow udp from $wip to any 53 out $ks
> # Ports
> $cmd 2104 allow tcp from $wip to any 80 out $ks
> $cmd 2105 allow tcp from $wip to any 433 out $ks
> # SSH
> $cmd 2106 allow tcp from $wip to any 22 out $ks
> $cmd 2107 allow tcp from $wip to any 65222 out $ks
> $cmd 2108 allow tcp from any to $wip 65222 in $ks
> # OpenNTP
> $cmd 2109 allow udp from $wip to any 123 out $ks
> ######################################################
> # Jails  =20
> # Out
> $cmd 3004 $skip ip4 from any to any out xmit $wif $ks
> # In
> $cmd 3000 $skip tcp from any to any dst-port 80 in via $wif $sks
> $cmd 3001 $skip tcp from any to any dst-port 443 in via $wif $sks
> $cmd 3002 $skip tcp from any to any dst-port 65444 in via $wif $sks
> $cmd 3003 $skip tcp from any to any dst-port 65432 in via $wif $sks
> $cmd 3004 $skip tcp from any to any dst-port 65333 in via $wif $sks
> ######################################################
>=20
> # jail.example.com
> . /usr/jails/jail.example.com/ipfw.rules
>=20
> $cmd 30000 nat 1 ip from $jnet to any out
>=20
> ######################################################
> # Deny Remainder and Log
> $cmd deny log all from any to any
> #####################################################
> As for me, ipfw is quite complex to configure in not trivial =
configurations where there are many interfaces. So my advise, you should =
use per-interface ACL and VIMAGE for jails for better traffic control. =
Small example below from the real machine.
>=20
> # tables with interfaces, in
> ipfw -fq table 10 flush
> ipfw table 10 add nfe0 3000
> ipfw table 10 add ue0 3500
> ipfw table 10 add lo0 4000
> ipfw table 10 add sk0 5000
> ipfw table 10 add epair1a 6000
> ipfw table 10 add epair2a 6000
> ipfw table 10 add epair3a 6000
> ipfw table 10 add epair4a 6000
> ipfw table 10 add epair999a 6000
> ipfw table 10 add tun1 6100
>=20
> # out
> ipfw -fq table 11 flush
> ipfw table 11 add nfe0 13000
> ipfw table 11 add ue0 13500
> ipfw table 11 add lo0 14000
> ipfw table 11 add sk0 15000
> ipfw table 11 add epair1a 16000
> ipfw table 11 add epair2a 16000
> ipfw table 11 add epair3a 16000
> ipfw table 11 add epair4a 16000
> ipfw table 11 add epair999a 16000
> ipfw table 11 add tun1 16100
>=20
> #nat rules .....
>=20
> $cmd 100 skipto tablearg log all from any to any in recv "table(10)"
> $cmd 110 skipto tablearg log all from any to any out xmit "table(11)"
>=20
> #rules for interfaces
>=20
> # rl0
> $cmd 1000 allow log ip4 from any to any
> $cmd 1099 deny log all from any to any
>=20
> $cmd 11000 allow log ip4 from any to any
> $cmd 11099 deny log all from any to any
>=20
> # epair[0-9]a in  (from jail)
> $cmd 6000 nat 2 log ip4 from "table(12)" 80,81,443,5555 to me
> $cmd 6001 allow log ip4 from "table(12)" 80,81,443,5555 to $nonroute =
1025-65535
> $cmd 6003 fwd 127.0.0.1,25 log ip4 from any to me 25
> $cmd 6010 check-state log
>=20
> $cmd 6020 allow log ip4 from 192.168.254.10 to not $nonroute $ks
> $cmd 6030 allow log ip4 from 192.168.254.2,192.168.254.254 to not =
$nonroute $ks    # allows requests to wild world from basejail & j1 =
only!
>=20
> $cmd 6099 deny log all from any to any
>=20
> # epair[0-9]a out    (to jail)
> $cmd 16000 nat 2 log ip4 from $nonroute 1025-65535 to "table(12)" =
80,81,443,5555
> $cmd 16010 allow log ip4 from me to "table(12)" $ks   # for www =
redirect ext_if -> jail_if and for $lan -> www jail
>=20
> $cmd 16090 check-state log
>=20
> $cmd 16099 deny log all from any to any
>=20
> ### IN ext_if sk0
> $cmd 5000 call 30000 log all from any to any
>=20
> $cmd 5010 nat 1 log ip4 from any to me in
> $cmd 5011 call 25000 log all from any to any
>=20
> $cmd 5020 check-state log
>=20
> $cmd 5100 allow log tcp from any to me 10001 setup limit src-addr 5
> $cmd 5122 allow log tcp from any to me 25 setup limit src-addr 100
> $cmd 5127 allow log tcp from any to me 2112 setup limit src-addr 10
> $cmd 5128 allow log tcp from any to me 49152-65535 limit src-addr 10
> $cmd 5150 allow log udp from any to me 11944 limit src-addr 5
> $cmd 5152 allow log ip4 from any to me 67,68
> $cmd 5200 allow log tcp from any to "table(12)" 80,81,443,5555 setup =
limit src-addr 20
> $cmd 5210 allow log tcp from any to 192.168.10.2 55551 setup limit =
src-addr 100                         # my torrent
> $cmd 5211 allow log udp from any to 192.168.10.2 55551 limit src-addr =
100                       # my torrent
> $cmd 5215 allow log udp from any to 192.168.254.10 11945 limit =
src-addr 2
>=20
> $cmd 5310 allow log icmp from any to any icmptypes 0,3,4,8,11
>=20
> $cmd 5999 deny log all from any to any
>=20
> ...
> and so on
> .....
>=20
>=20
> I massively use stateful feature and call subrequests. Hope this =
helps.
>=20
> Cheers,
> Vitaliy
>=20
>=20
>=20




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?C84335C7-3F27-42BC-A682-D1CB13321DC2>