Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 19 May 2015 00:41:22 -0700
From:      Wojciech Wojtyniak <wow@0x89.net>
To:        freebsd-net@freebsd.org
Subject:   Problem with ipfw, in-kernel NAT and port redirection to jails
Message-ID:  <A88A7FED-B5DD-4B1E-96A4-AE1F3EAB8E30@0x89.net>

next in thread | raw e-mail | index | archive | help
Hello,

I have a vps on vultr.com <http://vultr.com/>; running FreeBSD 10.1-p9 =
and a generic kernel:

% uname -a
FreeBSD tzar 10.1-RELEASE-p9 FreeBSD 10.1-RELEASE-p9 #0: Tue Apr  7 =
01:09:46 UTC 2015     =
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64

My goal is to run multiple services in jails (hopefully using ezjail or =
other convenient manager) and make them accessible from the Internet =
only on arbitrary ports (like 80 for http(s) server). So far my approach =
is as follows: I clone the lo0 interface and assign IP from 127.0.0.0/8 =
space to the jail and redirect port in ipfw nat definition to given =
address (example in configs below, I tried also with other addresses on =
vtnet0, which is my base network interface, with similar issues). =
Unfortunately this configuration doesn=E2=80=99t work for me.

I tested this for znc (IRC bouncer) and nginx. If I run them on main =
host (without NAT in front of them) everything works fine. However, if I =
run them in jail, behind NAT and send a HTTP(S) request to get some =
file, connections get dropped (znc has a web admin module, which is =
broken because of that). It works fine for small files but breaks for =
larger (I haven=E2=80=99t check the threshold but can do this if this is =
necessary). For example given curl command and znc service:

% curl http://$my_ip:6697/pub/jquery-1.11.2.min.js > /dev/null
# (stats cut out)
curl: (18) transfer closed with 58648 bytes remaining to read

if I tcpdump connection, transfer looks fine for some time and then ends =
with a following sequence (run on main host, jail master):

% sudo tcpdump port 6697
tcpdump: verbose output suppressed, use -v or -vv for full protocol =
decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 65535 =
bytes
23:37:28.621409 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: =
Flags [S], seq 3967654146, win 65535, options [mss 1460,nop,wscale =
5,nop,nop,TS val 601353780 ecr 0,sackOK,eol], length 0
23:37:28.621468 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: =
Flags [S.], seq 517055725, ack 3967654147, win 65535, options [mss =
1460,nop,wscale 6,sackOK,TS val 2553669008 ecr 601353780], length 0
23:37:28.635788 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: =
Flags [.], ack 1, win 4117, options [nop,nop,TS val 601353791 ecr =
2553669008], length 0
23:37:28.635865 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: =
Flags [P.], seq 1:109, ack 1, win 4117, options [nop,nop,TS val =
601353791 ecr 2553669008], length 108
23:37:28.636122 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: =
Flags [P.], seq 1:18, ack 109, win 1040, options [nop,nop,TS val =
2553669022 ecr 601353791], length 17
23:37:28.650153 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: =
Flags [.], ack 18, win 4117, options [nop,nop,TS val 601353805 ecr =
2553669022], length 0
23:37:29.123244 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: =
Flags [.], seq 18:1466, ack 109, win 1040, options [nop,nop,TS val =
2553669510 ecr 601353805], length 1448
(transfer goes normally)
23:37:35.519163 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: =
Flags [F.], seq 37666, ack 109, win 1040, options [nop,nop,TS val =
2553675906 ecr 601360615], length 0
23:37:35.531004 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: =
Flags [.], ack 33322, win 4096, options [nop,nop,TS val 601360640 ecr =
2553675880], length 0
23:37:36.165352 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: =
Flags [.], seq 33322:34770, ack 109, win 1040, options [nop,nop,TS val =
2553676552 ecr 601360640], length 1448
23:37:36.184582 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: =
Flags [.], ack 34770, win 4050, options [nop,nop,TS val 601361283 ecr =
2553676552], length 0
23:37:36.801437 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: =
Flags [.], seq 34770:36218, ack 109, win 1040, options [nop,nop,TS val =
2553677188 ecr 601361283], length 1448
23:37:36.910742 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: =
Flags [.], ack 36218, win 4050, options [nop,nop,TS val 601362012 ecr =
2553677188], length 0
23:37:36.910796 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: =
Flags [FP.], seq 36218:37666, ack 109, win 1040, options [nop,nop,TS val =
2553677297 ecr 601362012], length 1448
23:37:36.922685 IP $my_home_host.51256 > $my_vultr_host.vultr.com.6697: =
Flags [F.], seq 109, ack 37667, win 4096, options [nop,nop,TS val =
601362025 ecr 2553677297], length 0
23:37:36.922742 IP $my_vultr_host.vultr.com.6697 > $my_home_host.51256: =
Flags [.], ack 110, win 1040, options [nop,nop,TS val 2553677309 ecr =
601362025], length 0

My ipfw log doesn=E2=80=99t show any rejected packages in this case. For =
comparison when I run service on the main host (without NAT and port =
redirection) sending transfer is longer and ending sequence looks like =
follows:

% sudo tcpdump port 6696
tcpdump: verbose output suppressed, use -v or -vv for full protocol =
decode
listening on vtnet0, link-type EN10MB (Ethernet), capture size 65535 =
bytes
23:44:54.151840 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: =
Flags [S], seq 3928283896, win 65535, options [mss 1460,nop,wscale =
5,nop,nop,TS val 601797509 ecr 0,sackOK,eol], length 0
23:44:54.151879 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [S.], seq 3837375255, ack 3928283897, win 65535, options [mss =
1460,nop,wscale 6,sackOK,TS val 1729228577 ecr 601797509], length 0
23:44:54.167015 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: =
Flags [.], ack 1, win 4117, options [nop,nop,TS val 601797522 ecr =
1729228577], length 0
23:44:54.167052 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: =
Flags [P.], seq 1:109, ack 1, win 4117, options [nop,nop,TS val =
601797522 ecr 1729228577], length 108
23:44:54.167508 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [P.], seq 1:18, ack 109, win 1040, options [nop,nop,TS val =
1729228593 ecr 601797522], length 17
23:44:54.167573 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [P.], seq 18:4479, ack 109, win 1040, options [nop,nop,TS val =
1729228593 ecr 601797522], length 4461
23:44:54.167585 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [P.], seq 4479:8575, ack 109, win 1040, options [nop,nop,TS val =
1729228593 ecr 601797522], length 4096
23:44:54.167599 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [P.], seq 8575:12671, ack 109, win 1040, options [nop,nop,TS val =
1729228593 ecr 601797522], length 4096
23:44:54.167611 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [.], seq 12671:14481, ack 109, win 1040, options [nop,nop,TS val =
1729228593 ecr 601797522], length 1810
23:44:54.178628 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: =
Flags [.], ack 14481, win 4039, options [nop,nop,TS val 601797540 ecr =
1729228593], length 0
(transfer goes normally)
23:44:54.257686 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [P.], seq 85433:96314, ack 109, win 1040, options [nop,nop,TS val =
1729228683 ecr 601797608], length 10881
23:44:54.271581 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: =
Flags [.], ack 86881, win 4096, options [nop,nop,TS val 601797622 ecr =
1729228683], length 0
23:44:54.271614 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [F.], seq 96314, ack 109, win 1040, options [nop,nop,TS val =
1729228697 ecr 601797622], length 0
23:44:54.276925 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: =
Flags [F.], seq 109, ack 96314, win 4096, options [nop,nop,TS val =
601797624 ecr 1729228683], length 0
23:44:54.276939 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [F.], seq 96314, ack 110, win 1040, options [nop,nop,TS val =
1729228703 ecr 601797624], length 0
23:44:54.286060 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: =
Flags [F.], seq 109, ack 96315, win 4096, options [nop,nop,TS val =
601797633 ecr 1729228697], length 0
23:44:54.286085 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [.], ack 110, win 1040, options [nop,nop,TS val 1729228712 ecr =
601797624], length 0
23:44:54.291528 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: =
Flags [.], ack 96315, win 4096, options [nop,nop,TS val 601797637 ecr =
1729228697], length 0
23:44:54.720336 IP $my_vultr_host.vultr.com.6696 > $my_home_host.51368: =
Flags [.], ack 109, win 0, length 0
23:44:54.974955 IP $my_home_host.51368 > $my_vultr_host.vultr.com.6696: =
Flags [.], ack 96315, win 4096, options [nop,nop,TS val 601798310 ecr =
1729228712], length 0

Of course file transfer in the latter case works just fine. Connections =
initiated from the jail seem to work just fine (I am able to download =
larger files, like distfiles to build ports).=20

I=E2=80=99m grateful for every help. I=E2=80=99m happy to fix my configs =
or even change approach as long as I can achieve my goals. My configs =
are included below.

% cat /etc/rc.conf
hostname=3D=E2=80=9Cfoobar"
ifconfig_vtnet0=3D"dhcp"
sshd_enable=3D"YES"
static_routes=3Dlinklocal
route_linklocal=3D"-net 169.254.0.0/16 -interface vtnet0"
crypto_load=3DYES
cryptodev_load=3DYES
aesni_load=3DYES
virtio_random_load=3DYES
ifconfig_vtnet0_ipv6=3D"inet6 2001:aaaa:bbbb:cccc::64 prefixlen 64"
rtsold_enable=3DYES
ipv6_activate_all_interfaces=3DYES
rtsold_flags=3D"-aF"

# Firewall configuration
firewall_enable=3D"Yes"
firewall_script=3D"/etc/ipfw.rules"
firewall_logging=3D"YES"
firewall_nat_enable=3D"YES"
firewall_nat_interface=3D"vtnet0"

# Internal network configuration
cloned_interfaces=3D"${cloned_interfaces} lo1"
gateway_enable=3D"YES"		# enables the gateway

# (ezjail is not here as I run it with: service ezjail onestart command)

% cat /etc/sysctl.conf
net.inet.ip.fw.one_pass=3D0  # Allow multiple passes for packet, =
necessary for fw+nat to work
net.inet.ip.fw.verbose_limit=3D25

% cat /etc/ipfw.rules
#!/bin/sh
ipfw -q -f flush

# Set rules command prefix
cmd=3D"ipfw add"
pif=3D"vtnet0"     # interface name of NIC attached to Internet
skip=3D"skipto 10000"
mip=3D=E2=80=9C111.222.111.222"  # public ipv4 address

ipfw nat 100 config ip $mip log reset same_ports \
     redirect_port tcp 127.0.66.1:6697 6697 \
     redirect_port tcp 127.0.66.1:80 80=20

ipfw add 1 allow log ip from any to any via lo0

$cmd 0003 allow tcp from $my_home_ip/32 to me 22 in setup keep-state=20
$cmd 5 allow icmp6 from any to any via vtnet0

# NAT for incoming packets
$cmd 100 nat 100 log ip from any to $mip in
$cmd 101 check-state log

$cmd 500 $skip log tcp from 127.0.0.0/8 to any out setup keep-state
$cmd 501 allow log tcp from me to any out setup keep-state
$cmd 502 $skip log udp from 127.0.0.0/8 to any out keep-state
$cmd 503 allow log udp from me to any out keep-state
$cmd 504 $skip log icmp from 127.0.0.0/8 to any out keep-state
$cmd 505 allow log icmp from me to any out keep-state
$cmd 506 allow log icmp6 from me to any out keep-state

$cmd 610 $skip log tcp from any to me 6697 in setup keep-state
$cmd 620 $skip log tcp from any to me 80 in setup keep-state
$cmd 630 allow log tcp from any to me 6696 in setup keep-state

# Deny rest
$cmd 9998 deny log all from any to any in via $pif
$cmd 9999 deny log all from any to any out via $pif

$cmd 10000 nat 100 log ip4 from 127.0.0.0/8 to any out
$cmd 10001 allow log ip from any to any
$cmd 10002 deny log ip from any to any

% cat /usr/local/etc/ezjail/znc
export jail_znc_hostname=3D"znc"
export jail_znc_ip=3D"lo1|127.0.66.1"
# export =
jail_znc_ip=3D"lo1|127.0.66.1,lo1|::6697,vtnet0|10.0.0.66,vtnet0|2001:aaaa=
:bbbb:cccc::6697"
#export =
jail_znc_ip=3D"vtnet0|10.0.0.66/0xffffff00,vtnet0|10.0.0.66,vtnet0|2001:aa=
aa:bbbb:cccc::6697"
export jail_znc_rootdir=3D"/usr/jails/znc"
export jail_znc_exec_start=3D"/bin/sh /etc/rc"
export jail_znc_exec_stop=3D""
export jail_znc_mount_enable=3D"YES"
export jail_znc_devfs_enable=3D"YES"
export jail_znc_devfs_ruleset=3D"devfsrules_jail"
export jail_znc_procfs_enable=3D"YES"
export jail_znc_fdescfs_enable=3D"YES"
export jail_znc_image=3D""
export jail_znc_imagetype=3D""
export jail_znc_attachparams=3D""
export jail_znc_attachblocking=3D""
export jail_znc_forceblocking=3D""
export jail_znc_zfs_datasets=3D""
export jail_znc_cpuset=3D""
export jail_znc_fib=3D""
export jail_znc_parentzfs=3D""
export jail_znc_parameters=3D""
export jail_znc_post_start_script=3D""
export jail_znc_retention_policy=3D""

### Only for debugging
export jail_znc_parameters=3D"allow.raw_sockets=3D1=E2=80=9D

If you need any more configs or tests, please tell me. Thanks for your =
help!

--=20
BRs,
Wojciech Wojtyniak




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A88A7FED-B5DD-4B1E-96A4-AE1F3EAB8E30>