Date: Sun, 7 Aug 2016 10:26:52 +0200 From: Niklaas Baudet von Gersdorff <stdin@niklaas.eu> To: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org Subject: Re: Firewalling jails and lo0 Message-ID: <20160807082651.GA87754@box-hlm-03.niklaas.eu> In-Reply-To: <3C1C4822-17C2-42D9-A9BE-C3549B9B6F25@lists.zabbadoz.net> References: <20160806155411.GA5289@len-t420.klaas> <3C1C4822-17C2-42D9-A9BE-C3549B9B6F25@lists.zabbadoz.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Bjoern A. Zeeb [2016-08-06 20:02 +0000] :
> I am curious about this. Can you give me an (obfuscated) example? (if
> you want in private email)
-- $ jls -v
JID Hostname Path
Name State
CPUSetID
IP Address(es)
[...]
7 mx.box-hlm-03.niklaas.eu /usr/local/jails/smtp1
smtp1 ACTIVE
8
10.3.8.1
fd16:dcc0:f4cc:3::8:1
[...]
24 proxy1.box-hlm-03.niklaas.eu /usr/local/jails/proxy1
proxy1 ACTIVE
5
10.3.2.1
10.77.2.1
fd16:dcc0:f4cc:3::2:1
fd16:dcc0:f4cc:77::2:1
[...]
--
-- $ ifconfig lo1
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
[...]
inet 10.3.8.1 netmask 0xffff0000
inet6 fd16:dcc0:f4cc:3::8:1 prefixlen 64
[...]
inet 10.3.2.1 netmask 0xffff0000
inet 10.77.2.1 netmask 0xffff0000
inet6 fd16:dcc0:f4cc:3::2:1 prefixlen 64
inet6 fd16:dcc0:f4cc:77::2:1 prefixlen 64
[...]
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
--
The following is a /full/ output:
-- $ ifconfig lo0
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
--
So, as you can see, the jails only have IP addresses on lo1 and
none of them has one on lo0. To make that clear:
-- $ jexec smtp1 ifconfig
[...]
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet 10.3.8.1 netmask 0xffff0000
inet6 fd16:dcc0:f4cc:3::8:1 prefixlen 64
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
[...]
--
In my pf.conf I have the following. This is a simplified extract:
-- /etc/pf.conf
1 ext_if = vtnet0
2 jail_if = lo1
3
4 table <proxy> persist
5 table <mail> persist
6
7 set skip on lo0
8
9 nat on $ext_if from { <proxy> <mail> } to any -> <me>
10
11 block log all
12
13 pass out all keep state
14
15 pass in on $jail_if proto tcp from <proxy> to <mail> port { <some-ports> }
--
As you can see I have a principal block in line 11, and skip is
set on lo0 solely. That said, I block on lo1. Because of this,
I pass on lo1 in line 19. I thought this is necessary.
However, here comes the thing: Although the jails have IP addresses
attached to lo1 only, I can see traffic like the following:
-- $ tcpdump -nettti lo0 host 10.3.2.1
00:00:00.023424 AF IPv4 (2), length 64: 10.3.2.1.51096 > 10.3.8.1.9025: Flags [S], seq 4205430985, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 349909592 ecr 0], length 0
00:00:00.000064 AF IPv4 (2), length 64: 10.3.8.1.9025 > 10.3.2.1.51096: Flags [S.], seq 3921176095, ack 4205430986, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 3273771227 ecr 349909592], length 0
00:00:00.000023 AF IPv4 (2), length 56: 10.3.2.1.51096 > 10.3.8.1.9025: Flags [.], ack 1, win 1275, options [nop,nop,TS val 349909592 ecr 3273771227], length 0
--
-- $ tcpdump -nettti lo0 host fd16:dcc0:f4cc:3::8:1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes
00:00:00.000000 AF IPv6 (28), length 84: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [S], seq 3339315349, win 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 352469079 ecr 0], length 0
00:00:00.000035 AF IPv6 (28), length 84: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [S.], seq 3726000680, ack 3339315350, win 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 306734766 ecr 352469079], length 0
00:00:00.000044 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [.], ack 1, win 1274, options [nop,nop,TS val 352469079 ecr 306734766], length 0
00:00:05.060320 AF IPv6 (28), length 107: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [P.], seq 1:32, ack 1, win 1274, options [nop,nop,TS val 306739827 ecr 352469079], length 31
00:00:00.000113 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [F.], seq 32, ack 1, win 1274, options [nop,nop,TS val 306739827 ecr 352469079], length 0 00:00:00.000025 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [.], ack 33, win 1273, options [nop,nop,TS val 352474140 ecr 306739827], length 0 00:00:00.000413 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [F.], seq 1, ack 33, win 1274, options [nop,nop,TS val 352474140 ecr 306739827], length 0
--
As you can see, this is on lo0 although the jails don't have an
IP address on it. That said, restricting traffic on lo1 doesn't
make any sense because the jails use lo0 anyway.
> Are these ::1 connections, link-local addresses (unlikely as they should
> not be visible to jails), or full IP?
As you can see, they are full IP.
> And what’s the routing table entry in the base system for them?
Have a look at the following output of netstat (I removed some lines and cells):
-- $ netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default < > UGS vtnet0
10.0.0.0/8 link#4 U tap0
10.3.2.1 link#3 UH lo1
10.3.8.1 link#3 UH lo1
10.77.2.1 link#3 UH lo1
127.0.0.1 link#2 UH lo0
Internet6:
Destination Gateway Flags Netif Expire
::/96 ::1 UGRS lo0
default < > UGS vtnet0
::1 link#2 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
< > link#1 U vtnet0
< > link#1 UHS lo0
fd16:dcc0:f4cc:3::/64 link#3 U lo1
fd16:dcc0:f4cc:3::1 link#4 UHS lo0
fd16:dcc0:f4cc:3::2:1 link#3 UHS lo0
fd16:dcc0:f4cc:3::8:1 link#3 UHS lo0
fd16:dcc0:f4cc:77::/64 link#3 U lo1
fd16:dcc0:f4cc:77::2:1 link#3 UHS lo0
fe80::/10 ::1 UGRS lo0
fe80::%vtnet0/64 link#1 U vtnet0
fe80::< >%vtnet0 link#1 UHS lo0
fe80::%lo0/64 link#2 U lo0
fe80::1%lo0 link#2 UHS lo0
fe80::%tap0/64 link#4 U tap0
fe80::< >%tap0 link#4 UHS lo0
ff01::%vtnet0/32 < >%vtnet0 U vtnet0
ff01::%lo0/32 ::1 U lo0
ff01::%lo1/32 fd16:dcc0:f4cc:3::1:1 U lo1
ff01::%tap0/32 fd16:dcc0:f4cc:3::1 U tap0
ff02::/16 ::1 UGRS lo0
ff02::%vtnet0/32 < >%vtnet0 U vtnet0
ff02::%lo0/32 ::1 U lo0
ff02::%lo1/32 fd16:dcc0:f4cc:3::1:1 U lo1
ff02::%tap0/32 fd16:dcc0:f4cc:3::1 U tap0
--
> especially, do they have any IP address assigned to lo0 in them at all?
No, they don't.
Niklaas
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160807082651.GA87754>