Date: Sun, 07 Aug 2016 10:20:24 -0400 From: Ernie Luzar <luzar722@gmail.com> To: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org Subject: Re: Firewalling jails and lo0 Message-ID: <57A743A8.10005@gmail.com> In-Reply-To: <20160807082651.GA87754@box-hlm-03.niklaas.eu> References: <20160806155411.GA5289@len-t420.klaas> <3C1C4822-17C2-42D9-A9BE-C3549B9B6F25@lists.zabbadoz.net> <20160807082651.GA87754@box-hlm-03.niklaas.eu>
next in thread | previous in thread | raw e-mail | index | archive | help
Niklaas Baudet von Gersdorff wrote:
> Bjoern A. Zeeb [2016-08-06 20:02 +0000] :
>
>> I am curious about this. Can you give me an (obfuscated) example? (if
>> you want in private email)
>
> -- $ jls -v
> JID Hostname Path
> Name State
> CPUSetID
> IP Address(es)
> [...]
> 7 mx.box-hlm-03.niklaas.eu /usr/local/jails/smtp1
> smtp1 ACTIVE
> 8
> 10.3.8.1
> fd16:dcc0:f4cc:3::8:1
> [...]
> 24 proxy1.box-hlm-03.niklaas.eu /usr/local/jails/proxy1
> proxy1 ACTIVE
> 5
> 10.3.2.1
> 10.77.2.1
> fd16:dcc0:f4cc:3::2:1
> fd16:dcc0:f4cc:77::2:1
> [...]
> --
>
> -- $ ifconfig lo1
> lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> [...]
> inet 10.3.8.1 netmask 0xffff0000
> inet6 fd16:dcc0:f4cc:3::8:1 prefixlen 64
> [...]
> inet 10.3.2.1 netmask 0xffff0000
> inet 10.77.2.1 netmask 0xffff0000
> inet6 fd16:dcc0:f4cc:3::2:1 prefixlen 64
> inet6 fd16:dcc0:f4cc:77::2:1 prefixlen 64
> [...]
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> --
>
> The following is a /full/ output:
>
> -- $ ifconfig lo0
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
> inet 127.0.0.1 netmask 0xff000000
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> --
>
> So, as you can see, the jails only have IP addresses on lo1 and
> none of them has one on lo0. To make that clear:
>
> -- $ jexec smtp1 ifconfig
> [...]
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet 10.3.8.1 netmask 0xffff0000
> inet6 fd16:dcc0:f4cc:3::8:1 prefixlen 64
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> [...]
> --
>
> In my pf.conf I have the following. This is a simplified extract:
>
> -- /etc/pf.conf
> 1 ext_if = vtnet0
> 2 jail_if = lo1
> 3
> 4 table <proxy> persist
> 5 table <mail> persist
> 6
> 7 set skip on lo0
> 8
> 9 nat on $ext_if from { <proxy> <mail> } to any -> <me>
> 10
> 11 block log all
> 12
> 13 pass out all keep state
> 14
> 15 pass in on $jail_if proto tcp from <proxy> to <mail> port { <some-ports> }
> --
>
> As you can see I have a principal block in line 11, and skip is
> set on lo0 solely. That said, I block on lo1. Because of this,
> I pass on lo1 in line 19. I thought this is necessary.
>
> However, here comes the thing: Although the jails have IP addresses
> attached to lo1 only, I can see traffic like the following:
>
> -- $ tcpdump -nettti lo0 host 10.3.2.1
> 00:00:00.023424 AF IPv4 (2), length 64: 10.3.2.1.51096 > 10.3.8.1.9025: Flags [S], seq 4205430985, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 349909592 ecr 0], length 0
> 00:00:00.000064 AF IPv4 (2), length 64: 10.3.8.1.9025 > 10.3.2.1.51096: Flags [S.], seq 3921176095, ack 4205430986, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 3273771227 ecr 349909592], length 0
> 00:00:00.000023 AF IPv4 (2), length 56: 10.3.2.1.51096 > 10.3.8.1.9025: Flags [.], ack 1, win 1275, options [nop,nop,TS val 349909592 ecr 3273771227], length 0
> --
>
> -- $ tcpdump -nettti lo0 host fd16:dcc0:f4cc:3::8:1
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes
> 00:00:00.000000 AF IPv6 (28), length 84: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [S], seq 3339315349, win 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 352469079 ecr 0], length 0
> 00:00:00.000035 AF IPv6 (28), length 84: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [S.], seq 3726000680, ack 3339315350, win 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 306734766 ecr 352469079], length 0
> 00:00:00.000044 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [.], ack 1, win 1274, options [nop,nop,TS val 352469079 ecr 306734766], length 0
> 00:00:05.060320 AF IPv6 (28), length 107: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [P.], seq 1:32, ack 1, win 1274, options [nop,nop,TS val 306739827 ecr 352469079], length 31
> 00:00:00.000113 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [F.], seq 32, ack 1, win 1274, options [nop,nop,TS val 306739827 ecr 352469079], length 0 00:00:00.000025 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [.], ack 33, win 1273, options [nop,nop,TS val 352474140 ecr 306739827], length 0 00:00:00.000413 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [F.], seq 1, ack 33, win 1274, options [nop,nop,TS val 352474140 ecr 306739827], length 0
> --
>
> As you can see, this is on lo0 although the jails don't have an
> IP address on it. That said, restricting traffic on lo1 doesn't
> make any sense because the jails use lo0 anyway.
>
>> Are these ::1 connections, link-local addresses (unlikely as they should
>> not be visible to jails), or full IP?
>
> As you can see, they are full IP.
>
>> And what’s the routing table entry in the base system for them?
>
> Have a look at the following output of netstat (I removed some lines and cells):
>
> -- $ netstat -rn
> Routing tables
>
> Internet:
> Destination Gateway Flags Netif Expire
> default < > UGS vtnet0
> 10.0.0.0/8 link#4 U tap0
> 10.3.2.1 link#3 UH lo1
> 10.3.8.1 link#3 UH lo1
> 10.77.2.1 link#3 UH lo1
> 127.0.0.1 link#2 UH lo0
>
> Internet6:
> Destination Gateway Flags Netif Expire
> ::/96 ::1 UGRS lo0
> default < > UGS vtnet0
> ::1 link#2 UH lo0
> ::ffff:0.0.0.0/96 ::1 UGRS lo0
> < > link#1 U vtnet0
> < > link#1 UHS lo0
> fd16:dcc0:f4cc:3::/64 link#3 U lo1
> fd16:dcc0:f4cc:3::1 link#4 UHS lo0
> fd16:dcc0:f4cc:3::2:1 link#3 UHS lo0
> fd16:dcc0:f4cc:3::8:1 link#3 UHS lo0
> fd16:dcc0:f4cc:77::/64 link#3 U lo1
> fd16:dcc0:f4cc:77::2:1 link#3 UHS lo0
> fe80::/10 ::1 UGRS lo0
> fe80::%vtnet0/64 link#1 U vtnet0
> fe80::< >%vtnet0 link#1 UHS lo0
> fe80::%lo0/64 link#2 U lo0
> fe80::1%lo0 link#2 UHS lo0
> fe80::%tap0/64 link#4 U tap0
> fe80::< >%tap0 link#4 UHS lo0
> ff01::%vtnet0/32 < >%vtnet0 U vtnet0
> ff01::%lo0/32 ::1 U lo0
> ff01::%lo1/32 fd16:dcc0:f4cc:3::1:1 U lo1
> ff01::%tap0/32 fd16:dcc0:f4cc:3::1 U tap0
> ff02::/16 ::1 UGRS lo0
> ff02::%vtnet0/32 < >%vtnet0 U vtnet0
> ff02::%lo0/32 ::1 U lo0
> ff02::%lo1/32 fd16:dcc0:f4cc:3::1:1 U lo1
> ff02::%tap0/32 fd16:dcc0:f4cc:3::1 U tap0
> --
>
>> especially, do they have any IP address assigned to lo0 in them at all?
>
> No, they don't.
>
> Niklaas
I believe the loopback interface lo1 needs 127.0.0.0/8 ip address to
enable loopback functionally, and the ip address has to be a different
sub-net. IE 127.0.10.1 for lo1 while the hosts lo0 uses 127.0.0.1
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?57A743A8.10005>