From owner-freebsd-questions@freebsd.org Sun Aug 7 08:27:05 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B01C8BB1CA1; Sun, 7 Aug 2016 08:27:05 +0000 (UTC) (envelope-from stdin@niklaas.eu) Received: from mx.box-hlm-02.niklaas.eu (box-hlm-02.niklaas.eu [IPv6:2a02:2770:15:0:21a:4aff:feac:28f6]) by mx1.freebsd.org (Postfix) with ESMTP id 64AD917D0; Sun, 7 Aug 2016 08:27:05 +0000 (UTC) (envelope-from stdin@niklaas.eu) Received: from box-hlm-03.niklaas.eu (box-hlm-03.niklaas.eu [IPv6:2a02:2770:15:0:21a:4aff:feaa:e902]) by mx.box-hlm-02.niklaas.eu (Postfix) with ESMTPSA id ABDBA1C9782; Sun, 7 Aug 2016 10:26:56 +0200 (CEST) Date: Sun, 7 Aug 2016 10:26:52 +0200 From: Niklaas Baudet von Gersdorff To: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org Subject: Re: Firewalling jails and lo0 Message-ID: <20160807082651.GA87754@box-hlm-03.niklaas.eu> Reply-To: stdin@niklaas.eu Mail-Followup-To: freebsd-questions@freebsd.org, freebsd-pf@freebsd.org References: <20160806155411.GA5289@len-t420.klaas> <3C1C4822-17C2-42D9-A9BE-C3549B9B6F25@lists.zabbadoz.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <3C1C4822-17C2-42D9-A9BE-C3549B9B6F25@lists.zabbadoz.net> User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Aug 2016 08:27:05 -0000 Bjoern A. Zeeb [2016-08-06 20:02 +0000] : > I am curious about this. Can you give me an (obfuscated) example? (if > you want in private email) -- $ jls -v JID Hostname Path Name State CPUSetID IP Address(es) [...] 7 mx.box-hlm-03.niklaas.eu /usr/local/jails/smtp1 smtp1 ACTIVE 8 10.3.8.1 fd16:dcc0:f4cc:3::8:1 [...] 24 proxy1.box-hlm-03.niklaas.eu /usr/local/jails/proxy1 proxy1 ACTIVE 5 10.3.2.1 10.77.2.1 fd16:dcc0:f4cc:3::2:1 fd16:dcc0:f4cc:77::2:1 [...] -- -- $ ifconfig lo1 lo1: flags=8049 metric 0 mtu 16384 options=600003 [...] inet 10.3.8.1 netmask 0xffff0000 inet6 fd16:dcc0:f4cc:3::8:1 prefixlen 64 [...] inet 10.3.2.1 netmask 0xffff0000 inet 10.77.2.1 netmask 0xffff0000 inet6 fd16:dcc0:f4cc:3::2:1 prefixlen 64 inet6 fd16:dcc0:f4cc:77::2:1 prefixlen 64 [...] nd6 options=21 -- The following is a /full/ output: -- $ ifconfig lo0 lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 -- So, as you can see, the jails only have IP addresses on lo1 and none of them has one on lo0. To make that clear: -- $ jexec smtp1 ifconfig [...] lo0: flags=8049 metric 0 mtu 16384 options=600003 nd6 options=21 lo1: flags=8049 metric 0 mtu 16384 options=600003 inet 10.3.8.1 netmask 0xffff0000 inet6 fd16:dcc0:f4cc:3::8:1 prefixlen 64 nd6 options=21 [...] -- In my pf.conf I have the following. This is a simplified extract: -- /etc/pf.conf 1 ext_if = vtnet0 2 jail_if = lo1 3 4 table persist 5 table persist 6 7 set skip on lo0 8 9 nat on $ext_if from { } to any -> 10 11 block log all 12 13 pass out all keep state 14 15 pass in on $jail_if proto tcp from to port { } -- As you can see I have a principal block in line 11, and skip is set on lo0 solely. That said, I block on lo1. Because of this, I pass on lo1 in line 19. I thought this is necessary. However, here comes the thing: Although the jails have IP addresses attached to lo1 only, I can see traffic like the following: -- $ tcpdump -nettti lo0 host 10.3.2.1 00:00:00.023424 AF IPv4 (2), length 64: 10.3.2.1.51096 > 10.3.8.1.9025: Flags [S], seq 4205430985, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 349909592 ecr 0], length 0 00:00:00.000064 AF IPv4 (2), length 64: 10.3.8.1.9025 > 10.3.2.1.51096: Flags [S.], seq 3921176095, ack 4205430986, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 3273771227 ecr 349909592], length 0 00:00:00.000023 AF IPv4 (2), length 56: 10.3.2.1.51096 > 10.3.8.1.9025: Flags [.], ack 1, win 1275, options [nop,nop,TS val 349909592 ecr 3273771227], length 0 -- -- $ tcpdump -nettti lo0 host fd16:dcc0:f4cc:3::8:1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes 00:00:00.000000 AF IPv6 (28), length 84: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [S], seq 3339315349, win 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 352469079 ecr 0], length 0 00:00:00.000035 AF IPv6 (28), length 84: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [S.], seq 3726000680, ack 3339315350, win 65535, options [mss 16324,nop,wscale 6,sackOK,TS val 306734766 ecr 352469079], length 0 00:00:00.000044 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [.], ack 1, win 1274, options [nop,nop,TS val 352469079 ecr 306734766], length 0 00:00:05.060320 AF IPv6 (28), length 107: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [P.], seq 1:32, ack 1, win 1274, options [nop,nop,TS val 306739827 ecr 352469079], length 31 00:00:00.000113 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::8:1.25 > fd16:dcc0:f4cc:3::2:1.35851: Flags [F.], seq 32, ack 1, win 1274, options [nop,nop,TS val 306739827 ecr 352469079], length 0 00:00:00.000025 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [.], ack 33, win 1273, options [nop,nop,TS val 352474140 ecr 306739827], length 0 00:00:00.000413 AF IPv6 (28), length 76: fd16:dcc0:f4cc:3::2:1.35851 > fd16:dcc0:f4cc:3::8:1.25: Flags [F.], seq 1, ack 33, win 1274, options [nop,nop,TS val 352474140 ecr 306739827], length 0 -- As you can see, this is on lo0 although the jails don't have an IP address on it. That said, restricting traffic on lo1 doesn't make any sense because the jails use lo0 anyway. > Are these ::1 connections, link-local addresses (unlikely as they should > not be visible to jails), or full IP? As you can see, they are full IP. > And what’s the routing table entry in the base system for them? Have a look at the following output of netstat (I removed some lines and cells): -- $ netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default < > UGS vtnet0 10.0.0.0/8 link#4 U tap0 10.3.2.1 link#3 UH lo1 10.3.8.1 link#3 UH lo1 10.77.2.1 link#3 UH lo1 127.0.0.1 link#2 UH lo0 Internet6: Destination Gateway Flags Netif Expire ::/96 ::1 UGRS lo0 default < > UGS vtnet0 ::1 link#2 UH lo0 ::ffff:0.0.0.0/96 ::1 UGRS lo0 < > link#1 U vtnet0 < > link#1 UHS lo0 fd16:dcc0:f4cc:3::/64 link#3 U lo1 fd16:dcc0:f4cc:3::1 link#4 UHS lo0 fd16:dcc0:f4cc:3::2:1 link#3 UHS lo0 fd16:dcc0:f4cc:3::8:1 link#3 UHS lo0 fd16:dcc0:f4cc:77::/64 link#3 U lo1 fd16:dcc0:f4cc:77::2:1 link#3 UHS lo0 fe80::/10 ::1 UGRS lo0 fe80::%vtnet0/64 link#1 U vtnet0 fe80::< >%vtnet0 link#1 UHS lo0 fe80::%lo0/64 link#2 U lo0 fe80::1%lo0 link#2 UHS lo0 fe80::%tap0/64 link#4 U tap0 fe80::< >%tap0 link#4 UHS lo0 ff01::%vtnet0/32 < >%vtnet0 U vtnet0 ff01::%lo0/32 ::1 U lo0 ff01::%lo1/32 fd16:dcc0:f4cc:3::1:1 U lo1 ff01::%tap0/32 fd16:dcc0:f4cc:3::1 U tap0 ff02::/16 ::1 UGRS lo0 ff02::%vtnet0/32 < >%vtnet0 U vtnet0 ff02::%lo0/32 ::1 U lo0 ff02::%lo1/32 fd16:dcc0:f4cc:3::1:1 U lo1 ff02::%tap0/32 fd16:dcc0:f4cc:3::1 U tap0 -- > especially, do they have any IP address assigned to lo0 in them at all? No, they don't. Niklaas