Date: Fri, 19 Jun 2015 09:11:05 +0200 From: Thomas Steen Rasmussen <thomas@gibfest.dk> To: freebsd-net@freebsd.org, freebsd-pf@freebsd.org Subject: Issue with routing table entries, jails and pf filtering on loopback interfaces Message-ID: <5583C089.8090507@gibfest.dk>
next in thread | raw e-mail | index | archive | help
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 =20 Hello list, This will be a long post, apologies, but it is a complex issue. First I will explain how the server is configured, then I will explain the problem and the workaround I found. When I add one or more IP aliases to a non-loopback interface, lagg0 in this example, FreeBSD adds two routing table entries per IP: 185.96.88.18 link#6 UHS lo0 185.96.88.18/32 link#6 U lagg0 The first entry, the one with "lo0" as it's interface is the problem. The server also has an extra loopback interface with rfc1918 addresses for jails that do not need a real, public IP. A pretty common setup. I also run pf on the server, first filtering rule is "block log all" so I explicitly need to permit traffic on every interface (I do "set skip on lo0" though). I permit all outgoing traffic both on lo1 and on lagg0, so only incoming traffic needs to be permitted. The problem is that with this setup an rfc1918 jail _cannot_ establish a TCP connection to a service listening on an IP on the lagg0 interface. I think this is because the "lo0" routing table entry forces the traffic through an interface where it logically should not appear. The symptom: When I try to ssh from a jail with an rfc1918 IP (say 10.0.0.1) on lo1 to a jail with IP 185.96.88.18 on lagg0, then pf blocks the traffic, but not as I'd expect. I'd expect it to block an incoming packet on lagg0 from the rfc1918 ip to the jailhosts IP. Instead it blocks the syn/ack packet, but in the wrong direction, check this tcpdump from pflog= 0: 06:51:38.188170 rule 0..16777216/0(match): block out on lo1: 185.96.88.18.22 > 10.0.0.1.39228: Flags [S.], seq 573771477, ack 2565048197, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val 3503997415 ecr 53572243], length 0 This traffic should never appear in this direction on this interface. No wonder it gets blocked. The workaround: I've been able to workaround this issue by deleting the "lo0" routes for the IPs on lagg0. If I try SSH again everything looks correct again: 06:57:44.590356 rule 0..16777216/0(match): block in on lagg0: 10.0.0.1.48211 > 185.96.88.18.22: Flags [S], seq 3762725019, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 53938646 ecr 0], length 0 The syn packet is blocked where I'd expect it to get blocked, and if I permit 10.0.0.1 to SSH to 185.96.88.18 in pf.conf it works as expected. I've been asking around and it seems like I'm not the only one with this problem. People just do "set skip on lo1" and never notice it. From a security perspective though I really don't want traffic between jails to be unfirewalled. Can anyone shine a light on this? Thanks a bunch in advance :) Best regards Thomas Steen Rasmussen ps. For what it's worth OpenBSD adds an lo0 route for IP aliases on real interfaces too. No idea if the same problem is present though. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) =20 iQIcBAEBAgAGBQJVg8CJAAoJEHcv938JcvpY5gwQAIAvWW6WYbmhQcQK6CQ+Gqf4 jfkKasCoDFB7X4Fb+K2R7WViN2kt+u8IKO5iavnjQvz7A6Gfrn8LSfeHLCREDBRS IUROx2xPl0WDINhKeGgEdMHN9dldiG5XKJeZuZ0XSc2yPdP/nZGKcy0xBKc7GsPk uJ5CFt9xb9T+wngfySHHj39iKqQezEeYxJYrvtCB0ldq5Le1YQjGDr/fXm6xlV+Y 4UecXTMyU20T9ochJ4JIU+cVLVVjV/CZcnL5O4OWznIojt0y1FufMGpJ4ZPmXn8e xNz4efd5zlbA/e0BFO/OgIDlHlv09CELeudIitlnQniTyEmdRwpNF+Spw6hhsP20 6GEt3WXPbYFU22Y8/v4aN9Jb80kfRkY1Ts7naubLBc21JZZMOIWjKKeFfK8EjQlb wmduqqIdHcAqcEPkkZ5e3VLgqE6HrarbhaiEGHh9v90/BjMDXUoedUwa3DCTy0tO o6SiAGvphMU7s7UFjb6wvhfK7cAplbPNhyQkT8K2FIOU+WFzEvrqT9zXAwehI2pI sIk5v8K2cYevt+JHU7UPcun0NIR5iHfMTzhohs0TeEgPHcqZN9NgQji5i7seS9sN Rj9J4KwtctkGy0y/4yDf0I38IeL88n7NJBfpLr1cFxD8dv7t5mNXbKQSckfWvaqc sDso3GnRwzyvtrLj0zvA =3Dkt7G -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5583C089.8090507>