From owner-freebsd-jail@FreeBSD.ORG Sat Jan 21 08:57:39 2012 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37244106564A for ; Sat, 21 Jan 2012 08:57:39 +0000 (UTC) (envelope-from other@ahhyes.net) Received: from srv.ahhyes.net (srv.ahhyes.net [109.169.82.101]) by mx1.freebsd.org (Postfix) with ESMTP id 20DCB8FC0C for ; Sat, 21 Jan 2012 08:57:38 +0000 (UTC) Received: from [10.1.1.1] (helo=ahhyes.net) by srv.ahhyes.net with esmtpa (Exim 4.77 (FreeBSD)) (envelope-from ) id 1RoWOk-0002gv-5Q for freebsd-jail@freebsd.org; Sat, 21 Jan 2012 19:34:11 +1100 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 21 Jan 2012 19:34:06 +1100 From: other@ahhyes.net To: Message-ID: X-Sender: other@ahhyes.net User-Agent: Roundcube Webmail/0.7 X-SA-Exim-Connect-IP: 10.1.1.1 X-SA-Exim-Mail-From: other@ahhyes.net X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on mail.ahhyes.net X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=ham version=3.3.2 X-SA-Exim-Version: 4.2 X-SA-Exim-Scanned: Yes (on srv.ahhyes.net) Subject: nat + pf, network weirdness X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Jan 2012 08:57:39 -0000 Hi Guys, I am running 9.0-RELEASE on my VPS. I decided to jail a bunch of services that are public facing in an effort to improve security. Firstly a breakdown of how things are setup: srv# ifconfig pflog0: flags=0<> metric 0 mtu 33152 pfsync0: flags=0<> metric 0 mtu 1500 syncpeer: 0.0.0.0 maxupd: 128 lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff000000 xn0: flags=8843 metric 0 mtu 1500 options=503 ether 00:16:3e:85:8a:12 inet 109.IP.IP.IP netmask 0xffffff00 broadcast 109.169.82.255 media: Ethernet manual status: active lo1: flags=8049 metric 0 mtu 16384 options=3 inet 10.1.1.IP netmask 0xffffff00 inet 10.1.1.IP netmask 0xffffffff inet 10.1.1.IP netmask 0xffffffff inet 10.1.1.IP netmask 0xffffffff srv# jls JID IP Address Hostname Path 1 10.1.1.IP www.mydomain.net /somepath/jails/www 2 10.1.1.IP sql.mydomain.net /somepath/jails/db 3 10.1.1.IP ns.mydomain.net /somepath/jails/ns 5 10.1.1.IP mail.mydomain.net /somepath/jails/mail Interface xn0 is my public facing interface, with my public IP. Everything appears to work as it should, I have a PF running on the host with a default deny all policy. I have the following NAT rule in my pf.conf: nat on xn0 from 10.1.1.0/24 to any -> (xn0) This allows my jails to reach the outside world, and I have a bunch of port redirects to direct inbound traffic to the appropriate jail. The issue: There seems to be no ability to firewall the traffic between jails whilst the NAT rule is in place. For example, I can log into my jail for "ns" and telnet port 3306 on the jail for SQL and connect freely. If I remove the nat rule from PF, the jails cannot talk to each other over the network, which is what I expect because I have not specified any filter rules on my pf.conf to allow the traffic. According to the PF manual, the filter rules should still get run after the NAT translation takes place, but it does not. With NAT running, I can put a deny all from 10.1.1.0/24 as my very first filter rule and it will do nothing at all. I am unable to determine what the issue is as I cannot even run tcpdump -i lo1 on the host as it appears there is no traffic at all on that interface??? I have a suspicion that the NAT translation is causing traffic to hit a rule in PF that allows it to pass, but I cannot confirm this as I have no ability to see what's flowing over the lo1 interface, it's apparently silent (bull)... Any ideas? I really want to lock down the communication the jails have.