From owner-freebsd-net@freebsd.org Sat Jun 25 22:10:11 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60083B8028C for ; Sat, 25 Jun 2016 22:10:11 +0000 (UTC) (envelope-from org.freebsd.security@io7m.com) Received: from nov-007-i540.relay.mailchannels.net (nov-007-i540.relay.mailchannels.net [46.232.183.94]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 752C91706 for ; Sat, 25 Jun 2016 22:10:08 +0000 (UTC) (envelope-from org.freebsd.security@io7m.com) X-Sender-Id: _forwarded-from|212.69.61.187 Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 169276015D for ; Sat, 25 Jun 2016 22:01:41 +0000 (UTC) Received: from bs3-dallas.accountservergroup.com (ip-10-21-3-36.us-west-2.compute.internal [10.21.3.36]) by relay.mailchannels.net (Postfix) with ESMTPA id 3E4836195E for ; Sat, 25 Jun 2016 22:01:40 +0000 (UTC) X-Sender-Id: _forwarded-from|212.69.61.187 Received: from bs3-dallas.accountservergroup.com (bs3-dallas.accountservergroup.com [10.92.147.53]) (using TLSv1 with cipher DHE-RSA-AES256-SHA) by 0.0.0.0:2500 (trex/5.6.15); Sat, 25 Jun 2016 22:01:40 +0000 X-MC-Relay: Forwarding X-MailChannels-SenderId: _forwarded-from|212.69.61.187 X-MailChannels-Auth-Id: wwwh X-MC-Loop-Signature: 1466892132106:290862121 X-MC-Ingress-Time: 1466892132106 Received: from cust187-dsl61.idnet.net ([212.69.61.187]:65525 helo=copperhead.int.arc7.info) by bs3-dallas.accountservergroup.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.87) (envelope-from ) id 1bGvdz-000EgM-Bi for freebsd-net@freebsd.org; Sat, 25 Jun 2016 17:01:39 -0500 Date: Sat, 25 Jun 2016 22:01:37 +0000 From: To: freebsd-net@freebsd.org Subject: Filtering outbound traffic for private address jails? Message-ID: <20160625220137.1ed8de16@copperhead.int.arc7.info> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-PopBeforeSMTPSenders: io.github.lmax-exchange@io7m.com, com.the-blueprints@io7m.com, com.dropbox@io7m.com, com.rockstargames@io7m.com, org.openjdk@io7m.com, com.git-scm@io7m.com, com.bugsnag@io7m.com, com.jetbrains@io7m.com, com.apple@io7m.com, org.readium@io7m.com, com.google@io7m.com, com.slack@io7m.com, android-developers@io7m.com, com.skype@io7m.com, com.nexusmods@io7m.com, com.carpediemkravmaga@io7m.com, com.myfitnesspal@io7m.com, com.stronglifts@io7m.com, uk.co.discountsupplements@io7m.com, org.khanacademy@io7m.com, com.goodhempnutrition@io7m.com, org.freesound@io7m.com, org.mapdb@io7m.com, io.github.apitrace@io7m.com, org.codehaus@io7m.com, nu.xom@io7m.com, org.blender@io7m.com, org.jgrapht@io7m.com, org.eclipse@io7m.com, net.openvpn@io7m.com, org.freebsd.security@io7m.com, org.apache.commons@io7m.com, de.jflex.users@io7m.com, org.mesa3d.mesa-users@io7m.com, net.java@io7m.com, com.io7m.lists@io7m.com, org.codehaus.mojo@io7m.com, com.meetup@io7m.com, org.archlinux@io7m.com, com.steampowered@io7m.com, com.blendswap@ io7m.com, org.opengl@io7m.com, legalandgeneral@io7m.com, org.freedesktop@io7m.com, org.jogamp@io7m.com, org.junit@io7m.com, org.apache.maven.user@io7m.com, org.sonatype@io7m.com, org.dyn4j@io7m.com, com.creative.opensource.openal@io7m.com, org.fossil-scm.fossil-users@io7m.com, github@io7m.com, code@io7m.com, contact@io7m.com, mark-ext@io7m.com, mark@io7m.com X-AuthUser: X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jun 2016 22:10:11 -0000 Hello. I have been searching for the best part of a day for a solution to this problem and quite frankly cannot believe that I've spent this long on something that appears to be so simple and that used to be fairly easy to achieve. Many years ago, I solved this problem on FreeBSD 6, but the way I did it there seems to no longer work on modern releases. The problem is this: I have a single public IP address. I want to run multiple jails. Back in the days of FreeBSD 6.*, the accepted way to do this seemed to be to create a new loopback device: # ifconfig lo1 create ... and then add a lot of private 127.0.0.* addresses, one per jail. Then, the real network adapter and the new loopback device were both added to a bridge (if_bridge). Unfortunately, I can't remember the exact details, but I believe that NAT was then enabled on the real interface. In order to filter traffic to, from, and between jails, pf rules were written that filtered the bridge device. This meant that jails could correctly send outbound traffic and receive responses (via pf states), could correctly receive specific inbound traffic (via rdr rules), and traffic in both directions could be filtered based on packets entering and leaving the bridge. However (see my other mailing list post), it seems that now with FreeBSD 10, you just can't add loopback devices to bridges. I can find no evidence of anyone online doing this, or even using the old bridge method that I just described! I can find one post in russian that seems to have the same error that I encounter, but nobody has any idea why it's happening. I can find dozens of blog posts describing how to set up jails on private IP addresses. They all follow the same pattern: 1. Create a loopback device. 2. Create a 127.0.0.* address on the loopback device. 3. Create a jail using the address you just added. 4. Set up pf and enable NAT between the real network adapter and the new loopback device. Unfortunately, at this point, you completely lose the ability to filter outbound jail traffic; All packets sent from a jail will obviously have their source address changed to that of the host and therefore it's not possible to distinguish between outbound host traffic and outbound jail traffic in filter rules. As far as I can tell, people are just not filtering outbound traffic, which seems insane! Is it really impossible to do this with FreeBSD 10? M