From owner-freebsd-security@FreeBSD.ORG Mon Jul 2 18:11:40 2007 Return-Path: X-Original-To: freebsd-security@freebsd.org Delivered-To: freebsd-security@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2B1A516A473 for ; Mon, 2 Jul 2007 18:11:40 +0000 (UTC) (envelope-from matt@tnpi.net) Received: from toaster.tnpi.net (toaster.tnpi.net [72.29.111.133]) by mx1.freebsd.org (Postfix) with ESMTP id E806413C4BD for ; Mon, 2 Jul 2007 18:11:39 +0000 (UTC) (envelope-from matt@tnpi.net) Received: (qmail 20701 invoked by uid 89); 2 Jul 2007 17:43:37 -0000 Received: by simscan 1.2.0 ppid: 20697, pid: 20698, t: 0.0180s scanners: attach: 1.2.0 clamav: 0.88.7/m:42/d:2880 Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=tnpi.net; b=zWqBmz90Z178Cuq3c3Buun7lZlbr57dxBxWsI2whbsA5M8iM4jkVLAPIrZ9nBsjcG1xJelA8fd5IL6hW4gevAX0A92KUouGpPz0tTkT9/XFCS/6PNg3mIzKjreh20Hqk ; Received: from unknown (HELO ?10.0.1.207?) (matt@tnpi.biz@76.209.125.244) by 127.0.0.6 with ESMTPA; 2 Jul 2007 17:43:37 -0000 Mime-Version: 1.0 (Apple Message framework v752.3) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Matt Simerson Date: Mon, 2 Jul 2007 12:43:38 -0500 To: freebsd-security@freebsd.org X-Mailer: Apple Mail (2.752.3) X-Mailman-Approved-At: Mon, 02 Jul 2007 19:21:09 +0000 Cc: olli@lurza.secnetix.de Subject: Jails and loopback interfaces X-BeenThere: freebsd-security@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Security issues \[members-only posting\]" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jul 2007 18:11:40 -0000 I've got a server running FreeBSD 6.2 and PF. The server has a couple dozen jails on it. Previously, I had a few "private" services such as MySQL running on loopback IPs (127.0.0.2+) and the rest of the jails running on the public IPs. I have to renumber my machine with a new block of public IPs so I thought I'd be clever and move all the jails onto loopback IPs. Then I could use PF to redirect the new IPs and the old ones to the corresponding jails. The relevant parts of my PF config look like this: > ext_if="em0" > loop_if="lo0" > > ext_addr="72.29.111.130" > loop_addr="127.0.0.101" > > ext_net ="72.29.111.128/27" > ext_net2="208.75.180.64/27" > loop_net="127.0.0.0/8" > > lo_simerson = "127.0.0.4" > lo_toaster = "127.0.0.6" > lo_tnpi = "127.0.0.13" > > simerson = "72.29.111.131" > toaster = "72.29.111.133" > tnpi = "72.29.111.140" > > # nat the jails running on loopback to their public ip > nat on $ext_if from $lo_simerson to any -> $simerson > nat on $ext_if from $lo_toaster to any -> $toaster > nat on $ext_if from $lo_tnpi to any -> $tnpi > > # specific jail rules > rdr on $ext_if from any to $simerson -> $lo_simerson > rdr on $ext_if from any to $toaster -> $lo_toaster > rdr on $ext_if from any to $tnpi -> $lo_tnpi This works just great. Traffic from the outside world gets redirected to the correct jail via a rdr rule. The corresponding nat rule alters the source IP from each jail to its corresponding public IP. The problem I have with this arrangement is when a jail attempts to connect to the public IP of another jails, the connection fails. So, a client running in one jail can't send email to my mail server running in another jail. This problem is described in detail here: http://www.openbsd.org/faq/pf/rdr.html And I have attempted to configure the RDR and NAT combination solution as shown. But after trying a few dozen iterations, I can't get it to work. Based on my understand of the PF manual, conceptually, the problematic connection looks like this: simerson (127.0.0.4) -> toaster public IP (72.29.111.133) rewritten to toaster private IP (127.0.0.6) toaster (127.0.0.6) -> simerson (127.0.0.4) So the simerson jail gets returned a packet from the toasters loopback IP, which it ignores because it is not the IP it sent the connection attempt to. But no matter what I try, I can't make it work. PF is not denying any connections. Without resorting to split-horizon DNS, has anyone gotten this to work reliably on FreeBSD? I'd love to see a working configuration. Matt From owner-freebsd-security@FreeBSD.ORG Thu Jul 5 09:20:26 2007 Return-Path: X-Original-To: freebsd-security@freebsd.org Delivered-To: freebsd-security@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 588B116A400 for ; Thu, 5 Jul 2007 09:20:26 +0000 (UTC) (envelope-from bc@default.co.yu) Received: from kaitan.default.co.yu (kaitan.default.co.yu [87.237.201.133]) by mx1.freebsd.org (Postfix) with SMTP id E0D2213C44C for ; Thu, 5 Jul 2007 09:20:25 +0000 (UTC) (envelope-from bc@default.co.yu) Received: (qmail 66262 invoked by uid 89); 5 Jul 2007 10:53:43 +0200 Received: from unknown (HELO ?192.168.0.90?) (bc@default.co.yu@87.237.200.9) by 0 with SMTP; 5 Jul 2007 10:53:43 +0200 From: bc To: freebsd-security@freebsd.org In-Reply-To: References: Content-Type: text/plain Organization: Default Inc. Date: Thu, 05 Jul 2007 10:54:36 +0200 Message-Id: <1183625676.894.282.camel@serafim.b61.bg.wi> Mime-Version: 1.0 X-Mailer: Evolution 2.8.2.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Subject: Re: Jails and loopback interfaces X-BeenThere: freebsd-security@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Security issues \[members-only posting\]" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Jul 2007 09:20:26 -0000 On Mon, 2007-07-02 at 12:43 -0500, Matt Simerson wrote: > The problem I have with this arrangement is when a jail attempts to > connect to the public IP of another jails, the connection fails. So, > a client running in one jail can't send email to my mail server > running in another jail. You can try keeping up-to-date version of /etc/hosts with hostnames of public services pointing to you 127.0.0.2+ IPs. It's dirty, but at least keeps your pf.conf clean as much as possible. It works for me and it should for you if you dont move services around a lot. Then it requires lots of recursive changes in each jail if you move some service from one IP to another.