From owner-freebsd-current@FreeBSD.ORG Thu Nov 25 20:48:47 2010 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 850CA1065670 for ; Thu, 25 Nov 2010 20:48:47 +0000 (UTC) (envelope-from jamesbrandongooch@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 386538FC0A for ; Thu, 25 Nov 2010 20:48:46 +0000 (UTC) Received: by yxh35 with SMTP id 35so660110yxh.13 for ; Thu, 25 Nov 2010 12:48:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:references:in-reply-to :mime-version:content-transfer-encoding:content-type:message-id:cc :x-mailer:from:subject:date:to; bh=WzN3fdaGLKxDQ+b3gscmIuQEUTD38QW/q2BVMMIPZAA=; b=kudjHPEbKGM4VDJoo0qVLEE4G///BSW3fWp+8QiZPc8MRBBxYxShAG1ypxQ11GaHby ftCSLOD3B+duLYTJ2qKA6lrnRJFAHB37SrwrEkBUVFzzj3Wu8UeM1bRfp6hU7xsPnwSP JqDIKkenLGm+Vy5t0GYlUl3Cgt8lWViXyO9Iw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=references:in-reply-to:mime-version:content-transfer-encoding :content-type:message-id:cc:x-mailer:from:subject:date:to; b=ENujJKAPiYECxnQXWWblYmpte+7Os4jGV8UA6yN8Ss2PZSyTeQzDocLGYUEeFBMTy1 4zTuC5Y13T677zDh5yN3NVUXEqtYMbua7OrZci361Mewfb3DZZiuTD69dPR0ntr/UnFb 82AfsVOxXBGLIV02xTSQ8MmcvPrLtbbHPxHjo= Received: by 10.100.138.10 with SMTP id l10mr915644and.193.1290718126147; Thu, 25 Nov 2010 12:48:46 -0800 (PST) Received: from [10.93.8.63] ([166.205.9.92]) by mx.google.com with ESMTPS id c7sm1189633ana.37.2010.11.25.12.48.40 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 25 Nov 2010 12:48:44 -0800 (PST) References: <1832177924.3260.1290713938422.JavaMail.root@mrelmx10.mrec.ar> In-Reply-To: <1832177924.3260.1290713938422.JavaMail.root@mrelmx10.mrec.ar> Mime-Version: 1.0 (iPhone Mail 8A293) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Message-Id: <739DC3FD-84F4-4ECC-AD44-C9328B206884@gmail.com> X-Mailer: iPhone Mail (8A293) From: Brandon Gooch Date: Thu, 25 Nov 2010 14:50:22 -0600 To: Kevin Mai Cc: "freebsd-current@freebsd.org" Subject: Re: Issues with Jails/Routes/FIBs X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Nov 2010 20:48:47 -0000 On Nov 25, 2010, at 1:38 PM, Kevin Mai wrote: > Hi folks!=20 >=20 > I'm facing an issue here while trying to define separate routing tables fo= r each jail and host.=20 >=20 > Let me show you briefly how it's done:=20 >=20 > The server has 3 physical NICs, each one connected to a different network (= say, public network A, public network B, and LAN).=20 >=20 > Currently, the default gateway is set to be the LAN gateway, even though t= he two jails can see their own public network subnet.=20 >=20 > Internet:=20 > Destination Gateway Flags Refs Use Netif Expire=20 > default 172.16.2.1 UGS 1 3935 bce2=20 > 127.0.0.1 link#5 UH 0 0 lo0=20 > 172.16.2.0/24 link#3 U 0 0 bce2=20 > 172.16.2.127 link#3 UHS 0 0 lo0=20 > 100.16.97.0/24 link#1 U 0 0 bce0=20 > 100.16.97.5 link#1 UHS 0 0 lo0=20 > 100.16.98.0/24 link#2 U 0 0 bce1=20 > 100.16.98.5 link#2 UHS 0 0 lo0=20 >=20 > 100.16.97.0/24 and 100.16.98.0/24 are the two public networks and 172.16.2= .0/24 is the LAN.=20 >=20 > I have already tried removing devfs rules from the jails, setting securele= vel to -1 but I'm still out of luck..=20 >=20 > I know setfib can define alternate routing tables, and I even created a de= fault gateway for two fibs, 1 & 2:=20 >=20 > [root@mrefns09 ~]# setfib 2 netstat -rn=20 > Routing tables=20 >=20 > Internet:=20 > Destination Gateway Flags Refs Use Netif Expire=20 > default 100.16.98.100 UGS 14 906 bce1=20 > 127.0.0.1 link#5 UH 0 0 lo0=20 > 172.16.2.0/24 link#3 U 0 0 bce2=20 > 100.16.97.0/24 link#1 U 0 39 bce0=20 > 100.16.98.0/24 link#2 U 0 0 bce1=20 >=20 > [root@mrefns09 ~]# setfib 1 netstat -rn=20 > Routing tables=20 >=20 > Internet:=20 > Destination Gateway Flags Refs Use Netif Expire=20 > default 100.16.97.100 UGS 0 1758 bce0=20 > 127.0.0.1 link#5 UH 0 0 lo0=20 > 172.16.2.0/24 link#3 U 0 0 bce2=20 > 100.16.97.0/24 link#1 U 0 44 bce0=20 > 100.16.98.0/24 link#2 U 0 4 bce1=20 >=20 > And i've added the proper settings in rc.conf..=20 >=20 > jail_athea97_ip=3D"100.16.97.5 netmask 255.255.255.0"=20 > jail_athea97_fib=3D1=20 >=20 >=20 > jail_athea98_ip=3D"100.16.98.5 netmask 255.255.255.0"=20 > jail_athea98_fib=3D2=20 >=20 > Am I missing something? because once I get into the jail the routing table= is the same:=20 >=20 > [root@athea97 /]# netstat -rn=20 > Routing tables=20 >=20 > Internet:=20 > Destination Gateway Flags Refs Use Netif Expire=20 > default 172.16.2.1 UGS 13 6175 bce2=20 > 127.0.0.1 link#5 UH 0 0 lo0=20 > 172.16.2.0/24 link#3 U 0 0 bce2=20 > 172.16.2.127 link#3 UHS 0 0 lo0=20 > 100.16.97.0/24 link#1 U 0 0 bce0=20 > 100.16.97.5 link#1 UHS 0 0 lo0=20 > 100.16.98.0/24 link#2 U 0 0 bce1=20 > 100.16.98.5 link#2 UHS 0 0 lo0=20 >=20 > [root@athea97 /]# setfib 1 netstat -rn=20 > Routing tables=20 >=20 > Internet:=20 > Destination Gateway Flags Refs Use Netif Expire=20 > default 100.16.97.100 UGS 15 1814 bce0=20 > 127.0.0.1 link#5 UH 0 0 lo0=20 > 172.16.2.0/24 link#3 U 0 0 bce2=20 > 100.16.97.0/24 link#1 U 0 44 bce0=20 > 100.16.98.0/24 link#2 U 0 4 bce1=20 >=20 > The other jail is acting the same way. I know that since I'm doing a jexec= , the shell will have the host's route because, but, how can I know if it's g= etting the alternate routing table?=20 >=20 > Thanks,=20 >=20 > Kevin Try ssh'ing into one of the jails from the public side. The jail should hono= r the FIB configuration from that perspective. Are things behaving as you ex= pect in the jail at that point? As you've figured out, when jexec'ing into the jail from the host machine, y= ou inherit the FIB of your current shell. I think this due to the design of FreeBSD's multiple routing tables -- and n= ot a bug :) -Brandon=