From owner-freebsd-net@freebsd.org Mon Oct 16 18:37:20 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF193E41ADD for ; Mon, 16 Oct 2017 18:37:20 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (plan-b.pwste.edu.pl [89.188.221.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "plan-b.pwste.edu.pl", Issuer "plan-b.pwste.edu.pl" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 132576797F for ; Mon, 16 Oct 2017 18:37:19 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (zarychtam@localhost [127.0.0.1]) by plan-b.pwste.edu.pl (8.15.2/8.15.2) with ESMTPS id v9GI7Tcm034214 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 16 Oct 2017 20:07:29 +0200 (CEST) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: (from zarychtam@localhost) by plan-b.pwste.edu.pl (8.15.2/8.15.2/Submit) id v9GI7SDf034209; Mon, 16 Oct 2017 20:07:28 +0200 (CEST) (envelope-from zarychtam) Date: Mon, 16 Oct 2017 20:07:28 +0200 From: Marek Zarychta To: Marko =?utf-8?B?Q3VwYcSH?= Cc: freebsd-net@freebsd.org Subject: Re: setfib (ez)jails and wierd routing Message-ID: <20171016180728.GA32726@plan-b.pwste.edu.pl> References: <20171016162204.5d01a1b1@efreet-freebsd.kappastar.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="8t9RHnE3ZwKMSgU+" Content-Disposition: inline In-Reply-To: <20171016162204.5d01a1b1@efreet-freebsd.kappastar.com> User-Agent: Mutt/1.9.1 (2017-09-22) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Oct 2017 18:37:20 -0000 --8t9RHnE3ZwKMSgU+ Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 16, 2017 at 04:22:04PM +0200, Marko Cupa=C4=87 wrote: > Hi, >=20 > I have already asked this on -jail two weeks ago, but perhaps this is > better place to ask. >=20 > I notice wierd routing in my setfib (ez)jails setup. >=20 > I have a server with multiple NICs. setfib should ensure that LAN jails > (setfib 1) can not talk to DMZ jails (setfib 2) over loopbacks, but > need to go through firewalls as though they were physical boxes. >=20 > pacija@warden3:~ % sudo setfib 1 netstat -rn > Routing tables (fib: 1) >=20 > Internet: > Destination Gateway Flags Netif Expire > default 10.30.19.190 UGS bce0 > 10.30.19.160/27 00:1c:c4:de:0a:86 US bce0 > 127.0.0.1 lo0 UHS lo0 > 127.0.1.0/24 lo1 US lo1 >=20 > pacija@warden3:~ % sudo setfib 2 netstat -rn > Routing tables (fib: 2) >=20 > Internet: > Destination Gateway Flags Netif Expire > default 193.53.106.254 UGS bce1 > 127.0.0.1 lo0 UHS lo0 > 127.0.2.0/24 lo2 US lo2 > 193.53.106.0/24 00:1c:c4:de:0a:84 US bce1 >=20 > Host has the same default route as fib 1: >=20 > pacija@warden3:~ % sudo netstat -rn > Routing tables >=20 > Internet: > Destination Gateway Flags Netif Expire > default 10.30.19.190 UGS bce0 > ... >=20 > If I ssh from the Internet into DMZ jail, everything works as expected. > But if I ping DMZ jail from the Internet, I see reply packets leaving > not the interface they came from (bce1, public address space, DMZ), but > another one (bce0, private address space, LAN). This is kinda > understandable, because jail on fib2 does not have ICMP enabled, so > it is not DMZ jail, but the host (which is in fib 0) who replies to > packets via its default gateway (router on a private LAN). >=20 > Is there an easy and elegant way to solve this? Like binding IP address > to fib? I wouldn't like to have to fire up pf on host and meddle with > reply-to rules in order to achieve this, I'd rather revert to old setup > of separate physical servers for each network. >=20 Hi, try after to set "ifconfig bce1 fib 2" after disabling PF.=20 This should do the work. --=20 Marek Zarychta --8t9RHnE3ZwKMSgU+ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAABCAAdFiEEMOqvKm6wKvS1/ZeCdZ/s//1SjSwFAlnk9V0ACgkQdZ/s//1S jSzT7ggAmoEMMLJkCdiaRfPUWNnt5Kqs9M2Ui/msaZhCVn9aMCWC5J6w37aNGE1A To2sizmtITiQA46hKhjA4govkPmCyCtvs2IWOb5mL0ctpe54EeGfgeojHnkN8K5Y +nC1ne45O8dkMjijIMzq54I2q2jnAc+7LzBLgzBQwhwBsb7kTmItdoCGDY9ovuCw e0xjFnVQugNAG1lZ/nTwLF/iLBusY9xvK1Idx/tl31n3dA/U2X/3DKhlf1+kBU4S 9sj08XhoS/lHfbHa4MVtEXKF1FSiwTVMTniKufwHUhfWByEXsr2KU8lav2jH6GMb rocPS3iDjEqsOteG35h0v3BdSUqVZQ== =i8c6 -----END PGP SIGNATURE----- --8t9RHnE3ZwKMSgU+--