From owner-freebsd-net@freebsd.org Mon Oct 16 14:22:14 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB622E3BBF6 for ; Mon, 16 Oct 2017 14:22:14 +0000 (UTC) (envelope-from marko.cupac@mimar.rs) Received: from mail.mimar.rs (tazar.mimar.rs [193.53.106.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 98665828CE for ; Mon, 16 Oct 2017 14:22:13 +0000 (UTC) (envelope-from marko.cupac@mimar.rs) Received: from tazar.mimar.rs (localhost [127.0.2.132]) by mail.mimar.rs (Postfix) with ESMTP id BC248620BD5C for ; Mon, 16 Oct 2017 16:22:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mimar.rs; h= content-transfer-encoding:content-type:content-type:mime-version :x-mailer:organization:message-id:subject:subject:from:from:date :date:received:received; s=mimar-0901; t=1508163725; x= 1509978126; bh=eoCZgUsnlYZ99Iufwf6Rwy0Om9BAt/cE/XxHUIbPX/c=; b=K 2vuIoWLlL3FzYyi6yPXXYrdgg6SuxoSECVscZHZ2D1EUv0+DcMOeZ+K4XmYhA932 T6vwYNC4A/BX0cpSKGmXZFAnT55OZuY/ECajNiEHoXvpuKE0KEylOiRDOrD05Pey IqCdur/apABe8W80N4qyh7oKqus3yxB2oaIpFS3Cj8= X-Virus-Scanned: amavisd-new at mimar.rs Received: from mail.mimar.rs ([127.0.2.132]) by tazar.mimar.rs (amavis.mimar.rs [127.0.2.132]) (amavisd-new, port 10026) with LMTP id cupeZrZLOjMj for ; Mon, 16 Oct 2017 16:22:05 +0200 (CEST) Received: from efreet-freebsd.kappastar.com (nat-nat.kappastar.com [193.53.106.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: marko.cupac) by mail.mimar.rs (Postfix) with ESMTPSA id 8F96D620BD58 for ; Mon, 16 Oct 2017 16:22:05 +0200 (CEST) Date: Mon, 16 Oct 2017 16:22:04 +0200 From: Marko =?UTF-8?B?Q3VwYcSH?= To: freebsd-net@freebsd.org Subject: setfib (ez)jails and wierd routing Message-ID: <20171016162204.5d01a1b1@efreet-freebsd.kappastar.com> Organization: Mimar X-Mailer: Claws Mail 3.15.1 (GTK+ 2.24.31; amd64-portbld-freebsd11.1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Oct 2017 14:22:15 -0000 Hi, I have already asked this on -jail two weeks ago, but perhaps this is better place to ask. I notice wierd routing in my setfib (ez)jails setup. I have a server with multiple NICs. setfib should ensure that LAN jails (setfib 1) can not talk to DMZ jails (setfib 2) over loopbacks, but need to go through firewalls as though they were physical boxes. pacija@warden3:~ % sudo setfib 1 netstat -rn Routing tables (fib: 1) Internet: Destination Gateway Flags Netif Expire default 10.30.19.190 UGS bce0 10.30.19.160/27 00:1c:c4:de:0a:86 US bce0 127.0.0.1 lo0 UHS lo0 127.0.1.0/24 lo1 US lo1 pacija@warden3:~ % sudo setfib 2 netstat -rn Routing tables (fib: 2) Internet: Destination Gateway Flags Netif Expire default 193.53.106.254 UGS bce1 127.0.0.1 lo0 UHS lo0 127.0.2.0/24 lo2 US lo2 193.53.106.0/24 00:1c:c4:de:0a:84 US bce1 Host has the same default route as fib 1: pacija@warden3:~ % sudo netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 10.30.19.190 UGS bce0 ... If I ssh from the Internet into DMZ jail, everything works as expected. But if I ping DMZ jail from the Internet, I see reply packets leaving not the interface they came from (bce1, public address space, DMZ), but another one (bce0, private address space, LAN). This is kinda understandable, because jail on fib2 does not have ICMP enabled, so it is not DMZ jail, but the host (which is in fib 0) who replies to packets via its default gateway (router on a private LAN). Is there an easy and elegant way to solve this? Like binding IP address to fib? I wouldn't like to have to fire up pf on host and meddle with reply-to rules in order to achieve this, I'd rather revert to old setup of separate physical servers for each network. Thank you in advance, --=20 Before enlightenment - chop wood, draw water. After enlightenment - chop wood, draw water. Marko Cupa=C4=87 https://www.mimar.rs/