From owner-freebsd-jail@freebsd.org Wed May 31 08:40:08 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D297AFA69C for ; Wed, 31 May 2017 08:40:08 +0000 (UTC) (envelope-from marko.cupac@mimar.rs) Received: from mail.mimar.rs (mail1.mimar.rs [193.53.106.128]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B12A6EAA0 for ; Wed, 31 May 2017 08:40:07 +0000 (UTC) (envelope-from marko.cupac@mimar.rs) Received: from mail1.mimar.rs (localhost [127.0.1.128]) by mail.mimar.rs (Postfix) with ESMTP id 11CF99FA8B67 for ; Wed, 31 May 2017 10:33:53 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mimar.rs; h= content-transfer-encoding:content-type:content-type:mime-version :x-mailer:organization:message-id:subject:subject:from:from:date :date:received:received; s=mimar-0901; t=1496219630; x= 1498034031; bh=mvs4NS4qWZoMDgHZGU3yQHuYrn7rAa34yQZxOXU2FoQ=; b=3 KFDVsD1TWavQmSHlK3D2dbTTyBy66V3zZnUdjvz39BibM5AELN+hT2SLeVjs5/T7 W00kKeBrMfvShHn4DzBmRFxiRWBPduDw4mRA1clDDvysao7z6KuUcQzojcuQfdkF XclYWal1reiFzOfyz009byldJpB/DOJ+o+ydRuYypQ= X-Virus-Scanned: amavisd-new at mimar.rs Received: from mail.mimar.rs ([127.0.1.128]) by mail1.mimar.rs (amavis.mimar.rs [127.0.1.128]) (amavisd-new, port 10026) with LMTP id u4WSvt3JMsTc for ; Wed, 31 May 2017 10:33:50 +0200 (CEST) Received: from efreet-freebsd.kappastar.com (nat-nat.kappastar.com [193.53.106.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: marko.cupac) by mail.mimar.rs (Postfix) with ESMTPSA id 4251B9FA891B for ; Wed, 31 May 2017 10:33:50 +0200 (CEST) Date: Wed, 31 May 2017 10:33:49 +0200 From: Marko =?UTF-8?B?Q3VwYcSH?= To: freebsd-jail@freebsd.org Subject: setfib, jails and loopback interfaces Message-ID: <20170531103349.244f0fbf@efreet-freebsd.kappastar.com> Organization: Mimar X-Mailer: Claws Mail 3.15.0 (GTK+ 2.24.31; amd64-portbld-freebsd11.0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 May 2017 08:40:08 -0000 Hi, I'm not subscribed to the list, could you please keep me in CC? I'm using ezjail as instructed in Handbook, assigning jails lo1|127.0.0.X,bce0|10.66.66.X addresses, in order to keep jails' loopback traffic off host's, and in order to be able to keep internal services on lo1 (such as redis, mongodb, mysql etc.), and external on bce0 (such as apache, unifi5 etc.). Recently I got a server with multiple NICs, and I'd like to serve both LAN and DMZ services from it. I found some information on how to accomplish that with setfib: # cat /boot/loader.conf net.fibs=3D4 net.add_addr_allfibs=3D0 # cat /etc/rc.conf ... cloned_interfaces=3D"lo1" static_routes=3D"nix nixd" route_nix=3D"-net 10.66.66.0/24 -interface bce0 -fib 1" route_nixd=3D"default 10.66.66.254 -fib 1" ... In this setup, services bound to bce0 interface work fine, but they can't contact internal services on lo1. I guess it has something to do with jail routing, but can't figure out what. Thank you in advance for any hints. --=20 Before enlightenment - chop wood, draw water. After enlightenment - chop wood, draw water. Marko Cupa=C4=87 https://www.mimar.rs/ From owner-freebsd-jail@freebsd.org Fri Jun 2 16:17:40 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CCD28BFAB1C for ; Fri, 2 Jun 2017 16:17:40 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [199.192.165.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gritton.org", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id AEC15689DE for ; Fri, 2 Jun 2017 16:17:40 +0000 (UTC) (envelope-from jamie@freebsd.org) Received: from gritton.org (gritton.org [199.192.165.131]) by gritton.org (8.15.2/8.15.2) with ESMTP id v52G9OJ0085619; Fri, 2 Jun 2017 10:09:24 -0600 (MDT) (envelope-from jamie@freebsd.org) Received: (from www@localhost) by gritton.org (8.15.2/8.15.2/Submit) id v52G9ODp085617; Fri, 2 Jun 2017 10:09:24 -0600 (MDT) (envelope-from jamie@freebsd.org) X-Authentication-Warning: gritton.org: www set sender to jamie@freebsd.org using -f To: freebsd-jail@freebsd.org Subject: Re: setfib, jails and loopback interfaces X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Fri, 02 Jun 2017 10:09:24 -0600 From: James Gritton Cc: =?UTF-8?Q?Marko_Cupa=C4=87?= In-Reply-To: <20170531103349.244f0fbf@efreet-freebsd.kappastar.com> References: <20170531103349.244f0fbf@efreet-freebsd.kappastar.com> Message-ID: <2483b0d2a12f49924bf0e66bf7c48549@freebsd.org> X-Sender: jamie@freebsd.org User-Agent: Roundcube Webmail/1.2.3 X-Greylist: inspected by milter-greylist-4.6.2 (gritton.org [199.192.165.131]); Fri, 02 Jun 2017 10:09:25 -0600 (MDT) for IP:'199.192.165.131' DOMAIN:'gritton.org' HELO:'gritton.org' FROM:'jamie@freebsd.org' RCPT:'' X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (gritton.org [199.192.165.131]); Fri, 02 Jun 2017 10:09:25 -0600 (MDT) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Jun 2017 16:17:40 -0000 On 2017-05-31 02:33, Marko Cupać wrote: > Hi, > > I'm not subscribed to the list, could you please keep me in CC? > > I'm using ezjail as instructed in Handbook, assigning jails > lo1|127.0.0.X,bce0|10.66.66.X addresses, in order to keep jails' > loopback traffic off host's, and in order to be able to keep internal > services on lo1 (such as redis, mongodb, mysql etc.), and external on > bce0 (such as apache, unifi5 etc.). > > Recently I got a server with multiple NICs, and I'd like to serve both > LAN and DMZ services from it. I found some information on how to > accomplish that with setfib: > > # cat /boot/loader.conf > net.fibs=4 > net.add_addr_allfibs=0 > > # cat /etc/rc.conf > ... > cloned_interfaces="lo1" > static_routes="nix nixd" > route_nix="-net 10.66.66.0/24 -interface bce0 -fib 1" > route_nixd="default 10.66.66.254 -fib 1" > ... > > In this setup, services bound to bce0 interface work fine, but they > can't contact internal services on lo1. I guess it has something to do > with jail routing, but can't figure out what. > > Thank you in advance for any hints. I haven't done the lo1 trick before, but I have had jails with addresses on a different FIB. Note that the jail also has an FIB. You probably at least want to set the jail's fib to 1 (exec.fib in jail.conf, I suppose jail_*_fib or whatever in the old rc-based system ezjail still uses). The part I'm not sure about is you probably also want to have lo1's entries in the fib=1 routing table. I don't know the interaction between cloned_interfaces and fib though - that might take some exploring in rc, or a word or two from someone who knows that side of things more than I do. - Jamie