From owner-freebsd-jail@freebsd.org Sun Mar 10 19:40:30 2019 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC46A153F57A; Sun, 10 Mar 2019 19:40:29 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DD0E28B5FD; Sun, 10 Mar 2019 19:40:28 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id x2AJeKYW059964 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sun, 10 Mar 2019 19:40:20 GMT (envelope-from list1@gjunka.com) To: freebsd-net@freebsd.org, freebsd-jail@freebsd.org From: Grzegorz Junka Subject: exec.fib and a jail in two subnets Message-ID: Date: Sun, 10 Mar 2019 19:40:20 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.3.2 MIME-Version: 1.0 Content-Language: en-GB-large X-Rspamd-Queue-Id: DD0E28B5FD X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of list1@gjunka.com designates 88.98.225.149 as permitted sender) smtp.mailfrom=list1@gjunka.com X-Spamd-Result: default: False [-6.77 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:88.98.225.149]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[gjunka.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-3.64)[ip: (-9.53), ipnet: 88.98.192.0/18(-4.77), asn: 56478(-3.81), country: GB(-0.09)]; MX_GOOD(-0.01)[gjunka.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.81)[-0.815,0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:56478, ipnet:88.98.192.0/18, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2019 19:40:30 -0000 Hi, I am not sure if this question fits better to net or jail list so please delete one crosspost when replying. I have two routers in separate subnets (say 10.0.0.0/16 and 172.16.0.0/16). I have enabled multiple fibs on the host and I am trying to setup a jail so that packets from one router are returned to the same router. The second subnet is configured like this: setfib 1 route add -net 172.16.0.0/16 -iface lagg0 setfib 1 route add default 172.16.0.1 When the jail configuration is (differences in red): mta {   exec.fib=1;   ip4.addr = 172.16.0.2;   interface = lagg0; } router 172.16.0.1 is able to send to and receive packets from the jail as expected. When the jail configuration is: mta {   ip4.addr = 10.0.0.2,172.16.0.2;   interface = lagg0; } then router 10.0.0.1 is also able to send and receive packets from the jail as expected. However, when the configuration is: mta { exec.fib=1;   ip4.addr = 10.0.0.2,172.16.0.2;   interface = lagg0; } then router 172.16.0.1 is no longer able to receive a response from the jail. The router's event log shows entry similar to the following two about 2 minutes apart: IN: ACCEPT [54] Connection opened (Port Forwarding: TCP [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​ [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) IN: ACCEPT [57] Connection closed (Port Forwarding: TCP [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​ [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) My question is why the 10.0.0.1 router is able to communicate with the jail in the second configuration but 172.16.0.1 is not able to communicate with the jail in the third configuration. Is it because of order of IPs in ip4.addr? When the jail is started jls shows only the first IP from either of the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to 1). So my guess is that the first IP is somehow a default IP? Then my additional question is if it's possible for a jail to be in two subnets at the same time, i.e. so that when the jail responds to a packet received from router 10.0.0.1 it sends it to the default route from fib0 and when it responds to a packet received from 172.16.0.1 it sends it to the default route from fib1. What exec.fib should be in such a case? Any help would be greatly appreciated. Thanks! GrzegorzJ From owner-freebsd-jail@freebsd.org Mon Mar 11 10:45:38 2019 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A60A153C9FF for ; Mon, 11 Mar 2019 10:45:38 +0000 (UTC) (envelope-from SRS0=eqo3=RO=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BB40790528 for ; Mon, 11 Mar 2019 10:45:37 +0000 (UTC) (envelope-from SRS0=eqo3=RO=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id EC6D728437 for ; Mon, 11 Mar 2019 11:45:25 +0100 (CET) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id B94C128430 for ; Mon, 11 Mar 2019 11:45:22 +0100 (CET) Subject: Re: how to determine primary (source) IP address in jail To: freebsd-jail@freebsd.org References: <781124ab-ca3e-f410-1a60-649e216cded6@quip.cz> <0439c792-dbb7-49cb-aed2-37aa9e5e1689@www.fastmail.com> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <1179aae6-614e-b02e-9cf5-dc71003604f7@quip.cz> Date: Mon, 11 Mar 2019 11:45:19 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: <0439c792-dbb7-49cb-aed2-37aa9e5e1689@www.fastmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: BB40790528 X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [4.17 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: elsa.codelab.cz]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=eqo3=RO=quip.cz=000.fbsd@elsa.codelab.cz]; RECEIVED_SPAMHAUS_PBL(0.00)[209.16.49.86.zen.spamhaus.org : 127.0.0.11]; IP_SCORE(0.22)[ip: (0.55), ipnet: 94.124.104.0/21(0.28), asn: 42000(0.22), country: CZ(0.07)]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=eqo3=RO=quip.cz=000.fbsd@elsa.codelab.cz]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.92)[0.920,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.92)[0.921,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_LONG(0.92)[0.915,0]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; DMARC_NA(0.00)[quip.cz] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 10:45:38 -0000 Dave Cottlehuber wrote on 2019/03/01 12:43: > On Thu, 28 Feb 2019, at 11:59, Miroslav Lachman wrote: >> Is there some easy way to determine the primary (source) address which >> is used in jail with multiple IP addresses? > > I came to this problem with running local_unbound in jail. Unbound >> refuses queries originating in this jail because the do not come from >> real 127.0.0.1 (which is the only one allowed by default). Unbound in >> jail see requests come from jails IP. It is easy to determine (in shell >> script) if jail has only one IP. >> But what in case where jail has multiple IPs? Is there some sysctl or >> some call to ifconfig or any other util to get the IP which will be used >> as source address for queries on local services in jail? > > Specifically for unbound, try interface-automatic and see if that helps. > > interface-automatic: > Detect source interface on UDP queries and copy them to replies. > This feature is experimental, and needs support in your OS for > particular socket options. Default value is no. > > # /etc/unbound/conf.d/secure.conf > server: > interface-automatic: yes > access-control: 127.0.0.0/8 allow > access-control: 10.0.0.0/8 allow > access-control: 0.0.0.0/0 refuse > access-control: ::1/64 allow > access-control: ::/8 refuse > ... > > > I dont use it quite the same way as you though, and it doesn't solve the > generic problem. I run a single unbound instance in the host system, > and only allow jails to resolve via that. > > https://www.nlnetlabs.nl/documentation/unbound/unbound.conf/ Thanks to all for the replies. It seems that is easier to list all jail's IPs in the access-control with /32 mask than try to find the source address. I can live with it ;) Miroslav Lachman From owner-freebsd-jail@freebsd.org Tue Mar 12 19:47:05 2019 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9E6D3153C630 for ; Tue, 12 Mar 2019 19:47:05 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org (gritton.org [199.192.165.131]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8A8E082E3E for ; Tue, 12 Mar 2019 19:47:04 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org ([127.0.0.131]) by gritton.org (8.15.2/8.15.2) with ESMTP id x2CJJLIs012906; Tue, 12 Mar 2019 13:19:21 -0600 (MDT) (envelope-from jamie@gritton.org) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Tue, 12 Mar 2019 13:19:21 -0600 From: James Gritton To: freebsd-jail@freebsd.org Subject: Re: exec.fib and a jail in two subnets In-Reply-To: References: Message-ID: <6a245a1f51270c71d1da07c55ef51113@gritton.org> X-Sender: jamie@gritton.org User-Agent: Roundcube Webmail/1.3.8 X-Greylist: inspected by milter-greylist-4.6.2 (gritton.org [127.0.0.131]); Tue, 12 Mar 2019 13:19:22 -0600 (MDT) for IP:'127.0.0.131' DOMAIN:'[127.0.0.131]' HELO:'gritton.org' FROM:'jamie@gritton.org' RCPT:'' X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (gritton.org [127.0.0.131]); Tue, 12 Mar 2019 13:19:22 -0600 (MDT) X-Rspamd-Queue-Id: 8A8E082E3E X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of jamie@gritton.org designates 199.192.165.131 as permitted sender) smtp.mailfrom=jamie@gritton.org X-Spamd-Result: default: False [-0.90 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.05)[-0.051,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:199.192.165.128/28]; NEURAL_HAM_LONG(-0.86)[-0.856,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[gritton.org]; NEURAL_SPAM_SHORT(0.33)[0.333,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx5.googlemail.com,alt2.aspmx.l.google.com,aspmx2.googlemail.com]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-0.01)[country: US(-0.07)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:30247, ipnet:199.192.164.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Mar 2019 19:47:05 -0000 On 2019-03-10 13:40, Grzegorz Junka wrote: > Hi, > > I am not sure if this question fits better to net or jail list so > please delete one crosspost when replying. > > I have two routers in separate subnets (say 10.0.0.0/16 and > 172.16.0.0/16). I have enabled multiple fibs on the host and I am > trying to setup a jail so that packets from one router are returned to > the same router. The second subnet is configured like this: > > setfib 1 route add -net 172.16.0.0/16 -iface lagg0 > setfib 1 route add default 172.16.0.1 > > When the jail configuration is (differences in red): > > mta { >   exec.fib=1; >   ip4.addr = 172.16.0.2; >   interface = lagg0; > } > > router 172.16.0.1 is able to send to and receive packets from the jail > as expected. > > When the jail configuration is: > > mta { >   ip4.addr = 10.0.0.2,172.16.0.2; >   interface = lagg0; > } > > then router 10.0.0.1 is also able to send and receive packets from the > jail as expected. > > However, when the configuration is: > > mta { > exec.fib=1; >   ip4.addr = 10.0.0.2,172.16.0.2; >   interface = lagg0; > } > > then router 172.16.0.1 is no longer able to receive a response from > the jail. The router's event log shows entry similar to the following > two about 2 minutes apart: > > IN: ACCEPT [54] Connection opened (Port Forwarding: TCP > [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​ > [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) > IN: ACCEPT [57] Connection closed (Port Forwarding: TCP > [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​ > [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) > > My question is why the 10.0.0.1 router is able to communicate with the > jail in the second configuration but 172.16.0.1 is not able to > communicate with the jail in the third configuration. Is it because of > order of IPs in ip4.addr? > > When the jail is started jls shows only the first IP from either of > the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to > 1). So my guess is that the first IP is somehow a default IP? > > Then my additional question is if it's possible for a jail to be in > two subnets at the same time, i.e. so that when the jail responds to a > packet received from router 10.0.0.1 it sends it to the default route > from fib0 and when it responds to a packet received from 172.16.0.1 it > sends it to the default route from fib1. What exec.fib should be in > such a case? > > Any help would be greatly appreciated. Thanks! You're correct in your assumption that a jail's first IP address is its default: in the absence of binding a particular address for an outgoing connection, the first-listed address will be used. So then the problem with the third jail is you have a packing being sent from 10.0.0.2 with only the routing table that doesn't include 10.0/16. I can't say exactly why your second example *does* work, but at least from the jail side it has a default address that's reachable in its routing table. I'm thinking you're saying that the second jail works not only with 10.0 but also with 172.16 (it's the 172.16 part I'm unsure about). To answer your last question: sure, a jail can be in two subnets - but it will still use its first address by default for any outbound packets. Note that the FIB associated with the jail isn't *really* associated with the jail, but with the processes jail(8) starts for it - the reason for the "exec" in "exec.fib". You're still free to call setfib from inside the jail to access a different table. I haven't tried using two different routing tables in one jail at the same time; the closest I've come is one jail that routed on the non-default network. Outside of the jail world, I believe multiple routing tables implies multiple instances of servers, and that would be the same for inside a jail. Your router log shows port 80, so that would imply two different apache (or whatever) processes running the jail, each pointing to its own address, and rung under its own routing table. - Jamie From owner-freebsd-jail@freebsd.org Tue Mar 12 20:05:44 2019 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5E295153CF54 for ; Tue, 12 Mar 2019 20:05:44 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 31CF98384A for ; Tue, 12 Mar 2019 20:05:42 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id x2CK5YPS006992 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Tue, 12 Mar 2019 20:05:34 GMT (envelope-from list1@gjunka.com) Subject: Re: exec.fib and a jail in two subnets To: James Gritton , freebsd-jail@freebsd.org References: <6a245a1f51270c71d1da07c55ef51113@gritton.org> From: Grzegorz Junka Message-ID: Date: Tue, 12 Mar 2019 20:05:34 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.3.2 MIME-Version: 1.0 In-Reply-To: <6a245a1f51270c71d1da07c55ef51113@gritton.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-GB-large X-Rspamd-Queue-Id: 31CF98384A X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of list1@gjunka.com designates 88.98.225.149 as permitted sender) smtp.mailfrom=list1@gjunka.com X-Spamd-Result: default: False [-6.89 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:88.98.225.149]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[gjunka.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[gjunka.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.94)[-0.944,0]; IP_SCORE(-3.64)[ip: (-9.53), ipnet: 88.98.192.0/18(-4.77), asn: 56478(-3.81), country: GB(-0.09)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:56478, ipnet:88.98.192.0/18, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Mar 2019 20:05:44 -0000 On 12/03/2019 19:19, James Gritton wrote: > On 2019-03-10 13:40, Grzegorz Junka wrote: >> Hi, >> >> I am not sure if this question fits better to net or jail list so >> please delete one crosspost when replying. >> >> I have two routers in separate subnets (say 10.0.0.0/16 and >> 172.16.0.0/16). I have enabled multiple fibs on the host and I am >> trying to setup a jail so that packets from one router are returned to >> the same router. The second subnet is configured like this: >> >> setfib 1 route add -net 172.16.0.0/16 -iface lagg0 >> setfib 1 route add default 172.16.0.1 >> >> When the jail configuration is (differences in red): >> >> mta { >>   exec.fib=1; >>   ip4.addr = 172.16.0.2; >>   interface = lagg0; >> } >> >> router 172.16.0.1 is able to send to and receive packets from the jail >> as expected. >> >> When the jail configuration is: >> >> mta { >>   ip4.addr = 10.0.0.2,172.16.0.2; >>   interface = lagg0; >> } >> >> then router 10.0.0.1 is also able to send and receive packets from the >> jail as expected. >> >> However, when the configuration is: >> >> mta { >> exec.fib=1; >>   ip4.addr = 10.0.0.2,172.16.0.2; >>   interface = lagg0; >> } >> >> then router 172.16.0.1 is no longer able to receive a response from >> the jail. The router's event log shows entry similar to the following >> two about 2 minutes apart: >> >> IN: ACCEPT [54] Connection opened (Port Forwarding: TCP >> [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​ >> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) >> IN: ACCEPT [57] Connection closed (Port Forwarding: TCP >> [172.16.0.2]:80 <-​-​> [212.159.95.213]:80 -​ -​ -​ >> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) >> >> My question is why the 10.0.0.1 router is able to communicate with the >> jail in the second configuration but 172.16.0.1 is not able to >> communicate with the jail in the third configuration. Is it because of >> order of IPs in ip4.addr? >> >> When the jail is started jls shows only the first IP from either of >> the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to >> 1). So my guess is that the first IP is somehow a default IP? >> >> Then my additional question is if it's possible for a jail to be in >> two subnets at the same time, i.e. so that when the jail responds to a >> packet received from router 10.0.0.1 it sends it to the default route >> from fib0 and when it responds to a packet received from 172.16.0.1 it >> sends it to the default route from fib1. What exec.fib should be in >> such a case? >> >> Any help would be greatly appreciated. Thanks! > > You're correct in your assumption that a jail's first IP address is > its default: in the absence of binding a particular address for an > outgoing connection, the first-listed address will be used.  So then > the problem with the third jail is you have a packing being sent from > 10.0.0.2 with only the routing table that doesn't include 10.0/16.  I > can't say exactly why your second example *does* work, but at least > from the jail side it has a default address that's reachable in its > routing table.  I'm thinking you're saying that the second jail works > not only with 10.0 but also with 172.16 (it's the 172.16 part I'm > unsure about). > > To answer your last question: sure, a jail can be in two subnets - but > it will still use its first address by default for any outbound > packets.  Note that the FIB associated with the jail isn't *really* > associated with the jail, but with the processes jail(8) starts for it > - the reason for the "exec" in "exec.fib". You're still free to call > setfib from inside the jail to access a different table. > > I haven't tried using two different routing tables in one jail at the > same time; the closest I've come is one jail that routed on the > non-default network.  Outside of the jail world, I believe multiple > routing tables implies multiple instances of servers, and that would > be the same for inside a jail.  Your router log shows port 80, so that > would imply two different apache (or whatever) processes running the > jail, each pointing to its own address, and rung under its own routing > table. > Many thanks for your response. The second example works with 10.0.0.1 but not with 172.16.0.1, otherwise there would be no post. Following on your response, lets assume that a process (e.g. nginx) listens on both IPs, 10.0.0.2,172.16.0.2. Is it possible to configure fibs or default routes or whatever so that when a packet arrives from 10.0.0.1 it is send back to 10.0.0.1 and if it arrives from 172.16.0.1 it is send back to 172.16.0.1 (thus using default routes from either fib0 or fib1 depending if the packet came from a router in one of those network)? If not, would it be possible to do this with some iptables/pf rules (which I understand in FreeBSD 12 should work in a jail with VNET)? From owner-freebsd-jail@freebsd.org Wed Mar 13 21:55:14 2019 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2EE715430DF for ; Wed, 13 Mar 2019 21:55:14 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7292E723A6 for ; Wed, 13 Mar 2019 21:55:13 +0000 (UTC) (envelope-from jamie@gritton.org) Received: by mail-wm1-x329.google.com with SMTP id a188so753927wmf.3 for ; Wed, 13 Mar 2019 14:55:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gritton.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=YQImwn/O8eHlABc2NEGxnL2B76xe7BOW4maVDI86uPo=; b=mjBuMex2P8KSLfGrX6Px/51JVG57rhkPjpwkxzjflkQ7Bv/kcROMDwPgSHTy/LTJuE bMBYT0ja4ppLIQXP3AcNUOHrWWfVEiOpXhtEtOxYxmURDtI7FuNb/wRod9SpmAH4+SSy tukJJx/XWVR9ntKlSmo2GszoOkSE8pz7Ksawc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=YQImwn/O8eHlABc2NEGxnL2B76xe7BOW4maVDI86uPo=; b=E7YYvUzeTx4my6TNDattcrE9ubIfsDQAMOIGBePcnqeB8dtgD02xYjEbEsoxURCR5P h1C3NP8hnWSC3oMpbMke8doLCCm93q+VIVe8WQJSx59CMGOgrTSgn/ZtFucB6iJBpVkr D8WwWp5FrRPd5uy4ljRyghL9TY4eZuGJKbStJ+4qOawtGxXog7yOc/z2/CEfq8H4snyC vNl1PJuXf87t0CM699EDzZBX17iMFhDKG7CJn0t0houOhVaJ+Ls8FUY6Wje4cVc19EXw nxHVTEFcydS6DgkxX0DhMWzyez4l/Uqe+kcCMpnWQq1lTWZF52BJvWEXGqTqEIX+O1rR bIdw== X-Gm-Message-State: APjAAAUDxYeMR+T9BIuvTDYHW7iul4KYbVpETiD8Cr37I49TOcnsKHlc wkbe8pCfMOIXMzHbTikTqsuHEgA5yfmMJqxBzjzgzXC7Q1A= X-Google-Smtp-Source: APXvYqwod1/i5CMydIj1qxsSz8NEB21lwLOxzm0FjJSEPWCEfYSyp4eg15YjVlW5B/EbVs0zD8fqWkyApNFydhJAdM0= X-Received: by 2002:a1c:c3c3:: with SMTP id t186mr228543wmf.8.1552514111110; Wed, 13 Mar 2019 14:55:11 -0700 (PDT) MIME-Version: 1.0 References: <6a245a1f51270c71d1da07c55ef51113@gritton.org> In-Reply-To: From: James Gritton Date: Wed, 13 Mar 2019 15:54:59 -0600 Message-ID: Subject: Re: exec.fib and a jail in two subnets To: Grzegorz Junka Cc: freebsd-jail@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 7292E723A6 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gritton.org header.s=google header.b=mjBuMex2; spf=pass (mx1.freebsd.org: domain of jamie@gritton.org designates 2a00:1450:4864:20::329 as permitted sender) smtp.mailfrom=jamie@gritton.org X-Spamd-Result: default: False [-6.11 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[gritton.org:s=google]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; DMARC_NA(0.00)[gritton.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gritton.org:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[9.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx2.googlemail.com,alt2.aspmx.l.google.com,aspmx5.googlemail.com]; IP_SCORE(-2.67)[ip: (-8.84), ipnet: 2a00:1450::/32(-2.35), asn: 15169(-2.08), country: US(-0.07)]; NEURAL_HAM_SHORT(-0.93)[-0.933,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Mar 2019 21:55:14 -0000 On Tue, Mar 12, 2019 at 2:05 PM Grzegorz Junka wrote: > > On 12/03/2019 19:19, James Gritton wrote: > > On 2019-03-10 13:40, Grzegorz Junka wrote: > >> Hi, > >> > >> I am not sure if this question fits better to net or jail list so > >> please delete one crosspost when replying. > >> > >> I have two routers in separate subnets (say 10.0.0.0/16 and > >> 172.16.0.0/16). I have enabled multiple fibs on the host and I am > >> trying to setup a jail so that packets from one router are returned to > >> the same router. The second subnet is configured like this: > >> > >> setfib 1 route add -net 172.16.0.0/16 -iface lagg0 > >> setfib 1 route add default 172.16.0.1 > >> > >> When the jail configuration is (differences in red): > >> > >> mta { > >> exec.fib=1; > >> ip4.addr = 172.16.0.2; > >> interface = lagg0; > >> } > >> > >> router 172.16.0.1 is able to send to and receive packets from the jail > >> as expected. > >> > >> When the jail configuration is: > >> > >> mta { > >> ip4.addr = 10.0.0.2,172.16.0.2; > >> interface = lagg0; > >> } > >> > >> then router 10.0.0.1 is also able to send and receive packets from the > >> jail as expected. > >> > >> However, when the configuration is: > >> > >> mta { > >> exec.fib=1; > >> ip4.addr = 10.0.0.2,172.16.0.2; > >> interface = lagg0; > >> } > >> > >> then router 172.16.0.1 is no longer able to receive a response from > >> the jail. The router's event log shows entry similar to the following > >> two about 2 minutes apart: > >> > >> IN: ACCEPT [54] Connection opened (Port Forwarding: TCP > >> [172.16.0.2]:80 <--> [212.159.95.213]:80 - - - > >> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) > >> IN: ACCEPT [57] Connection closed (Port Forwarding: TCP > >> [172.16.0.2]:80 <--> [212.159.95.213]:80 - - - > >> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) > >> > >> My question is why the 10.0.0.1 router is able to communicate with the > >> jail in the second configuration but 172.16.0.1 is not able to > >> communicate with the jail in the third configuration. Is it because of > >> order of IPs in ip4.addr? > >> > >> When the jail is started jls shows only the first IP from either of > >> the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to > >> 1). So my guess is that the first IP is somehow a default IP? > >> > >> Then my additional question is if it's possible for a jail to be in > >> two subnets at the same time, i.e. so that when the jail responds to a > >> packet received from router 10.0.0.1 it sends it to the default route > >> from fib0 and when it responds to a packet received from 172.16.0.1 it > >> sends it to the default route from fib1. What exec.fib should be in > >> such a case? > >> > >> Any help would be greatly appreciated. Thanks! > > > > You're correct in your assumption that a jail's first IP address is > > its default: in the absence of binding a particular address for an > > outgoing connection, the first-listed address will be used. So then > > the problem with the third jail is you have a packing being sent from > > 10.0.0.2 with only the routing table that doesn't include 10.0/16. I > > can't say exactly why your second example *does* work, but at least > > from the jail side it has a default address that's reachable in its > > routing table. I'm thinking you're saying that the second jail works > > not only with 10.0 but also with 172.16 (it's the 172.16 part I'm > > unsure about). > > > > To answer your last question: sure, a jail can be in two subnets - but > > it will still use its first address by default for any outbound > > packets. Note that the FIB associated with the jail isn't *really* > > associated with the jail, but with the processes jail(8) starts for it > > - the reason for the "exec" in "exec.fib". You're still free to call > > setfib from inside the jail to access a different table. > > > > I haven't tried using two different routing tables in one jail at the > > same time; the closest I've come is one jail that routed on the > > non-default network. Outside of the jail world, I believe multiple > > routing tables implies multiple instances of servers, and that would > > be the same for inside a jail. Your router log shows port 80, so that > > would imply two different apache (or whatever) processes running the > > jail, each pointing to its own address, and rung under its own routing > > table. > > > > Many thanks for your response. The second example works with 10.0.0.1 > but not with 172.16.0.1, otherwise there would be no post. Following on > your response, lets assume that a process (e.g. nginx) listens on both > IPs, 10.0.0.2,172.16.0.2. Is it possible to configure fibs or default > routes or whatever so that when a packet arrives from 10.0.0.1 it is > send back to 10.0.0.1 and if it arrives from 172.16.0.1 it is send back > to 172.16.0.1 (thus using default routes from either fib0 or fib1 > depending if the packet came from a router in one of those network)? If > not, would it be possible to do this with some iptables/pf rules (which > I understand in FreeBSD 12 should work in a jail with VNET)? > My understanding (which I admit is imperfect) is that it's not > possible with default routes alone. At the application level, it > would be possible if nginx was either fib-aware, or if it explicitly > bound the source address of its replies - but neither of those are > things typically done at the application level. > It is possible however at the firewall level; At least I know it's > possible for ipfw (the small corner of the firewall world that I > use). A quick check of ipf and ipfilter man pages didn't show "fib" > anywhere, but don't take my word on those. It also may require a > VNET jail; I've never run a system with your exact setup so I'm > unsure whether the binding to the first (non-vnet) jail address > happens before or after the ipfilter rules. - Jamie From owner-freebsd-jail@freebsd.org Wed Mar 13 23:13:09 2019 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 251D2154435F for ; Wed, 13 Mar 2019 23:13:09 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9F63B74782 for ; Wed, 13 Mar 2019 23:13:07 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id x2DND0Hr034348 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 13 Mar 2019 23:13:00 GMT (envelope-from list1@gjunka.com) Subject: Re: exec.fib and a jail in two subnets To: James Gritton Cc: freebsd-jail@freebsd.org References: <6a245a1f51270c71d1da07c55ef51113@gritton.org> From: Grzegorz Junka Message-ID: <8fc3d6df-9f8f-f07c-4e13-1ced74f5c0ad@gjunka.com> Date: Wed, 13 Mar 2019 23:13:00 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.3.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-GB-large X-Rspamd-Queue-Id: 9F63B74782 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of list1@gjunka.com designates 88.98.225.149 as permitted sender) smtp.mailfrom=list1@gjunka.com X-Spamd-Result: default: False [-6.80 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:88.98.225.149]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[gjunka.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[gjunka.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.85)[-0.855,0]; IP_SCORE(-3.64)[ip: (-9.53), ipnet: 88.98.192.0/18(-4.77), asn: 56478(-3.81), country: GB(-0.09)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:56478, ipnet:88.98.192.0/18, country:GB]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Mar 2019 23:13:09 -0000 >> Many thanks for your response. The second example works with 10.0.0.1 >> but not with 172.16.0.1, otherwise there would be no post. Following on >> your response, lets assume that a process (e.g. nginx) listens on both >> IPs, 10.0.0.2,172.16.0.2. Is it possible to configure fibs or default >> routes or whatever so that when a packet arrives from 10.0.0.1 it is >> send back to 10.0.0.1 and if it arrives from 172.16.0.1 it is send back >> to 172.16.0.1 (thus using default routes from either fib0 or fib1 >> depending if the packet came from a router in one of those network)? If >> not, would it be possible to do this with some iptables/pf rules (which >> I understand in FreeBSD 12 should work in a jail with VNET)? > My understanding (which I admit is imperfect) is that it's not > possible with default routes alone. At the application level, it > would be possible if nginx was either fib-aware, or if it explicitly > bound the source address of its replies - but neither of those are > things typically done at the application level. > > It is possible however at the firewall level; At least I know it's > possible for ipfw (the small corner of the firewall world that I > use). A quick check of ipf and ipfilter man pages didn't show "fib" > anywhere, but don't take my word on those. It also may require a > VNET jail; I've never run a system with your exact setup so I'm > unsure whether the binding to the first (non-vnet) jail address > happens before or after the ipfilter rules. > > - Jamie I am just playing with this now and what I see is that a jail can't be in two fibs at the same time. It looks like the host is able to select the default route depending on the subnet in which is the IP I want to reach, but in the jail, telneting or otherwise trying to reach any IP that isn't in the same subnet as the fib specified in exec.fib, is not working. For example, in jail this works: telnet 172.16.0.1 80 but this doesn't telnet 10.0.0.1 80 On the host both works. And both, the host and the jail have an IP and an alias in both subnets.