From owner-freebsd-jail@freebsd.org Wed Mar 13 21:55:14 2019 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2EE715430DF for ; Wed, 13 Mar 2019 21:55:14 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7292E723A6 for ; Wed, 13 Mar 2019 21:55:13 +0000 (UTC) (envelope-from jamie@gritton.org) Received: by mail-wm1-x329.google.com with SMTP id a188so753927wmf.3 for ; Wed, 13 Mar 2019 14:55:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gritton.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=YQImwn/O8eHlABc2NEGxnL2B76xe7BOW4maVDI86uPo=; b=mjBuMex2P8KSLfGrX6Px/51JVG57rhkPjpwkxzjflkQ7Bv/kcROMDwPgSHTy/LTJuE bMBYT0ja4ppLIQXP3AcNUOHrWWfVEiOpXhtEtOxYxmURDtI7FuNb/wRod9SpmAH4+SSy tukJJx/XWVR9ntKlSmo2GszoOkSE8pz7Ksawc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=YQImwn/O8eHlABc2NEGxnL2B76xe7BOW4maVDI86uPo=; b=E7YYvUzeTx4my6TNDattcrE9ubIfsDQAMOIGBePcnqeB8dtgD02xYjEbEsoxURCR5P h1C3NP8hnWSC3oMpbMke8doLCCm93q+VIVe8WQJSx59CMGOgrTSgn/ZtFucB6iJBpVkr D8WwWp5FrRPd5uy4ljRyghL9TY4eZuGJKbStJ+4qOawtGxXog7yOc/z2/CEfq8H4snyC vNl1PJuXf87t0CM699EDzZBX17iMFhDKG7CJn0t0houOhVaJ+Ls8FUY6Wje4cVc19EXw nxHVTEFcydS6DgkxX0DhMWzyez4l/Uqe+kcCMpnWQq1lTWZF52BJvWEXGqTqEIX+O1rR bIdw== X-Gm-Message-State: APjAAAUDxYeMR+T9BIuvTDYHW7iul4KYbVpETiD8Cr37I49TOcnsKHlc wkbe8pCfMOIXMzHbTikTqsuHEgA5yfmMJqxBzjzgzXC7Q1A= X-Google-Smtp-Source: APXvYqwod1/i5CMydIj1qxsSz8NEB21lwLOxzm0FjJSEPWCEfYSyp4eg15YjVlW5B/EbVs0zD8fqWkyApNFydhJAdM0= X-Received: by 2002:a1c:c3c3:: with SMTP id t186mr228543wmf.8.1552514111110; Wed, 13 Mar 2019 14:55:11 -0700 (PDT) MIME-Version: 1.0 References: <6a245a1f51270c71d1da07c55ef51113@gritton.org> In-Reply-To: From: James Gritton Date: Wed, 13 Mar 2019 15:54:59 -0600 Message-ID: Subject: Re: exec.fib and a jail in two subnets To: Grzegorz Junka Cc: freebsd-jail@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 7292E723A6 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gritton.org header.s=google header.b=mjBuMex2; spf=pass (mx1.freebsd.org: domain of jamie@gritton.org designates 2a00:1450:4864:20::329 as permitted sender) smtp.mailfrom=jamie@gritton.org X-Spamd-Result: default: False [-6.11 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[gritton.org:s=google]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-jail@freebsd.org]; DMARC_NA(0.00)[gritton.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gritton.org:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[9.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx2.googlemail.com,alt2.aspmx.l.google.com,aspmx5.googlemail.com]; IP_SCORE(-2.67)[ip: (-8.84), ipnet: 2a00:1450::/32(-2.35), asn: 15169(-2.08), country: US(-0.07)]; NEURAL_HAM_SHORT(-0.93)[-0.933,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Mar 2019 21:55:14 -0000 On Tue, Mar 12, 2019 at 2:05 PM Grzegorz Junka wrote: > > On 12/03/2019 19:19, James Gritton wrote: > > On 2019-03-10 13:40, Grzegorz Junka wrote: > >> Hi, > >> > >> I am not sure if this question fits better to net or jail list so > >> please delete one crosspost when replying. > >> > >> I have two routers in separate subnets (say 10.0.0.0/16 and > >> 172.16.0.0/16). I have enabled multiple fibs on the host and I am > >> trying to setup a jail so that packets from one router are returned to > >> the same router. The second subnet is configured like this: > >> > >> setfib 1 route add -net 172.16.0.0/16 -iface lagg0 > >> setfib 1 route add default 172.16.0.1 > >> > >> When the jail configuration is (differences in red): > >> > >> mta { > >> exec.fib=1; > >> ip4.addr = 172.16.0.2; > >> interface = lagg0; > >> } > >> > >> router 172.16.0.1 is able to send to and receive packets from the jail > >> as expected. > >> > >> When the jail configuration is: > >> > >> mta { > >> ip4.addr = 10.0.0.2,172.16.0.2; > >> interface = lagg0; > >> } > >> > >> then router 10.0.0.1 is also able to send and receive packets from the > >> jail as expected. > >> > >> However, when the configuration is: > >> > >> mta { > >> exec.fib=1; > >> ip4.addr = 10.0.0.2,172.16.0.2; > >> interface = lagg0; > >> } > >> > >> then router 172.16.0.1 is no longer able to receive a response from > >> the jail. The router's event log shows entry similar to the following > >> two about 2 minutes apart: > >> > >> IN: ACCEPT [54] Connection opened (Port Forwarding: TCP > >> [172.16.0.2]:80 <--> [212.159.95.213]:80 - - - > >> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) > >> IN: ACCEPT [57] Connection closed (Port Forwarding: TCP > >> [172.16.0.2]:80 <--> [212.159.95.213]:80 - - - > >> [111.202.101.2]:34172 CLOSED/SYN_SENT ppp3 NAPT) > >> > >> My question is why the 10.0.0.1 router is able to communicate with the > >> jail in the second configuration but 172.16.0.1 is not able to > >> communicate with the jail in the third configuration. Is it because of > >> order of IPs in ip4.addr? > >> > >> When the jail is started jls shows only the first IP from either of > >> the configuration list above (i.e. 10.0.0.2 even if exec.fib is set to > >> 1). So my guess is that the first IP is somehow a default IP? > >> > >> Then my additional question is if it's possible for a jail to be in > >> two subnets at the same time, i.e. so that when the jail responds to a > >> packet received from router 10.0.0.1 it sends it to the default route > >> from fib0 and when it responds to a packet received from 172.16.0.1 it > >> sends it to the default route from fib1. What exec.fib should be in > >> such a case? > >> > >> Any help would be greatly appreciated. Thanks! > > > > You're correct in your assumption that a jail's first IP address is > > its default: in the absence of binding a particular address for an > > outgoing connection, the first-listed address will be used. So then > > the problem with the third jail is you have a packing being sent from > > 10.0.0.2 with only the routing table that doesn't include 10.0/16. I > > can't say exactly why your second example *does* work, but at least > > from the jail side it has a default address that's reachable in its > > routing table. I'm thinking you're saying that the second jail works > > not only with 10.0 but also with 172.16 (it's the 172.16 part I'm > > unsure about). > > > > To answer your last question: sure, a jail can be in two subnets - but > > it will still use its first address by default for any outbound > > packets. Note that the FIB associated with the jail isn't *really* > > associated with the jail, but with the processes jail(8) starts for it > > - the reason for the "exec" in "exec.fib". You're still free to call > > setfib from inside the jail to access a different table. > > > > I haven't tried using two different routing tables in one jail at the > > same time; the closest I've come is one jail that routed on the > > non-default network. Outside of the jail world, I believe multiple > > routing tables implies multiple instances of servers, and that would > > be the same for inside a jail. Your router log shows port 80, so that > > would imply two different apache (or whatever) processes running the > > jail, each pointing to its own address, and rung under its own routing > > table. > > > > Many thanks for your response. The second example works with 10.0.0.1 > but not with 172.16.0.1, otherwise there would be no post. Following on > your response, lets assume that a process (e.g. nginx) listens on both > IPs, 10.0.0.2,172.16.0.2. Is it possible to configure fibs or default > routes or whatever so that when a packet arrives from 10.0.0.1 it is > send back to 10.0.0.1 and if it arrives from 172.16.0.1 it is send back > to 172.16.0.1 (thus using default routes from either fib0 or fib1 > depending if the packet came from a router in one of those network)? If > not, would it be possible to do this with some iptables/pf rules (which > I understand in FreeBSD 12 should work in a jail with VNET)? > My understanding (which I admit is imperfect) is that it's not > possible with default routes alone. At the application level, it > would be possible if nginx was either fib-aware, or if it explicitly > bound the source address of its replies - but neither of those are > things typically done at the application level. > It is possible however at the firewall level; At least I know it's > possible for ipfw (the small corner of the firewall world that I > use). A quick check of ipf and ipfilter man pages didn't show "fib" > anywhere, but don't take my word on those. It also may require a > VNET jail; I've never run a system with your exact setup so I'm > unsure whether the binding to the first (non-vnet) jail address > happens before or after the ipfilter rules. - Jamie