From nobody Thu Oct 20 05:42:35 2022 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MtGhm1RwMz4fbQ1 for ; Thu, 20 Oct 2022 05:42:48 +0000 (UTC) (envelope-from freebsd@gushi.org) Received: from prime.gushi.org (prime.gushi.org [IPv6:2620:137:6000:10::142]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "prime.gushi.org", Issuer "RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MtGhl0X94z3wh3 for ; Thu, 20 Oct 2022 05:42:46 +0000 (UTC) (envelope-from freebsd@gushi.org) Received: from smtpclient.apple ([IPv6:2601:602:87f:b05d:ad8e:614d:6e70:974b]) (authenticated bits=0) by prime.gushi.org (8.16.1/8.16.1) with ESMTPSA id 29K5gf9f053197 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 19 Oct 2022 22:42:41 -0700 (PDT) (envelope-from freebsd@gushi.org) DKIM-Filter: OpenDKIM Filter v2.10.3 prime.gushi.org 29K5gf9f053197 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gushi.org; s=prime2014; t=1666244562; bh=Zc48kYyHs4rlVaAVVAH7E0SWTMw9BazSYpfwhz0uzJQ=; h=From:Subject:Date:In-Reply-To:Cc:To:References; z=From:=20Dan=20Mahoney=20|Subject:=20Re:=20Inte rface=20routes=20and=20multiple=20fibs|Date:=20Wed,=2019=20Oct=202 022=2022:42:35=20-0700|In-Reply-To:=20|Cc:=20questions@freebsd .org|To:=20Paul=20Procacci=20|References:=20< 354F1536-D803-472A-933C-8B6D9EAED1F1@gushi.org>=0D=0A=20; b=dm3QOBbgRvfz/J0YCqfdZhhZ8e1Sds4X7yU4BVfsvmFneQIuowDoUjN2HqqWMtv9w ixGJOgo2OUBPC589ubT53xrRw4Bs65SaLTR++GOITLSpkSxsCOAaRl958yanOyOlN3 Hih4tVa2klDq+Jl39UsCBIYqrt7AbphntU+PdUP5q+aziMCYq7KV/lg57QnhgjUu7u wPO90aRkj/3P2iAC82vzFdBhRwFJ9TJHVXV/zWJFyB9bgqvcHdIvJJFLMnYfXVb6sm GwghuMxjERCOWR3lasJyiLKEeeU9/HajRmOWI1PNp2gsL7D2HwryTIJ0Nzantynde7 5hUs6UIo7qaAQ== X-Authentication-Warning: prime.gushi.org: Host [IPv6:2601:602:87f:b05d:ad8e:614d:6e70:974b] claimed to be smtpclient.apple From: Dan Mahoney Message-Id: <70D0A3A6-7967-4C2D-A165-BF9A7084A706@gushi.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_FD14590B-0A75-4DF3-BE1D-3D1183F3EA69" List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: Interface routes and multiple fibs Date: Wed, 19 Oct 2022 22:42:35 -0700 In-Reply-To: Cc: questions@freebsd.org To: Paul Procacci References: <354F1536-D803-472A-933C-8B6D9EAED1F1@gushi.org> X-Mailer: Apple Mail (2.3696.120.41.1.1) X-Rspamd-Queue-Id: 4MtGhl0X94z3wh3 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gushi.org header.s=prime2014 header.b=dm3QOBbg; dmarc=pass (policy=none) header.from=gushi.org; spf=pass (mx1.freebsd.org: domain of freebsd@gushi.org designates 2620:137:6000:10::142 as permitted sender) smtp.mailfrom=freebsd@gushi.org X-Spamd-Result: default: False [-5.20 / 15.00]; DWL_DNSWL_MED(-2.00)[gushi.org:dkim]; HTTP_TO_IP(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[gushi.org,none]; RCVD_IN_DNSWL_MED(-0.20)[2620:137:6000:10::142:from]; R_DKIM_ALLOW(-0.20)[gushi.org:s=prime2014]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[questions@freebsd.org]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:393507, ipnet:2620:137:6000::/44, country:US]; RCPT_COUNT_TWO(0.00)[2]; HAS_XAW(0.00)[]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[gushi.org:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_FD14590B-0A75-4DF3-BE1D-3D1183F3EA69 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Oct 18, 2022, at 15:16, Paul Procacci wrote: >=20 >=20 >=20 > On Tue, Oct 18, 2022 at 5:12 PM Dan Mahoney > wrote: > All, >=20 > Maybe a question for the -net or -rc people. If I should ask there, = let me know. >=20 > I'm running with multiple fibs. One fib is just box management, ssh, = etc. The other fib (which takes BGP routes from peers via BIRD) does = DNS anycast things. The DNS server runs in fib 1. Our default route is = added to both fibs. >=20 > My fib0 routing table looks like this: >=20 > Internet: > Destination Gateway Flags Netif Expire > default 192.159.249.233 UGS bge0 > 127.0.0.1 link#5 UH lo0 > 182.159.249.232/29 link#1 U = bge0 > 182.159.249.236 link#1 UHS lo0 >=20 > Fib 1 is missing that final route.: >=20 > default 182.159.249.233 UGS bge0 > 127.0.0.1 link#5 UH lo0 > 182.159.249.232/29 link#1 U = bge0 >=20 > I've noticed that when I try to do a query (with dig) against it from = fib 0, it sends over lo0 to the named process, but the reply packet just = gets sent out ON BGE0, and is never received, since dig is listening on = the interface it sent the packet over (lo0) to hear the response, which, = near as I can tell with tcpdump -i bge0, just goes out on the wire >=20 > Obviously, we can add the static route to that second fib with: route = add -host 182.159.249.236 -interface lo0 -fib 1. >=20 > Yes, we can also make this stick useing default_routes in rc.conf. >=20 > But it feels like we shouldn't have to. This feels like a glitch, and = that if all fibs get the SUBNET route , they should also get the = loopback. >=20 > -Dan >=20 >=20 > Why would you not expect to add a route for it? > The same subnets can exist in different fibs and be part of different = lan segments a la vlans. Routes are required. But...the same route is added for the SUBNET on both fibs automatically, = even though bge0 is in fib 1. Just not for the actual host. This feels = woefully inconsistent. --Apple-Mail=_FD14590B-0A75-4DF3-BE1D-3D1183F3EA69 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=us-ascii

On Oct 18, 2022, at 15:16, Paul Procacci <pprocacci@gmail.com> wrote:

On Tue, Oct 18, 2022 at 5:12 PM Dan Mahoney <freebsd@gushi.org> wrote:
All,

Maybe a question for the -net or -rc people. If I should ask there, let me know.

I'm running with multiple fibs. One fib is just box management, ssh, etc. The other fib (which takes BGP routes from peers via BIRD) does DNS anycast things. The DNS server runs in fib 1. Our default route is added to both fibs.

My fib0 routing table looks like this:

Internet:
Destination Gateway Flags Netif Expire
default 192.159.249.233 UGS bge0
127.0.0.1 link#5 UH lo0
182.159.249.232/29 link#1 U bge0
182.159.249.236 link#1 UHS lo0

Fib 1 is missing that final route.:

default 182.159.249.233 UGS bge0
127.0.0.1 link#5 UH lo0
182.159.249.232/29 link#1 U bge0

I've noticed that when I try to do a query (with dig) against it from fib 0, it sends over lo0 to the named process, but the reply packet just gets sent out ON BGE0, and is never received, since dig is listening on the interface it sent the packet over (lo0) to hear the response, which, near as I can tell with tcpdump -i bge0, just goes out on the wire

Obviously, we can add the static route to that second fib with: route add -host 182.159.249.236 -interface lo0 -fib 1.

Yes, we can also make this stick useing default_routes in rc.conf.

But it feels like we shouldn't have to. This feels like a glitch, and that if all fibs get the SUBNET route , they should also get the loopback.

-Dan

Why would you not expect to add a route for it?
The same subnets can exist in different fibs and be part of different lan segments a la vlans. Routes are required.

But...the same route is added for the SUBNET on both fibs automatically, even though bge0 is in fib 1. Just not for the actual host. This feels woefully inconsistent.

--Apple-Mail=_FD14590B-0A75-4DF3-BE1D-3D1183F3EA69--