From owner-freebsd-net@freebsd.org Tue Aug 18 08:17:05 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 348F63B3555 for ; Tue, 18 Aug 2020 08:17:05 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [88.98.225.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BW3fm0Cp7z4v5r for ; Tue, 18 Aug 2020 08:17:03 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from venus.yoonka.com (venus.yoonka.com [10.70.7.24]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id 07I8GuKu019297 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Tue, 18 Aug 2020 08:16:56 GMT (envelope-from list1@gjunka.com) Subject: Re: net.add_addr_allfibs=1 behaviour deprecation To: freebsd-net@freebsd.org References: <236161595078191@mail.yandex.ru> <348771597489519@mail.yandex.ru> <0f321bdc-240e-0f85-915b-c9bdd7daa01b@freebsd.org> From: Grzegorz Junka Message-ID: Date: Tue, 18 Aug 2020 08:16:55 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 In-Reply-To: <0f321bdc-240e-0f85-915b-c9bdd7daa01b@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1597738624; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1Vgx+VBqHX/JMrDRiCmXpnHjkHf4thGd119XqLF8E70=; b=kyII+0O2I+T09vfhPUSfdEvYsWxtflUzAHMZhmp8PoYgigCfsj6eQeUZBt9hZgY9nL/mXK 1RGpcou3Juio/5ET7gnHi8kSlIM/w0ogsB7XBapFfbFMR6oOhkG6amE68Q9WgBY0V1uyuL yF99nXRpHdWnNpn+y/+BchTikZ4NTJMLqj/4oxSkwr7NppOgE7TLrGkNemC0gharW5Z+O6 FYFwZKnFkNohzlDvNhYK4f+C4hCiDLZPiVsKUE1wHFbtyU5JAYoP1ZZNj+yyCWfekigLBR bk3Ut0g/cY0rDY/HP8GuHnBUVfirVs6cNoaj0Rgk659iGIIIYAWpwstZpFDB+w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1597738624; a=rsa-sha256; cv=none; b=nsjwCd8cHAIhqCRdmJxmknOYMaRhQV/pLoBveFy8u2t1oFBB7iWBfkgkqXpygMHphus/UQ XyvAwcU1JmIyRlv9ILFsom9ewSMgfgddL0SIo7Fjku3QgX6gYH2fegNnXkbpmImeHsYFgH R7x775Yp3CYD4H3DmeZtv4xKfamw8u8OXZp9OUeDYFFvaLThv4VJ8dg6imp7ztfauJZE/f bFJu2YBautEiIDcjOavf91zVqYYD6W5RxuYRQn3WpGoeBrX8+EOBeEZ3E7Kb7CA0ilcjx8 AM6Ou8tZQBU1w2Aw8hIVdKYA5WryHqRMVddtDg6LGR8/5KUHUQ3VeQWvHQTEsA== ARC-Authentication-Results: i=1; mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of list1@gjunka.com designates 88.98.225.149 as permitted sender) smtp.mailfrom=list1@gjunka.com X-Rspamd-Queue-Id: 4BW3fm0Cp7z4v5r X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of list1@gjunka.com designates 88.98.225.149 as permitted sender) smtp.mailfrom=list1@gjunka.com X-Spamd-Result: default: False [-2.14 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:88.98.225.149]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.02)[-1.021]; ARC_SIGNED(0.00)[i=1]; NEURAL_HAM_SHORT(-0.03)[-0.034]; DMARC_NA(0.00)[gjunka.com]; NEURAL_HAM_MEDIUM(-0.78)[-0.783]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:56478, ipnet:88.98.192.0/18, country:GB]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Aug 2020 08:17:05 -0000 On 18/08/2020 07:54, Julian Elischer wrote: > The reason for the two behaviours is that there are two ways that the > previous behaviour of  "add addresses to the only FIB" could be > interpreted and extended once multiple fibs became available. The > single fib case could be interpreted as either of: > > "Add to All N fibs where N == 1"    or     "add to the first (of 1) > fibs". > I decided to do both :-) > > At Ironport where I wrote it we had a scenario where I didn't want to > have wrong entries in all the fibs when a new interface was brought > up. Even for a moment. An other scenarios where  for example a tunnel > uses fib 1 but the rest of the system uses fib0 (which points through > the tunnel) The addition of new routes into the tunnel's route when a > new virtual interface is brought up pointing through the tunnel to the > same address, leads in the tunnel immediately redirecting packets > through itself which ends in tears.     SO the obvious thing to do was > to make it possible to only add the entry in the fib that was the > default fib or in the case of Ironport, the fib that was the default > fib of the process adding the interface. > > If you had to make a choice I think the '0' choice is the way to go. > All other fibs need to be populated deliberately.. > > On 8/15/20 4:24 AM, Alexander V. Chernikov wrote: >> 18.07.2020, 14:22, "Alexander V. Chernikov" : >>> Dear FreeBSD users, >>> >>> I would like to make net.add_addr_allfibs=0 as the default system >>> behaviour and remove net.add_addr_allfibs. >>> To do so, I would like to collect use cases with >>> net.add_addr_allfibs=1 and multiple fibs, to ensure they can still >>> be supported after removal. >>> >>> Background: >>> >>> Multi-fib support was added in r178888 [1], 12 years ago. Addition >>> of interface addresses to all fibs was a feature from day 1. >>> The `net.add_addr_allfibs` sysctl  was added in r180840 [2], 12 >>> years ago. >>> >>> Problem: >>> The goal of the fib support is to provide multiple independent >>> routing tables, isolated from each other. >>> `net.add_addr_allfibs` default tries to shift gears in the opposite >>> direction, unconditionally inserting all addresses to all of the fibs. >>> >>> It complicates the logic, kernel code and makes control plane >>> performance decrease with the number of fibs. >>> It make impossible to use the same prefixes in multiple fibs, which >>> may be desired given shortage of IPv4 address space. >>> >>> I do understand that there are some cases where such behaviour is >>> desired. >>> For example, it can be used to achieve VRF route leaking or binding >>> on address from different fibs. >>> I would like to collect such cases to consider supporting them in a >>> different way. >>> >>> The goal is to make net.add_addr_allfibs=0 default behaviour and >>> remove net.add_addr_allfibs. >>> It will simplify kernel fib-related code and allow bringing more >>> fib-related features. It will also improve fib scaling. >> No objections has been received. >> Next steps: >> * Switch net.add_addr_allfibs to 0 ( >> https://reviews.freebsd.org/D26076 ) >> * Provide an ability to use nexthops from different fibs >> * Remove net.add_addr_allfibs >>> Timeline: >>> Aug 1: summarising feedback and the usecases, decision on proceeding >>> further >>> Aug 20 (tentative):  patches for supported usecases >>> Sep 15 (tentative):  net.add_addr_allfibs removal. >>> >>> [1]: [base Contents of >>> /head/sys/net/route.c](https://svnweb.freebsd.org/base/head/sys/net/route.c?revision=178888&view=markup) >>> [2]: [base Diff of >>> /head/sys/net/route.c](https://svnweb.freebsd.org/base/head/sys/net/route.c?r1=180839&r2=180840&) >>> >>> /Alexander >> Agree completely, defaulting "add_addr_allfibs" to 1 broke many existing installations, which goes against the least surprise principle so many times advocated on FreeBSD lists. This is just one example: https://forums.freebsd.org/threads/strange-behavior-of-setfib-since-freebsd-12-0.73348/ Now, changing the default again might again break existing installations, which shouldn't be a reason for not doing it, but might be a reason to better communicate it this time around. GrzegorzJ