From owner-freebsd-arch@freebsd.org Sun Nov 1 18:39:28 2020 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9CA03455F5F for ; Sun, 1 Nov 2020 18:39:28 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CPPwH358Hz4RWc for ; Sun, 1 Nov 2020 18:39:27 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 0A1IdJAo075993 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 1 Nov 2020 20:39:22 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 0A1IdJAo075993 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 0A1IdJrq075992; Sun, 1 Nov 2020 20:39:19 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 1 Nov 2020 20:39:19 +0200 From: Konstantin Belousov To: "Alexander V. Chernikov" Cc: freebsd-arch Subject: Re: Versioning support for kernel<>userland sysctl interface Message-ID: <20201101183919.GK2654@kib.kiev.ua> References: <356181604233241@mail.yandex.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <356181604233241@mail.yandex.ru> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4CPPwH358Hz4RWc X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-0.25 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.38)[-0.381]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-0.00)[-0.001]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; NEURAL_SPAM_SHORT(0.13)[0.135]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-arch]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Nov 2020 18:39:28 -0000 On Sun, Nov 01, 2020 at 12:47:17PM +0000, Alexander V. Chernikov wrote: > > Hey folks, > > I would like to propose a change [1] that introduces versioning support for the data structures exposed to userland by sysctl interface. > > We have dozens of interfaces exposing various statistics and control data by filling in and exporting structures. > net.inet6.icmp6.stats or net.inet6.icmp6.nd6_prlist can be a good examples of such interaction. > > Most of these structure do not have version information embedded, which requires us to break compatibility when changing them. > > The idea behind the change is really simple: append current structure version number to the sysctl OID to get the desired version of the structure. > > For example, fetching "net.inet6.icmp6.stats" becomes "net.inet6.icmp6.stats.1" (or, code-wise, something like "net.inet6.icmp6.stats." __XSTRING(ICMP6STAT_VER)). > > The interface satistifes the following properties: > 1) preserving backward compatibility > 2) allowing for low-cost kernel ABI maintenance > 2) allowing for forward compatibility - application can fetch list of all supported versions of a structure. > > > Example: > 11:25 [1] m@devel0 sysctl -o net.inet6.icmp6.stats > net.inet6.icmp6.stats.0: Format:S Length:4328 Dump:0x00000000000000000000000000000000... > net.inet6.icmp6.stats.1: Format:S Length:4624 Dump:0x00000000000000000000000000000000... > > 12:42 [1] m@devel0 ~/test net.inet6.icmp6.stats > sysctlnametomib("net.inet6.icmp6.stats")=0 -> 4.28.58.1. > sysctl("net.inet6.icmp6.stats")=-1 sz=512 > > 12:43 [1] m@devel0 ~/test net.inet6.icmp6.stats.1 > sysctlnametomib("net.inet6.icmp6.stats.1")=0 -> 4.28.58.1.1. > sysctl("net.inet6.icmp6.stats.1")=-1 sz=512 > > > > > Some downside of this change would be the potential need to duplicate structures definitions to be 100% sure we don't break API. For example, rebuilding & running 3rd-party software may result in error fetching the necessary structure. Unmodified application build with the latest structure version will request an oldest version of a structure. > > I see multiple approaches to address it: > 1) duplicate structure with a new name (appending postfix like _v) - works the best for small structure > 2) do nothing specific - will mostly work for append-only statistics structures > 3) rely on kernel warning on calling unversioned sysctls to identify & fix the problematic customers > > Please take a look at [1] for a more detailed technical description of a change. > > Any feedback is highly appreciated. > > [1] https://reviews.freebsd.org/D27035 There was some desire to provide backward ABI-compat shims for sysctls during ino64 work, https://reviews.freebsd.org/D10439. Most prominent idea from that time, AFAIR, was to have another MIB tree, that would be have all the same MIBs but rooted with osrel. In other words, if you accessed e.g. MIB 1.2.3.4, libc internally translates that to MIB 1024..1.2.3.4, and kernel applies whatever shims it knows about that osrel version. If there is no compat, call goes directly to 1.2.3.4 handler. The osrel value can be taken from the binary ABI note, as an example. There was some discussion, but after more work done on this, it appeared that not much sysctls need ABI shims at all, and interesting cases could be adequately handled simply by checking passed buffer length. The osrel approach has a drawback that it ignores possibly different ABI of the loaded shared library which might make the call. On the other hand, it avoids introducing additional burden of requiring consumers to learn new MIBs and manually handle versions.