From owner-freebsd-arch@freebsd.org Mon Nov 9 23:15:40 2020 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 03F6246FFA7 for ; Mon, 9 Nov 2020 23:15:40 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CVRgH0f3cz4Vwl for ; Mon, 9 Nov 2020 23:15:38 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 0A9NFThZ007399 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 9 Nov 2020 15:15:29 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 0A9NFTMK007398; Mon, 9 Nov 2020 15:15:29 -0800 (PST) (envelope-from jmg) Date: Mon, 9 Nov 2020 15:15:29 -0800 From: John-Mark Gurney To: "Alexander V. Chernikov" Cc: freebsd-arch Subject: Re: Versioning support for kernel<>userland sysctl interface Message-ID: <20201109231529.GH31099@funkthat.com> Mail-Followup-To: "Alexander V. Chernikov" , freebsd-arch References: <356181604233241@mail.yandex.ru> <20201102221330.GS31099@funkthat.com> <428251604959994@mail.yandex.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <428251604959994@mail.yandex.ru> X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Mon, 09 Nov 2020 15:15:29 -0800 (PST) X-Rspamd-Queue-Id: 4CVRgH0f3cz4Vwl X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [-1.80 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[jmg]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[208.87.223.18:from]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; AUTH_NA(1.00)[]; SPAMHAUS_ZRD(0.00)[208.87.223.18:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; MAILMAN_DEST(0.00)[freebsd-arch] X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Nov 2020 23:15:40 -0000 Alexander V. Chernikov wrote this message on Mon, Nov 09, 2020 at 22:28 +0000: > 02.11.2020, 22:13, "John-Mark Gurney" : > > Alexander V. Chernikov wrote this message on Sun, Nov 01, 2020 at 12:47 +0000: > >>  I would like to propose a change [1] that introduces versioning support for the data structures exposed to userland by sysctl interface. > >> > >>  We have dozens of interfaces exposing various statistics and control data by filling in and exporting structures. > >>  net.inet6.icmp6.stats or net.inet6.icmp6.nd6_prlist can be a good examples of such interaction. > > > > We also need to decide the policy on dealing w/ support for these > > data structures going forward... Because if we do the simple, default > > policy of all userland apps can handle all structures, and kernel can > > produce all structures, we now have an unbounded growth of complexity > > and testing... > I totally agree. While backward compatibility is important, it should not impose notable technical debt. I had the following as my mental model: > * the code should be organised to support output for the latest version. > * There should be a separate, isolatable, piece of code that converts from latest to n-1 (which can be chained: from n-1 to n-2 and so on) > * when introducing changes we should garden older versions by COMPAT_X defines. Yeah, if we restrict the code to COMPAT_x for the existing versions, and ensure that it doesn't change, it isn't TOO terrible, but still, the likelyhood of people writing tests and verifying that they work to make sure that the compat code works for all n-x versions isn't great... it's doable, but I dobut most people are going to put in the effort.. > > I do understand the desire to solve this problem, but IMO, this solution > > is too simple, and dangerous to unbounded growth above. > > > > While I do like it's simplicity, one idea that I've had, while being a > > bit more complex, has the ability to handle modification in a more > > compatible way. > > > > Since we have dtrace, one of the outputs of dtrace is ctf, which allows > > use to convey the type and structure information in a machine parseable > > format. The idea is that each sysctl oid (that supports this) would > > have the ability to fetch the ctf data for that oid. The userland would > > then be able to convert the members to the local members of a similar > > struct. A set of defaults could also be provided, allowing new fields > > to have sane initial values. > > > > As long as the name of a structure member is never reused for a different > > meaning, this will get us most of the way there, in a much cleaner > > method... > > > > I do realize that this isn't the easiest thing, but the tools to do this > > are in the tree, and would solve this problem, IMO, in a way that is a > > lot more maintainable, and long term than the current proposal. > > > > Other solution, use ctf data to produce nvlist generation/consumption > > code for a structure... The data transfered would be larger, but also > > more compatible... > I do like idea on the self-documenting approach. It addresses append-only case nicely, but that's not always the case. > For example, in the initially-discussed icmp6 stats we have 256 64-bit counters representing icmp6 protocol historgram, resulting in 4k frame being allocated on stack for the current kernel implementation. If in the future our icmp6 kernel implementation changes and we won't be able to provide this counters, eventually we would want to remove all these counters from the structure. I'm not sure how can this be addressed without some sort of versioning scheme. So the bit that gets the ctf data would also have an nvlist (or something) that contains the defaults for when fields are removed... So, initally: struct foo { int x; int y; }; Then it gets changed to: struct foo { int x; int y; int z; }; This is easy, the z will be included, and transmitted, but be ignored by older code, then when it changes to: struct { int y; int z; }; The ctf data would be something like: , Where nvlist defaults is: x: -1 So, the consuming code would set the defaults from the nvlist first, then set the fetch data, so that x gets set to some default value. With a few simple rules like this, handling deletions is not a problem when the older code is expecting it. If a variable must always have a value that "must" be correct (and a default cannot be set), then another member needs to be added (I think ctf handles bit fields properly) that says if that member is valid... when it gets removed, that is valid flag gets set to zero, and then the old code knows not to handle it. > > Overall, using bare structures is an ABI compatibility nightmare that > > should be fixed in a better method. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."