From owner-freebsd-pkgbase@freebsd.org Mon Apr 29 13:37:21 2019 Return-Path: Delivered-To: freebsd-pkgbase@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88136158D8B4 for ; Mon, 29 Apr 2019 13:37:21 +0000 (UTC) (envelope-from kris@ixsystems.com) Received: from mail-yw1-xc2c.google.com (mail-yw1-xc2c.google.com [IPv6:2607:f8b0:4864:20::c2c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 691D280448 for ; Mon, 29 Apr 2019 13:37:20 +0000 (UTC) (envelope-from kris@ixsystems.com) Received: by mail-yw1-xc2c.google.com with SMTP id t79so3685487ywc.7 for ; Mon, 29 Apr 2019 06:37:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ixsystems-com.20150623.gappssmtp.com; s=20150623; h=from:to:references:in-reply-to:subject:date:message-id:mime-version :content-transfer-encoding:thread-index:content-language; bh=u/KR4+k4F1V0MDA9+SoOrAt7A1oBgeNTG/zyJUCfdfw=; b=HQcPz3YCNCkK7xIFxh3m4HeS+/Xqnai0w1GjGCHZTKoZBLk3tCqD0KFOqFHA8e39+y Qal9ScY8VjRAmvF5r4TqCoeU/IZSXUVwdW/9VKZpZJkdVb57c4DzMDfYqeChH4DGs5qY g4EZu7pm50+QoL9wEj5LF2cn8sjgiN6gCUXdMh+ogw+9gZeXD+Er+yMKQN0BwYI/aIO7 Rw7/jcyF7vPEUlnkFUm8k2/r5FRfJ18TF52ww9/YSIUvhEtd2FcGjqPaufjfazD1foea THCAnQ0qkuq6FDl98RgUJdqj6evr8b1Ab9pSJB7hoxJ8Va3lu6ubHXjG1yYVUzvVCKIy gbIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:references:in-reply-to:subject:date :message-id:mime-version:content-transfer-encoding:thread-index :content-language; bh=u/KR4+k4F1V0MDA9+SoOrAt7A1oBgeNTG/zyJUCfdfw=; b=KXRCwNvn57EDJTqB6mwRuG501d5eVHAaWdF2aBXmygd4PBdFfWfPE4BWnp2CSjDlgG NF9IIfxbtTthl4ZGxKysSuu0+xO8oTc6rze0d9PVMAOMeGtbMHs2xiz05SVq2JZ3KhL6 25xIJKghr8hcSdSqIlEf80AGoA6PtdgLQWlirtNwdwRqfgH8dbQ1feJbJH5R6RW3WQNE nR/R6lcxde4Jfc+7RqrV+VWaiG+fOkMNVDF+cCiQP2Ww/6DXH1NRLrcuOaWfj6jcxo1F DVKbsCjkr5UnUuxxqowlDQixXQi5hHA3RQIHXmryhVjDsfY7L50yqAiLOVmkhjVrWhle w9Cg== X-Gm-Message-State: APjAAAUDDwTqBi50lAPWQlMGwFc+hUhpjolFqUSxZ/rYQsN96ygtZZXv uDPGny/+FY2Yz4Yd59cbyzfComyYqmTkGAn/I5bFITu4tJVPK4z2Tf5wwQzPU6yCuhIpQhAz6oA oabAKZoIi3iUuy2wWZ2POw2KaHmotxaoi5VNNb2PwrRvYDvXz+tzsppaWmpD3UmOZkoMkDYntVw == X-Google-Smtp-Source: APXvYqz5smcg4tkjpZvGhVO7hlKzBR+EwyAe6gc96qfKCr6G0O3dXilKmqKxyKz2XVQfeGvBAjdETA== X-Received: by 2002:a81:2f93:: with SMTP id v141mr48639190ywv.209.1556545039124; Mon, 29 Apr 2019 06:37:19 -0700 (PDT) Received: from KrisYogaC930 ([12.189.233.130]) by smtp.gmail.com with ESMTPSA id a4sm11538927ywa.7.2019.04.29.06.37.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 06:37:17 -0700 (PDT) From: To: "'Ken Moore'" , References: <002901d4fdfb$e52eb890$af8c29b0$@ixsystems.com> <20190429120808.GI85201@kib.kiev.ua> In-Reply-To: Subject: RE: CFT: FreeBSD Package Base Date: Mon, 29 Apr 2019 09:37:16 -0400 Message-ID: <02ef01d4fe90$a997a360$fcc6ea20$@ixsystems.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQFBRupbev+o6AHqgfoJLZwOgBKxNQH06tQyAf5q95SnWvhLcA== Content-Language: en-us X-Rspamd-Queue-Id: 691D280448 X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ixsystems-com.20150623.gappssmtp.com header.s=20150623 header.b=HQcPz3YC; dmarc=pass (policy=none) header.from=ixsystems.com; spf=pass (mx1.freebsd.org: domain of kris@ixsystems.com designates 2607:f8b0:4864:20::c2c as permitted sender) smtp.mailfrom=kris@ixsystems.com X-Spamd-Result: default: False [-7.00 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[ixsystems-com.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-3.01)[ip: (-9.58), ipnet: 2607:f8b0::/32(-3.17), asn: 15169(-2.24), country: US(-0.06)]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-pkgbase@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[ixsystems-com.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: ALT3.ASPMX.L.GOOGLE.com]; FROM_NO_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[c.2.c.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.98)[-0.981,0]; DMARC_POLICY_ALLOW(-0.50)[ixsystems.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-pkgbase@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Packaging the FreeBSD base system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Apr 2019 13:37:21 -0000 Just echo'ing what Ken has stated here. This is part of the reason we implemented this style with the less granular pkgs. The entire 'userland-base' is one single archive, minus docs/tests/debug files. This means a single 'pkg upgrade' of userland-base will be able to finish extraction in one pass, ensuring that libc/libthr/libelf and friends all are splatted on disk in the same pass. -- Kris Moore Vice President of Engineering iXsystems, Inc Ph: (408) 943-4100 Ph: (408) 943-4101 The Groundbreaking TrueNAS M-Series - Enterprise Storage & Servers Driven By Open Source -----Original Message----- From: owner-freebsd-pkgbase@freebsd.org On Behalf Of Ken Moore Sent: Monday, April 29, 2019 8:55 AM To: freebsd-pkgbase@freebsd.org Subject: Re: CFT: FreeBSD Package Base On Monday, April 29, 2019 8:08:08 AM EDT, Konstantin Belousov wrote: > Cc: list trimmed to relevant. Very long essey below, be warned. > > On Sun, Apr 28, 2019 at 03:52:21PM -0400, kris@ixsystems.com wrote: >> FreeBSD Community, >> >> >> >> I'm pleased to announce a CFT for builds of FreeBSD 12-stable and >> 13-current using "TrueOS-inspired" packaged base. These are stock >> FreeBSD images which will allow users to perform all updating via the >> 'pkg' command directly. ... > > I do not know what are design decisions for trueos pkgbase are, but I > do know something about in-tree split and why some packaging decisions > where made. I cannot attend your WG, but I believe the reasoning used > for the in-tree is important enough to represent it intact from the > source. I have to start with some explanatory long text to put it > into the proper perspective. > > There are two knots of interdependinces which are critical for > correctness of any upgrade where the target system cannot be simply > discarded on failure: > 1. C runtime > 2. Minimal boot path to prompt. > Let me elaborate both, starting from point 1, which is typically very > obscure despite having the fundamental nature for anything related to > upgrades. > > The basic execution environment for any program executed by the > FreeBSD kernel is formed by combination of kernel' syscall interface > and some system userspace code which makes the expected environment > over the bare-bone image state after execve. The environment is > typically named C runtime environment since C language ABI is directly > tied into it, and normal C programs only get whatever is provided by > the C runtime unless additional libraries are linked in. Trully, it is > not just C runtime, any other execution environment on top of the OS > is based on this one, but since almost every 'advanced' language > runtime is backed by C language and its runtime, the name stuck. > > FreeBSD C runtime, arguably, is provided by the following four objects: > /libexec/ld-elf.so.1 > /lib/libc.so.7 > /lib/libthr.so.3 > /lib/libm.so.5 > There, we do *guarantee* that the external ABI of the whole pack of > these four objects is backward compatible, i.e. if the binary was > compiled against set if base libraries at earlier date (may be also on > earlier branch), then the binary behaviour would be same when executed > on newer C runtime pack. This is not trivial to achieve, besides > technical measures that helps there, like backward-compatible syscall > interface, symbol versioning, providing fall-back code for older > interface, a lot of overhead in the development is enforced, like > carefull reviews of the changes, the policy and related discipline of > versioning, following published ABI standards, and so on. > > But, internal ABI of the C runtime pack, i.e. interfaces which make > rtld work with libc and libthr, or way by which libthr, when loaded, > makes libc thread-aware, are not stable, and more, they are often > changed in backward-incompatible way. Requiring backward-compatibility > there would stop our ability to evolve the system. Answering some > questions in advance, yes, rtld delves into libc, libthr patches libc > on load, libc has hooks to control some libthr behaviour. > > The only provision that we make is that ld-elf.so.1 is required to > work with older libc/libthr combination, but even then libc and libthr > must be built from the same sources with the same options set. > > Now, returning to pkgbase, if you look at what libs are packed into > clibs, you see: > ld-elf.so.1 > libc.so.7 (and modules like iconv tables or nss, if any) > libthr.so.3 > libdl.so.1 > libgcc{, _eh, _s}.so.1 > libm.so.5 > libedit.so.7 > libncurses{, w}.so.8 > libc++.so.1 > It adds very popular libs like libncurses/libedit, and C++ runtime. > The basic reasoning is that this package is small and chances of > something going wrong while installing it are small as result. Put it > other way, the small clibs package organization makes it highly > probable that system is left in the consistent state (either all new > libs, or all old > libs) after the upgrade, whetever the outcome is. > > If the C runtime pack is not split from the whole 700MB+ update blob, > libthr update has almost certain chance to occur long after or before > libc update, so failures do tend to leave inconsistent > rtld/libc/libthr set. At best, it gives you strange glitches, at > worst you get unusable system that cannot be repaired without external media. > > Now, the second item, the minimal boot path. By definition, it > consists of everything that is required to get bare-bone shell prompt > in single user mode, and where user can repair failed upgrade. > Arguably, it should also include the tools to configure the network > and fix filesystems. So it should consists of > loader (including forth/lua scripts) > kernel > C runtime > /sbin/init > /bin/sh > newfs/fsck/tunefs for UFS > zfs/zfspool and libs for ZFS > ifconfig/route/ping > In this set, zfs and network management tools must be synced with the > kernel, since ABI of the management syscalls is not guaranteed to be > stable even on stable branches. > > The above brain dump is at least partial enumeration of things that > were discussed between me and Glen when Glen created the current > in-tree packaging code. Konstantin: Please read the pkgbase documentation that Kris posted in the CFT (https://trueos.github.io/pkgbase-docs/). Your issues/questions keeps referencing the packaging used in the current FreeBSD base-package implementation instead of the pkgbase system proposed in this CFT. TLDR: The package format proposed here does not follow the current/experimental base package format, but rather is a new ports-based implementation which tries to mimic the traditional distfile outputs of FreeBSD in package form. Because this new base package system is governed by ports instead of in-tree changes to the freebsd source tree itself, this allows for the same base package implementation to be used on almost any version of FreeBSD that you like: which is how 12-STABLE and 13-CURRENT package repos were both trivially created for this CFT. -- ~~ Ken Moore ~~ ken@ixsystems.com _______________________________________________ freebsd-pkgbase@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase To unsubscribe, send any mail to "freebsd-pkgbase-unsubscribe@freebsd.org"