Date: Mon, 29 Apr 2019 09:37:16 -0400 From: <kris@ixsystems.com> To: "'Ken Moore'" <ken@ixsystems.com>, <freebsd-pkgbase@freebsd.org> Subject: RE: CFT: FreeBSD Package Base Message-ID: <02ef01d4fe90$a997a360$fcc6ea20$@ixsystems.com> In-Reply-To: <b40ea945-0f4e-4d59-83fe-57e973e9b829@ixsystems.com> References: <002901d4fdfb$e52eb890$af8c29b0$@ixsystems.com> <20190429120808.GI85201@kib.kiev.ua> <b40ea945-0f4e-4d59-83fe-57e973e9b829@ixsystems.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Just echo'ing what Ken has stated here. This is part of the reason we
implemented this style with the less granular pkgs. The entire
'userland-base' is one single archive, minus docs/tests/debug files. This
means a single 'pkg upgrade' of userland-base will be able to finish
extraction in one pass, ensuring that libc/libthr/libelf and friends all are
splatted on disk in the same pass.
--
Kris Moore
Vice President of Engineering
iXsystems, Inc
Ph: (408) 943-4100
Ph: (408) 943-4101
The Groundbreaking TrueNAS M-Series -
Enterprise Storage & Servers Driven By Open Source
-----Original Message-----
From: owner-freebsd-pkgbase@freebsd.org <owner-freebsd-pkgbase@freebsd.org>
On Behalf Of Ken Moore
Sent: Monday, April 29, 2019 8:55 AM
To: freebsd-pkgbase@freebsd.org
Subject: Re: CFT: FreeBSD Package Base
On Monday, April 29, 2019 8:08:08 AM EDT, Konstantin Belousov wrote:
> Cc: list trimmed to relevant. Very long essey below, be warned.
>
> On Sun, Apr 28, 2019 at 03:52:21PM -0400, kris@ixsystems.com wrote:
>> FreeBSD Community,
>>
>>
>>
>> I'm pleased to announce a CFT for builds of FreeBSD 12-stable and
>> 13-current using "TrueOS-inspired" packaged base. These are stock
>> FreeBSD images which will allow users to perform all updating via the
>> 'pkg' command directly. ...
>
> I do not know what are design decisions for trueos pkgbase are, but I
> do know something about in-tree split and why some packaging decisions
> where made. I cannot attend your WG, but I believe the reasoning used
> for the in-tree is important enough to represent it intact from the
> source. I have to start with some explanatory long text to put it
> into the proper perspective.
>
> There are two knots of interdependinces which are critical for
> correctness of any upgrade where the target system cannot be simply
> discarded on failure:
> 1. C runtime
> 2. Minimal boot path to prompt.
> Let me elaborate both, starting from point 1, which is typically very
> obscure despite having the fundamental nature for anything related to
> upgrades.
>
> The basic execution environment for any program executed by the
> FreeBSD kernel is formed by combination of kernel' syscall interface
> and some system userspace code which makes the expected environment
> over the bare-bone image state after execve. The environment is
> typically named C runtime environment since C language ABI is directly
> tied into it, and normal C programs only get whatever is provided by
> the C runtime unless additional libraries are linked in. Trully, it is
> not just C runtime, any other execution environment on top of the OS
> is based on this one, but since almost every 'advanced' language
> runtime is backed by C language and its runtime, the name stuck.
>
> FreeBSD C runtime, arguably, is provided by the following four objects:
> /libexec/ld-elf.so.1
> /lib/libc.so.7
> /lib/libthr.so.3
> /lib/libm.so.5
> There, we do *guarantee* that the external ABI of the whole pack of
> these four objects is backward compatible, i.e. if the binary was
> compiled against set if base libraries at earlier date (may be also on
> earlier branch), then the binary behaviour would be same when executed
> on newer C runtime pack. This is not trivial to achieve, besides
> technical measures that helps there, like backward-compatible syscall
> interface, symbol versioning, providing fall-back code for older
> interface, a lot of overhead in the development is enforced, like
> carefull reviews of the changes, the policy and related discipline of
> versioning, following published ABI standards, and so on.
>
> But, internal ABI of the C runtime pack, i.e. interfaces which make
> rtld work with libc and libthr, or way by which libthr, when loaded,
> makes libc thread-aware, are not stable, and more, they are often
> changed in backward-incompatible way. Requiring backward-compatibility
> there would stop our ability to evolve the system. Answering some
> questions in advance, yes, rtld delves into libc, libthr patches libc
> on load, libc has hooks to control some libthr behaviour.
>
> The only provision that we make is that ld-elf.so.1 is required to
> work with older libc/libthr combination, but even then libc and libthr
> must be built from the same sources with the same options set.
>
> Now, returning to pkgbase, if you look at what libs are packed into
> clibs, you see:
> ld-elf.so.1
> libc.so.7 (and modules like iconv tables or nss, if any)
> libthr.so.3
> libdl.so.1
> libgcc{, _eh, _s}.so.1
> libm.so.5
> libedit.so.7
> libncurses{, w}.so.8
> libc++.so.1
> It adds very popular libs like libncurses/libedit, and C++ runtime.
> The basic reasoning is that this package is small and chances of
> something going wrong while installing it are small as result. Put it
> other way, the small clibs package organization makes it highly
> probable that system is left in the consistent state (either all new
> libs, or all old
> libs) after the upgrade, whetever the outcome is.
>
> If the C runtime pack is not split from the whole 700MB+ update blob,
> libthr update has almost certain chance to occur long after or before
> libc update, so failures do tend to leave inconsistent
> rtld/libc/libthr set. At best, it gives you strange glitches, at
> worst you get unusable system that cannot be repaired without external
media.
>
> Now, the second item, the minimal boot path. By definition, it
> consists of everything that is required to get bare-bone shell prompt
> in single user mode, and where user can repair failed upgrade.
> Arguably, it should also include the tools to configure the network
> and fix filesystems. So it should consists of
> loader (including forth/lua scripts)
> kernel
> C runtime
> /sbin/init
> /bin/sh
> newfs/fsck/tunefs for UFS
> zfs/zfspool and libs for ZFS
> ifconfig/route/ping
> In this set, zfs and network management tools must be synced with the
> kernel, since ABI of the management syscalls is not guaranteed to be
> stable even on stable branches.
>
> The above brain dump is at least partial enumeration of things that
> were discussed between me and Glen when Glen created the current
> in-tree packaging code.
Konstantin:
Please read the pkgbase documentation that Kris posted in the CFT
(https://trueos.github.io/pkgbase-docs/). Your issues/questions keeps
referencing the packaging used in the current FreeBSD base-package
implementation instead of the pkgbase system proposed in this CFT.
TLDR: The package format proposed here does not follow the
current/experimental base package format, but rather is a new ports-based
implementation which tries to mimic the traditional distfile outputs of
FreeBSD in package form.
Because this new base package system is governed by ports instead of in-tree
changes to the freebsd source tree itself, this allows for the same base
package implementation to be used on almost any version of FreeBSD that you
like: which is how 12-STABLE and 13-CURRENT package repos were both
trivially created for this CFT.
--
~~ Ken Moore ~~
ken@ixsystems.com
_______________________________________________
freebsd-pkgbase@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-pkgbase
To unsubscribe, send any mail to "freebsd-pkgbase-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?02ef01d4fe90$a997a360$fcc6ea20$>