From owner-freebsd-pkgbase@freebsd.org Wed Mar 2 23:54:31 2016 Return-Path: Delivered-To: freebsd-pkgbase@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58C43AC12E8; Wed, 2 Mar 2016 23:54:31 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 4AA7D1E2B; Wed, 2 Mar 2016 23:54:31 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from FreeBSD.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by freefall.freebsd.org (Postfix) with ESMTP id BEF2C1A3A; Wed, 2 Mar 2016 23:54:30 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Date: Wed, 2 Mar 2016 23:54:29 +0000 From: Glen Barber To: freebsd-current@freebsd.org, freebsd-pkgbase@freebsd.org Subject: [CFT] packaging the base system with pkg(8) Message-ID: <20160302235429.GD75641@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="MIdTMoZhcV1D07fI" Content-Disposition: inline X-Operating-System: FreeBSD 11.0-CURRENT amd64 X-SCUD-Definition: Sudden Completely Unexpected Dataloss X-SULE-Definition: Sudden Unexpected Learning Event X-PEKBAC-Definition: Problem Exists, Keyboard Between Admin/Computer User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-pkgbase@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Packaging the FreeBSD base system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 23:54:31 -0000 --MIdTMoZhcV1D07fI Content-Type: multipart/mixed; boundary="d9ADC0YsG2v16Js0" Content-Disposition: inline --d9ADC0YsG2v16Js0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline For those who have missed the initial email surrounding this topic, we are planning on packaging the base system with pkg(8) for 11.0-RELEASE. https://lists.freebsd.org/pipermail/freebsd-pkgbase/2016-January/000000.html At this time, I believe the major blockers and critical issues have been resolved where it is time for an official call-for-testing. Please note, as with any development branch, this is not yet intended for production environments. Testing on virtual machines or dedicated testing machines is strongly encouraged. Also note (as repeated below), running 'pkg delete -a' will implicitly remove base system packages after they are installed. To obtain the sources for testing, please use the projects/release-pkg branch: # svn co svn://svn.freebsd.org/base/projects/release-pkg /usr/src The projects/release-pkg branch is (at this time) in sync with head revision r296327. After checking out the project branch, build the userland and kernel as normal with the 'buildworld' and 'buildkernel' targets. Afterward, packages can be created with the 'packages' target. # cd /usr/src # make [make flags] buildworld # make [make flags] buildkernel # make packages At present, the base system consists of 755 packages with the default build (empty src.conf(5) and make.conf(5)) for amd64. The number of packages depends on several factors, but for most cases a runtime binary is split into several components. In particular, most shared libraries are individually packaged, in addition to debugging symbols, profiling libraries, and 32-bit packaged separately. The package repository will be created within /usr/obj/usr/src/repo by default. To enable the repository, create /usr/local/etc/pkg/repos/base.conf with the following contents: # FreeBSD base system repository FreeBSD-base: { url: "file:///usr/obj/usr/src/repo/${ABI}/latest", mirror_type: "none", enabled: yes } To initially bootstrap the 'FreeBSD-*' packages, they must be forcibly installed. Package registration is not performed during 'installworld' or 'installkernel', and there are no immediate plans to do this. This can be done by running: # pkg update -r FreeBSD-base # pkg install -g 'FreeBSD-*' Please note the following: 1) The pkg(8) binary is required for this to work, however an additional patch against the ports-mgmt/pkg port is required to properly track base system shared libraries. The patch against the ports-mgmt/pkg port is attached. In my testing, excluding this patch has not caused anything horrible to happen, however applying both patches is suggested at this point. The main noticeable effect of excluding the patch is system binary packages and their dependent library packages are not directly linked, which makes it possible to delete a library package that is required by a runtime binary. 2) At present, running 'pkg delete -a' will implicitly remove the 'FreeBSD-*' packages, leaving the system in an unusable state. There are valid use cases for removing all packages, such as test chroot(8) or jail(8) environments, so a solution to avoid accidental foot shooting is still being investigated. 3) With the attached patch, /lib32 and /usr/lib32 shared libraries are not tracked. Since they are optional and do not affect the default running userland, this should not prevent testing, however it is worth noting. 4) For kernel packages, the first listed kernel in KERNCONF is installed as /boot/kernel, and subsequent kernels in KERNCONF are installed as /boot/kernel.${KERNEL}. Building GENERIC is not required, as each kernel package is named with the kernel name included. For example, if 'KERNCONF=MYLOCALKERNEL' is set in make.conf(5), the resulting kernel package will be 'FreeBSD-kernel-mylocalkernel-release', and the debug symbols as 'FreeBSD-kernel-mylocalkernel-debug'. 5) There are still a few outstanding issues with configuration file merging, which is still being investigated. Please follow up on the freebsd-pkgbase@ mailing list with problems (and successes). Many suggestions were made, such as further granularity between runtime binaries and daemons (rwho(1) and rwhod(8) is one example I recall off-hand), that have not been implemented yet (and also not forgotten). Thank you in advance to everyone that can test this, so we can get this completed in time for 11.0-RELEASE. Glen --d9ADC0YsG2v16Js0 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="pkgpatch.diff.txt" Content-Transfer-Encoding: quoted-printable Index: ports-mgmt/pkg/files/patch-libpkg_pkg__config.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- ports-mgmt/pkg/files/patch-libpkg_pkg__config.c (nonexistent) +++ ports-mgmt/pkg/files/patch-libpkg_pkg__config.c (working copy) @@ -0,0 +1,15 @@ +--- libpkg/pkg_config.c.orig 2016-01-26 23:32:05 UTC ++++ libpkg/pkg_config.c +@@ -390,6 +390,12 @@ static struct config_entry c[] =3D { + "VALID_URL_SCHEME", + "pkg+http,pkg+https,https,http,ftp,file,ssh", + }, ++ { ++ PKG_BOOL, ++ "ALLOW_BASE_SHLIBS", ++ "NO", ++ "Enable base libraries analysis", ++ }, + }; +=20 + static bool parsed =3D false; Index: ports-mgmt/pkg/files/patch-libpkg_pkg__elf.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- ports-mgmt/pkg/files/patch-libpkg_pkg__elf.c (nonexistent) +++ ports-mgmt/pkg/files/patch-libpkg_pkg__elf.c (working copy) @@ -0,0 +1,40 @@ +--- libpkg/pkg_elf.c.orig 2015-09-21 08:53:23 UTC ++++ libpkg/pkg_elf.c +@@ -85,23 +85,28 @@ static int + filter_system_shlibs(const char *name, char *path, size_t pathlen) + { + const char *shlib_path; ++ bool packaging_base =3D pkg_object_bool(pkg_config_get("ALLOW_BASE_SHLIB= S")); +=20 +- shlib_path =3D shlib_list_find_by_name(name); +- if (shlib_path =3D=3D NULL) { +- /* dynamic linker could not resolve */ +- return (EPKG_FATAL); ++ if (!packaging_base) { ++ shlib_path =3D shlib_list_find_by_name(name); ++ if (shlib_path =3D=3D NULL) { ++ /* dynamic linker could not resolve */ ++ return (EPKG_FATAL); ++ } + } +=20 +- /* match /lib, /lib32, /usr/lib and /usr/lib32 */ +- if (strncmp(shlib_path, "/lib", 4) =3D=3D 0 || +- strncmp(shlib_path, "/usr/lib", 8) =3D=3D 0) +- return (EPKG_END); /* ignore libs from base */ ++ if (!packaging_base) { ++ /* match /lib, /lib32, /usr/lib and /usr/lib32 */ ++ if (strncmp(shlib_path, "/lib", 4) =3D=3D 0 || ++ strncmp(shlib_path, "/usr/lib", 8) =3D=3D 0) ++ return (EPKG_END); /* ignore libs from base */ ++ } +=20 + if (path !=3D NULL) + strncpy(path, shlib_path, pathlen); +=20 + return (EPKG_OK); +-}=20 ++} +=20 + /* ARGSUSED */ + static int --d9ADC0YsG2v16Js0-- --MIdTMoZhcV1D07fI Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJW1301AAoJEAMUWKVHj+KT+rwP/3NogwgInBqdjhcsfjE8mCVa ZfrIm8JB6Rrlq53W3T6TTvcGMCZSzSowXL7l5doxJO9FbToU9md1A0iELv0VTQks lgSzQOuArYzRINI9H/PLlmIaMlCJPvk4KJyvZkIj9QgY8Bhmr1AQKNG3KvXBzxo+ NvaKET8GDnbriALn4zaAWNxquPIWCmjsvTg+/B8C40yvyr0roEZbMCH7MFP//lBv B3Ah3UYp8jdtQ5x7d5tCVy919W4cZsHSWWxMdSA6GPbdbxvScVT14v6fTi/jWdQP WaewRsOK22vmoZj0F/oVaQmg9tBCHTiSN5J7cXd7zWNO6LQr+Rfk5xQBgi2SRYdq l5bkr3XmJC3TggYecPNqUDw+CAq/U+KkVqw4Zc8I2aZJW/e1EqMO8fTKUu5Ce8L9 iYKkymHjTPgye/o3rAfHWb7iK3N0vNtdUEShyYStAK3UMOD0NJwDsjrhuQy8P453 2tXxm9JZ2Sdo+Psnf62NgzBwEprkZdENhuFnvYxIOYbij3x5Sf6R0CYgw0TodF6f 9M4J4iINh7fuNORhutjmroSTfGffx5SNQ9fjtk1YV4txRCVi6NsLqedePEQ1S5K0 lEH2ozwr4nv/UNb4RPN/YanbPXOjlbQltDN+3OqWBNwyWM/966zR4KKlCHN9yKnx iR5GzudtG3KrxSAdBUAu =HWlk -----END PGP SIGNATURE----- --MIdTMoZhcV1D07fI--