From owner-freebsd-current@freebsd.org Tue Mar 8 13:18:49 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0C02AC7C9F; Tue, 8 Mar 2016 13:18:49 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id E16792E3; Tue, 8 Mar 2016 13:18:49 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from FreeBSD.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by freefall.freebsd.org (Postfix) with ESMTP id 568391CC8; Tue, 8 Mar 2016 13:18:49 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Date: Tue, 8 Mar 2016 13:18:47 +0000 From: Glen Barber To: Slawa Olhovchenkov Cc: freebsd-current@freebsd.org, freebsd-pkgbase@freebsd.org Subject: Re: [CFT] packaging the base system with pkg(8) Message-ID: <20160308131847.GP1531@FreeBSD.org> References: <20160302235429.GD75641@FreeBSD.org> <20160308124016.GA70809@zxy.spb.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="BZziOT8Kz25R/m/E" Content-Disposition: inline In-Reply-To: <20160308124016.GA70809@zxy.spb.ru> X-Operating-System: FreeBSD 11.0-CURRENT amd64 X-SCUD-Definition: Sudden Completely Unexpected Dataloss X-SULE-Definition: Sudden Unexpected Learning Event X-PEKBAC-Definition: Problem Exists, Keyboard Between Admin/Computer User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Mar 2016 13:18:50 -0000 --BZziOT8Kz25R/m/E Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 08, 2016 at 03:40:16PM +0300, Slawa Olhovchenkov wrote: > On Wed, Mar 02, 2016 at 11:54:29PM +0000, Glen Barber wrote: >=20 > > To obtain the sources for testing, please use the projects/release-pkg > > branch: > >=20 > > # svn co svn://svn.freebsd.org/base/projects/release-pkg /usr/src > >=20 > > The projects/release-pkg branch is (at this time) in sync with head > > revision r296327. > >=20 > > After checking out the project branch, build the userland and kernel as > > normal with the 'buildworld' and 'buildkernel' targets. Afterward, > > packages can be created with the 'packages' target. > >=20 > > # cd /usr/src > > # make [make flags] buildworld > > # make [make flags] buildkernel > > # make packages > >=20 > > At present, the base system consists of 755 packages with the default > > build (empty src.conf(5) and make.conf(5)) for amd64. The number of > > packages depends on several factors, but for most cases a runtime binary > > is split into several components. In particular, most shared libraries > > are individually packaged, in addition to debugging symbols, profiling > > libraries, and 32-bit packaged separately. > >=20 > > The package repository will be created within /usr/obj/usr/src/repo by > > default. >=20 > I am get snapshot .iso for install test setup in VirtualBox and using > projects/release-pkg for sources. After make buildworld buildkernel > packages and pkg install '*' I am have some words. >=20 > I am not developer, I am like maintenance services. I am do maintaing > systems more ten 20 years, some systems maintainig more ten 10 year > continuous, some systems got for maintenance after years unmaintening. > All of this give some requirement and vision different from > developers. Please, do not reject this! >=20 > First, you do collocal work, thanks! >=20 > I am don't check all, but already found some stranges: >=20 > Package FreeBSD-clibs-development contain /usr/lib/libthr.so, > /usr/lib/libedit.so and etc (and same in other packages). >=20 Correct. The libraries in the 'clibs' package must be installed before the rest of the system can be safely upgraded, to ensure consistency with the core runtime libraries. > Misspeling FreeBSD-debug, FreeBSD-development and FreeBSD-profile as > FreeBSD-runtime-debug, FreeBSD-runtime-development and FreeBSD-runtime-pr= ofile? >=20 I am not sure why this was originally named as they are, but this can be fixed. > I am reseach spliting to package and try some aggregation: >=20 > NumPkgs tarSize(MB) flatSize(MB) Aggregation > 1 30.7 102 FreeBSD-kernel-generic-release > 1 57.1 331 FreeBSD-kernel-generic-debug > 1 2.8 5.4 FreeBSD-clibs > 1 3.5 24 FreeBSD-clibs-development > 1 2.4 11 FreeBSD-clibs-debug > 1 1.3 9.8 FreeBSD-clibs-profile > 1 20.7 103 FreeBSD-runtime > 1 5.9 38.1 FreeBSD-development > 1 2.9 2.8 FreeBSD-runtime-manuals > 1 14.9 65 FreeBSD-debug > 1 2.2 12.5 FreeBSD-profile > 1 24.3 93 FreeBSD-clang > 1 8.7 66 FreeBSD-clang-debug > 116 19.0 80 FreeBSD-* > 89 3.2 14 FreeBSD-*-development > 110 12.5 61 FreeBSD-*-debug > 85 2.8 13 FreeBSD-*-profile > 85 6.0 18 FreeBSD-*-lib32-* > 88 7.4 30 FreeBSD-*-lib32-development > 84 11.6 43 FreeBSD-*-lib32-debug > 85 5.8 24 FreeBSD-*-lib32-profile >=20 > I.e -development is substantially less of main package and don't need > separatly (and many .so incorrectly packaging into -development). > Same as -profile vs -debug (and -profile useless w/o -debug). >=20 Regarding the .so files, I am not clear on the original intent behind separating the actual shared library from the installed symbolic link to the real shared library, but in my investigation into this, only the symlinks are provided by the '-development' package. For example: root@pkgbase:/ # file /usr/lib/libjail.so /usr/lib/libjail.so: symbolic link to ../../lib/libjail.so.1 root@pkgbase:/ # pkg shlib libjail.so No packages provide libjail.so. No packages require libjail.so. root@pkgbase:/ # pkg shlib libjail.so.1 libjail.so.1 is provided by the following packages: FreeBSD-jail-11.0.s20160303044402 FreeBSD-jail-lib32-11.0.s20160303044402 libjail.so.1 is linked to by the following packages: FreeBSD-runtime-11.0.s20160303044402 Moving them to the package that installs the shared library itself should be fairly easy to do. Regarding '-profile' and '-debug' package separation, it is possible to install the debugging files without requiring the profiling libraries now, so I think keeping them as separate packages is the best way to achieve this. (Note, profiling libraries will not be installed with WITHOUT_PROFILE=3D1 in src.conf(5)). > Manual must be installed always, IMHO (size is small and version of > manual must matcj version of utility). >=20 This is similarly broken up by package, at least where it has been found so far. The jail.conf(5) manual is only installed if the 'FreeBSD-jail' package is installed, for example. root@pkgbase:/ # pkg which /usr/share/man/man5/jail.conf.5.gz /usr/share/man/man5/jail.conf.5.gz was installed by package FreeBSD-jail-11.0.s20160303044402 > Packaging of individual utilites is useless (total 19MB vs > 30.7+2.8+20.7+2.9) and incorrect (for example, WITHOUT_ACCT not only > don't build accton/lastcomm/sa but also cut off accaunting code from > kernel for space saving and perforamce). >=20 Packaging individual utilities is not useless, depending on who you ask. One of the first replies I received when starting separating userland utilities into separate packages was further splitting rwho(1) and rwhod(8) into different packages, the use case being not necessarily needing (or wanting) the rwho(1) utility on systems where rwhod(8) runs. > I am propose don't distinct profile and debug, development and main > package. >=20 > I am propose divide only to FreeBSD-kernel, FreeBSD-clibs (clibs, > runtime and manuals), FreeBSD-clang, FreeBSD-lib32. >=20 This will make updates for SAs and ENs too large. This is part of the reason the packages were split up as they have been so far (and will continue to be further split as progress is made). If the argument is simply "there are too many packages", see one of the previous replies in this thread that discusses the background on why this decision was made. But that aside, trying to make everyone happy will turn out to make no one happy. > Dividing to many packages is anoyning on install and maintancing (what > exact keys of this utilites this version?! stupid admin don't install > manuals!) >=20 > About use cases. I am try to imagine different use cases and don't > found answer how do this: >=20 > 1. package building as `make packages` witch version as timestamp of > start buildworld. I.e. on every buildworld every package will be > rebuild, take new version and will be reinstaled. Where is profit of > package spliting? >=20 This is the case for 11-CURRENT. The PKG_VERSION evaluates the BRANCH =66rom Makefile.inc1 to determine if it should use the timestamp. (Since -CURRENT is a fast-moving target, and we recommend keeping the userland and kernel in sync, this makes sense, at least to me.) > 2. After src.conf change some package don't build. Where analog of > `make delete-old delete-old-libs`? >=20 I believe 'pkg autoremove' should handle this, but I will check. > 3. After src.conf chanege some (WITHOUT_ACCT for example) some > packages can't be installed. How handle this? >=20 Have you run into this? If so, it needs to be investigated. But otherwise, the generated packages respect make.conf(5) and src.conf(5), so this should not happen in theory. > 4. How install debug symbols after installing separately set of > packages? Not all *-debug*- and don't selecting all 200 packages > individualy? >=20 Could you clarify what you mean a bit more? Specifically, I am unclear of the order of events you mean. > 5. Take system installed by unknow person (ex: ISP support 5 years > ago). Try to write program. Don't find nothing for this. Version is > 11.0-BETA3. How to install required packages? For system w/o inetrnet > connectivity and with lost install media? >=20 The plan at the moment is to include pkg(8) in the repository with the base system. This will also be required for architectures we do not have an upstream package repository (powerpc, for example). This is handled by the 'package-pkg' target at the moment. > As for windows (please found in garbage CD with exact version and > insert in lost CD-ROM)? >=20 > May be preserve for this all packages in some places on HDD? >=20 I am having trouble parsing these two sentences. Could you please clarify? > 6. pkg which /etc/ssh/ssh_config > /etc/ssh/ssh_config was not found in the database >=20 There are may occurrences of this at the moment. Configuration file merging needs to be resolved in pkg(8) before we can safely add files like sshd_config(8) to the 'FreeBSD-ssh' package. > What?! How to handle this? What is instead of mergemaster? > (etcupdate currently to dangerous, I am totaly destroed one host by > etcupdate. Also, current database of etcupdate is very strange). We use etcupdate(8) in the FreeBSD infrastructure, and have not run into such problems. Glen --BZziOT8Kz25R/m/E Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJW3tExAAoJEAMUWKVHj+KT3pYP/3suTcQjgzGTFouvk5sqlGpr JTR3rkC1/SxWY7gymHJaHTWpohaWQGv0omCkSXW38/P1UH6Udi2Z3gUDjo47ViC7 M3D4eggxZB8DZJ5fhUU2If1aeoLaEST4CVVHGy4aYiVgyfU5J0YraAaNuhrp6Lao Q9eM9xXhB7M5cfoth+TFIFezRFIqQuORpxBUqQfYg7b19QTIx2uOGminJ2ShvAZW tp6ZWURIkGMcL+0OxbJFYMKqbWeM6GHYAPzx81le3L3dVcWcT3VNpJbHrkmOoLqG 3Wx/mR194Zes24WRlmyAFCcMDHrdSUtqeeC2jqU2icGMjiqkOBQExty3siYiRh9y q0MUpt83g0VTHjex70LR4q0PS7k4WzqwCLj/+PtloNmVVEaeR+WqgddxASM6z1Jt IzHQwcIzJsttzR4Tcgeax5KKsgrdOkj5HAz4HajuZ43Op+DdL2CkmonHiqWQG9IS 5m1ItvR1gvoP/R37wBFyPR7IQhHCwEkxXyxFzKYZznvUxEsMegPOs0DHl+/mF4um a/NWyFoQCUY0wBEJ4BPhnVsl6ECJYy/v4NTycyVdW4g8M+Pgr1LNLPF6NyMk3NWX sDmPQ8ppS8AhPmR4YgDt8pH+LN97YSqh5dHirvToJzZmzq/Pz7NRcviGgaofMHKZ P3A6nH0Qfq8gSDN2l6D9 =XRT/ -----END PGP SIGNATURE----- --BZziOT8Kz25R/m/E--