From owner-freebsd-pkgbase@freebsd.org Wed Sep 27 16:30:15 2017 Return-Path: Delivered-To: freebsd-pkgbase@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F541E0884E for ; Wed, 27 Sep 2017 16:30:15 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7CC487DBD7; Wed, 27 Sep 2017 16:30:15 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Received: from FreeBSD.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id 8A10E9D06; Wed, 27 Sep 2017 16:30:14 +0000 (UTC) (envelope-from gjb@FreeBSD.org) Date: Wed, 27 Sep 2017 16:30:12 +0000 From: Glen Barber To: freebsd-pkgbase@freebsd.org Subject: Problems with packaging /etc and etcupdate(8) Message-ID: <20170927163012.GA99112@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="DKU6Jbt7q3WqK7+M" Content-Disposition: inline User-Agent: Mutt/1.8.2 (2017-04-18) X-BeenThere: freebsd-pkgbase@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Packaging the FreeBSD base system." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Sep 2017 16:30:15 -0000 --DKU6Jbt7q3WqK7+M Content-Type: multipart/mixed; boundary="Nq2Wo0NMKNjxTN9z" Content-Disposition: inline --Nq2Wo0NMKNjxTN9z Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, Over the past two weeks, I spent quite a bit of time looking into using etcupdate(8) to bootstrap /var/db/etcupdate for use with pkg(8), and here are some of the issues I have run into. The most current diff tested is attached. 1) A new 'create-packages-files' target was added, which invokes the etcupdate(8) 'extract' command using WSTAGEDIR for the DESTDIR. Despite passing 'TAGS=package=etc' and 'PACKAGE=etc' to etcupdate(8) '-M' flag, in addition to '-DNO_ROOT', some files still fail to be tagged in WSTAGEDIR/var/db/etcupdate/current/METALOG. To work around this, a hack was added to release/scripts/mtree-to-plist.awk to trap and tag files when '-v etcupdate=yes' is passed to awk(1) in this target. 2) Some files do not automatically get created by etcupdate(8), as noted in the manual page. For example, pwd_mkdb(8), cap_mkdb(1), newaliases(1), tzsetup(8), and services_mkdb(8) are invoked after merging files with the newly-created tree. This causes, for example, files like /etc/passwd to be nonexistent when creating a package. To work around this, another hack was added to the mtree-to-plist.awk script to skip nonexistent files in WSTAGEDIR/var/db/etcupdate. This is still non-ideal, I think. 3) Setting 'prefix = /var/db/etcupdate' in the release/packages/etc.ucl file appears to be ignored. Files in the 'etc' package conflict with files installed by other packages, such as runtime, autofs, and so on, with conflicting files being things like /etc/autofs/include_ldap or other similarly-installed files within /etc. To work around this when packaging files, a hack was added to the mtree-to-plist.awk script (again) to prepend '/var/db/etcupdate' to the file path in order to get the correct files included in the package. (This is somewhat in parallel with (2) above.) 4) When running etcupdate(8) after hacking around some of the issues above, I discovered that /var/db/etcupdate/current in the root of the system in which it is being run disappears. Additionally, I had to restore files within /etc (such as master.passwd(5), passwd(5), etc.) in order to make my test system usable again. In addition to replacing files that should have been merged, 'pkg check' against the 'FreeBSD-etc' package reports all of the files from the 'current' directory have vanished. In addition to the above items, I also did some (albeit minimal) testing with mergemaster(8), as it creates /var/db/mergemaster.mtree, however in my brief testing it seems the mtree(8) file created is not compatible with pkg(8) or the mtree-to-plist.awk script. (Note: the mergemaster(8) bits from my testing are not included in the diff, as I had to revert a number of changes to ensure the correct diff was included in this email. I am at the point now where I am out of ideas on how to proceed, and could use some help on figuring this out. Any help on the above issues, or anything in the Wiki would be greatly appreciated. https://wiki.freebsd.org/PkgBase Thanks, Glen --Nq2Wo0NMKNjxTN9z Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="etc.txt" Content-Transfer-Encoding: quoted-printable Index: Makefile =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- Makefile (revision 324062) +++ Makefile (working copy) @@ -132,6 +132,7 @@ builddtb xdev xdev-build xdev-install \ xdev-links native-xtools stageworld stagekernel stage-packages \ create-packages-world create-packages-kernel create-packages \ + create-files-packages \ packages installconfig real-packages sign-packages package-pkg \ print-dir test-system-compiler =20 Index: Makefile.inc1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- Makefile.inc1 (revision 324063) +++ Makefile.inc1 (working copy) @@ -1555,6 +1555,7 @@ =20 .ORDER: stage-packages create-packages .ORDER: create-packages create-world-packages +.ORDER: create-packages create-files-packages .ORDER: create-packages create-kernel-packages .ORDER: create-packages sign-packages =20 @@ -1596,6 +1597,12 @@ DESTDIR=3D${WSTAGEDIR} \ PKG_VERSION=3D${PKG_VERSION} create-world-packages =20 +create-packages-files: _pkgbootstrap _repodir .PHONY + ${_+_}@cd ${.CURDIR}; \ + ${MAKE} -f Makefile.inc1 \ + DESTDIR=3D${WSTAGEDIR} \ + PKG_VERSION=3D${PKG_VERSION} create-files-packages + create-packages-kernel: _pkgbootstrap _repodir .PHONY ${_+_}@cd ${.CURDIR}; \ ${MAKE} -f Makefile.inc1 \ @@ -1603,8 +1610,25 @@ PKG_VERSION=3D${PKG_VERSION} DISTDIR=3Dkernel \ create-kernel-packages =20 -create-packages: .PHONY create-packages-world create-packages-kernel +create-packages: .PHONY create-packages-world create-packages-files create= -packages-kernel =20 +create-files-packages: _pkgbootstrap .PHONY + mkdir -p ${WSTAGEDIR}/var/db/etcupdate/current + cd ${WSTAGEDIR} ; \ + ${WSTAGEDIR}/usr/sbin/etcupdate extract -B -M \ + "TARGET=3D${TARGET} TARGET_ARCH=3D${TARGET_ARCH} -DNO_ROOT TAGS=3Dpackag= e=3Detc PACKAGE=3Detc" \ + -s ${SRCDIR} -d ${WSTAGEDIR}/var/db/etcupdate + awk -f ${SRCDIR}/release/scripts/mtree-to-plist.awk \ + -v etcupdate=3Dyes -v ws=3D${WSTAGEDIR} \ + ${WSTAGEDIR}/var/db/etcupdate/current/METALOG + sh ${SRCDIR}/release/packages/generate-ucl.sh -o etc \ + -s ${SRCDIR} -u ${WSTAGEDIR}/etc.ucl + ${PKG_CMD} -o ABI_FILE=3D${WSTAGEDIR}/bin/sh -o ALLOW_BASE_SHLIBS=3Dyes \ + create -M ${WSTAGEDIR}/etc.ucl \ + -p ${WSTAGEDIR}/etc.plist \ + -r ${WSTAGEDIR} \ + -o ${REPODIR}/$$(${PKG_CMD} -o ABI_FILE=3D${WSTAGEDIR}/bin/sh config ABI= )/${PKG_VERSION} + create-world-packages: _pkgbootstrap .PHONY @rm -f ${WSTAGEDIR}/*.plist 2>/dev/null || : @cd ${WSTAGEDIR} ; \ Index: release/packages/etc.ucl =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- release/packages/etc.ucl (nonexistent) +++ release/packages/etc.ucl (working copy) @@ -0,0 +1,22 @@ +# +# $FreeBSD: head/release/packages/runtime.ucl 322327 2017-08-09 19:16:54Z = gjb $ +# + +name =3D "FreeBSD-etc" +origin =3D "base" +version =3D "%VERSION%" +comment =3D "etc package" +categories =3D [ base ] +maintainer =3D "re@FreeBSD.org" +www =3D "https://www.FreeBSD.org" +prefix =3D "/var/db/etcupdate" +licenselogic =3D "single" +licenses =3D [ BSD2CLAUSE ] +desc =3D < output + next + } else { + next + } + next + } + print "@"type"("uname","gname","mode","flags") " $1 > output } --Nq2Wo0NMKNjxTN9z-- --DKU6Jbt7q3WqK7+M Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEVVuz/A7vpH93hEhKuWzd6q+LXtAFAlnL0hQACgkQuWzd6q+L XtBAYA/8Dai/lJ7VM+7uks8oULtpSz3IZFfkApxBXGYFvS+Ptpryjjk681x/s+23 vGf7ua1x+E8U0t68YZCWNyc1fBtj9aoQmnTA66Dm8SJu/i1c4jzbZmRP3dl31RFU NUkCrewIfTkhMoJbvFyv5tdFVcIpSDrwtAKVTDCJ4UQJXfSFcOtGfAToPB9x1EOy oYHOOh8TxoSGPE3sZv4mV87nJRKoIzlw2ueRKFB6IghKHvWpBb0ROK37g5HhPS4g F8kvVMmhYBPnFF+wZtI99ZH3Zv4ekGj0V9gNi2WyV3XYNkRAsSmCJfdpTuBYpcux s0HpCOEDH8/wa3rKuLq93/2v/dQ6yX6i0ZwmkYR+zbZQJaievs1q0PHq9zVctxp/ wSiljzSixGNmFZe+S/Gi4juBLTzK+dpl4qHVJZJKArla5A2j1xfAimo3WrcE0W/C OBVMEiRWk43BNWgsBy9Cz2N7od+6ffRKAmg8mFZ5N5hkK7omH8E4vOWlY5PjFwwO iXNtwji16b0bbDSKqS2I0nXF7ouWa/MLXoMsuJzvM88TovVRQ95Etgx+wD7s5Y3t k46ib7l/8ZK/nXJtxgI4PfPOfBcHpzJxskkS6j9af+KIQ7OUbQi5RjowuEWtSuuT YftNdz3eFr5lxA3cFKlU4RnqnMoJ3tvKfgSp2BbgMcjDy0JmYbo= =E4QT -----END PGP SIGNATURE----- --DKU6Jbt7q3WqK7+M--