Date: Mon, 13 Nov 2017 17:20:54 +0000 From: Glen Barber <gjb@FreeBSD.org> To: freebsd-pkgbase@freebsd.org Cc: John Baldwin <jhb@FreeBSD.org> Subject: Re: Problems with packaging /etc and etcupdate(8) Message-ID: <20171113172054.GA23670@FreeBSD.org> In-Reply-To: <20170927163012.GA99112@FreeBSD.org> References: <20170927163012.GA99112@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--2fHTh5uZTiUOsy+g Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Adding jhb, in case I'm doing something wrong with etcupdate(8) in this approach, but gratuitously pinging the list to solicit ideas. Glen On Wed, Sep 27, 2017 at 04:30:12PM +0000, Glen Barber wrote: > Hi, >=20 > Over the past two weeks, I spent quite a bit of time looking into using > etcupdate(8) to bootstrap /var/db/etcupdate for use with pkg(8), and > here are some of the issues I have run into. The most current diff > tested is attached. >=20 > 1) A new 'create-packages-files' target was added, which invokes the > etcupdate(8) 'extract' command using WSTAGEDIR for the DESTDIR. > Despite passing 'TAGS=3Dpackage=3Detc' and 'PACKAGE=3Detc' to etcupdat= e(8) > '-M' flag, in addition to '-DNO_ROOT', some files still fail to be > tagged in WSTAGEDIR/var/db/etcupdate/current/METALOG. To work around > this, a hack was added to release/scripts/mtree-to-plist.awk to trap > and tag files when '-v etcupdate=3Dyes' is passed to awk(1) in this > target. >=20 > 2) Some files do not automatically get created by etcupdate(8), as noted > in the manual page. For example, pwd_mkdb(8), cap_mkdb(1), > newaliases(1), tzsetup(8), and services_mkdb(8) are invoked after > merging files with the newly-created tree. This causes, for example, > files like /etc/passwd to be nonexistent when creating a package. To > work around this, another hack was added to the mtree-to-plist.awk > script to skip nonexistent files in WSTAGEDIR/var/db/etcupdate. This > is still non-ideal, I think. >=20 > 3) Setting 'prefix =3D /var/db/etcupdate' in the release/packages/etc.ucl > file appears to be ignored. Files in the 'etc' package conflict with > files installed by other packages, such as runtime, autofs, and so > on, with conflicting files being things like /etc/autofs/include_ldap > or other similarly-installed files within /etc. To work around this > when packaging files, a hack was added to the mtree-to-plist.awk > script (again) to prepend '/var/db/etcupdate' to the file path in > order to get the correct files included in the package. (This is > somewhat in parallel with (2) above.) >=20 > 4) When running etcupdate(8) after hacking around some of the issues > above, I discovered that /var/db/etcupdate/current in the root of the > system in which it is being run disappears. Additionally, I had to > restore files within /etc (such as master.passwd(5), passwd(5), etc.) > in order to make my test system usable again. In addition to > replacing files that should have been merged, 'pkg check' against the > 'FreeBSD-etc' package reports all of the files from the 'current' > directory have vanished. >=20 > In addition to the above items, I also did some (albeit minimal) testing > with mergemaster(8), as it creates /var/db/mergemaster.mtree, however in > my brief testing it seems the mtree(8) file created is not compatible > with pkg(8) or the mtree-to-plist.awk script. (Note: the mergemaster(8) > bits from my testing are not included in the diff, as I had to revert > a number of changes to ensure the correct diff was included in this > email. >=20 > I am at the point now where I am out of ideas on how to proceed, and > could use some help on figuring this out. Any help on the above issues, > or anything in the Wiki would be greatly appreciated. >=20 > https://wiki.freebsd.org/PkgBase >=20 > Thanks, >=20 > Glen >=20 > Index: Makefile > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- Makefile (revision 324062) > +++ Makefile (working copy) > @@ -132,6 +132,7 @@ > builddtb xdev xdev-build xdev-install \ > xdev-links native-xtools stageworld stagekernel stage-packages \ > create-packages-world create-packages-kernel create-packages \ > + create-files-packages \ > packages installconfig real-packages sign-packages package-pkg \ > print-dir test-system-compiler > =20 > Index: Makefile.inc1 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- Makefile.inc1 (revision 324063) > +++ Makefile.inc1 (working copy) > @@ -1555,6 +1555,7 @@ > =20 > .ORDER: stage-packages create-packages > .ORDER: create-packages create-world-packages > +.ORDER: create-packages create-files-packages > .ORDER: create-packages create-kernel-packages > .ORDER: create-packages sign-packages > =20 > @@ -1596,6 +1597,12 @@ > DESTDIR=3D${WSTAGEDIR} \ > PKG_VERSION=3D${PKG_VERSION} create-world-packages > =20 > +create-packages-files: _pkgbootstrap _repodir .PHONY > + ${_+_}@cd ${.CURDIR}; \ > + ${MAKE} -f Makefile.inc1 \ > + DESTDIR=3D${WSTAGEDIR} \ > + PKG_VERSION=3D${PKG_VERSION} create-files-packages > + > create-packages-kernel: _pkgbootstrap _repodir .PHONY > ${_+_}@cd ${.CURDIR}; \ > ${MAKE} -f Makefile.inc1 \ > @@ -1603,8 +1610,25 @@ > PKG_VERSION=3D${PKG_VERSION} DISTDIR=3Dkernel \ > create-kernel-packages > =20 > -create-packages: .PHONY create-packages-world create-packages-kernel > +create-packages: .PHONY create-packages-world create-packages-files crea= te-packages-kernel > =20 > +create-files-packages: _pkgbootstrap .PHONY > + mkdir -p ${WSTAGEDIR}/var/db/etcupdate/current > + cd ${WSTAGEDIR} ; \ > + ${WSTAGEDIR}/usr/sbin/etcupdate extract -B -M \ > + "TARGET=3D${TARGET} TARGET_ARCH=3D${TARGET_ARCH} -DNO_ROOT TAGS=3Dpack= age=3Detc PACKAGE=3Detc" \ > + -s ${SRCDIR} -d ${WSTAGEDIR}/var/db/etcupdate > + awk -f ${SRCDIR}/release/scripts/mtree-to-plist.awk \ > + -v etcupdate=3Dyes -v ws=3D${WSTAGEDIR} \ > + ${WSTAGEDIR}/var/db/etcupdate/current/METALOG > + sh ${SRCDIR}/release/packages/generate-ucl.sh -o etc \ > + -s ${SRCDIR} -u ${WSTAGEDIR}/etc.ucl > + ${PKG_CMD} -o ABI_FILE=3D${WSTAGEDIR}/bin/sh -o ALLOW_BASE_SHLIBS=3Dyes= \ > + create -M ${WSTAGEDIR}/etc.ucl \ > + -p ${WSTAGEDIR}/etc.plist \ > + -r ${WSTAGEDIR} \ > + -o ${REPODIR}/$$(${PKG_CMD} -o ABI_FILE=3D${WSTAGEDIR}/bin/sh config A= BI)/${PKG_VERSION} > + > create-world-packages: _pkgbootstrap .PHONY > @rm -f ${WSTAGEDIR}/*.plist 2>/dev/null || : > @cd ${WSTAGEDIR} ; \ > Index: release/packages/etc.ucl > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- release/packages/etc.ucl (nonexistent) > +++ release/packages/etc.ucl (working copy) > @@ -0,0 +1,22 @@ > +# > +# $FreeBSD: head/release/packages/runtime.ucl 322327 2017-08-09 19:16:54= Z gjb $ > +# > + > +name =3D "FreeBSD-etc" > +origin =3D "base" > +version =3D "%VERSION%" > +comment =3D "etc package" > +categories =3D [ base ] > +maintainer =3D "re@FreeBSD.org" > +www =3D "https://www.FreeBSD.org" > +prefix =3D "/var/db/etcupdate" > +licenselogic =3D "single" > +licenses =3D [ BSD2CLAUSE ] > +desc =3D <<EOD > +etc package > +EOD > +scripts: { > + post-install =3D <<EOD > + echo "Please use etcupdate(8) or mergemaster(8) to update" > + echo "files under /etc." > +EOD > Index: release/scripts/mtree-to-plist.awk > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- release/scripts/mtree-to-plist.awk (revision 324062) > +++ release/scripts/mtree-to-plist.awk (working copy) > @@ -28,6 +28,13 @@ > tags=3Dtags""_kernconf > } > } > + if (etcupdate !=3D "") { > + if (length(gsub(/tags=3D/, "", tags) =3D=3D 0)) > + tags=3D"package=3Detc" > + if ((length(type) !=3D 0) && (type =3D=3D "dir")) { > + next > + } > + } > if (length(tags) =3D=3D 0) > next > if (tags ~ /package=3D/) { > @@ -76,5 +83,17 @@ > output=3Dpkgname".plist" > } > =20 > + if (etcupdate !=3D "") { > + pfx=3D"/var/db/etcupdate/current" > + filename=3Dws""pfx""$1 > + if (system("test -e " filename) =3D=3D 0) { > + print "@"type"("uname","gname","mode","flags") " pfx""$1 > output > + next > + } else { > + next > + } > + next > + } > + > print "@"type"("uname","gname","mode","flags") " $1 > output > } --2fHTh5uZTiUOsy+g Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAloJ1HYACgkQAxRYpUeP 4pO/mA//Zd87xE02szNGyV0b2CqbxJUhxO4shRJIkTBS7hI0nrDSFysbm4/zQ/s6 pX+iZqmwwD7N/YdhdsuWHUBJgmo5NchioXMDy0JkuMj/Cj5B6MjjUws72zcgEtXn Ajk9iEgnucuPzuF9c8d20R86I7Q0sDJMD12M1fki2YsgX+Ly5L3D5RR8gk8zNZbQ 5xV6tLKaWRaETKdcq118DkDTqM/XM1aHtQA+4D4dVx42Ne2LtML9QTqrQ/xN07pq A6XL5m6p36lGOxZ2z3KoLXrDg2zCMDan1Z+oOEIvxTIHckMjHePh6ovUVmYk6dA1 2SF0jU2DploZF/qufv2xleRjgslDPsfx8NEcHeGBXfuWgQZP0q93aXXDPB4kAkKh YwcXxZ7KMLCWv0U3PNVIOz2WZOZfAadfyyyg1+MOo+nD7lWkDBRZQWrNw8E4dceG ucl9RfVdDopWCL74in7UVnbMAzTUeJVdQY7nSHs5qSKVW611P0xC3FhZZHYOCdL/ WTsm078Zq7jaz2d5WvU61W01/mntXFsOFj3N3uRuG264fLs9F3ccu86Wx4MUzPVS AEA3/B46CHJ2qMIQVB6XeFz0Ny+lPcBVqJJ4rj4OD+O0zYgRC5agxWFabRyMq64x RyWVLQTwhgxVGYqZGtH7W5/O9ee/WyUt8b6w8YlJrfdlpRWrrEY= =fWZi -----END PGP SIGNATURE----- --2fHTh5uZTiUOsy+g--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20171113172054.GA23670>