Date: Mon, 13 Nov 2017 17:20:54 +0000 From: Glen Barber <gjb@FreeBSD.org> To: freebsd-pkgbase@freebsd.org Cc: John Baldwin <jhb@FreeBSD.org> Subject: Re: Problems with packaging /etc and etcupdate(8) Message-ID: <20171113172054.GA23670@FreeBSD.org> In-Reply-To: <20170927163012.GA99112@FreeBSD.org> References: <20170927163012.GA99112@FreeBSD.org>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] Adding jhb, in case I'm doing something wrong with etcupdate(8) in this approach, but gratuitously pinging the list to solicit ideas. Glen On Wed, Sep 27, 2017 at 04:30:12PM +0000, Glen Barber wrote: > Hi, > > Over the past two weeks, I spent quite a bit of time looking into using > etcupdate(8) to bootstrap /var/db/etcupdate for use with pkg(8), and > here are some of the issues I have run into. The most current diff > tested is attached. > > 1) A new 'create-packages-files' target was added, which invokes the > etcupdate(8) 'extract' command using WSTAGEDIR for the DESTDIR. > Despite passing 'TAGS=package=etc' and 'PACKAGE=etc' to etcupdate(8) > '-M' flag, in addition to '-DNO_ROOT', some files still fail to be > tagged in WSTAGEDIR/var/db/etcupdate/current/METALOG. To work around > this, a hack was added to release/scripts/mtree-to-plist.awk to trap > and tag files when '-v etcupdate=yes' is passed to awk(1) in this > target. > > 2) Some files do not automatically get created by etcupdate(8), as noted > in the manual page. For example, pwd_mkdb(8), cap_mkdb(1), > newaliases(1), tzsetup(8), and services_mkdb(8) are invoked after > merging files with the newly-created tree. This causes, for example, > files like /etc/passwd to be nonexistent when creating a package. To > work around this, another hack was added to the mtree-to-plist.awk > script to skip nonexistent files in WSTAGEDIR/var/db/etcupdate. This > is still non-ideal, I think. > > 3) Setting 'prefix = /var/db/etcupdate' in the release/packages/etc.ucl > file appears to be ignored. Files in the 'etc' package conflict with > files installed by other packages, such as runtime, autofs, and so > on, with conflicting files being things like /etc/autofs/include_ldap > or other similarly-installed files within /etc. To work around this > when packaging files, a hack was added to the mtree-to-plist.awk > script (again) to prepend '/var/db/etcupdate' to the file path in > order to get the correct files included in the package. (This is > somewhat in parallel with (2) above.) > > 4) When running etcupdate(8) after hacking around some of the issues > above, I discovered that /var/db/etcupdate/current in the root of the > system in which it is being run disappears. Additionally, I had to > restore files within /etc (such as master.passwd(5), passwd(5), etc.) > in order to make my test system usable again. In addition to > replacing files that should have been merged, 'pkg check' against the > 'FreeBSD-etc' package reports all of the files from the 'current' > directory have vanished. > > In addition to the above items, I also did some (albeit minimal) testing > with mergemaster(8), as it creates /var/db/mergemaster.mtree, however in > my brief testing it seems the mtree(8) file created is not compatible > with pkg(8) or the mtree-to-plist.awk script. (Note: the mergemaster(8) > bits from my testing are not included in the diff, as I had to revert > a number of changes to ensure the correct diff was included in this > email. > > I am at the point now where I am out of ideas on how to proceed, and > could use some help on figuring this out. Any help on the above issues, > or anything in the Wiki would be greatly appreciated. > > https://wiki.freebsd.org/PkgBase > > Thanks, > > Glen > > Index: Makefile > =================================================================== > --- Makefile (revision 324062) > +++ Makefile (working copy) > @@ -132,6 +132,7 @@ > builddtb xdev xdev-build xdev-install \ > xdev-links native-xtools stageworld stagekernel stage-packages \ > create-packages-world create-packages-kernel create-packages \ > + create-files-packages \ > packages installconfig real-packages sign-packages package-pkg \ > print-dir test-system-compiler > > Index: Makefile.inc1 > =================================================================== > --- Makefile.inc1 (revision 324063) > +++ Makefile.inc1 (working copy) > @@ -1555,6 +1555,7 @@ > > .ORDER: stage-packages create-packages > .ORDER: create-packages create-world-packages > +.ORDER: create-packages create-files-packages > .ORDER: create-packages create-kernel-packages > .ORDER: create-packages sign-packages > > @@ -1596,6 +1597,12 @@ > DESTDIR=${WSTAGEDIR} \ > PKG_VERSION=${PKG_VERSION} create-world-packages > > +create-packages-files: _pkgbootstrap _repodir .PHONY > + ${_+_}@cd ${.CURDIR}; \ > + ${MAKE} -f Makefile.inc1 \ > + DESTDIR=${WSTAGEDIR} \ > + PKG_VERSION=${PKG_VERSION} create-files-packages > + > create-packages-kernel: _pkgbootstrap _repodir .PHONY > ${_+_}@cd ${.CURDIR}; \ > ${MAKE} -f Makefile.inc1 \ > @@ -1603,8 +1610,25 @@ > PKG_VERSION=${PKG_VERSION} DISTDIR=kernel \ > create-kernel-packages > > -create-packages: .PHONY create-packages-world create-packages-kernel > +create-packages: .PHONY create-packages-world create-packages-files create-packages-kernel > > +create-files-packages: _pkgbootstrap .PHONY > + mkdir -p ${WSTAGEDIR}/var/db/etcupdate/current > + cd ${WSTAGEDIR} ; \ > + ${WSTAGEDIR}/usr/sbin/etcupdate extract -B -M \ > + "TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} -DNO_ROOT TAGS=package=etc PACKAGE=etc" \ > + -s ${SRCDIR} -d ${WSTAGEDIR}/var/db/etcupdate > + awk -f ${SRCDIR}/release/scripts/mtree-to-plist.awk \ > + -v etcupdate=yes -v ws=${WSTAGEDIR} \ > + ${WSTAGEDIR}/var/db/etcupdate/current/METALOG > + sh ${SRCDIR}/release/packages/generate-ucl.sh -o etc \ > + -s ${SRCDIR} -u ${WSTAGEDIR}/etc.ucl > + ${PKG_CMD} -o ABI_FILE=${WSTAGEDIR}/bin/sh -o ALLOW_BASE_SHLIBS=yes \ > + create -M ${WSTAGEDIR}/etc.ucl \ > + -p ${WSTAGEDIR}/etc.plist \ > + -r ${WSTAGEDIR} \ > + -o ${REPODIR}/$$(${PKG_CMD} -o ABI_FILE=${WSTAGEDIR}/bin/sh config ABI)/${PKG_VERSION} > + > create-world-packages: _pkgbootstrap .PHONY > @rm -f ${WSTAGEDIR}/*.plist 2>/dev/null || : > @cd ${WSTAGEDIR} ; \ > Index: release/packages/etc.ucl > =================================================================== > --- release/packages/etc.ucl (nonexistent) > +++ release/packages/etc.ucl (working copy) > @@ -0,0 +1,22 @@ > +# > +# $FreeBSD: head/release/packages/runtime.ucl 322327 2017-08-09 19:16:54Z gjb $ > +# > + > +name = "FreeBSD-etc" > +origin = "base" > +version = "%VERSION%" > +comment = "etc package" > +categories = [ base ] > +maintainer = "re@FreeBSD.org" > +www = "https://www.FreeBSD.org" > +prefix = "/var/db/etcupdate" > +licenselogic = "single" > +licenses = [ BSD2CLAUSE ] > +desc = <<EOD > +etc package > +EOD > +scripts: { > + post-install = <<EOD > + echo "Please use etcupdate(8) or mergemaster(8) to update" > + echo "files under /etc." > +EOD > Index: release/scripts/mtree-to-plist.awk > =================================================================== > --- release/scripts/mtree-to-plist.awk (revision 324062) > +++ release/scripts/mtree-to-plist.awk (working copy) > @@ -28,6 +28,13 @@ > tags=tags""_kernconf > } > } > + if (etcupdate != "") { > + if (length(gsub(/tags=/, "", tags) == 0)) > + tags="package=etc" > + if ((length(type) != 0) && (type == "dir")) { > + next > + } > + } > if (length(tags) == 0) > next > if (tags ~ /package=/) { > @@ -76,5 +83,17 @@ > output=pkgname".plist" > } > > + if (etcupdate != "") { > + pfx="/var/db/etcupdate/current" > + filename=ws""pfx""$1 > + if (system("test -e " filename) == 0) { > + print "@"type"("uname","gname","mode","flags") " pfx""$1 > output > + next > + } else { > + next > + } > + next > + } > + > print "@"type"("uname","gname","mode","flags") " $1 > output > } [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEjRJAPC5sqwhs9k2jAxRYpUeP4pMFAloJ1HYACgkQAxRYpUeP 4pO/mA//Zd87xE02szNGyV0b2CqbxJUhxO4shRJIkTBS7hI0nrDSFysbm4/zQ/s6 pX+iZqmwwD7N/YdhdsuWHUBJgmo5NchioXMDy0JkuMj/Cj5B6MjjUws72zcgEtXn Ajk9iEgnucuPzuF9c8d20R86I7Q0sDJMD12M1fki2YsgX+Ly5L3D5RR8gk8zNZbQ 5xV6tLKaWRaETKdcq118DkDTqM/XM1aHtQA+4D4dVx42Ne2LtML9QTqrQ/xN07pq A6XL5m6p36lGOxZ2z3KoLXrDg2zCMDan1Z+oOEIvxTIHckMjHePh6ovUVmYk6dA1 2SF0jU2DploZF/qufv2xleRjgslDPsfx8NEcHeGBXfuWgQZP0q93aXXDPB4kAkKh YwcXxZ7KMLCWv0U3PNVIOz2WZOZfAadfyyyg1+MOo+nD7lWkDBRZQWrNw8E4dceG ucl9RfVdDopWCL74in7UVnbMAzTUeJVdQY7nSHs5qSKVW611P0xC3FhZZHYOCdL/ WTsm078Zq7jaz2d5WvU61W01/mntXFsOFj3N3uRuG264fLs9F3ccu86Wx4MUzPVS AEA3/B46CHJ2qMIQVB6XeFz0Ny+lPcBVqJJ4rj4OD+O0zYgRC5agxWFabRyMq64x RyWVLQTwhgxVGYqZGtH7W5/O9ee/WyUt8b6w8YlJrfdlpRWrrEY= =fWZi -----END PGP SIGNATURE-----help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20171113172054.GA23670>