Date: Mon, 13 Nov 2017 09:26:49 -0800 From: John Baldwin <jhb@freebsd.org> To: Glen Barber <gjb@freebsd.org> Cc: freebsd-pkgbase@freebsd.org Subject: Re: Problems with packaging /etc and etcupdate(8) Message-ID: <1972909.hTPcQf8OeH@ralph.baldwin.cx> In-Reply-To: <20171113172054.GA23670@FreeBSD.org> References: <20170927163012.GA99112@FreeBSD.org> <20171113172054.GA23670@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
I had assumed that in a pkgbase setup, one wouldn't use etcupdate or mergemaster. Instead, my understanding is that pkg was going to "learn" about configuration files and would grow etcupdate-like functionality to merge in changes to configuration files during upgrades (and would also support 'pkg confdiff', etc. so that 'pkg confdiff -a' would replace 'etcupdate diff' for example, but would also support packages built from ports, not just base system conf files). Each pkgbase package would contain its associated /etc files as configuration files in this scheme. On Monday, November 13, 2017 05:20:54 PM Glen Barber wrote: > Adding jhb, in case I'm doing something wrong with etcupdate(8) in this > approach, but gratuitously pinging the list to solicit ideas. > > Glen > > On Wed, Sep 27, 2017 at 04:30:12PM +0000, Glen Barber wrote: > > Hi, > > > > Over the past two weeks, I spent quite a bit of time looking into using > > etcupdate(8) to bootstrap /var/db/etcupdate for use with pkg(8), and > > here are some of the issues I have run into. The most current diff > > tested is attached. > > > > 1) A new 'create-packages-files' target was added, which invokes the > > etcupdate(8) 'extract' command using WSTAGEDIR for the DESTDIR. > > Despite passing 'TAGS=package=etc' and 'PACKAGE=etc' to etcupdate(8) > > '-M' flag, in addition to '-DNO_ROOT', some files still fail to be > > tagged in WSTAGEDIR/var/db/etcupdate/current/METALOG. To work around > > this, a hack was added to release/scripts/mtree-to-plist.awk to trap > > and tag files when '-v etcupdate=yes' is passed to awk(1) in this > > target. > > > > 2) Some files do not automatically get created by etcupdate(8), as noted > > in the manual page. For example, pwd_mkdb(8), cap_mkdb(1), > > newaliases(1), tzsetup(8), and services_mkdb(8) are invoked after > > merging files with the newly-created tree. This causes, for example, > > files like /etc/passwd to be nonexistent when creating a package. To > > work around this, another hack was added to the mtree-to-plist.awk > > script to skip nonexistent files in WSTAGEDIR/var/db/etcupdate. This > > is still non-ideal, I think. > > > > 3) Setting 'prefix = /var/db/etcupdate' in the release/packages/etc.ucl > > file appears to be ignored. Files in the 'etc' package conflict with > > files installed by other packages, such as runtime, autofs, and so > > on, with conflicting files being things like /etc/autofs/include_ldap > > or other similarly-installed files within /etc. To work around this > > when packaging files, a hack was added to the mtree-to-plist.awk > > script (again) to prepend '/var/db/etcupdate' to the file path in > > order to get the correct files included in the package. (This is > > somewhat in parallel with (2) above.) > > > > 4) When running etcupdate(8) after hacking around some of the issues > > above, I discovered that /var/db/etcupdate/current in the root of the > > system in which it is being run disappears. Additionally, I had to > > restore files within /etc (such as master.passwd(5), passwd(5), etc.) > > in order to make my test system usable again. In addition to > > replacing files that should have been merged, 'pkg check' against the > > 'FreeBSD-etc' package reports all of the files from the 'current' > > directory have vanished. > > > > In addition to the above items, I also did some (albeit minimal) testing > > with mergemaster(8), as it creates /var/db/mergemaster.mtree, however in > > my brief testing it seems the mtree(8) file created is not compatible > > with pkg(8) or the mtree-to-plist.awk script. (Note: the mergemaster(8) > > bits from my testing are not included in the diff, as I had to revert > > a number of changes to ensure the correct diff was included in this > > email. > > > > I am at the point now where I am out of ideas on how to proceed, and > > could use some help on figuring this out. Any help on the above issues, > > or anything in the Wiki would be greatly appreciated. > > > > https://wiki.freebsd.org/PkgBase > > > > Thanks, > > > > Glen > > > > > Index: Makefile > > =================================================================== > > --- Makefile (revision 324062) > > +++ Makefile (working copy) > > @@ -132,6 +132,7 @@ > > builddtb xdev xdev-build xdev-install \ > > xdev-links native-xtools stageworld stagekernel stage-packages \ > > create-packages-world create-packages-kernel create-packages \ > > + create-files-packages \ > > packages installconfig real-packages sign-packages package-pkg \ > > print-dir test-system-compiler > > > > Index: Makefile.inc1 > > =================================================================== > > --- Makefile.inc1 (revision 324063) > > +++ Makefile.inc1 (working copy) > > @@ -1555,6 +1555,7 @@ > > > > .ORDER: stage-packages create-packages > > .ORDER: create-packages create-world-packages > > +.ORDER: create-packages create-files-packages > > .ORDER: create-packages create-kernel-packages > > .ORDER: create-packages sign-packages > > > > @@ -1596,6 +1597,12 @@ > > DESTDIR=${WSTAGEDIR} \ > > PKG_VERSION=${PKG_VERSION} create-world-packages > > > > +create-packages-files: _pkgbootstrap _repodir .PHONY > > + ${_+_}@cd ${.CURDIR}; \ > > + ${MAKE} -f Makefile.inc1 \ > > + DESTDIR=${WSTAGEDIR} \ > > + PKG_VERSION=${PKG_VERSION} create-files-packages > > + > > create-packages-kernel: _pkgbootstrap _repodir .PHONY > > ${_+_}@cd ${.CURDIR}; \ > > ${MAKE} -f Makefile.inc1 \ > > @@ -1603,8 +1610,25 @@ > > PKG_VERSION=${PKG_VERSION} DISTDIR=kernel \ > > create-kernel-packages > > > > -create-packages: .PHONY create-packages-world create-packages-kernel > > +create-packages: .PHONY create-packages-world create-packages-files create-packages-kernel > > > > +create-files-packages: _pkgbootstrap .PHONY > > + mkdir -p ${WSTAGEDIR}/var/db/etcupdate/current > > + cd ${WSTAGEDIR} ; \ > > + ${WSTAGEDIR}/usr/sbin/etcupdate extract -B -M \ > > + "TARGET=${TARGET} TARGET_ARCH=${TARGET_ARCH} -DNO_ROOT TAGS=package=etc PACKAGE=etc" \ > > + -s ${SRCDIR} -d ${WSTAGEDIR}/var/db/etcupdate > > + awk -f ${SRCDIR}/release/scripts/mtree-to-plist.awk \ > > + -v etcupdate=yes -v ws=${WSTAGEDIR} \ > > + ${WSTAGEDIR}/var/db/etcupdate/current/METALOG > > + sh ${SRCDIR}/release/packages/generate-ucl.sh -o etc \ > > + -s ${SRCDIR} -u ${WSTAGEDIR}/etc.ucl > > + ${PKG_CMD} -o ABI_FILE=${WSTAGEDIR}/bin/sh -o ALLOW_BASE_SHLIBS=yes \ > > + create -M ${WSTAGEDIR}/etc.ucl \ > > + -p ${WSTAGEDIR}/etc.plist \ > > + -r ${WSTAGEDIR} \ > > + -o ${REPODIR}/$$(${PKG_CMD} -o ABI_FILE=${WSTAGEDIR}/bin/sh config ABI)/${PKG_VERSION} > > + > > create-world-packages: _pkgbootstrap .PHONY > > @rm -f ${WSTAGEDIR}/*.plist 2>/dev/null || : > > @cd ${WSTAGEDIR} ; \ > > Index: release/packages/etc.ucl > > =================================================================== > > --- release/packages/etc.ucl (nonexistent) > > +++ release/packages/etc.ucl (working copy) > > @@ -0,0 +1,22 @@ > > +# > > +# $FreeBSD: head/release/packages/runtime.ucl 322327 2017-08-09 19:16:54Z gjb $ > > +# > > + > > +name = "FreeBSD-etc" > > +origin = "base" > > +version = "%VERSION%" > > +comment = "etc package" > > +categories = [ base ] > > +maintainer = "re@FreeBSD.org" > > +www = "https://www.FreeBSD.org" > > +prefix = "/var/db/etcupdate" > > +licenselogic = "single" > > +licenses = [ BSD2CLAUSE ] > > +desc = <<EOD > > +etc package > > +EOD > > +scripts: { > > + post-install = <<EOD > > + echo "Please use etcupdate(8) or mergemaster(8) to update" > > + echo "files under /etc." > > +EOD > > Index: release/scripts/mtree-to-plist.awk > > =================================================================== > > --- release/scripts/mtree-to-plist.awk (revision 324062) > > +++ release/scripts/mtree-to-plist.awk (working copy) > > @@ -28,6 +28,13 @@ > > tags=tags""_kernconf > > } > > } > > + if (etcupdate != "") { > > + if (length(gsub(/tags=/, "", tags) == 0)) > > + tags="package=etc" > > + if ((length(type) != 0) && (type == "dir")) { > > + next > > + } > > + } > > if (length(tags) == 0) > > next > > if (tags ~ /package=/) { > > @@ -76,5 +83,17 @@ > > output=pkgname".plist" > > } > > > > + if (etcupdate != "") { > > + pfx="/var/db/etcupdate/current" > > + filename=ws""pfx""$1 > > + if (system("test -e " filename) == 0) { > > + print "@"type"("uname","gname","mode","flags") " pfx""$1 > output > > + next > > + } else { > > + next > > + } > > + next > > + } > > + > > print "@"type"("uname","gname","mode","flags") " $1 > output > > } > > > -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1972909.hTPcQf8OeH>