From owner-freebsd-hackers@FreeBSD.ORG Sun Jul 15 17:14:15 2007 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A548916A406 for ; Sun, 15 Jul 2007 17:14:15 +0000 (UTC) (envelope-from fb-hackers@psconsult.nl) Received: from ps226.psconsult.nl (ps226.psconsult.nl [213.222.19.226]) by mx1.freebsd.org (Postfix) with ESMTP id 2704013C48E for ; Sun, 15 Jul 2007 17:14:14 +0000 (UTC) (envelope-from fb-hackers@psconsult.nl) Received: from phuket.psconsult.nl (localhost [127.0.0.1]) by phuket.psconsult.nl (8.13.1/8.13.1) with ESMTP id l6FDUG8o058668; Sun, 15 Jul 2007 15:30:16 +0200 (CEST) (envelope-from fb-hackers@psconsult.nl) Received: (from paul@localhost) by phuket.psconsult.nl (8.13.1/8.13.1/Submit) id l6FDUGcV058667; Sun, 15 Jul 2007 15:30:16 +0200 (CEST) (envelope-from fb-hackers@psconsult.nl) Date: Sun, 15 Jul 2007 15:30:16 +0200 From: Paul Schenkeveld To: freebsd-hackers@freebsd.org, hackers@freebsd.org Message-ID: <20070715133016.GA53853@psconsult.nl> Mail-Followup-To: freebsd-hackers@freebsd.org, hackers@freebsd.org References: <46992FFF.7010906@kientzle.com> <20070714223853.GF16579@britannica.bec.de> <469992CA.6000104@freebsd.org> <4699BE75.2090808@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4699BE75.2090808@freebsd.org> User-Agent: Mutt/1.5.6i Cc: Subject: Re: Tar output mode for installworld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Jul 2007 17:14:15 -0000 On Sat, Jul 14, 2007 at 11:28:05PM -0700, Tim Kientzle wrote: > >>>This is easy to implement ... just build > >>>a description of the final archive in a nice verbose > >>>text format such as: > >> > >>...which is done by NetBSD for the unprivileged release building via > >>build.sh. Anyone interested in working on this should possibly have a > >>look there. > > Here's a rough implementation of my core idea. Add the > attached archive_read_support_format_ntree.c to libarchive > and patch archive_read_support_format_all.c, then > rebuild libarchive and bsdtar. You'll then be able > to read, extract, etc, a format I'm calling "ntree" > for now. This similar to NetBSD's "metalog" format, except: > > 1) First line must be "#%ntree". This is used as a file signature. > > 2) Blank lines and lines beginning with '#' are ignored. > > 3) All other lines have the following format: > > = = ... > > Where key is one of: > time: decimal seconds since beginning of epoch > gid,uid: decimal group/user ID > gname,uname: textual group/user name > mode: octal > type: as in mtree, defaults to 'file' > content: name of a file on disk Great! This is exactly the next BIG step in (Free)BSD build/distribute/install that I have been waiting for. In the mean time I've been playing a lot with automating and customizing the build/distribute/install process of FreeBSD, partly inspired by nanobsd. Basically the process is something like: __MAKE_CONF=my_make.conf make buildworld __MAKE_CONF=my_make.conf make installworld DESTDIR=/some/place $CUSTOMIZATIONS /some/place tar czf tarball.tgz -C /some/place # Find a way to get the tarball.tgz to its destination tar xpf tarball.tgz # at the destination For the $CUSTOMIZATIONS to work it would be very nice to extend your proposal with the following features: - Allow scattered lines describing attributes of the same file but retain the order in which they appear so that $CUSTOMIZATIONS can override attributes set by make installworld. - Implement something like: remove to undo the installation of files (embedded systems hardly ever need stuff like documentation, NLS and so on but the make.conf and src.conf knobs are not fine-grained enough for all situations) - Implement something like: move= to allow $CUSTOMIZATIONS to move things around. A special case here should be observed: /bin/foo ... ... /bin/foo move=/bin/foo.orig /bin/foo ... here the $CUSTOMIZATIONS move some file so another place and installs a wrapper. Probably a sorting algorithm that retains the order in which /bin/foo lines appear does half the trick and libarchive should collect and accumulate lines with the same filename and only emit or extract a file after the last line with the same name OR when an move= key appears and reset file info immediately after this line. Having a file describing everything that gets installed would also benefit later upgrades to a system. In my experiments I've implemented a manifest that gets shipped with the tarball.tgs (still mean to put it inside but got some ENOTIME errors). My manifest also includes md5 and/or sha256 checksums for each file so the upgrade tool can easily detect which files were altered/moved/removed by the sysadmin an sensibly act accordingly. Again ENOTIME prevented me so far from writing the update tool but I can certainly help out here. If all works out well, this could also benefit ports where a package could be created without installing the port on the build system. Many hurdles to take, I know but certainly a goal to consider. -- Paul Schenkeveld