From owner-freebsd-hackers@FreeBSD.ORG Sun Jul 15 18:46:12 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A9DC116A401; Sun, 15 Jul 2007 18:46:12 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from kientzle.com (h-66-166-149-50.snvacaid.covad.net [66.166.149.50]) by mx1.freebsd.org (Postfix) with ESMTP id 503EA13C4C5; Sun, 15 Jul 2007 18:46:11 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from [10.0.0.222] (p54.kientzle.com [66.166.149.54]) by kientzle.com (8.12.9/8.12.9) with ESMTP id l6FIkBH7009292; Sun, 15 Jul 2007 11:46:11 -0700 (PDT) (envelope-from kientzle@freebsd.org) Message-ID: <469A6B73.1030004@freebsd.org> Date: Sun, 15 Jul 2007 11:46:11 -0700 From: Tim Kientzle User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20060422 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Paul Schenkeveld References: <46992FFF.7010906@kientzle.com> <20070714223853.GF16579@britannica.bec.de> <469992CA.6000104@freebsd.org> <4699BE75.2090808@freebsd.org> <20070715133016.GA53853@psconsult.nl> In-Reply-To: <20070715133016.GA53853@psconsult.nl> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, hackers@freebsd.org Subject: Re: Tar output mode for installworld X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Jul 2007 18:46:12 -0000 Paul Schenkeveld wrote: >>... read, extract, etc, a format I'm calling "ntree" >>for now. ... lines have the following format: >> = = ... >>Where key is one of: >> time ... gid,uid ... gname,uname ... mode ... content ... > ... I've been playing a lot with automating and customizing > the build/distribute/install process of FreeBSD ... Yes, there are a lot of interesting games you can play if you can build a description of a .tgz file and then handily generate a .tgz file from that description. I was pleased that bsdtar's already-existing "archive conversion" feature made this so simple to implement. > For the $CUSTOMIZATIONS to work it would be very nice to extend your > proposal with the following features: > - Allow scattered lines describing attributes of the same file but > retain the order in which they appear so that $CUSTOMIZATIONS can > override attributes set by make installworld. I already intend to support multiple lines for the same file to allow attributes to be specified separately: file1 type=file mode=04666 file1 uname=root gname=wheel file1 flags=noschg It's trivial to ensure that later options override earlier ones. The tricky part is the word "scattered." This either requires that libarchive read the entire description file into memory up front (awkward, but not particularly unreasonable) or that we have a separate tool that can move those scattered lines so they occur together. (My initial idea was that /usr/bin/sort would suffice, but I don't think it will. In particular, tar requires that hardlinks follow the object being linked to, which is not something that /usr/bin/sort is going to preserve.) > - Implement something like: remove Hmmm... Interesting idea. It would also be interesting to extend the tar format with a "whiteout" entry that erases an entry already on disk. Hmmm.... does this raise any security issues? I'll have to think about that one. Question: Should file1 type=file file1 remove only remove the file from this archive (i.e., not create file1) or should it also look on disk and remove file1 if it's there. Obviously, the latter can only be carried through a .tgz installation file if the tar format supports whiteouts. > - Implement something like: > move= > to allow $CUSTOMIZATIONS to move things around. > A special case here should be observed: > /bin/foo ... > ... > /bin/foo move=/bin/foo.orig > /bin/foo ... Implementable if libarchive read the entire description into memory up front, but otherwise very tricky. Looks like it will be mandatory for the libarchive support to read the entire description into memory so it can preprocess it before returning complete entries. > Having a file describing everything that gets installed would also benefit > later upgrades to a system. One of my questions: Does my proposed format suffice for these other purposes? If not, what other features would be required? Is it worth trying to design a single format that handles these various cases? > In my experiments I've implemented a manifest > that gets shipped with the tarball.tgs ... includes md5 and/or sha256 ... There is some real value to having md5 and/or sha256 checks in generic tar archives as well. I have a couple of ideas for supporting this using pax extensions, but it's tricky to implement robustly, so it's not going to happen for a little while. (In particular, the error check in the tar archive should include all of the headers, and you want to avoid a separate scan of the file to collect an error check before archiving it.) Tim Kientzle