Date: Thu, 15 Jan 2004 13:28:31 -0800 From: Tim Kientzle <kientzle@acm.org> To: David Schultz <das@FreeBSD.ORG> Cc: freebsd-arch@FreeBSD.ORG Subject: Re: Request for Comments: libarchive, bsdtar Message-ID: <400705FF.7090709@acm.org> In-Reply-To: <20040115204458.GA27244@VARK.homeunix.com> References: <4004D445.7020205@acm.org> <20040115182532.GA26149@VARK.homeunix.com> <4006E337.7000404@acm.org> <20040115204458.GA27244@VARK.homeunix.com>
next in thread | previous in thread | raw e-mail | index | archive | help
David Schultz wrote: > > ... the package tools don't have an archive format > that supports random access, so they wind up extracting a > temporary copy of the whole archive, then processing the parts > that they want. <grin> Funny you should mention this. As it turns out, random access is not necessary to avoid the temporary copy. To prove it, I've written a pkg_add replacement that extracts directly to the final location in a single pass with no temporary directory and no need for random access to the package file. In particular, it can extract and install a package directly from stdin or an FTP site with no disk overhead at all. How? Easy. Use libarchive to extract the packing list---which is always the first archive entry---directly into memory. Parse the packing list to find out where everything goes, then extract files from the package directly to their final location. Anything that needs to be ignored just gets skipped over. No random access necessary. My prototype pkg_add works and it's about three times faster than the pkg_add in the tree. However, it's also a bit of a mess. libarchive is a big step towards cleaning up that mess by separating out the tarfile reading code. My next project is to separate out some of the generic package management functions into a libpkg library. At that point, I should be able to overhaul my prototype to use these libraries and finally have something production-quality. It is certainly possible that cleaner handling of package files might allow us to eventually use package-management tools for the sysinstall distribution files, but that's certainly not a short-term concern. My goal right now is just to clean up the existing package tools. In particular, my new package tools will: * have the same names as the current tools * use the same command-line options (as far as possible <sigh>) * use the exact same file format as the current tools * be built from general-purpose modular library components This last item is key. This will make it easier to maintain and improve these tools over time with new capabilities (better conflict management, for example) and even new kinds of package tools. As I said, libarchive is just the first major product of this program. There's still a lot of work to do, but I have a roadmap and I'll keep chugging down it as my limited time allows. Tim
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?400705FF.7090709>