Date: Fri, 08 Feb 2008 09:41:09 +0100 From: Erik Cederstrand <erik@cederstrand.dk> To: Brooks Davis <brooks@freebsd.org> Cc: freebsd-performance@freebsd.org, kris@FreeBSD.org Subject: Re: Performance Tracker project update Message-ID: <47AC15A5.5020009@cederstrand.dk> In-Reply-To: <20080130190000.GA18333@lor.one-eyed-alien.net> References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> <47A0BFE7.4070708@cederstrand.dk> <20080130190000.GA18333@lor.one-eyed-alien.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Brooks Davis skrev: > On Wed, Jan 30, 2008 at 07:20:23PM +0100, Erik Cederstrand wrote: >> >> I'd like a situation where I can very quickly set up a slave with a >> specific version of FreeBSD to run additional tests or provide shell access >> to a developer. This currently involves adding an entry to a queue, >> rebooting and waiting 2 minutes. Quick and easy, but the archiving strategy >> is obviously very inefficient. >> >> I'm thinking of a couple of options: >> 1. Having one full install per month and archiving the rest as diffs >> against that by recursively bsdiff'ing every file in the tree (I >> could bsdiff a whole tarball, but bsdiff is very memory-intensive). >> Quick test: 25 mins. >> 2. Make a hash of all files and only store the binaries where the hash >> is different from the monthly tarball. Faster than 1., but less >> effective. Quick test: 5 mins. >> 3. Use some kind of VCS. My experience with Subversion and binary files >> is that it's very slow. >> 4. Throw hardware at the problem. >> >> I'd say it should not take more than 10 mins to recreate an archived >> version. Any thoughts? > > It seems like you should be able to combine 1 and 2 with checksums to > decide if you need to run diffs. I'd think that would be quite fast. I finally got around to testing this, and with a combination of mtree comparing md5 hashes, bsdiff compacting changed files and hardlinking unchanged files I get a reduction in size from 256MB to 10MB. Pretty good, and the whole operation only takes a few minutes. I have one peculiarity, though. I install python2.5 into the directory containing the build, and even though the python version has not changed, I still get mismatching md5 sums on every .pyo and .pyc file. Any thoughts on this? Erik
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?47AC15A5.5020009>