Date: Wed, 30 Jan 2008 13:00:00 -0600 From: Brooks Davis <brooks@freebsd.org> To: Erik Cederstrand <erik@cederstrand.dk> Cc: freebsd-performance@freebsd.org Subject: Re: Performance Tracker project update Message-ID: <20080130190000.GA18333@lor.one-eyed-alien.net> In-Reply-To: <47A0BFE7.4070708@cederstrand.dk> References: <4796C717.9000507@cederstrand.dk> <20080123193400.N63024@fledge.watson.org> <4797A245.7080202@cederstrand.dk> <20080123202433.E63024@fledge.watson.org> <4797A802.8060509@FreeBSD.org> <47A0BFE7.4070708@cederstrand.dk>
next in thread | previous in thread | raw e-mail | index | archive | help
--Dxnq1zWXvFF0Q93v Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jan 30, 2008 at 07:20:23PM +0100, Erik Cederstrand wrote: > Kris Kennaway wrote: >> Robert Watson wrote: >> One thing I am looking at is how to best create a library of world=20 >> tarballs that can be used to populate a nfsroot (or hybrid of periodic= =20 >> tarballs + binary diffs to save space). Then you could provide your=20 >> benchmark in a standardized format (start/end/cleanup scripts, etc) and= =20 >> tell a machine "go and run this benchmark on every daily snapshot for th= e=20 >> last year and give me the numbers". >=20 > Actually, this just bit me. To make packages for installation, I do a=20 > standard installworld/kernel/distribution into a directory and a chroot= =20 > install of the necessary ports. I create a .tgz and the end result is=20 > around 90MB (takes ca. 2 hours). Now I have 186 of those, and my file=20 > system is full (it's only 15GB, but even a 500GB disk will fill up in 2-3= =20 > months with a fast build server). >=20 > I'd like a situation where I can very quickly set up a slave with a=20 > specific version of FreeBSD to run additional tests or provide shell acce= ss=20 > to a developer. This currently involves adding an entry to a queue,=20 > rebooting and waiting 2 minutes. Quick and easy, but the archiving strate= gy=20 > is obviously very inefficient. >=20 > I'm thinking of a couple of options: > 1. Having one full install per month and archiving the rest as diffs > against that by recursively bsdiff'ing every file in the tree (I > could bsdiff a whole tarball, but bsdiff is very memory-intensive). > Quick test: 25 mins. > 2. Make a hash of all files and only store the binaries where the hash > is different from the monthly tarball. Faster than 1., but less > effective. Quick test: 5 mins. > 3. Use some kind of VCS. My experience with Subversion and binary files > is that it's very slow. > 4. Throw hardware at the problem. >=20 > I'd say it should not take more than 10 mins to recreate an archived=20 > version. Any thoughts? It seems like you should be able to combine 1 and 2 with checksums to decide if you need to run diffs. I'd think that would be quite fast. Another option would be to use ZFS snapshots. As long as you avoid replaci= ng identical files unnecessicairly, you should have pretty good effencies and little overhead. -- Brooks --Dxnq1zWXvFF0Q93v Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (FreeBSD) iD8DBQFHoMksXY6L6fI4GtQRAuKAAJ91Oleqalwg69wLylvaVqvgD9VivgCgs0xZ AazDOiwKFZzG8OIm8kPPjbw= =hZ0V -----END PGP SIGNATURE----- --Dxnq1zWXvFF0Q93v--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080130190000.GA18333>