Date: Sun, 10 Nov 2013 22:05:39 +0100 From: Erik Cederstrand <erik+lists@cederstrand.dk> To: Julian Elischer <julian@freebsd.org> Cc: George Neville-Neil <gnn@freebsd.org>, FreeBSD current <freebsd-current@freebsd.org> Subject: Re: freebsd perf testing Message-ID: <2034DAAF-F264-42B7-A497-4B2CB846381A@cederstrand.dk> In-Reply-To: <527F47AC.50705@freebsd.org> References: <527C462F.9040707@elischer.org> <F148BE2A-2662-47C9-8B7B-E6E61FBE53DE@cederstrand.dk> <527F47AC.50705@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Den 10/11/2013 kl. 09.45 skrev Julian Elischer <julian@freebsd.org>: >=20 > it would be interesting to know what you did. and what conclusions you = came to.. I created a system with a build server, a set of slaves, and a website = to collect and publish the benchmark data in graphs and raw data and = highlight significant changes. The build server built FreeBSD and any required ports and benchmark = software, and bundled it in installation images complete with a script = to run the benchmarks on the slave and push data to the website. The slaves were dedicated machines that booted via PXE, wiped the hard = drive, fetched and installed an image and proceeded to run the specified = benchmarks. The website published graphs, and the graphs were linked to useful = related data like raw measurements, slave hardware, commit messages, = changed source files and binaries etc. It actually worked pretty well. I only ran benchmarks over a couple of = months of commits, but I did identify some significant regressions in = MySQL and some of the unixbench microbenchmarks. I did not spend much = time designing the benchmarking strategy, and =93symbolics=94 has many = good points there. I did find out that all the current continuous = integration / performance benchmark frameworks (Jenkins etc) did not = play well with a situation where the entire operating system is = reinstalled. A wide range of useful benchmarks can be collected in just 10-20 = minutes, but building FreeBSD and all the ports takes much longer than = that (1-2 hours using the default procedure from the handbook). Most of = my work went into optimizing build times, installation image sizes and = slave installation times, so I could build an image in just 10-15 = minutes and use ca. 10MB disk space amortized, and install an image on a = slave in just 2 minutes (cheap 2008-era hardware). But having a complete archive of ready-to-install images for each commit = is a real advantage, not just for performance benchmarking. Imagine = being able to fetch a VirtualBox disk image for a random SVN commit, = booting it and start debugging right away. Or you could write a script = to test if a bug is present in a FreeBSD installation, and then let an = automated process do a binary search to find the offending commit in = maybe less than an hour. My work was completed in 2008 and would (at least) require updating the = scripts to handle the upgrade from GCC to Clang, and from CVS to SVN. Erik=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2034DAAF-F264-42B7-A497-4B2CB846381A>