From owner-freebsd-current@FreeBSD.ORG Sun Nov 10 21:05:46 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5B6F9DD; Sun, 10 Nov 2013 21:05:46 +0000 (UTC) (envelope-from erik+lists@cederstrand.dk) Received: from csmtp2.one.com (csmtp2.one.com [91.198.169.22]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 198EC2206; Sun, 10 Nov 2013 21:05:45 +0000 (UTC) Received: from bigmac.router9fbd7c.com (unknown [176.222.238.90]) by csmtp2.one.com (Postfix) with ESMTPA id E7F3E20E8D; Sun, 10 Nov 2013 21:05:37 +0000 (UTC) Received: from bigmac.router9fbd7c.com ([UNAVAILABLE]. [176.222.238.90]) (using TLSv1 with cipher AES128-SHA) by 0.0.0.0:2500 (trex/4.8.87); Sun, 10 Nov 2013 21:05:38 GMT Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Re: freebsd perf testing From: Erik Cederstrand In-Reply-To: <527F47AC.50705@freebsd.org> Date: Sun, 10 Nov 2013 22:05:39 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <2034DAAF-F264-42B7-A497-4B2CB846381A@cederstrand.dk> References: <527C462F.9040707@elischer.org> <527F47AC.50705@freebsd.org> To: Julian Elischer X-Mailer: Apple Mail (2.1822) Cc: George Neville-Neil , FreeBSD current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Nov 2013 21:05:46 -0000 Den 10/11/2013 kl. 09.45 skrev Julian Elischer : >=20 > it would be interesting to know what you did. and what conclusions you = came to.. I created a system with a build server, a set of slaves, and a website = to collect and publish the benchmark data in graphs and raw data and = highlight significant changes. The build server built FreeBSD and any required ports and benchmark = software, and bundled it in installation images complete with a script = to run the benchmarks on the slave and push data to the website. The slaves were dedicated machines that booted via PXE, wiped the hard = drive, fetched and installed an image and proceeded to run the specified = benchmarks. The website published graphs, and the graphs were linked to useful = related data like raw measurements, slave hardware, commit messages, = changed source files and binaries etc. It actually worked pretty well. I only ran benchmarks over a couple of = months of commits, but I did identify some significant regressions in = MySQL and some of the unixbench microbenchmarks. I did not spend much = time designing the benchmarking strategy, and =93symbolics=94 has many = good points there. I did find out that all the current continuous = integration / performance benchmark frameworks (Jenkins etc) did not = play well with a situation where the entire operating system is = reinstalled. A wide range of useful benchmarks can be collected in just 10-20 = minutes, but building FreeBSD and all the ports takes much longer than = that (1-2 hours using the default procedure from the handbook). Most of = my work went into optimizing build times, installation image sizes and = slave installation times, so I could build an image in just 10-15 = minutes and use ca. 10MB disk space amortized, and install an image on a = slave in just 2 minutes (cheap 2008-era hardware). But having a complete archive of ready-to-install images for each commit = is a real advantage, not just for performance benchmarking. Imagine = being able to fetch a VirtualBox disk image for a random SVN commit, = booting it and start debugging right away. Or you could write a script = to test if a bug is present in a FreeBSD installation, and then let an = automated process do a binary search to find the offending commit in = maybe less than an hour. My work was completed in 2008 and would (at least) require updating the = scripts to handle the upgrade from GCC to Clang, and from CVS to SVN. Erik=