From owner-freebsd-performance@FreeBSD.ORG Mon Jan 31 15:45:39 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BE7B116A4CF; Mon, 31 Jan 2005 15:45:39 +0000 (GMT) Received: from otter3.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id D650743D31; Mon, 31 Jan 2005 15:45:38 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by otter3.centtech.com (8.12.3/8.12.3) with ESMTP id j0VFjaOJ016927; Mon, 31 Jan 2005 09:45:36 -0600 (CST) (envelope-from anderson@centtech.com) Message-ID: <41FE529E.7080601@centtech.com> Date: Mon, 31 Jan 2005 09:45:34 -0600 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.3) Gecko/20041110 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Robert Watson References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: freebsd-performance@freebsd.org cc: "Jim C. Nasby" Subject: Re: Automated performance testing X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Jan 2005 15:45:40 -0000 Robert Watson wrote: > On Sun, 30 Jan 2005, Jim C. Nasby wrote: > > >>With all the discussion of performance testing between 4.11, 5.3, and >>Linux, would it be useful to make performance testing part of the >>automated testing that already occurs (via tinderbox, iirc). Doing so >>might make it easier to detect performance impacting changes, as well as >>making performance testing easier in general. > > > Yes, it would be quite valuable. I've been hoping to set up something > like this for a while, but have never found the opportunity. I have been > tracking the long term behavior of MySQL performance as part of the > netperf work, but because testing is fairly hardware and time consuming, > the polling intervals are uneven, and not quite close enough to nail down > culprits. I'd really like to see a small and fairly well-defined set of > tests run every couple of days so we can show long term graphs, and catch > regressions quickly. Unfortunately, this is a bit harder than > tinder-boxing, because it involves swapping out whole system > configurations, recovering from the inevitable failure modes, etc, which > proves to be the usual sticking point in implementing this. However, I'd > love to see someone work on it :-). Maybe it would help to come up with a list of tests, or even just things to be tested initially. Then one could get a sense of what tests need to be run, and how to do it.. I wonder if it's worth building a little app that runs a suite of other included apps (found in the ports collection), grabs system hardware and configuration information, runs the tests, and sends them to a central location. Then we can have OS, hardware, software config, etc information, and compare system vs system, OS ver, etc, and people just build the fbsdperftest port (which installs a slew of other ports, like iozone, bonnie, apache (HTTP testing), mysql (DB testing?), etc), then they run the program, it gathers the data, asks a couple of questions, and sends the data off to a central spot for analyzing later. Then, one could set up a machine that just does the cvsup, buildworld, ..., fbsdperftest, repeat.. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology I have seen the future and it is just like the present, only longer. ------------------------------------------------------------------------