From owner-freebsd-performance@FreeBSD.ORG Tue Feb 1 04:59:59 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3913C16A4CE; Tue, 1 Feb 2005 04:59:59 +0000 (GMT) Received: from flake.decibel.org (flake.decibel.org [66.143.173.58]) by mx1.FreeBSD.org (Postfix) with ESMTP id E995843D3F; Tue, 1 Feb 2005 04:59:58 +0000 (GMT) (envelope-from decibel@decibel.org) Received: by flake.decibel.org (Postfix, from userid 1001) id 5E8D21C916; Mon, 31 Jan 2005 22:59:58 -0600 (CST) Date: Mon, 31 Jan 2005 22:59:58 -0600 From: "Jim C. Nasby" To: Robert Watson Message-ID: <20050201045958.GD32356@decibel.org> References: <20050130230527.GR64304@decibel.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 4.10-RELEASE-p3 i386 X-Distributed: Join the Effort! http://www.distributed.net User-Agent: Mutt/1.5.6i cc: freebsd-performance@freebsd.org Subject: Re: Automated performance testing X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Feb 2005 04:59:59 -0000 On Mon, Jan 31, 2005 at 03:24:39PM +0000, Robert Watson wrote: > > On Sun, 30 Jan 2005, Jim C. Nasby wrote: > > > With all the discussion of performance testing between 4.11, 5.3, and > > Linux, would it be useful to make performance testing part of the > > automated testing that already occurs (via tinderbox, iirc). Doing so > > might make it easier to detect performance impacting changes, as well as > > making performance testing easier in general. > > Yes, it would be quite valuable. I've been hoping to set up something > like this for a while, but have never found the opportunity. I have been > tracking the long term behavior of MySQL performance as part of the > netperf work, but because testing is fairly hardware and time consuming, > the polling intervals are uneven, and not quite close enough to nail down > culprits. I'd really like to see a small and fairly well-defined set of > tests run every couple of days so we can show long term graphs, and catch > regressions quickly. Unfortunately, this is a bit harder than > tinder-boxing, because it involves swapping out whole system > configurations, recovering from the inevitable failure modes, etc, which > proves to be the usual sticking point in implementing this. However, I'd > love to see someone work on it :-). FWIW, I'd suggest something less complicated than a database for performance testing. For starters, there's no way to isolate what part of the OS (if any) is responsible for a performance change. Databases also continually improve their own performance, so it's very much a moving standard. -- Jim C. Nasby, Database Consultant decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?"