Date: Mon, 31 Jan 2005 09:45:34 -0600 From: Eric Anderson <anderson@centtech.com> To: Robert Watson <rwatson@freebsd.org> Cc: "Jim C. Nasby" <decibel@decibel.org> Subject: Re: Automated performance testing Message-ID: <41FE529E.7080601@centtech.com> In-Reply-To: <Pine.NEB.3.96L.1050131152227.35704J-100000@fledge.watson.org> References: <Pine.NEB.3.96L.1050131152227.35704J-100000@fledge.watson.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Robert Watson wrote: > On Sun, 30 Jan 2005, Jim C. Nasby wrote: > > >>With all the discussion of performance testing between 4.11, 5.3, and >>Linux, would it be useful to make performance testing part of the >>automated testing that already occurs (via tinderbox, iirc). Doing so >>might make it easier to detect performance impacting changes, as well as >>making performance testing easier in general. > > > Yes, it would be quite valuable. I've been hoping to set up something > like this for a while, but have never found the opportunity. I have been > tracking the long term behavior of MySQL performance as part of the > netperf work, but because testing is fairly hardware and time consuming, > the polling intervals are uneven, and not quite close enough to nail down > culprits. I'd really like to see a small and fairly well-defined set of > tests run every couple of days so we can show long term graphs, and catch > regressions quickly. Unfortunately, this is a bit harder than > tinder-boxing, because it involves swapping out whole system > configurations, recovering from the inevitable failure modes, etc, which > proves to be the usual sticking point in implementing this. However, I'd > love to see someone work on it :-). Maybe it would help to come up with a list of tests, or even just things to be tested initially. Then one could get a sense of what tests need to be run, and how to do it.. I wonder if it's worth building a little app that runs a suite of other included apps (found in the ports collection), grabs system hardware and configuration information, runs the tests, and sends them to a central location. Then we can have OS, hardware, software config, etc information, and compare system vs system, OS ver, etc, and people just build the fbsdperftest port (which installs a slew of other ports, like iozone, bonnie, apache (HTTP testing), mysql (DB testing?), etc), then they run the program, it gathers the data, asks a couple of questions, and sends the data off to a central spot for analyzing later. Then, one could set up a machine that just does the cvsup, buildworld, ..., fbsdperftest, repeat.. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology I have seen the future and it is just like the present, only longer. ------------------------------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?41FE529E.7080601>