From owner-freebsd-performance@FreeBSD.ORG Wed Jan 23 13:49:34 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2233B16A46B; Wed, 23 Jan 2008 13:49:34 +0000 (UTC) (envelope-from erik@cederstrand.dk) Received: from mx2.itu.dk (unknown [130.226.142.29]) by mx1.freebsd.org (Postfix) with ESMTP id D421D13C457; Wed, 23 Jan 2008 13:49:33 +0000 (UTC) (envelope-from erik@cederstrand.dk) Received: from wimac.littlebit.dk (unknown [85.233.238.191]) by mx2.itu.dk (Postfix) with ESMTP id 815A0F4806A; Wed, 23 Jan 2008 14:49:32 +0100 (CET) Message-ID: <479745DA.8010003@cederstrand.dk> Date: Wed, 23 Jan 2008 14:49:14 +0100 From: Erik Cederstrand User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Kris Kennaway References: <4796C717.9000507@cederstrand.dk> <47972895.4050005@FreeBSD.org> In-Reply-To: <47972895.4050005@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-performance@freebsd.org" Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jan 2008 13:49:34 -0000 Kris Kennaway wrote: > > This is coming along very nicely indeed! > > One suggestion I have is that as more metrics are added it becomes > important for an "at a glance" overview of changes so we can monitor for > performance improvements and regressions among many workloads. > > One way to do this would be a matrix of each metric with its change > compared to recent samples. e.g. you could do a student's T comparison > of today's numbers with those from yesterday, or from a week ago, and > colour-code those that show a significant deviation from "no change". > This might be a bit noisy on short timescales, so you could aggregrate > data into larger bins and compare e.g. moving 1-week aggregates. > Fluctuations on short timescales won't stand out, but if there is a real > change then it will show up less than a week later. I agree that there's a need for an overview and some sort of notification. I've been collecting historical data to get a baseline for the statistics and I'll try to see what I can do over the next weeks. > These significant events could also be graphed themselves and/or a > history log maintained (or automatically annotated on the individual > graphs) so historical changes can also be pinpointed. > > At some point the ability to annotate the data will become important > (e.g. "We understand the cause of this, it was r1.123 of foo.c, which > was corrected in r1.124. The developer responsible has been shot.") There's a field in the database for this sort of thing. I just think it needs some sort of authentication. That'll have to wait a bit. > P.S. If I understand correctly, the float test shows a regression? The > metric is calculations/second, so higher = better? The documentation on Unixbench is scarce, but I would think so. BTW if anyone's interested my SVN repo is online at: svn://littlebit.dk/website/trunk (Pylons project) svn://littlebit.dk/tracker/trunk (sh/Python scripts for runnning the server and slaves) Be careful with your eyes - this is my first attempt at both shell scripting and Python :-) Erik