From owner-freebsd-performance@FreeBSD.ORG Fri Feb 1 06:38:17 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5FF2116A41A for ; Fri, 1 Feb 2008 06:38:17 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from outbound0.mx.meer.net (outbound0.mx.meer.net [209.157.153.23]) by mx1.freebsd.org (Postfix) with ESMTP id 460CE13C4E5 for ; Fri, 1 Feb 2008 06:38:17 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mail.meer.net (mail.meer.net [209.157.152.14]) by outbound0.mx.meer.net (8.12.10/8.12.6) with ESMTP id m115t37T067225; Thu, 31 Jan 2008 21:55:03 -0800 (PST) (envelope-from gnn@neville-neil.com) Received: from mail2.meer.net (mail2.meer.net [64.13.141.16]) by mail.meer.net (8.13.3/8.13.3/meer) with ESMTP id m115t2Iu076669; Thu, 31 Jan 2008 21:55:02 -0800 (PST) (envelope-from gnn@neville-neil.com) Received: from minion.local.neville-neil.com (61.204.211.246.customerlink.pwd.ne.jp [61.204.211.246]) (authenticated bits=0) by mail2.meer.net (8.14.1/8.14.1) with ESMTP id m115t1Yv024529; Thu, 31 Jan 2008 21:55:02 -0800 (PST) (envelope-from gnn@neville-neil.com) Date: Fri, 01 Feb 2008 14:55:01 +0900 Message-ID: From: gnn@freebsd.org To: Erik Cederstrand In-Reply-To: <479745DA.8010003@cederstrand.dk> References: <4796C717.9000507@cederstrand.dk> <47972895.4050005@FreeBSD.org> <479745DA.8010003@cederstrand.dk> User-Agent: Wanderlust/2.15.5 (Almost Unreal) SEMI/1.14.6 (Maruoka) FLIM/1.14.8 (=?ISO-8859-4?Q?Shij=F2?=) APEL/10.7 Emacs/22.1.50 (i386-apple-darwin8.10.1) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII Cc: Kris Kennaway , "freebsd-performance@freebsd.org" Subject: Re: Performance Tracker project update X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Feb 2008 06:38:17 -0000 At Wed, 23 Jan 2008 14:49:14 +0100, Erik Cederstrand wrote: > > Kris Kennaway wrote: > > > > This is coming along very nicely indeed! > > > > One suggestion I have is that as more metrics are added it becomes > > important for an "at a glance" overview of changes so we can monitor for > > performance improvements and regressions among many workloads. > > > > One way to do this would be a matrix of each metric with its change > > compared to recent samples. e.g. you could do a student's T comparison > > of today's numbers with those from yesterday, or from a week ago, and > > colour-code those that show a significant deviation from "no change". > > This might be a bit noisy on short timescales, so you could aggregrate > > data into larger bins and compare e.g. moving 1-week aggregates. > > Fluctuations on short timescales won't stand out, but if there is a real > > change then it will show up less than a week later. > > I agree that there's a need for an overview and some sort of > notification. I've been collecting historical data to get a baseline for > the statistics and I'll try to see what I can do over the next weeks. > > > These significant events could also be graphed themselves and/or a > > history log maintained (or automatically annotated on the individual > > graphs) so historical changes can also be pinpointed. > > > > At some point the ability to annotate the data will become important > > (e.g. "We understand the cause of this, it was r1.123 of foo.c, which > > was corrected in r1.124. The developer responsible has been shot.") > > There's a field in the database for this sort of thing. I just think it > needs some sort of authentication. That'll have to wait a bit. > > > P.S. If I understand correctly, the float test shows a regression? The > > metric is calculations/second, so higher = better? > > The documentation on Unixbench is scarce, but I would think so. > > BTW if anyone's interested my SVN repo is online at: > > svn://littlebit.dk/website/trunk (Pylons project) > svn://littlebit.dk/tracker/trunk (sh/Python scripts for runnning the > server and slaves) > > Be careful with your eyes - this is my first attempt at both shell > scripting and Python :-) > BTW This is excellent work and, if I get the time, I'll be using some of this at my day job. Thanks, George