From owner-freebsd-performance@FreeBSD.ORG Sat Nov 10 20:18:31 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 453EB16A41B for ; Sat, 10 Nov 2007 20:18:31 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.180]) by mx1.freebsd.org (Postfix) with ESMTP id 8330B13C4D9 for ; Sat, 10 Nov 2007 20:18:30 +0000 (UTC) (envelope-from josh.carroll@gmail.com) Received: by py-out-1112.google.com with SMTP id u77so386148pyb for ; Sat, 10 Nov 2007 12:18:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=8RZhtvNMTxR32h5oGU77qSGb+vpb9vw717Qvr9GiK+c=; b=d6UnhlPFt7Io+xYaFX0hy9g9Z/EHC+DAq9WidkSGMdKsz/KJvxc2x6TPUffgmWK0JAqe7WJKPsOPWH3kKaCR2HZnWNRi+lJ3vy9yZYcceNc2HJn0lKw492BUMWtgbVH+2E+LBPa81abLW8TEgMduZ+/ST7Ur+aUr0t07IIRqG4o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=PnFtJqPVUySIHsIquCsdJ2FE7rqa2pS372vbnwiHMSy+AKIRSpz4j+HTIuAhOpJSErgq5hqDz7c7arGg7n1Cp9XbGMfZ+6DB3B+CwjVENXtaei9Yzj+OKkr2vhqzKiXPJ38wx930dcS8984Ag4doizorr30Rh2ecJJoIxTbvEEs= Received: by 10.35.90.1 with SMTP id s1mr3776492pyl.1194725903794; Sat, 10 Nov 2007 12:18:23 -0800 (PST) Received: by 10.35.57.12 with HTTP; Sat, 10 Nov 2007 12:18:23 -0800 (PST) Message-ID: <8cb6106e0711101218l532fbd27xb21a03f71cb85975@mail.gmail.com> Date: Sat, 10 Nov 2007 15:18:23 -0500 From: "Josh Carroll" To: "Kris Kennaway" In-Reply-To: <47361134.3050301@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <8cb6106e0710230902x4edf2c8eu2d912d5de1f5d4a2@mail.gmail.com> <8cb6106e0711041243s37ef92e3i9c5c79827f547dbb@mail.gmail.com> <20071104151722.M544@10.0.0.1> <8cb6106e0711050141g674a7733h53f7e6a20e75ad07@mail.gmail.com> <20071106171059.M544@10.0.0.1> <8cb6106e0711062136r18ca8e67sbdb4d4372e10e086@mail.gmail.com> <20071109145827.W639@10.0.0.1> <8cb6106e0711091855x1305ee2bt6d57696df7c7564@mail.gmail.com> <8cb6106e0711101158o16887999rf7f7302ad44d0de8@mail.gmail.com> <47361134.3050301@FreeBSD.org> Cc: Jeff Roberson , freebsd-performance@freebsd.org Subject: Re: ULE vs. 4BSD in RELENG_7 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: josh.carroll@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Nov 2007 20:18:31 -0000 > BTW, it doesn't make much sense to be measuring to millisecond precision > a quantity which has variation that is unknown but probably much larger > :) When trying to make comparisons to identify performance changes, a > careful statistical approach is necessary. > > Try the /usr/src/tools/tools/ministat utility for a simple and effective > way to compare these kinds of noisy measurements and extract reliable > comparisons. Thanks Kris! I have been taking an average of 3 values, but this looks extremely useful. I'll go ahead and build the necessary input files for ministat for stock 4BSD and stock ULE from RELENG_7, so I can generate something like the ministat README for future benchmark runs. I'll also increase my number of runs to 5 to provide a better statistical set of data. Appreciate the pointer, I wasn't aware of ministat. :) Thanks! Josh