From owner-freebsd-hackers@FreeBSD.ORG Tue Aug 5 10:22:42 2003 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7001B37B401 for ; Tue, 5 Aug 2003 10:22:42 -0700 (PDT) Received: from phk.freebsd.dk (phk.freebsd.dk [212.242.86.175]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6430043FBD for ; Tue, 5 Aug 2003 10:22:41 -0700 (PDT) (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by phk.freebsd.dk (8.12.8/8.12.8) with ESMTP id h75HMdV3014193 for ; Tue, 5 Aug 2003 17:22:39 GMT (envelope-from phk@phk.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.12.9/8.12.9) with ESMTP id h75HMcPI033256 for ; Tue, 5 Aug 2003 19:22:39 +0200 (CEST) (envelope-from phk@phk.freebsd.dk) To: hackers@freebsd.org From: Poul-Henning Kamp Date: Tue, 05 Aug 2003 19:22:38 +0200 Message-ID: <33255.1060104158@critter.freebsd.dk> Subject: small statistics tool in the tree ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Aug 2003 17:22:42 -0000 I written up a small statistics tool which is targeted at showing if a benchmark proves anything or not. critter phk> cat before 29195 29232 29226 29203 29211 critter phk> cat after 29179 29203 29195 29194 29196 critter phk> ./ministat before after x before + after +--------------------------------------------------------------------------+ |+ +*+ * x x x| | |___________A_M___|_____|___________M__A_____________________| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 5 29195 29232 29211 29213.4 15.469324 + 5 29179 29203 29195 29193.4 8.7920419 Difference at 95.0% confidence -20 ± 18.3497 -0.0684617% ± 0.0628127% (Student's t, pooled s = 12.5817) In this case the benchmark shows a very small improvement has obtained with 95% probability Here is a more convincing result: x before + after +--------------------------------------------------------------------------+ | + | | + | |x x + | |x x x + +| ||_________A__M______| |_____M____A_________| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 5 0.132 0.134 0.133 0.1328 0.00083666003 + 5 0.136 0.138 0.136 0.1364 0.00089442719 Difference at 99.5% confidence 0.0036 ± 0.0024642 2.71084% ± 1.85558% (Student's t, pooled s = 0.000866025) The program is a <500 line .c program and I was wondering if it belongs in the tree so we have an easy to use tool to point people at when they run benchmarks. Source: http://phk.freebsd.dk/patch/ministat Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.