From owner-freebsd-current@FreeBSD.ORG  Sat Mar 12 13:35:40 2011
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: freebsd-current@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 38EAE106566C;
	Sat, 12 Mar 2011 13:35:40 +0000 (UTC)
	(envelope-from phk@critter.freebsd.dk)
Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222])
	by mx1.freebsd.org (Postfix) with ESMTP id F09F98FC19;
	Sat, 12 Mar 2011 13:35:39 +0000 (UTC)
Received: from critter.freebsd.dk (critter.freebsd.dk [192.168.61.3])
	by phk.freebsd.dk (Postfix) with ESMTP id 714295E2D;
	Sat, 12 Mar 2011 13:35:38 +0000 (UTC)
Received: from critter.freebsd.dk (localhost [127.0.0.1])
	by critter.freebsd.dk (8.14.4/8.14.4) with ESMTP id p2CDZbj8060072;
	Sat, 12 Mar 2011 13:35:38 GMT (envelope-from phk@critter.freebsd.dk)
To: Martin Matuska <mm@FreeBSD.org>
From: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
In-Reply-To: Your message of "Sat, 12 Mar 2011 11:02:23 +0100."
	<4D7B44AF.7040406@FreeBSD.org> 
Date: Sat, 12 Mar 2011 13:35:37 +0000
Message-ID: <60071.1299936937@critter.freebsd.dk>
Sender: phk@critter.freebsd.dk
Cc: freebsd-performance@FreeBSD.org, freebsd-current@FreeBSD.org
Subject: Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang 
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
	<freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>, 
	<mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 12 Mar 2011 13:35:40 -0000

In message <4D7B44AF.7040406@FreeBSD.org>, Martin Matuska writes:


Thanks a lot for doing this properly.

>What significance level should I take?

I think I set ministat(1) to use 95 % confidence level by default
and that is in general a pretty safe bet (1 in 20 chance)

>I hope this approach is better :)

Much, much better.

As I said, this was not to go after you personally, but to point
out that we need to be more rigorous with benchmarks in general.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.