From owner-freebsd-current@FreeBSD.ORG  Fri Mar 11 16:46:20 2011
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: freebsd-current@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 13390106566C;
	Fri, 11 Mar 2011 16:46:20 +0000 (UTC)
	(envelope-from phk@critter.freebsd.dk)
Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222])
	by mx1.freebsd.org (Postfix) with ESMTP id C8E7B8FC1A;
	Fri, 11 Mar 2011 16:46:19 +0000 (UTC)
Received: from critter.freebsd.dk (critter.freebsd.dk [192.168.61.3])
	by phk.freebsd.dk (Postfix) with ESMTP id 799755DB8;
	Fri, 11 Mar 2011 16:46:18 +0000 (UTC)
Received: from critter.freebsd.dk (localhost [127.0.0.1])
	by critter.freebsd.dk (8.14.4/8.14.4) with ESMTP id p2BGkIup098497;
	Fri, 11 Mar 2011 16:46:18 GMT (envelope-from phk@critter.freebsd.dk)
To: Martin Matuska <mm@FreeBSD.org>
From: "Poul-Henning Kamp" <phk@phk.freebsd.dk>
In-Reply-To: Your message of "Fri, 11 Mar 2011 16:42:04 +0100."
	<4D7A42CC.8020807@FreeBSD.org> 
Date: Fri, 11 Mar 2011 16:46:18 +0000
Message-ID: <98496.1299861978@critter.freebsd.dk>
Sender: phk@critter.freebsd.dk
Cc: freebsd-performance@FreeBSD.org, freebsd-current@FreeBSD.org
Subject: Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang 
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
	<freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>, 
	<mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 11 Mar 2011 16:46:20 -0000

In message <4D7A42CC.8020807@FreeBSD.org>, Martin Matuska writes:

>But what I can say, e.g. for the Intel Atom processor, if there are
>performance gains in all but one test (that falls 2% behind), generic
>perl code (the routines benchmarked) on this processor is very likely to
>run faster with that setup.

No, actually you cannot say that, unless you run all the tests at
least three times for each compiler(+flag), calculate the average
and standard deviation of all the tests, and see which, if any of
the results are statistically significant.

Until you do that, you numbers are meaningless, because we have no
idea what the signal/noise ratio is.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk@FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.