Date: Fri, 11 Mar 2011 14:01:36 +0000 From: "Poul-Henning Kamp" <phk@phk.freebsd.dk> To: Martin Matuska <mm@FreeBSD.org> Cc: freebsd-performance@FreeBSD.org, freebsd-current@FreeBSD.org Subject: Re: FreeBSD Compiler Benchmark: gcc-base vs. gcc-ports vs. clang Message-ID: <90325.1299852096@critter.freebsd.dk> In-Reply-To: Your message of "Thu, 10 Mar 2011 22:33:37 %2B0100." <4D7943B1.1030604@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
In message <4D7943B1.1030604@FreeBSD.org>, Martin Matuska writes: >More information, detailed test results and test configuration are at >our blog: >http://blog.vx.sk/archives/25-FreeBSD-Compiler-Benchmark-gcc-base-vs-gcc-ports-vs-clang.html Please don't take this personally Martin, but you have triggered my periodic rant about proper running, evaluation and reporting of benchmarks. These results are not published at a level of detail that allows anybody to draw any kind of conclusions from them. In particular, your use of "overall best" result selection is totally bogus from a statistical point of view. At the very least, we need to see standard-deviations on your numbers, and preferably, when you claim that "X is N% better than Y", you should also provide the confidence interval on that judgment, "Student's T" being the canonical test. The ministat(1) program does both of these things, and is now in FreeBSD/src, so there is absolutely no excuse for not using it. In practice this means that you have to run each test at least three times, to get a standardeviation, and you have to make sure that your testconditions are as identical as possible. Therefore, proper benchmarking procedure is something like: (boot machine single-user // Improves reproducibility) (mount md(4)/malloc filesystem // ditto) (newfs test-partition // ditto) for at least 4 iterations: run test A run test B run test C ... Throw first result away for all tests Run remaining results through ministat(1) This was a public service announcement. Poul-Henning PS: Recommended reading: http://www.larrygonick.com/html/pub/books/sci7.html -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?90325.1299852096>