Date: Mon, 16 Jul 2018 20:26:42 +0100 From: Edward Tomasz =?utf-8?Q?Napiera=C5=82a?= <trasz@freebsd.org> To: CL Moonriver <clmoonriver@equinedreams.art> Cc: Erich Dollansky <freebsd.ed.lists@sumeritec.com>, freebsd-advocacy@freebsd.org Subject: Re: CoC does not help in benchmarks Message-ID: <20180716192642.GA3806@brick> In-Reply-To: <88e6b96d-1f89-6774-09b8-83afcfc27a82@equinedreams.art> References: <20180714064429.36c6bc43.freebsd.ed.lists@sumeritec.com> <88e6b96d-1f89-6774-09b8-83afcfc27a82@equinedreams.art>
next in thread | previous in thread | raw e-mail | index | archive | help
It's not just that - part of the problem is that Phoronix Test Suite seems be focused more on producing pretty graphs than obtaining any meaningful results. It fails at basic statistics - error bars are nowhere to be found. Even if we assume the differences in numbers are not just measurement errors, they might come from the compiler differences, or just comparing completely different pieces of software: the "gzip compression", which shows an order of magnitude of difference between FreeBSD and Clear Linux, mentions "patches its carrying for threading"; the "git benchmark" is at least clear that it compares different git versions - ranging from 1.7.1 to 2.18.0. Then there's the PHPBench, where there's 4x difference between Linux versions (Scientific Linux and Clear Linux). There are also differences in compiler options and mount flags. Then there's the fact that when they actually try to compare the same piece of software, they build it from vanilla sources instead of using official ports/packages. And they use the default configuration, ignoring basic guidelines that are generally known to people who use the software - disabling update_process_titles, for example. Which means they run something different from what people actually use. To put it bluntly: Phoronix benchmarks are utterly useless for anything apart from generating page views. On 0715T1258, CL Moonriver wrote: > Can you clarify what you mean? Performance benchmarks like that are, > after all, often of limited value when it comes to why one might choose > one OS over another. A lot of other things factor into the decision. > Security, stability, consistency, ease of configuration and > administration, etc. > > > On 07/13/18 17:44, Erich Dollansky wrote: > > Hi, > > > > here are the consequences of putting a CoC up high on the priority list: > > > > https://www.phoronix.com/scan.php?page=article&item=windows-freebsd112-8linux&num=1 > > > > Focusing on software would have made FreeBSD do better. > > > > Erich
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180716192642.GA3806>