Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 04 Jan 2012 15:52:06 -0800
From:      Matthew Tippett <matthew@phoronix.com>
To:        Alexander Kabaev <kabaev@gmail.com>
Cc:        Adrian Chadd <adrian@freebsd.org>, FreeBSD Stable Mailing List <freebsd-stable@freebsd.org>, Joe Holden <lists@rewt.org.uk>, Michael Larabel <Michael@phoronix.com>, Current FreeBSD <freebsd-current@freebsd.org>, Arnaud Lacombe <lacombar@gmail.com>
Subject:   Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server
Message-ID:  <4F04E626.5040406@phoronix.com>
In-Reply-To: <20120104184910.0d604240@kan.dyndns.org>
References:  <CACqU3MXz-vCt8Agkq=z7zyr7ptMgthRBsETxyaFQahX9X1uzPg@mail.gmail.com> <20120104223158.911B11065678@hub.freebsd.org> <20120104184910.0d604240@kan.dyndns.org>

next in thread | previous in thread | raw e-mail | index | archive | help
Hmm... No sure what happened there again.  What I sent (pulled from my 
"Sent" folder...
===

Thanks for the comment Arnaud.   For comparative benchmarking on 
Phoronix.com <http://Phoronix.com>, Michael invariable leaves it in the 
default configuration 'in the way the developers or vendor wanted it for 
production'.  This is by rule.

However, invariable the community or vendor for platforms that post poor 
scores on benchmark cry foul about using the default config.  'it should 
be tuned, no-one deploys an untuned system' or 'the system is configured 
for a different workload'.

The response from us to this comes in two forms.

1) If it is the wrong workload for the platform, do a public post 
explaining and analysing the results.  Highlighting the rationale for 
the concious reduction in performance (ie: journaling filesystems with 
barriers suffer in some write benchmarks for the sake of filesystem 
integrity.

2) If tuning can have a material impact on the results, post a tuning 
guide with step by step and rationale.  Ie: educate the community and users.

Michael and I have had many discussions with vendors and communities on 
this.  In almost all cases, the vendor has either changed the default 
configuration or accepted the results as valid.

As a service to the community or vendor that publishes the tuning guide, 
Michael is more than willing to redo a tuned vs untuned comparison.  To 
date, the communities have never taken us up on that offer.  In part, 
this affects Phoronix.com <http://Phoronix.com>'s perception in the 
public, but that is more of a result of a one sided discussion by a 
party external to a particular community (with a healthy touch of 
journalisticly pumped compare & contrast).  For the FreeBSD community, 
who else outside of the FreeBSD community actually runs public 
comparisons of FreeBSD against anything?

Matthew
===

On 01/04/2012 03:49 PM, Alexander Kabaev wrote:
> On Wed, 04 Jan 2012 14:31:55 -0800
> <matthew@phoronix.com>  wrote:
>
>>     Thanks for the comment Arnaud.   For comparative benchmarking
>> on    [1]Phoronix.com, Michael inva   configuration 'in the way the
>> developers or   production'.  This is by rule. However, i   poor
>> scores on be   'it should be tuned,   is configured for a diffe   The
>> response from us to this comes in two forms.&nb   1) If it is the
>> wrong workload for the platform, do a public pos   explaining and
>> analysing the results.  Highlighting the rationale fo   r the
>> concious reduction in performance (ie: journaling filesystems with
>> ba   filesystem integrity   2) If tuning can have a material impact
>> on the results, post a t   uning guide with step by step and
>> rationale.  Ie: educate the communit   Michael and I have had many
>> discussions with vendors an   on this.  In almost all cases, the
>> vendor has either cha   default configuration or accepted the results
>> as valid. As    guide, Micha   comparison.  To dat   offer.  In part,
>> thi   public, but that is more of a result of a one sided d   party
>> external to a particular community (with a healthy tou
>> journalisticly pumped compare&  contrast).  For the FreeBSD
>> community, who else outside of the FreeBSD community actually runs
>> public c   Matthew
> Not really related to the discussion on hand, but the above about the
> most unreadable email I am yet to read on the public mailing list.
>




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F04E626.5040406>