Date: Tue, 28 Jun 2005 14:29:56 -0400 From: Paul Mather <paul@gromit.dlib.vt.edu> To: Roman Neuhauser <neuhauser@sigpipe.cz> Cc: Michael Schuh <michael.schuh@gmail.com>, freebsd-stable@freebsd.org Subject: Re: FreeBSD MySQL still WAY slower than Linux Message-ID: <1119983396.7900.72.camel@zappa.Chelsea-Ct.Org> In-Reply-To: <20050628171739.GC51923@isis.sigpipe.cz> References: <1dbad315050621051525f4c6fc@mail.gmail.com> <200506211451.j5LEpA2W024350@lurza.secnetix.de> <20050628092126.GB48140@isis.sigpipe.cz> <1119973124.7900.20.camel@zappa.Chelsea-Ct.Org> <20050628163928.GA51923@isis.sigpipe.cz> <1119978184.7900.36.camel@zappa.Chelsea-Ct.Org> <20050628171739.GC51923@isis.sigpipe.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 2005-06-28 at 19:17 +0200, Roman Neuhauser wrote:
> # paul@gromit.dlib.vt.edu / 2005-06-28 13:03:04 -0400:
> > What you see as being defensive I see as being rigorous. If someone is
> > making a claim based upon a performance benchmark, people will quiz the
> > person conducting the benchmark to ascertain exactly how it has been
> > undertaken. To put any stock in a benchmark result, it is important to
> > be able to convince yourself it is a meaningful result. Well, at least
> > most people I've encountered believe that to be the case.
>
> Say I install FreeBSD (using default partitions), install MySQL from
> a package on the CD, run a stress test, collect numbers, then
> repeat the process with a Linux installed over the previous FreeBSD
> installation, and find out that FreeBSD allows the MySQL server
> process 1/3 queries less, what (if anything) will be wrong in my
> claim that MySQL/FreeBSD is slower than MySQL/Linux?
To make the specific claim above, it would be okay, at least in my book.
To make a more general claim, pedantically speaking, you should, e.g.,
replicate your benchmark using various different hardware combinations,
to rule out the possibility of a pathological case affecting one or
other OS (e.g., where one OS has much better driver support for some
specific hardware aspect than the other).
Also, you would need to be careful how you stated your claim. For
example, it would be better to say something like "untuned
MySQL/Particular-FreeBSD-Version on a default install is slower than
untuned MySQL/Particular-Linux-Distribution on a default install." If
you test many Linux distributions and find they beat out all the
FreeBSDs, then a more general "MySQL/FreeBSD" and "MySQL/Linux" might be
appropriate. Similarly, if no amount of tuning lets FreeBSD MySQL
compete with Linux, I'd say your original statement would be defensible.
However, if some combinations of MySQL and FreeBSD tuning perform better
than some well-tuned MySQL/Linux distributions, then it's not so
straightforward to claim MySQL/FreeBSD is slower than MySQL/Linux. (It
may be that default tunings favour one or the other. I'm sure the Linux
people would gripe that you're using filesystem X instead of filesystem
Y, which would give better performance.:)
If you just want to make a benchmark statement about untuned installs on
default distributions, that's fine, but I'm not sure how illuminating
that is for the real world given that production systems are normally
tuned for performance.
Although this might seem like splitting hairs to the extreme, I guess
the point I'm making is that benchmarks can be highly subjective.
Unless you learn the context and point of view in which it was
performed, you can't really tell if the results apply to you. In fact,
even a very good benchmark may not yield the expected performance in the
real world when run in an environment containing other systems (e.g.,
Apache, Squid, Postfix, etc.) that interact and contend for resources
and affect performance in a way not measured when the systems are
benchmarked in isolation.
I guess the preferred colour for the consumers of benchmarks is black
and white, when in reality what you get are subtle shades of grey. :-)
Cheers,
Paul.
--
e-mail: paul@gromit.dlib.vt.edu
"Without music to decorate it, time is just a bunch of boring production
deadlines or dates by which bills must be paid."
--- Frank Vincent Zappa
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1119983396.7900.72.camel>