Date: Wed, 21 Dec 2011 16:22:41 +1100 (EST) From: Ian Smith <smithi@nimnet.asn.au> To: "Samuel J. Greear" <sjg@evilcode.net> Cc: "O. Hartmann" <ohartman@zedat.fu-berlin.de>, Adrian Chadd <adrian@freebsd.org>, lev@freebsd.org, FreeBSD Stable Mailing List <freebsd-stable@freebsd.org>, Jeremy Chadwick <freebsd@jdc.parodius.com> Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server Message-ID: <20111220033328.I64681@sola.nimnet.asn.au> In-Reply-To: <CANY-Wm9-JTN0gvjoRv4XFMDaweoPSoZ4erTUto3Z-s1LxqGzhg@mail.gmail.com> References: <4EE1EAFE.3070408@m5p.com> <CAJ-FndDniGH8QoT=kUxOQ%2BzdVhWF0Z0NKLU0PGS-Gt=BK6noWw@mail.gmail.com> <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <CAFHbX1%2B5PttyZuNnYot8emTn_AWkABdJCvnpo5rcRxVXj0ypJA@mail.gmail.com> <4EE933C6.4020209@zedat.fu-berlin.de> <CAPjTQNEJDE17TLH-mDrG_-_Qa9R5N3mSeXSYYWtqz_DFidzYQw@mail.gmail.com> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <CAJ-VmomWnAvsVPcK0mfFECvFw_FKcja1m3NE9ue=TOkF%2Bx14Xg@mail.gmail.com> <CANY-Wm8jbtr3tiwdGQMDx8SVZKEBspGwTV7Q0wziYWsV%2Bf3BSQ@mail.gmail.com> <6140271.20111219122721@serebryakov.spb.ru> <CANY-Wm9-JTN0gvjoRv4XFMDaweoPSoZ4erTUto3Z-s1LxqGzhg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This message is in MIME format. The first part should be readable text,
while the remaining parts are likely unreadable without MIME-aware tools.
--0-2065991131-1324317421=:64681
Content-Type: TEXT/PLAIN; CHARSET=ISO-8859-1
Content-Transfer-Encoding: 8BIT
Content-ID: <20111221152941.C64681@sola.nimnet.asn.au>
[performance@ & current@ ccs trimmed, I'm not subscribed. Feel free ..]
On Mon, 19 Dec 2011, Samuel J. Greear wrote:
> 2011/12/19 Lev Serebryakov <lev@freebsd.org>:
> > Hello, Samuel.
> > You wrote 15 ˙˙˙˙˙˙˙˙˙˙˙˙˙˙ 2011 ˙˙., 16:32:47:
> >
> >> Other benchmarks in the Phoronix suite and their representations are
> >> similarly flawed, _ALL_ of these results should be ignored and no time
> >> should be wasted by any FreeBSD committer further evaluating this
> >> garbage. (Yes, I have been down this rabbit hole).
I downloaded the sources the other night, poked around a bit trying to
suss out the test environment and FreeBSD dependencies. Gobs of PHP and
shell scripts for those with time on their hands, but I concentrated on
*BSD installation and such for a couple of hours. Observations below.
> > Here is one problem: we have choice from three items:
> >
> > (1) Make FreeBSD looks good on benchmarks by "fixing" FreeBSD
Or use benchmarks and kernel tuning to suit, where FreeBSD can shine :)
> > (2) Make FreeBSD looks good on benchmarks by "fixing" Phoronix
> > (communication with them, convincing, that they benchamrks are unfare
> > / meaningless, ets)
I've no idea whether GPLv3 really allows us to fix it ourselves, but the
general orientation is entirely Linux, with {free,net}bsd as 'distros',
so to speak. No blame there, just so long as that emphasis is clear.
> > (3) Lose [potential] userbase.
> >
> > You know, that these benchmarks are bad. I know. But potential (and
> > even some current!) user doesn't. And it seems, that these benchmarks
> > become popular over Internet.
> >
> > --
> > // Black Lion AKA Lev Serebryakov <lev@FreeBSD.org>
Self-selected, like a 'Standard & Poors' of the OS 'market'? :) People
who choose OS by fan base have already made their choice, and were never
'ours' to lose. Recall the Benchmark Battles between Windows and Linux?
> Here is where you completely derail the train, let me paste again what
> I said before.
>
> ...
> Take the first test as an example, Blogbench read. This doesn't raise
> any red flags, right? At least not until you realize that Blogbench
> isn't a read test, it's a read/write test. So what they have done here
> is run a read/write test and then thrown away the write results for
> both platforms and reported only the read results. If you dig down
> into the actual results,
> http://openbenchmarking.org/result/1112113-AR-ORACLELIN37 -- you will
> see two Blogbench numbers, one for read and another for write. These
> were both taken from the same Blogbench run, so FreeBSD optimizes
> writes over reads, that's probably a good thing for your data but a
> bad thing when someone totally misrepresents benchmark results.
> ...
>
> FreeBSD actually does _BETTER_ (subjectively) in this test than the
> Linux system when you look at what is really going on. FreeBSD is
> favoring writes, which is _GOOD_. FreeBSD does not need to be fixed,
> the benchmarks need to be fixed to represent reality rather than
> throwing half of the results in the trash. To be quite frank, "fixing"
> FreeBSD to look good on this benchmark will make it a worse real-world
> OS. But you guys go ahead and foot-shoot over these ridiculous
> benchmarks all you want.
>
> Sam
I think the notion that installing FreeBSD with no tuning at all for
particular types of work can give comparable results is flawed, when
optimising for widely varying types of workload is normally expected.
Noone expects a database, file or web server, probably headless, to be
configured anything like the same as, say, a scientific workstation or a
multimedia box or a high-performance router or ..
I've only installed Linux twice, Debian Etch and Lenny. I soon gave up
trying to install Lenny sans X and Gnome. I'm sure it can be done, by
fighting the line of least resistance. My point is that out of the box,
basic configuration and (I suspect) tuning of FreeBSD and Linux systems
has quite a different emphasis, and likely expected workload/s.
One thing I'd like to see is even 'ps -auxww' listings of these setups
while actually running these tests. Not only PHP and X but all sorts of
stuff gets installed and some are presumed to be running on top of the
benchmarks per se, NetBSD even having a jdk dependency; I was a little
unnerved to see the substantial list of packages to install seen in:
./pts-core/external-test-dependencies/xml/freebsd-packages.xml
presumably, where the listed executables are not found, installed by:
./pts-core/external-test-dependencies/scripts/install-freebsd-packages.sh:
#!/bin/sh
# FreeBSD package installation
echo "Please enter your root password below:" 1>&2
su root -c "PACKAGESITE=\"ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7-stable/Latest/\" pkg_add -r $*"
exit
Hmm. Would the 'ordinary user' of this software be expected to notice
and adjust PACKAGESITE for later versions? I admit to not having read
the substantial docs - it's an admirably large body of work, no mistake
- but I've spent too long 'down this rabbit hole' already.
I find the results on this page very strange, but perhaps indicative:
http://www.phoronix.com/scan.php?page=article&item=debian_kfreebsd_h210&num=1
Here we see scant difference in results between Debian running FreeBSD
7.3 or 8.0 or Linux 2.6.32 kernels, yet native FreeBSD 7.3 and 8.0
installations apparently run far slower, especially on the gzip test!
Does this imply that given the similar kernel speed, Debian GNU userland
performs so dramatically better than FreeBSD userland? Or does it
perhaps point to the default tuning of the FreeBSD systems compared to
(here) Debian, for these particular tests? Indeed, `which gzip`?
And yes, FreeBSD could sure use some sort of tuning 'profiles' mechanism
to be able to preconfigure systems for at least several vastly different
types of workload. Nate Lawson used to talk about this, then in respect
to simple 'laptop vs desktop' scenarios, but we've since seen volumes
written, mostly in lists but some wikis, parts of the Handbook, guides
for performance tuning etc, scarcely accessible to J. Random Installer.
A set of tunings for these Phoronix benchmarks might be a good start?
cheers, Ian
--0-2065991131-1324317421=:64681--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20111220033328.I64681>