From owner-freebsd-current@FreeBSD.ORG Wed Dec 21 00:28:08 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 239181065670; Wed, 21 Dec 2011 00:28:08 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 8E64C8FC0C; Wed, 21 Dec 2011 00:28:07 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1RdA2Q-0002E4-Ou>; Wed, 21 Dec 2011 01:28:06 +0100 Received: from e178016253.adsl.alicedsl.de ([85.178.16.253] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1RdA2Q-0003rz-DI>; Wed, 21 Dec 2011 01:28:06 +0100 Message-ID: <4EF12815.2090805@zedat.fu-berlin.de> Date: Wed, 21 Dec 2011 01:28:05 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111109 Thunderbird/8.0 MIME-Version: 1.0 To: Jeremy Chadwick References: <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <4EE933C6.4020209@zedat.fu-berlin.de> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <4EF1121F.9010209@zedat.fu-berlin.de> <20111220232925.GA55953@icarus.home.lan> In-Reply-To: <20111220232925.GA55953@icarus.home.lan> X-Enigmail-Version: undefined Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig3B0CE417BCECAA1895E10BDC" X-Originating-IP: 85.178.16.253 Cc: "Samuel J. Greear" , freebsd-performance@freebsd.org, Current FreeBSD , FreeBSD Stable Mailing List , Igor Mozolevsky Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Dec 2011 00:28:08 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig3B0CE417BCECAA1895E10BDC Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 12/21/11 00:29, Jeremy Chadwick wrote: > On Tue, Dec 20, 2011 at 11:54:23PM +0100, O. Hartmann wrote: >> On 12/20/11 22:45, Samuel J. Greear wrote: >>> http://www.osnews.com/story/25334/DragonFly_BSD_MP_Performance_Signif= icantly_Improved >>> >>> PostgreSQL tests, see the linked PDF for #'s on FreeBSD, DragonFly, L= inux >>> and Solaris. Steps to reproduce these benchmarks provided. >>> >>> Sam >>> >>> On Tue, Dec 20, 2011 at 1:20 PM, Igor Mozolevsky wrote: >>> >>>> Interestingly, while people seem to be (arguably rightly) focused on= >>>> criticising Phoronix's benchmarking, nobody has offered an alternati= ve >>>> benchmark; and while (again, arguably rightly) it is important to >>>> benchmark real world performance, equally, nobody has offered any >>>> numbers in relation to, for example, HTTP or SMTP, or any other "rea= l >>>> world"-application torture tests done on the aforementioned two >>>> platforms... IMO, this just goes to show that "doing is hard" and >>>> "criticising is much easier" (yes, I am aware of the irony involved = in >>>> making this statement, but someone has to!) >>>> >>>> >>>> Cheers, >>>> Igor M :-) >>>> _______________________________________________ >>>> freebsd-current@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current >>>> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebs= d.org" >>>> >> >> Thanks for those numbers. >> Impressive how Matthew Dillon's project jumps forward now. And it is >> still impressive to see that the picture is still in the right place >> when it comes to a comparison to Linux. >> Also, OpenIndiana shows an impressive performance. >=20 > Preface to my long post below: >=20 > The things being discussed here are benchmarks, as in "how much work > can you get out of Thing". This is VERY DIFFERENT from testing > interactivity in a scheduler, which is more of a test that says "when > Thing X is executed while heavier-Thing Y is also being executed, how > much interaction is lost in Thing X". >=20 > The reason people notice this when using Xorg is because it's visual, > in an environment where responsiveness is absolutely mandatory above al= l > else. Nobody is going to put up with a system where during a buildworl= d > they go to move a window or click a mouse button or type a key and find= > that the window doesn't move, the mouse click is lost, or the key typed= > has gone into the bit bucket -- or, that those things are SEVERELY > delayed, to the point where interactivity is crap. I whitnessed sticky, jumpy and non-responsive-for seconds FreeBSD servers (serving homes, NFS/SAMBA and PostgreSQL database (small)). Those "seconds" where enough to cut a ssh line. Not funny. Network traffic droped significantly. X/Desktop makes the problem visible, indeed. But not seeing it does not mean it isn't there. This might be the reason why FreeBSD is so much behind when it comes to X= ? >=20 > I just want to make that clear to folks. This immense thread has been > with regards to the latter -- bad interactivity/responsiveness on a > system which was undergoing load that SHOULD be distributed "more > evenly" across the system *while* keeping interactivity/responsiveness > high. Historically nice/renice has been used for this task, but that > was when kernels were a little less complex and I/O subsystems were les= s > complex. Remember: we've now got schedulers for each type of thing, > and who gets what priority? You get my point I'm sure. >=20 > So remember: this was to discuss that aspect, with regards to ULE vs. > 4BSD schedulers. >=20 > Now, back to the benchmarks: >=20 > This also interested me: >=20 > * Linux system crashed > http://leaf.dragonflybsd.org/mailarchive/kernel/2011-11/msg00008.html= >=20 > * OpenIndiana system crashed same way as Linux system > http://leaf.dragonflybsd.org/mailarchive/kernel/2011-11/msg00017.html= >=20 > I cannot help but wonder if the Linux and OpenIndiana installations wer= e > more stressful on the hardware -- getting more out of the system, maybe= > resulting in increased power/load, which in turn resulted in the system= s > locking up (shoddy PSU, unstable mainboard, MCH problems, etc.). Is FreeBSD supposed to run on dumpyard equipment? In former times, freeBSD was used on high value hardware, not the decomissioned crap with shoddy PSUs or whatsoever. If I need a server, I care about quality hardware as I do for my lab's box and my own box at home. I expect a "server garde" hardware to act like that and I expect the operating system to get the maximum out of that hardware. Otherwise it is not worth one shot. >=20 > My point is that Francois states these things in such a way to imply > that "DragonflyBSD was more stable", when in fact I happen to wonder th= e > opposite point -- that is to say, Linux and OpenIndiana were trying to > use the hardware more-so than DragonflyBSD, thus tickled what may be a > hardware-level problem. >=20 >> But this is only one suite of testing. Scientific Linux is supposed to= >> give the best performance for scientifi purposes, i.e. for longhaul >> calculations, much numerical stuff. It outperforms in a typical server= >> application FreeBSd, were "FreeBSD shoulkd have the power to serve". >> >> Is the postgresql benchmark the only way to benchmark? >=20 > I sure hope not. But you know what's equally as interesting? This: >=20 > http://people.freebsd.org/~kris/scaling/ >=20 > Specifically circa 2008: >=20 > http://people.freebsd.org/~kris/scaling/4cpu-pgsql.png > http://people.freebsd.org/~kris/scaling/pgsql-16cpu-2.png > http://people.freebsd.org/~kris/scaling/pgsql-16cpu.png >=20 > Now, I don't know if what was used in those ("pgsql sysbench") was the > same thing as "pg_bench" in the DragonflyBSD tests, but if so, the > numbers are different to a point that is preposterous. >=20 > There's also this: >=20 > http://people.freebsd.org/~kris/scaling/pgsql-ncpu.png >=20 > Now, compare those numbers to the TPS numbers shown here: >=20 > http://dl.wolfpond.org/Pg-benchmarks.pdf >=20 > So um... yeah. Now, if someone here is going to say "well, what > was tested by Kris was FreeBSD 7.0, while what was tested by Francois > was FreeBSD 9.0, and there have been improvements", then I ask that > someone show me where the improvements are that would exhibit a 4-8x > performance increase in some cases. >=20 > This rambling of mine is the same rambling I posted earlier in this > thread. There needs to be a consistent, standardised way of testing > this stuff. Every system tested tuned the exact same way, software > configured the same way, absolutely no quirks applied, etc.. Otherwise= > we end up with "mixed results" as shown above. Didn't got M. Larabel at Phoronix this half the way, except the ZFS fault= ? >=20 > Much to the disapproval of others, the Phoronix test suite is supposed > to be that "standard". Meaning, it's a suite you're supposed to be abl= e > to install and thus ensures that, aside from compiler used and any > system tests, that the same code is being used regardless of what syste= m > and OS it's on. Have I ever used it? No. And it's important that I > admit that up front, because being honest is necessary. >=20 >> Well, this inspires me to gather together all the benchmarks someone >> could find. There were lots of compalins about FreeBSD's poor >> performance with BIND - once a domain of FreeBSD. Network performance >> seems also to be an issue if it comes to scalability. >> It would be nice to see what portion of the raw CPU/GPU power the OS >> (FreeBSD, Linux ...) delivers to scientific applications. >=20 > Kris Kenneway's "BIND benchmark" that was released a long time ago > touched base on this. Remember: these plots show nothing other than > number of queries per second correlated with number of DNS server > threads (since BIND does have a 1:1 thread-to-CPU creation ratio): >=20 > http://people.freebsd.org/~kris/scaling/bind-pt.png > http://people.freebsd.org/~kris/scaling/bind-pt-2.png > http://people.freebsd.org/~kris/scaling/bind-pt-gige.png >=20 >> I only know some kind of benchmarks, BYTE UNIX benchmark, LINPACK test= >> ... Does someone know a site to look for a couple of benchmarks to tes= t >> >> a) memory system >> b) scalability (apart from pgbench) >> c) network performance/throughput/network scalability >> d) portion of CPU performance the system delivers for numerical >> applications to the user apart from the system's own consumption >> e) disk I/O performance and scalability >> >> it would also be nice to discuss some nice settings and performance >> tunings for FreeBSD for several scenarios. I guess, starting developin= g >> benchmarking test scenarios for several purposes would lead faster to >> real numbers and non polemic than weird discussions ... >=20 > All I wish is that we had some kind of "test suite" of our own, maybe a= s > a port, maybe in the base system, which could really help with all of > this. Something consistent. Why not supporting those guys at Phoronix? If we start with "our own", then we end up as you described above - not comparable, different numbers on different platforms, no normalization possible. >=20 > Now I'm switching back to discussing interactivity/responsiveness tests= : >=20 > Attilio Rao did comment in this thread to me, giving me some test > methodologies for testing interactivity during two types of simultaneou= s > loads -- but one involves dnetc, which I imagine means I'd need to get > familiar with that whole thing. >=20 > http://lists.freebsd.org/pipermail/freebsd-stable/2011-December/064936.= html >=20 > I haven't responded to his post yet (this thread is so long and tedious= > that I'm having serious problems following it + remembering all the > details -- am I the only one who feels daunted by this? God I hope > not), but his insights are, as always, beneficial, but also > overwhelming. Furthermore, I do not have 16-core or 24-core systems > to test on -- I have single-CPU, quad-core and dual-core systems to tes= t > on. I am a firm believer these are going to make up the majority of th= e > FreeBSD userbase (desktop and server environments). Extreme hardware (= e.g. > quad CPU with 12 cores per CPU) can be tested too, but let's at least > pick a demographic to start with. >=20 > Again: the FreeBSD users and administrative community want to help. Al= l > of us do. We just need to know exactly what we should be doing to test= , > and what exactly we're testing for. I'll be blunt while choosing to > play the Idiot Admin for a moment: I'd be much happier if someone had a= > tarball of shell scripts and things which could be used to test these > things. Lots of things need to be kept in mind, such as if someone is > running the "client" test on the same box as the "server" test, and > things like "the test data is written to a local filesystem, with > echo/printf statements constantly flushed" (great, now we're causing I/= O > load on top of our tests!), which to me means we should probably be > using something like mdconfig(8) to create a temporary filesystem to > store logs/data results. >=20 > The KTR stuff Atillio and many others have requested, I think, will be > the most beneficial way to get the developers the data they need. I ha= d > no idea about it until I found out that KTR was something completely > different than ktrace. >=20 > I still haven't found the time to do all of this, BTW, and for that I > apologise. The reason has to do with time at work + personal desire to= > do it. When I get a daunting task, I tend to get... well, not > depressed, but "scared" of the massive undertaking since it involves > lots of recurring tests, reboots, etc. -- hours of work -- and if I get= > that wrong, it's wasted effort (thus wasted developer time). I want to= > get it right. :-) >=20 --------------enig3B0CE417BCECAA1895E10BDC Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iQEcBAEBAgAGBQJO8SgVAAoJEOgBcD7A/5N8hgIH/3OXj5P3KAZ7bAx0Xdcg93EP qU1qvXqmbf/H4QCYxWIEk7NG+TT5MoTIoRN9393F1n1+nkn8QRjU27NLqnPwFsbx cRk8VWQXtJ58DImkfHnAhmuScfScPrS5u5CDRs5LXS7cPT9r6NPZ6uyNr6qcEA7i Y5urmT8RLtPMZIcJRJx4i8BU0Wg314QxfH2AiOjiiS8vbIcXk3gYB+2C7VnPM2Le nvo2OYPW5T3t6KSwW/T2ikXSTNgKqy6YL9hX6Q/xZOvG/ZvXS/QkVNmnKuUgroqU A28ILznYmqnACfXrV4yzJAb04SrzxxCN8BgOwOs/zdzzzuQximfETIDNn44GYpY= =rq59 -----END PGP SIGNATURE----- --------------enig3B0CE417BCECAA1895E10BDC--