Date: Tue, 20 Dec 2011 15:29:25 -0800 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: "O. Hartmann" <ohartman@zedat.fu-berlin.de> Cc: FreeBSD Stable Mailing List <freebsd-stable@freebsd.org>, freebsd-performance@freebsd.org, Current FreeBSD <freebsd-current@freebsd.org>, "Samuel J. Greear" <sjg@evilcode.net>, Igor Mozolevsky <igor@hybrid-lab.co.uk> Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server Message-ID: <20111220232925.GA55953@icarus.home.lan> In-Reply-To: <4EF1121F.9010209@zedat.fu-berlin.de> References: <4EE2AE64.9060802@m5p.com> <4EE88343.2050302@m5p.com> <CAFHbX1%2B5PttyZuNnYot8emTn_AWkABdJCvnpo5rcRxVXj0ypJA@mail.gmail.com> <4EE933C6.4020209@zedat.fu-berlin.de> <CAPjTQNEJDE17TLH-mDrG_-_Qa9R5N3mSeXSYYWtqz_DFidzYQw@mail.gmail.com> <20111215024249.GA13557@icarus.home.lan> <4EE9A2A0.80607@zedat.fu-berlin.de> <CADWvR2hpooV4Sdkzor6oHpXWkLCz0z0UxBB_-nnrnYLeUY5oow@mail.gmail.com> <CANY-Wm8DmFxm7xwXfRcyBSu4z6fTOap%2By=Mhx%2BuBP=uJN4g3fw@mail.gmail.com> <4EF1121F.9010209@zedat.fu-berlin.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Dec 20, 2011 at 11:54:23PM +0100, O. Hartmann wrote: > On 12/20/11 22:45, Samuel J. Greear wrote: > > http://www.osnews.com/story/25334/DragonFly_BSD_MP_Performance_Significantly_Improved > > > > PostgreSQL tests, see the linked PDF for #'s on FreeBSD, DragonFly, Linux > > and Solaris. Steps to reproduce these benchmarks provided. > > > > Sam > > > > On Tue, Dec 20, 2011 at 1:20 PM, Igor Mozolevsky <igor@hybrid-lab.co.uk>wrote: > > > >> Interestingly, while people seem to be (arguably rightly) focused on > >> criticising Phoronix's benchmarking, nobody has offered an alternative > >> benchmark; and while (again, arguably rightly) it is important to > >> benchmark real world performance, equally, nobody has offered any > >> numbers in relation to, for example, HTTP or SMTP, or any other "real > >> world"-application torture tests done on the aforementioned two > >> platforms... IMO, this just goes to show that "doing is hard" and > >> "criticising is much easier" (yes, I am aware of the irony involved in > >> making this statement, but someone has to!) > >> > >> > >> Cheers, > >> Igor M :-) > >> _______________________________________________ > >> freebsd-current@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-current > >> To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > >> > > Thanks for those numbers. > Impressive how Matthew Dillon's project jumps forward now. And it is > still impressive to see that the picture is still in the right place > when it comes to a comparison to Linux. > Also, OpenIndiana shows an impressive performance. Preface to my long post below: The things being discussed here are benchmarks, as in "how much work can you get out of Thing". This is VERY DIFFERENT from testing interactivity in a scheduler, which is more of a test that says "when Thing X is executed while heavier-Thing Y is also being executed, how much interaction is lost in Thing X". The reason people notice this when using Xorg is because it's visual, in an environment where responsiveness is absolutely mandatory above all else. Nobody is going to put up with a system where during a buildworld they go to move a window or click a mouse button or type a key and find that the window doesn't move, the mouse click is lost, or the key typed has gone into the bit bucket -- or, that those things are SEVERELY delayed, to the point where interactivity is crap. I just want to make that clear to folks. This immense thread has been with regards to the latter -- bad interactivity/responsiveness on a system which was undergoing load that SHOULD be distributed "more evenly" across the system *while* keeping interactivity/responsiveness high. Historically nice/renice has been used for this task, but that was when kernels were a little less complex and I/O subsystems were less complex. Remember: we've now got schedulers for each type of thing, and who gets what priority? You get my point I'm sure. So remember: this was to discuss that aspect, with regards to ULE vs. 4BSD schedulers. Now, back to the benchmarks: This also interested me: * Linux system crashed http://leaf.dragonflybsd.org/mailarchive/kernel/2011-11/msg00008.html * OpenIndiana system crashed same way as Linux system http://leaf.dragonflybsd.org/mailarchive/kernel/2011-11/msg00017.html I cannot help but wonder if the Linux and OpenIndiana installations were more stressful on the hardware -- getting more out of the system, maybe resulting in increased power/load, which in turn resulted in the systems locking up (shoddy PSU, unstable mainboard, MCH problems, etc.). My point is that Francois states these things in such a way to imply that "DragonflyBSD was more stable", when in fact I happen to wonder the opposite point -- that is to say, Linux and OpenIndiana were trying to use the hardware more-so than DragonflyBSD, thus tickled what may be a hardware-level problem. > But this is only one suite of testing. Scientific Linux is supposed to > give the best performance for scientifi purposes, i.e. for longhaul > calculations, much numerical stuff. It outperforms in a typical server > application FreeBSd, were "FreeBSD shoulkd have the power to serve". > > Is the postgresql benchmark the only way to benchmark? I sure hope not. But you know what's equally as interesting? This: http://people.freebsd.org/~kris/scaling/ Specifically circa 2008: http://people.freebsd.org/~kris/scaling/4cpu-pgsql.png http://people.freebsd.org/~kris/scaling/pgsql-16cpu-2.png http://people.freebsd.org/~kris/scaling/pgsql-16cpu.png Now, I don't know if what was used in those ("pgsql sysbench") was the same thing as "pg_bench" in the DragonflyBSD tests, but if so, the numbers are different to a point that is preposterous. There's also this: http://people.freebsd.org/~kris/scaling/pgsql-ncpu.png Now, compare those numbers to the TPS numbers shown here: http://dl.wolfpond.org/Pg-benchmarks.pdf So um... yeah. Now, if someone here is going to say "well, what was tested by Kris was FreeBSD 7.0, while what was tested by Francois was FreeBSD 9.0, and there have been improvements", then I ask that someone show me where the improvements are that would exhibit a 4-8x performance increase in some cases. This rambling of mine is the same rambling I posted earlier in this thread. There needs to be a consistent, standardised way of testing this stuff. Every system tested tuned the exact same way, software configured the same way, absolutely no quirks applied, etc.. Otherwise we end up with "mixed results" as shown above. Much to the disapproval of others, the Phoronix test suite is supposed to be that "standard". Meaning, it's a suite you're supposed to be able to install and thus ensures that, aside from compiler used and any system tests, that the same code is being used regardless of what system and OS it's on. Have I ever used it? No. And it's important that I admit that up front, because being honest is necessary. > Well, this inspires me to gather together all the benchmarks someone > could find. There were lots of compalins about FreeBSD's poor > performance with BIND - once a domain of FreeBSD. Network performance > seems also to be an issue if it comes to scalability. > It would be nice to see what portion of the raw CPU/GPU power the OS > (FreeBSD, Linux ...) delivers to scientific applications. Kris Kenneway's "BIND benchmark" that was released a long time ago touched base on this. Remember: these plots show nothing other than number of queries per second correlated with number of DNS server threads (since BIND does have a 1:1 thread-to-CPU creation ratio): http://people.freebsd.org/~kris/scaling/bind-pt.png http://people.freebsd.org/~kris/scaling/bind-pt-2.png http://people.freebsd.org/~kris/scaling/bind-pt-gige.png > I only know some kind of benchmarks, BYTE UNIX benchmark, LINPACK test > ... Does someone know a site to look for a couple of benchmarks to test > > a) memory system > b) scalability (apart from pgbench) > c) network performance/throughput/network scalability > d) portion of CPU performance the system delivers for numerical > applications to the user apart from the system's own consumption > e) disk I/O performance and scalability > > it would also be nice to discuss some nice settings and performance > tunings for FreeBSD for several scenarios. I guess, starting developing > benchmarking test scenarios for several purposes would lead faster to > real numbers and non polemic than weird discussions ... All I wish is that we had some kind of "test suite" of our own, maybe as a port, maybe in the base system, which could really help with all of this. Something consistent. Now I'm switching back to discussing interactivity/responsiveness tests: Attilio Rao did comment in this thread to me, giving me some test methodologies for testing interactivity during two types of simultaneous loads -- but one involves dnetc, which I imagine means I'd need to get familiar with that whole thing. http://lists.freebsd.org/pipermail/freebsd-stable/2011-December/064936.html I haven't responded to his post yet (this thread is so long and tedious that I'm having serious problems following it + remembering all the details -- am I the only one who feels daunted by this? God I hope not), but his insights are, as always, beneficial, but also overwhelming. Furthermore, I do not have 16-core or 24-core systems to test on -- I have single-CPU, quad-core and dual-core systems to test on. I am a firm believer these are going to make up the majority of the FreeBSD userbase (desktop and server environments). Extreme hardware (e.g. quad CPU with 12 cores per CPU) can be tested too, but let's at least pick a demographic to start with. Again: the FreeBSD users and administrative community want to help. All of us do. We just need to know exactly what we should be doing to test, and what exactly we're testing for. I'll be blunt while choosing to play the Idiot Admin for a moment: I'd be much happier if someone had a tarball of shell scripts and things which could be used to test these things. Lots of things need to be kept in mind, such as if someone is running the "client" test on the same box as the "server" test, and things like "the test data is written to a local filesystem, with echo/printf statements constantly flushed" (great, now we're causing I/O load on top of our tests!), which to me means we should probably be using something like mdconfig(8) to create a temporary filesystem to store logs/data results. The KTR stuff Atillio and many others have requested, I think, will be the most beneficial way to get the developers the data they need. I had no idea about it until I found out that KTR was something completely different than ktrace. I still haven't found the time to do all of this, BTW, and for that I apologise. The reason has to do with time at work + personal desire to do it. When I get a daunting task, I tend to get... well, not depressed, but "scared" of the massive undertaking since it involves lots of recurring tests, reboots, etc. -- hours of work -- and if I get that wrong, it's wasted effort (thus wasted developer time). I want to get it right. :-) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20111220232925.GA55953>