Date: Fri, 7 Mar 2008 04:12:23 -0800 From: "Ted Mittelstaedt" <tedm@toybox.placo.com> To: "Peter Losher" <Peter_Losher@isc.org> Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: RE: FreeBSD bind performance in FreeBSD 7 Message-ID: <BMEDLGAENEKCJFGODFOCOEHJCFAA.tedm@toybox.placo.com> In-Reply-To: <47CCE982.4060201@isc.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> -----Original Message----- > From: Peter Losher [mailto:Peter_Losher@isc.org] > Sent: Monday, March 03, 2008 10:18 PM > To: Ted Mittelstaedt > Cc: freebsd-performance@freebsd.org; freebsd-questions@freebsd.org > Subject: Re: FreeBSD bind performance in FreeBSD 7 > > > Yeah, ISC just hates FreeBSD... <rolls eyes> This final report here: ftp://ftp.isc.org/isc/dns_perf/ISC-TN-2008-1.pdf is LIGHTYEARS different than the draft here: http://new.isc.org/proj/dnsperf/OStest.html The draft contains the conclusion: "...We will use Linux Gentoo 2.6.20.7 for further production testing. We brought these numbers to the attention of the FreeBSD development team, and will re-test when FreeBSD 7.1 is released..." This is completely missing in the final. Added is a bunch of praise of bind on commodity hardware. And also added is the line: "...All computers in the testbed were loaded with identical copies of FreeBSD 7.0-RELEASE..." which is missing in the draft. So in other words, it certainly appears that the final is 180 degrees opposite of it's discussion of FreeBSD. The draft appears to suggest to avoid it - the final appears to suggest to embrace it. So what, exactly may I ask, were you expecting after writing that draft? Everyone here to be happy? It almost seems to me like the draft was a trial balloon floated to get the FreeBSD developers to jump in and do some coding for you at the last minute. But, I'll say no more about that and turn towards the report - because it has some significant problems. I'll start with the beginning: "...We have been particularly interested in the performance of DNSSEC mechanisms within the DNS protocols and on the use of BIND as a production server for major zones..." OK, fine and good. However, the conclusion is rather different: "...Commodity hardware with BIND 9 software is fast enough by a wide margin to run large production name service..." What is going on here? This project started out as purely observational - merely interested in BIND performance - and ended up being a proof for the hypothesis that BIND is good enough to run large nameservers on commodity hardware. In short, the report is moving from an objective view to a subjective goal of proving BIND is kick-ass. It is interesting how the original draft conclusion IS NOT subjective with regards to BIND (it is with regards to FreeBSD of course) and uses the phrase "further production testing" implying that BIND is still under development, while the final report uses the language: "...open-source software and commodity hardware are up to the task of providing full-scale production name..." which definitely implies that BIND is "done" and ready for production. Another thing of interest concernes the OS. Microsoft Windows 2003 server is included in the first breaking point test. It is absent from the other tests. And the version chosen is old, old, it is NOT even Server 2003 R2, nor the RC of Server 2008 which is available. Why were the Windows test results even left in the published report at all? What purpose do they serve other than as a feel-good "bash Windows". If you really were interested in the results of testing, you would have wanted to know how BIND did under Windows for the other tests. But, as I pointed out, by the time the later tests were run the goal has stopped being the pure objective observational goal, and become the subjective "prove BIND is the best" goal. And as the Windows results for the breaking test were so low, it was an embarassment to keep bothering with it, so it was dropped. The report also suffers from NOT listing out the components of the HP servers and instead offering a link to HP. Yeah, how long is that link going to be valid? HP changes it's website and changes it's product line up as often as I change my underpants - a year from now, that product will be gone and a new reader will have a snowball's chance in Hell of getting the actual server specs, and I mean the chipsets in use for the disk controller, nic card, video, etc. You know, the stuff that actually -affects- the performance of different operating systems. But the biggest hole is the report conclusion and this shift from objective, to subjective, reporting. The conclusion claims BIND is great on commodity hardware but what it ONLY has proven is that BIND is great on this one specific hardware platform running a couple specific operating systems. If you really wanted to merely objectively observe BIND on commodity hardware you should have had your testers stay out of the setup of the OS and platform. You should have called up the developers of the various operating systems you were going to use - Microsoft among them - and told them to each send in a group that would build a server to their spec. You should have merely set a maximum limit that the server could cost that was in line with commodity server hardware costs - something like $2K and it had to be name-brand, for example - and let all of the vested interest groups do their best to create a server that would run as fast as they could in those constraints. In short, if the testers are setting out to prove BIND is really powerful, they are essentially trying to write a benchmark. And the way you do that is by deliberatly pulling all the stops out to make your stuff run as lickety-split as possible - then you document the crap out of everything you did to make it run lickety-split, so that anyone else can come along, set up the stuff the same way you did, and then get the same results. Benchmarks are subjective and they are expected to be subjective - but when you write them, your admitting your testers are being subjective. In that case, there is no point in having an OS bake-off since your going to have your testers select the OS that will give the best shine to your product. The report needs to make up it's mind what it's actually trying to accomplish, objective, or subjective, reporting. Ted
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BMEDLGAENEKCJFGODFOCOEHJCFAA.tedm>