Date: Sun, 17 Jun 2001 22:34:57 +1000 From: "Andrew Reilly" <areilly@bigpond.net.au> To: Garance A Drosihn <drosih@rpi.edu> Cc: "Albert D. Cahalan" <acahalan@cs.uml.edu>, freebsd-hackers@FreeBSD.ORG, dillon@earth.backplane.com, mhagerty@voyager.net Subject: Re: Article: Network performance by OS Message-ID: <20010617223457.A10913@gurney.reilly.home> In-Reply-To: <p05100e03b7517ec6dacf@[128.113.24.47]>; from drosih@rpi.edu on Sat, Jun 16, 2001 at 05:39:49PM -0400 References: <200106162031.f5GKVfm16209@saturn.cs.uml.edu> <p05100e03b7517ec6dacf@[128.113.24.47]>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Jun 16, 2001 at 05:39:49PM -0400, Garance A Drosihn wrote: > Mind you, I do agree that it would be very nice if we ["the > industry"] could figure out benchmarking tactics which did > not depend on the knowledge level of the person doing the > benchmark. It would also be really nice to see lasting > peace in every corner of the globe, but that also isn't > going to happen without divine intervention. Getting back > to benchmarks, the problem is that as soon as someone designs > a benchmark, some members of the competition (the "competition" > in whatever field is being benchmarked) sits down and figures > out how to "look good on that benchmark". The way that the SPEC organisation manages it (and has been doing a pretty good job on CPU/memory benchmarks over the years) is to work hard to make sure that the work done by the benchmark _is_ representative of the sorts of work that real people do with the sorts of systems where you're interested in CPU performance. That way, companies that figure out how to look good on the benchmark tend to actually make their platform behave well for a wide variety of applications. The other thing about SPEC benchmarks is that the interested parties benchmark themselves, and disclose their configurations, so that those reading the results can (a) reproduce them and (b) know what sorts of things that they should do if they want to make their own applications run as well as that. I remember that the last time this question arose, someone suggested raising a fund to buy (yes, the sad part of SPEC is that they sell the test suite to fund future developments) SpecWEB. How is that effort going, and do the folk who have volunteered to run it have access to suitably impressive hardware? (That last point is likely a stumbling block for either FreeBSD _or_ Linux. You can bet that if Microsoft wanted to win SpecWEB races they'd be able to buy better hardware than any group that's scratching to get $800 together...) For OS benchmarking, what we probably need is something like SpecWEB with Laser association(the sailing variety) one-design rules for the hardware. -- Andrew To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20010617223457.A10913>