From owner-freebsd-hackers  Mon Dec  2 20:36:01 1996
Return-Path: owner-hackers
Received: (from root@localhost)
          by freefall.freebsd.org (8.7.5/8.7.3) id UAA06945
          for hackers-outgoing; Mon, 2 Dec 1996 20:36:01 -0800 (PST)
Received: from brasil.moneng.mei.com (brasil.moneng.mei.com [151.186.109.160])
          by freefall.freebsd.org (8.7.5/8.7.3) with ESMTP id UAA06928
          for <hackers@freebsd.org>; Mon, 2 Dec 1996 20:35:58 -0800 (PST)
Received: (from jgreco@localhost) by brasil.moneng.mei.com (8.7.Beta.1/8.7.Beta.1) id WAA25907; Mon, 2 Dec 1996 22:35:01 -0600
From: Joe Greco <jgreco@brasil.moneng.mei.com>
Message-Id: <199612030435.WAA25907@brasil.moneng.mei.com>
Subject: Re: TCP/IP bandwidth bragging
To: proff@suburbia.net (Julian Assange)
Date: Mon, 2 Dec 1996 22:35:00 -0600 (CST)
Cc: hackers@freebsd.org
In-Reply-To: <199612030420.PAA15467@suburbia.net> from "Julian Assange" at Dec 3, 96 03:20:06 pm
X-Mailer: ELM [version 2.4 PL24]
Content-Type: text
Sender: owner-hackers@freebsd.org
X-Loop: FreeBSD.org
Precedence: bulk

> > Look, folks, benchmarks are benchmarks.  They are not
> > real world performance indicators.  They are simply
> > relative artificial performance evaluators, and as 
> > such can be influenced by a wide variety of factors,
> > including OS tweaks.  I never make the mistake of 
> > taking a benchmark's results as an absolute comparison
> > of apples and oranges.
> 
> Bench marks are not totally useless. Quite often it is hard to find
> bottle necks without them. "The system feels slower" isn't going
> to do you much good as a diagnostic tool with a monolithic kernel.
> This does not imply that they find all bottle-necks.

Hi Julian,

Well, of course!  That goes without saying.  Benchmarks exist because
they are useful in many cases.

In fact, benchmarks are particularly significant when used on the same
OS and hardware platform.  Benchmarks are less significant when
comparing different OS's on the same platform, or the same OS on 
different platforms, without some careful analysis and interpretation 
of the results.  (I think John Dyson has repeatedly talked about this). 

I do not think benchmarks are useless.  They certainly do an excellent
job of testing performance under artificial circumstances, and for many
purposes, this is useful information.  However, when comparing across
operating systems, I would tend to agree that John Dyson is correct when
stating that a direct comparison may not be particularly meaningful.

I will confess that I typically use a very unscientific set of
"benchmarks" to evaluate a new platform and to give me a rough idea
how it measures up to what I already know.  This is sufficient for
me because often I am only interested in order-of-magnitude comparisons.
I am aware of this, and I interpret the results accordingly.

... JG