Date: Mon, 20 Sep 1999 19:59:11 +0930 (CST) From: Mark Newton <newton@atdot.dotat.org> To: freebsd-hackers@freebsd.org Subject: RE: Some more commentary and results on 'postmark' (fwd) Message-ID: <199909201029.TAA48175@atdot.dotat.org>
next in thread | raw e-mail | index | archive | help
I forwarded Brad Knowles' comments about the PostMark benchmarking suite
to a NetApp engineer I happen to have known since my teens. Selected
portions of his comments are included below (with permission) for your
entertainment and elucidation.
- mark
A NetApp person wrote:
> Brad Knowles wrote:
>
> > Both the client and server systems were 90%+ *IDLE* during all tests.
>
> I/O bound is fun! :)
>
> > The benchmark has a number of problems. The 'postmark' program
> > isn't forking at all, so there is a serious bottleneck in the process
> > itself, especially whenever a read is issued. It doesn't really give
> > us an accurate representation of a multi-tasking load. Most
> > NFS servers have a multitasking load so it isn't really a fair test.
>
> I don't think Jeff was really all that interested in accurate comparisons of
> multi-tasking NFS performance -- for that, we've got SPEC SFS, which has
> been the standard method of comparing NFS performance for more than a
> decade.
>
> > The benchmark shows pretty clearly the inefficiency of large UFS
> > directories. Putting 20000 files in a single directory is not fun,
> > and it seriously skews the test results considering what the benchmark
> > is supposed to be testing.
>
> Jeff's favourite problem domain is mail services, which have traditionally
> been lots-of-small-files-in-a-directory stuff. His benchmark reflects that
> focus. Thankfully, you can tweak the options to make it reflect some (but
> not all) other problem domains.
>
> > It seems pretty clear to me that this benchmark has been designed
> > to show-off the netapp in the best possible light and its competitors
> > in the worst possible light. Well, ok, that may be an overly-harsh
> > assessment, but it is still true to some degree.
>
> Actually, it's not true to *any* degree, and I know because I talked to Jeff
> Katcher (and his boss at the time, Andy Watson) whilst he was developing it
> and writing the white paper.
>
> PostMark was written to shame Sun and HP into improving their single-task
> NFS CLIENT performance by showing them how much better FreeBSD and Linux
> performed. Pure and simple.
>
> Jeff stopped coding as soon as he had something which was vaguely tunable to
> reflect different application loads (you don't have to have a fixed ratio of
> files to transactions, guys) and which showed the kinds of performance
> problems we'd seen with real live applications and the latest revisions of
> the commercial NFS clients which, thanks to introduction of a few more
> internal abstraction layers, were considerably slower than their
> predecessors.
>
> > The benchmark is seriously flawed.
>
> <laughter> To paraphrase a hundred posts in freebsd-security: don't complain
> unless you're willing to write the code that fixes the problem, or at least
> suggest implementable solutions to the author. I'm sure Jeff will be more
> than happy to revise the benchmark if time permits, and I'll be forwarding
> the posts to him so he's got some impetus. :)
--------------------------------------------------------------------
I tried an internal modem, newton@atdot.dotat.org
but it hurt when I walked. Mark Newton
----- Voice: +61-4-1620-2223 ------------- Fax: +61-8-82231777 -----
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199909201029.TAA48175>