From owner-freebsd-hackers Sat Apr 13 16:54:27 1996 Return-Path: owner-hackers Received: (from root@localhost) by freefall.freebsd.org (8.7.3/8.7.3) id QAA08189 for hackers-outgoing; Sat, 13 Apr 1996 16:54:27 -0700 (PDT) Received: from phaeton.artisoft.com (phaeton.Artisoft.COM [198.17.250.211]) by freefall.freebsd.org (8.7.3/8.7.3) with SMTP id QAA08181 for ; Sat, 13 Apr 1996 16:54:25 -0700 (PDT) Received: (from terry@localhost) by phaeton.artisoft.com (8.6.11/8.6.9) id QAA05968; Sat, 13 Apr 1996 16:52:00 -0700 From: Terry Lambert Message-Id: <199604132352.QAA05968@phaeton.artisoft.com> Subject: Re: Unices are created equal, but ... To: groudier@iplus.fr (Gerard Roudier) Date: Sat, 13 Apr 1996 16:52:00 -0700 (MST) Cc: hackers@FreeBSD.org, linux-kernel@vger.rutgers.edu In-Reply-To: from "Gerard Roudier" at Apr 13, 96 10:44:24 pm X-Mailer: ELM [version 2.4 PL24] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-hackers@FreeBSD.org X-Loop: FreeBSD.org Precedence: bulk > I was implementing some performances enhancement for "Unix A" kernel. > Seems to work fine. > I had a look for another Unix in order to compare performances. > I had luck, since "Unix B" is installed on my machine on the same > hard disk. > Unix B is installed at the beginning of the disk media and Unix A at the end. > Unix B should have better IO throughput (see below if that's ok or not ok). > > Then I run the first benchmark I found to prepare the tests. > > I get the following results: Amusing. You didn't anonymize the OS version numbers; even so, I could tell the old version of FreeBSD being used because of: Unix A: > Pipe-based Context Switching Test 1318.5 22788.7 17.3 Unix B: > Pipe-based Context Switching Test 1318.5 6304.2 4.8 It is obvious that you are running very old pipe code. Because pipes are use for IPC in the tests, this pretty much invalidates your results. Running a single sample for all cases is also statistically invalid... what happen, cache effeects make "Unix A" look bad? 8-). The integer artimatic is also a dead-giveaway for the old code. The exec and process creation tests are seriously impacted by other factors which are, in my opinion, inapropriately weighted in the benchmark and thus inherently favor "Unix A". Even so, the numbers indicate old code; even though what is being tested is not in fact a "figure of merit", "Unix B" should be able to easily beat "Unix A" on both using current releases of both OS's. The File "I/O" numbers are obviously wieghted on the basis of metadata -- as the "File Copy", which involves simultaneous read and write, clearly shows. This must have been a large number of small files so as to favor playing "fast and loose" with metadata (a well known attribute of "Unix A", and a well known implementation issue in "Unix B", if you use the default FS type in both systems and do not configure their FS mount settings comparably). You should reconfigure one or the other systems away from their default settings: increase the FS robustness defaults for "Unix A" or decrease the FS robustness defaults for "Unix B". The compiler test is similarly affected by pipe overhead (if -pipe is use) or FS settings (if /tmp files are used). In either case, the comparison is invalid, since the compiler code is otherwise identical. Did you use the same shell (ash vs. bash), linked the same way (shared libs vs. static) for the shell tests? This is a case of increasing benchmark performance at the cost of standards compliance. What are the compiler versions and compilation defaults for inlining trancendental functions for the sqrt() test? The recusrion test ws probably unfair to "Unix A", assuming all other comparisons were fair (which they weren't). > Even if this benchmark is a little questionnable, I invite people who say or > write that Unix B is FASTER than Unix A to stop, or to say or write > the OPPOSITE. Or the third option: declaring your procedures scientifically suspect and therefore the results lacking in information content? Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers.