From owner-freebsd-current@FreeBSD.ORG Wed Dec 21 22:03:02 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 11059106566B for ; Wed, 21 Dec 2011 22:03:02 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id C3B2E8FC0C for ; Wed, 21 Dec 2011 22:03:01 +0000 (UTC) Received: by vbbfr13 with SMTP id fr13so10758410vbb.13 for ; Wed, 21 Dec 2011 14:03:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=js6bD65PmfdGiIL5sFF18wlEcKAkAHoriR2JU1nhS18=; b=ExNfkqLj4OvpcuOtkxN3y0rx3y5bxGrfLBrFFFo+69r6unqqpS9wGeyNe5BCNeXQ+k tq/Q8BEaQuuf75JwGe3lJNTgOZGyWAoVi/xwP7i/B+6u9gaiS3Baz4c579Mjzvdjw6+Y tYIBOVDV+KUr8KH7iLeq7tM50RFw7T+6L/dSU= MIME-Version: 1.0 Received: by 10.52.94.6 with SMTP id cy6mr5220490vdb.28.1324504981115; Wed, 21 Dec 2011 14:03:01 -0800 (PST) Received: by 10.220.194.131 with HTTP; Wed, 21 Dec 2011 14:03:01 -0800 (PST) In-Reply-To: <4EF25468.9040204@gmail.com> References: <4EF25468.9040204@gmail.com> Date: Wed, 21 Dec 2011 14:03:01 -0800 Message-ID: From: Freddie Cash To: FreeBSD Content-Type: text/plain; charset=UTF-8 Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Dec 2011 22:03:02 -0000 On Wed, Dec 21, 2011 at 1:49 PM, Johan Hendriks wrote: > Nice page, but one thing i do not get is the following. > > [quote] > If you compare FreeBSD / GCC 4.2.1 against, for example, Ubuntu / GCC 4.7 > then the results are unlikely to tell you anything meaningful about FreeBSD > vs Ubuntu. > [/quote] > > That is a little strange in my opinion. > It tells me that FreeBSD falls more and more behind on Linux. > The reason is or could be that FreeBSD cannot or will not include GCC 4.7 > and that FreeBSD will not be on par with Linux anymore. When benchmarking two systems, you need to make sure that everything possible is the same (constants) and that the only differences between the two systems are what you want to benchmark (variables). For example, if you want to compare the performance of GCC-compiled binaries, then you would use the same hardware host, the same OS install, the same source code, and only change the compiler versions used to compile the "benchmark" binaries. That way, the only variable is "version of GCC", everything else is constant, and thus the benchmark is actually testing the performance of GCC. Likewise, if you want to benchmark the performance of two OSes, you need to eliminate as many variables as possible: - same hardware - running the same benchmark binaries - using the same versions of GCC - using the same filesystems - etc That gives you the starting point. Then, you modify one of the constants above, and re-run the benchmarks. Then you modify one more of the constants above, and re-run the benchmarks. Etc. Each time, you vary only 1 thing, so that you can measure the impact of that *ONE* thing. Comparing "random binary compile with GCC X on FreeBSD Y on filesystem Z on hardware config A" against "random binary built with GCC Q on Linux R on fileystem S on hardware config B" doesn't show anything. Was the performance difference due to hardware? Filesystem? OS? GCC version? Something else? You can't use a shotgun the thread a needle. :) > And what if in the future LLVM gets on par with Linux, is it stil fair to > compare FreeBSD with Linux? Then you add "compiler suite" to the list of variables, and you make it a constant in the first run, and then vary it one piece at a time in later runs, to isolate whether or not it affects performance. In order to do a proper comparison of any two "things", you have to first make them as equal as possible, and then vary things one bit at a time until you are at the "default" configuration for each. Only then can you really, truly, empirically say why A is better/faster/more-uber than B. Unfortunately, doing it "right" requires a lot of time, effort, time, and more time. -- Freddie Cash fjwcash@gmail.com