From owner-freebsd-performance@FreeBSD.ORG Mon Dec 17 23:13:51 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18C8A16A417 for ; Mon, 17 Dec 2007 23:13:51 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by mx1.freebsd.org (Postfix) with ESMTP id A049113C45A for ; Mon, 17 Dec 2007 23:13:50 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by koef.zs64.net (8.14.1/8.14.1) with ESMTP id lBHMusZF098267; Mon, 17 Dec 2007 23:56:54 +0100 (CET) (envelope-from cracauer@koef.zs64.net) Received: (from cracauer@localhost) by koef.zs64.net (8.14.2/8.14.1/Submit) id lBHMusvt098266; Mon, 17 Dec 2007 17:56:54 -0500 (EST) (envelope-from cracauer) Date: Mon, 17 Dec 2007 17:56:54 -0500 From: Martin Cracauer To: Shantanu Ghosh Message-ID: <20071217225654.GA97600@cons.org> References: <963530.77689.qm@web54505.mail.re2.yahoo.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <963530.77689.qm@web54505.mail.re2.yahoo.com> User-Agent: Mutt/1.4.2.3i Cc: freebsd-performance@freebsd.org Subject: Re: freebsd vs linux: performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2007 23:13:51 -0000 Shantanu Ghosh wrote on Thu, Dec 13, 2007 at 04:07:50AM -0800: > Hi, > > I am running FreeBSD 7.0 Beta1 and Linux FC6 on two identical pieces of > hardware - Dell poweredge with intel core2 duo. Each system has 4 CPUs. I assume that means 2 CPUs with two cores each, aka socket 771 woodcrests? Please be more specific. /proc/cpuinfo > Now, in simple memory access operations, I see the freebsd system being > noticably slower than the linux system. A simple C program that copies > from one memory buffer to another, when executed in a loop executes > between 10-30% slower on freebsd, as compared to linux. The assembly > code of the program used for testing is identical in both the cases. Please provide that simple C program. Below I assume that your assembly doesn't ever call memcpy() or similar. Please let us know which Linux kernel version, I gave up on FC and don't know the FC to kernel map. Anyway... This is most likely something I experienced myself: sometime between Linux 2.6.17 and 2.6.20 they were teaching the kernel about Core2 and about the shared cache in particular. Memory task performance such as piping around gzip output used to be horrible on Core2 systems that had some system cores sharing L2 cache and others don't, such as a dual Woodcrests system which has 4 cores total of which two and two share the L2 cache. A socket 775 system with just a Core2Duo (which means all cores in the system share the single L2 cache) used to be much better than the dual Woodcrest in 2.6.17 but in 2.6.20 it was fixed. I assume this is very simply a scheduler change that now knows which cores share L2 cache and sets affinity appropriately. On a loaded system with mixed random stuff doing on this is likely not a factor anymore (because the scheduler has too many other constraints to babysit one process), but benchmarking and single-tasking can expose it. > One observation is that freebsd system performance decreases as the > size of the buffer increases. If the buffer is under 1k, both the > sytems give the same performance. freebsd performance is about 10% > slower if the buffer size is around 4k, and about 30% slower if the > buffer is around 1Mb. A benchmark like sysbench memory read operation > performs miserably on the freebsd system, compared to linux. "buffer" here means you first read bytes, then write bytes elsewhere? How do you allocate the buffer to hold this data? Alignment plays a big role here. If you can, please give us the C program, otherwise I'd like you to print the address of the buffer in both cases. > As far as I can see, the BIOS settings are identical on both the > machines. Any idea what could be going on? Make double sure that the hardware readahead that some of the socket 771 chipsets is set in an identical manner. Also, the snoop filter in 5000x chipset suc^Hffers from underengineering and should be turned off for most applications. Also, please run the stream.c benchmark on both, including the Linux binary on FreeBSD using the Linuxulator as a third run. I put a copy on http://www.cons.org/stream.c Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer http://www.cons.org/cracauer/ FreeBSD - where you want to go, today. http://www.freebsd.org/