From owner-freebsd-stable@FreeBSD.ORG Wed Dec 26 20:13:53 2007 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3275E16A421 for ; Wed, 26 Dec 2007 20:13:53 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by mx1.freebsd.org (Postfix) with ESMTP id A9C3113C4E3 for ; Wed, 26 Dec 2007 20:13:52 +0000 (UTC) (envelope-from cracauer@koef.zs64.net) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by koef.zs64.net (8.14.1/8.14.1) with ESMTP id lBQJeIcm030312; Wed, 26 Dec 2007 20:40:18 +0100 (CET) (envelope-from cracauer@koef.zs64.net) Received: (from cracauer@localhost) by koef.zs64.net (8.14.2/8.14.1/Submit) id lBQJeIRc030311; Wed, 26 Dec 2007 14:40:18 -0500 (EST) (envelope-from cracauer) Date: Wed, 26 Dec 2007 14:40:18 -0500 From: Martin Cracauer To: Brett Glass Message-ID: <20071226194018.GA30004@cons.org> References: <200712220531.WAA09277@lariat.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200712220531.WAA09277@lariat.net> User-Agent: Mutt/1.4.2.3i Cc: stable@freebsd.org Subject: Re: SMP on FreeBSD 6.x and 7.0: Worth doing? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Dec 2007 20:13:53 -0000 Brett Glass wrote on Fri, Dec 21, 2007 at 10:31:24PM -0700: > > As has been reported in some other messages on this list, Linux is > currently blowing FreeBSD away. It's taking as much as 20% less > time to get through the benchmark, depending on exactly how the > random shuffle came out. This is with 4 GB RAM, the GENERIC FreeBSD > SMP kernel (using SCHED_ULE), and aufs as the storage schema for Squid. How much CPU load is there on the server during this benchmark? As I have measured and reported since 1997, if you have CPU load and do throughput timing at the same time, then FreeBSD always defaults to satisfy the CPU eating processes and starves network (at least TCP) and Linux is the other way round, keeps the throughput up but CPU needing processes lose out. Kind of opposite of what you'd expect from the ninja macho networker OS (FreeBSD) and the desktop friendly OS (Linux). Anyway, the real question is whether you get anything in return, aka is there anything running on the server that uses CPU but is not directly bound to the network throughput and hence gets work done quicker. > It appears, though I'd need to instrument the code more to be sure, > that the slowdown is coming from file I/O. I could measure the above effects with things coming out of nowhere, aka data made up by a process. If you are observing the same phaenomenon then it has nothing to do with file I/O. > Could it be that there > less concurrency or more overhead in FreeBSD file operations than > there is in Linux? Even with SoftUpdates turned on, the cache > volume mounted with -noatime, and aufs (which uses kqueues -- a > FreeBSD invention -- to optimize multithreaded disk access), the > benchmark shows FreeBSD losing out. Why? All that async and softupdates stuff only matters if you use a lot of small files, which you shouldn't do in the first place if you are building a cache of some sorts. In all likelyness the cache that you are building is satisfying requests out of the RAM, even if the cache is file-backed. You need to verify that before you can blame file I/O. So the real questions right now are: - how much CPU load (post top output) - how much disk activity (post iostat or vmstat output) - can't hurt to have a netstat throughput output, too Martin -- %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Martin Cracauer http://www.cons.org/cracauer/ FreeBSD - where you want to go, today. http://www.freebsd.org/