From owner-freebsd-current@FreeBSD.ORG Mon Feb 6 13:00:37 2006 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 679BC16A420 for ; Mon, 6 Feb 2006 13:00:37 +0000 (GMT) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6754F43D48 for ; Mon, 6 Feb 2006 13:00:36 +0000 (GMT) (envelope-from andre@freebsd.org) Received: (qmail 61624 invoked from network); 6 Feb 2006 12:58:30 -0000 Received: from dotat.atdotat.at (HELO [62.48.0.47]) ([62.48.0.47]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 6 Feb 2006 12:58:30 -0000 Message-ID: <43E74872.7000002@freebsd.org> Date: Mon, 06 Feb 2006 14:00:34 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8b) Gecko/20050217 MIME-Version: 1.0 To: Andrew Gallatin References: <17379.56708.421007.613310@grasshopper.cs.duke.edu> In-Reply-To: <17379.56708.421007.613310@grasshopper.cs.duke.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-current@freebsd.org Subject: Re: machdep.cpu_idle_hlt and SMP perf? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Feb 2006 13:00:37 -0000 Andrew Gallatin wrote: > Why dooes machdep.cpu_idle_hlt=1 drop my 10GbE network rx > performance by a considerable amount (7.5Gbs -> 5.5Gbs)? > > I've (blindly) tried leaving machdep.cpu_idle_hlt=1 enabled > and playing with the vast array of kern.sched.ipiwakeup.* sysctls, > but receive performance remains limited to ~5.5Gb/sec or less. > > This is an 'AMD Athlon(tm) 64 X2 Dual Core Processor 3800+' running > FreeBSD-current as of about one week ago. The interrupt load is > about 22,000 device interrupts/sec (ithreaded). Interestingly, > the more I decrease the interrupt load by increasing the interrupt > coalescing timer, the worse the machdep.cpu_idle_hlt=1 case does. > > Is this just a case of the wakeup IPI taking a long time or blocking > on some lock? This may be the same problem OpenBSD has fixed last year in the handling of the idle loop. From the kerneltrap posting: % "In the following article, Bob provides a first-person account of tracking % down what began simply as a RAID performance issue, but ultimately turned % out to be a problem with the idle loop that when fixed resulted in an % impressive performance boost. Bob noted, "the idle loop is where the kernel % spins when there is no work to do in userland, because of this, it's also % where we catch and service many of our interrupts from drivers that may queue % work to the device and then tsleep waiting for an interrupt from the card % saying the work is done." Bob went on to explain that prior to today's fix, % interrupts were handled appropriately when there was userland work happening, % but not when there was nothing happening in userland and the kernel was simply % waiting for device input/output. Read on for Bob's full account of the day, % leading up to the discovery of the problem and the implementation of the fix, % including performance numbers." Here is the more specific information: http://kerneltrap.org/node/5169 First commit message: http://marc.theaimsgroup.com/?l=openbsd-cvs&m=111692513727274&w=2 The MFC with all changes in one commit message: http://marc.theaimsgroup.com/?l=openbsd-cvs&m=111859519015510&w=2 http://www.openbsd.org/cgi-bin/cvsweb/src/sys/arch/i386/i386/locore.s http://www.openbsd.org/cgi-bin/cvsweb/src/sys/arch/i386/i386/apm.c -- Andre