From owner-freebsd-performance@FreeBSD.ORG Tue Jun 13 19:34:54 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 51A6316A41A for ; Tue, 13 Jun 2006 19:34:54 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from nz-out-0102.google.com (nz-out-0102.google.com [64.233.162.207]) by mx1.FreeBSD.org (Postfix) with ESMTP id 067C543D70 for ; Tue, 13 Jun 2006 19:34:45 +0000 (GMT) (envelope-from kip.macy@gmail.com) Received: by nz-out-0102.google.com with SMTP id 13so1588274nzn for ; Tue, 13 Jun 2006 12:34:45 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=GSwUzKjwGtXzh1ZjzGJPfvhhku7JwAHM/r/siISETSBROle55LtPml718AKMKjvU2mLekZ392gR8ajhTvsc3wq9QtuJRCVIdFdFX/IffvNK0vPm4X2PZeDUqdfP0mcSQ66drQPH4gYOyjQYx4yoeKYSoIj6GTHrFHtRdVfyIsFw= Received: by 10.65.59.4 with SMTP id m4mr3359480qbk; Tue, 13 Jun 2006 12:34:44 -0700 (PDT) Received: by 10.65.231.11 with HTTP; Tue, 13 Jun 2006 12:34:44 -0700 (PDT) Message-ID: Date: Tue, 13 Jun 2006 12:34:44 -0700 From: "Kip Macy" To: "Robert Watson" In-Reply-To: <20060613105930.N34121@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <20060612195754.72452.qmail@web33306.mail.mud.yahoo.com> <20060612210723.K26068@fledge.watson.org> <20060612203248.GA72885@xor.obsecurity.org> <200606130715.52425.davidxu@freebsd.org> <20060613105930.N34121@fledge.watson.org> X-Mailman-Approved-At: Tue, 13 Jun 2006 20:51:03 +0000 Cc: Scott Long , kmacy@freebsd.org, David Xu , Kris Kennaway , freebsd-performance@freebsd.org, danial_thom@yahoo.com Subject: Re: Initial 6.1 questions X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: kmacy@fsmware.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jun 2006 19:34:54 -0000 I have a number of issues with our current locking regime and our propensity for disabling interrupts. I have in mind some ideas for reducing interrupt disabling and eliminating scheduling contention except in the case of one cpu stealing a thread from another cpu's runqueue. I'll try to dash that off early this evening. This should also greatly reduce the overhead of timer interrupts. -Kip On 6/13/06, Robert Watson wrote: > > On Tue, 13 Jun 2006, David Xu wrote: > > > On Tuesday 13 June 2006 04:32, Kris Kennaway wrote: > >> On Mon, Jun 12, 2006 at 09:08:12PM +0100, Robert Watson wrote: > >>> On Mon, 12 Jun 2006, Scott Long wrote: > >>>> I run a number of high-load production systems that do a lot of network > >>>> and filesystem activity, all with HZ set to 100. It has also been shown > >>>> in the past that certain things in the network area where not fixed to > >>>> deal with a high HZ value, so it's possible that it's even more > >>>> stable/reliable with an HZ value of 100. > >>>> > >>>> My personal opinion is that HZ should gop back down to 100 in 7-CURRENT > >>>> immediately, and only be incremented back up when/if it's proven to be > >>>> the right thing to do. And, I say that as someone who (errantly) pushed > >>>> for the increase to 1000 several years ago. > >>> > >>> I think it's probably a good idea to do it sooner rather than later. It > >>> may slightly negatively impact some services that rely on frequent timers > >>> to do things like retransmit timing and the like. But I haven't done any > >>> measurements. > >> > >> As you know, but for the benefit of the list, restoring HZ=100 is often an > >> important performance tweak on SMP systems with many CPUs because of all > >> the sched_lock activity from statclock/hardclock, which scales with HZ and > >> NCPUS. > > > > sched_lock is another big bottleneck, since if you 32 CPUs, in theory you > > have 32X context switch speed, but now it still has only 1X speed, and there > > are code abusing sched_lock, the M:N bits dynamically inserts a thread into > > thread list at context switch time, this is a bug, this causes thread list > > in a proc has to be protected by scheduler lock, and delivering a signal to > > process has to hold scheduler lock and find a thread, if the proc has many > > threads, this will introduce long scheduler latency, a proc lock is not > > enough to find a thread, this is a bug, there are other code abusing > > scheduler lock which really can use its own lock. > > I've added Kip Macy to the CC, who is working with a patch for Sun4v that > eliminates sched_lock. Maybe he can comment some more on this thread? > > Robert N M Watson > Computer Laboratory > Universty of Cambridge >