From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 29 11:54:37 2007 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 44AE616A404; Thu, 29 Mar 2007 11:54:37 +0000 (UTC) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (comp.chem.msu.su [158.250.32.97]) by mx1.freebsd.org (Postfix) with ESMTP id 38EA213C468; Thu, 29 Mar 2007 11:54:33 +0000 (UTC) (envelope-from yar@comp.chem.msu.su) Received: from comp.chem.msu.su (localhost [127.0.0.1]) by comp.chem.msu.su (8.13.4/8.13.4) with ESMTP id l2TBQCgb077214; Thu, 29 Mar 2007 15:26:12 +0400 (MSD) (envelope-from yar@comp.chem.msu.su) Received: (from yar@localhost) by comp.chem.msu.su (8.13.4/8.13.4/Submit) id l2TBQ7W7077210; Thu, 29 Mar 2007 15:26:07 +0400 (MSD) (envelope-from yar) Date: Thu, 29 Mar 2007 15:26:07 +0400 From: Yar Tikhiy To: Robert Watson Message-ID: <20070329112606.GA72497@comp.chem.msu.su> References: <20070328082620.GA1052@dwpc.dwlabs.ca> <20070328102027.T1185@fledge.watson.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20070328102027.T1185@fledge.watson.org> User-Agent: Mutt/1.5.9i Cc: freebsd-hackers@freebsd.org, Duane Whitty Subject: Re: Locking etc. (Long, boring, redundant, newbie questions) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2007 11:54:37 -0000 On Wed, Mar 28, 2007 at 10:40:58AM +0100, Robert Watson wrote: > > Spin locks are, FYI, slower than default mutexes. The reason is that they > have to do more work: they not only perform an atomic operation/memory > barrier to set the cross-CPU lock state, but they also have to disable > interrupts to synchronize with fast interrupt handlers. In general, you > are right: you should only use a spin mutex if you are running in a fast > handler, or synchronizing with a fast handler. The one general exception > is the scheduler itself, which must protect its data structures with spin > locks in order to implement sleeping primitives. As such, the scheduler > lock and various other low level locks (such as in turnstiles) are > implemented with spin locks and not default mutexes. Since default mutexes > spin adaptively, the reduced overhead of contention experienced with spin > locks (i.e., no scheduler overhead for simple contention cases) is also > experienced with default mutexes. By the way, could the following observation of mine be related to the high cost of spin mutexes? While doing some measurements of the performance of our vlan driver in a router, I found that with RELENG_6 the pps rate through the router degraded considerably (by ~100Kpps) if its physical interfaces used hardware vlan tagging. I attributed that to the overhead of allocating and freeing a mbuf tag for each packet as it entered and then left the router. I used hwpmc and pmcstat to see which kernel functions took most time and found that critical_exit() made top 5 in the list of CPU time eaters if the network interfaces were using hardware vlan tagging. The machine was an UP amd64, but it ran FreeBSD/i386, with an UP kernel. As I can see from the code, critical_exit() may grab and later release a spin mutex. I've got a hazy recollection that our kernel memory allocator uses critical sections to protect its per-CPU structures. That's why I suspect that the effect I observed may have its roots in the behaviour of spin mutexes. Could it be so? Thanks! -- Yar