From owner-freebsd-hackers@FreeBSD.ORG Sat Apr 11 21:39:00 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E7881065676 for ; Sat, 11 Apr 2009 21:39:00 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 15D008FC1C for ; Sat, 11 Apr 2009 21:39:00 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id A647446B39; Sat, 11 Apr 2009 17:38:59 -0400 (EDT) Date: Sat, 11 Apr 2009 22:38:59 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Andrew Brampton In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers@freebsd.org Subject: Re: FreeBSD memguard + spinlocks X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2009 21:39:01 -0000 On Sat, 11 Apr 2009, Andrew Brampton wrote: > Thanks very much for your detailed reply. I'm slowly understanding how > everything in FreeBSD fits together, and I appreciate your help. > > I've been given a project to take over, and all of the design decisions were > made before I started working on it, thus I'm playing catch up. One of the > decisions was to implement their own version of a spin lock, which literally > looks something like this: > > lock_aquire() { > critical_enter(); > while (! lockHeld ) {} > lockHeld++; > } > > This was actually the code tripping up MemGuard, as it is inside a critical > section, which MemGuard is unable to sleep within. This is all running > inside a kthread_create thread (I'm unsure of the proper name of this type > of thread). > > Anyway, that is why I also asked about a lighter weight spin lock (perhaps > similar to this one). I tempted to replace this custom spinlock with the > standard MTX_DEF, however I'm unsure of its impact. The custom spin lock > seems quick and light to acquire, and it does not concern me that a > interrupt can potentially interrupt the code. > > On a related note, if you change the lock in memguard to a MTX_SPIN, it > panics the kernel during boot. So that is not an option :) I was only using > memguard because I suspected memory being used after it was freed. However, > I think I will either change my locks to MTX_DEF or live without memguard. > > I realise I've not really asked any questions, but I would be grateful for > any insights anyone may have. Andrew My advice, unless you're definitely executing code in fast interrupt contexts, is to simply use the FreeBSD default mutex primitive instead of either a custom-build spinlock or a FreeBSD MTX_SPIN mutex. The default mutex is adaptive, and will spin when contending the lock unless the thread holding the lock isn't executing, in which case it will fall back on a context switch. Our mutexes also make correct use of memory barriers, which the above example code doesn't appear to, so will work on systems that have weaker memory ordering properties. Using the default mutex scheme also allows you to take advantage of WITNESS, our lock order verifier, which proves a really useful tool when a lot of locks are in flight. The critical sections you're using above may not have the effect you intend: they prevent preemption by another thread, and they prevent migration to another CPU, but they don't prevent fast interrupt handlers from executing. Any synchronization with a fast interrupt handler needs to be done either using spinlocks, or other atomic operations. Robert N M Watson Computer Laboratory University of Cambridge