From owner-freebsd-hackers@FreeBSD.ORG Sat Apr 11 21:04:00 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2085B106564A for ; Sat, 11 Apr 2009 21:04:00 +0000 (UTC) (envelope-from brampton@gmail.com) Received: from mail-bw0-f164.google.com (mail-bw0-f164.google.com [209.85.218.164]) by mx1.freebsd.org (Postfix) with ESMTP id 473C68FC1C for ; Sat, 11 Apr 2009 21:03:58 +0000 (UTC) (envelope-from brampton@gmail.com) Received: by bwz8 with SMTP id 8so1580895bwz.43 for ; Sat, 11 Apr 2009 14:03:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=hgRXqnvuc3XdAnUoHF1Xy2mMmIeq+B1KboSgP91IBWM=; b=bge49vxVfL54JgyPxY4+qzJbBfNJvTygYN7bTADUgPLM9xo8+63vUsPc8hSuigBgIU NFgcjNIkDrscrJXX2HB8XXENxOJPXfco3FNLKyw1MDZHbfIwZ/YsJKGzvCpBV0ysKFhD 3ySYazNVl4+08DlEMM6FwtRPXniDrhMyZZ0eE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=PqghV51fzwtpFQgdF4Y1auNq6AqI0qK1MviYQwd8BHwsI4kuaLVVMpn3Bg/NcYXuNG iezbvanF22YiEeF8RhnC7qi9S2yQKX28jkqs5nkT39ncSMbak11icUPONO02rnJFpkx0 u6eWakQc1CVjdTBCuHbotXLUIdd1IN1eAdQfs= MIME-Version: 1.0 Sender: brampton@gmail.com Received: by 10.223.126.66 with SMTP id b2mr1383714fas.3.1239483838034; Sat, 11 Apr 2009 14:03:58 -0700 (PDT) In-Reply-To: References: Date: Sat, 11 Apr 2009 22:03:58 +0100 X-Google-Sender-Auth: e4daf6e033b17634 Message-ID: From: Andrew Brampton To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Robert Watson Subject: Re: FreeBSD memguard + spinlocks X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2009 21:04:00 -0000 2009/4/11 Robert Watson : > On Sat, 11 Apr 2009, Andrew Brampton wrote: > > Your understanding is mostly right. =C2=A0The missing bit is this: there = are two > kinds of interrupt contexts -- fast/filter interrupt handlers, which borr= ow > the stack and execution context of the kernel thread they preempt, and > interrupt threads, which get their own complete thread context. > > Fast interrupt handlers are allowed unlock to acquire spinlocks so as to > avoid deadlock because of the borrowed context. =C2=A0This means they can= 't > perform any sort of sleep, or acquire any locks that might sleep, since t= he > thread they've preempted may hold conflicting locks, or be the one that > would have woken up the sleep that the handler performed. =C2=A0Almost no= code > will run in fast handlers -- perhaps checking some device registers, doin= g > work on a lockless or spinlock-protected queue, and waking up a worker > thread. > > This is why, BTW, spin locks disable interrupt: they need to control > preemption by other interrupt handlers to avoid deadlock, but they are no= t > intended for use except when either in the scheduler, in a few related IP= I > contexts, or when synchronizing between normal kernel code and a fast > handler. > > Full interrupt thread contexts are permitted to perform short lock sleeps= , > such as those performed when contending default mutexes, rwlocks, and > rmlocks. They are permitted to invoke kernel services such as malloc(9), > UMA(9), the network stack, etc, as long as they use M_NOWAIT and don't > invoke msleep(9) or similar unbounded sleeps -- again to avoid the > possibility of deadlocks, since you don't want an interrupt thread sleepi= ng > waiting for an event that only it can satisfy. > > So the first question, really, is whether you are or mean to be using > fast/filter interrupt handler. =C2=A0Device drivers will never call memor= y > allocation, free, etc, from there, but will defer it to an ithread using = the > filter mechanism in 8.x, or to a task queue or other worker in 7.x and > earlier. =C2=A0If you're using a regular INTR_MPSAFE ithread, you should = be able > to use only default mutexes (a single atomic operation if uncontended) > without disabling interrupts, etc. > > Robert N M Watson > Computer Laboratory > University of Cambridge > Thanks very much for your detailed reply. I'm slowly understanding how everything in FreeBSD fits together, and I appreciate your help. I've been given a project to take over, and all of the design decisions were made before I started working on it, thus I'm playing catch up. One of the decisions was to implement their own version of a spin lock, which literally looks something like this: lock_aquire() { critical_enter(); while (! lockHeld ) {} lockHeld++; } This was actually the code tripping up MemGuard, as it is inside a critical section, which MemGuard is unable to sleep within. This is all running inside a kthread_create thread (I'm unsure of the proper name of this type of thread). Anyway, that is why I also asked about a lighter weight spin lock (perhaps similar to this one). I tempted to replace this custom spinlock with the standard MTX_DEF, however I'm unsure of its impact. The custom spin lock seems quick and light to acquire, and it does not concern me that a interrupt can potentially interrupt the code. On a related note, if you change the lock in memguard to a MTX_SPIN, it panics the kernel during boot. So that is not an option :) I was only using memguard because I suspected memory being used after it was freed. However, I think I will either change my locks to MTX_DEF or live without memguard. I realise I've not really asked any questions, but I would be grateful for any insights anyone may have. Andrew