From owner-freebsd-smp@FreeBSD.ORG Wed Apr 23 05:35:57 2003 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0AA1537B401 for ; Wed, 23 Apr 2003 05:35:57 -0700 (PDT) Received: from mailman.zeta.org.au (mailman.zeta.org.au [203.26.10.16]) by mx1.FreeBSD.org (Postfix) with ESMTP id C658543FDD for ; Wed, 23 Apr 2003 05:35:55 -0700 (PDT) (envelope-from bde@zeta.org.au) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailman.zeta.org.au (8.9.3p2/8.8.7) with ESMTP id WAA24506; Wed, 23 Apr 2003 22:35:49 +1000 Date: Wed, 23 Apr 2003 22:35:48 +1000 (EST) From: Bruce Evans X-X-Sender: bde@gamplex.bde.org To: Aniruddha Bohra In-Reply-To: <3EA5E29F.9000801@cs.rutgers.edu> Message-ID: <20030423214814.P21296@gamplex.bde.org> References: <3EA5E29F.9000801@cs.rutgers.edu> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-smp@freebsd.org Subject: Re: Question about interrupt threads X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Apr 2003 12:35:57 -0000 On Tue, 22 Apr 2003, Aniruddha Bohra wrote: > Reading the implementation of interrupt handling on i386, > and reading Greg Lehey's Usenix paper on FreeBSD 5.0 SMP > implementation, I understand that interrupt handling is done in > process context. This is needed partly to support sleeping while > handling the interrupt because GIANT needs to be locked and > a sleep is possible there. > > My question is after all subsystems that lock giant are made > INTR_MPSAFE will the implementation go back to non-process > context interrupt handling (as in older versions) ? I would like that, but don't see how it could work right. > Moreover, will the locking be per-IRQ or there will be a > global IRQ lock for ensuring atomic access to the ISRs ? Neither. Each interrupt handler would lock whatever it accesses, and hopefully not much more, and not too often (zillions of locks and unlocks per interrupt), and without much lock contention and without any deadlocks. Locking using Giant basically locks everything, which is far too much. One advantage of Giant locking is that it tends to require only 1 lock/unlock per interrupt, but I think that advantage is small compared with the disadvantages of extra lock contention (almost everything competes for Giant, and switches to an interrupt handler that does nothing at first except block on Giant and switch back are common). Otherwise, Giant locking of interrupt handlers is little different from correct locking (if any). Both require a context to hold the state while blocked on a (non-spin) lock. In -current, that context is an ordinary thread context which is too heavy weight, but this is simplest and good for debugging. > Finally why do we need to create a kernel thread for > each IRQ - why can't one kernel thread handle all the > IRQs ? First, multiple threads are useful because they can be prioritized and scheduled independently. This is probably not very important for interrupts provided there is no slow PIO hardware for critical devices so that no interrupt handler wants more than a few usec at a time (just run the critical devices round-robin and defer processing for the non-critical ones to timeout routines). Second, multiple threads are needed so that they can block on locks independently. I think we actually use 1 thread per device driver to irq attachment, so we have multiple threads per irq in the shared irqs case. This is probably needed to let independent device drivers of device that are attached to the same irq block independently. We don't seem to have any examples of 1 thread per device giving multiple threads per irq withing 1 driver, perhaps because few drivers are fully SMP-aware internally. I would like at most 1 kernel interrupt thread per CPU, but don't see how it could work right. Bruce