From owner-freebsd-current@FreeBSD.ORG Wed Mar 18 05:34:18 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CFD591065672 for ; Wed, 18 Mar 2009 05:34:17 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id 75CEE8FC0C for ; Wed, 18 Mar 2009 05:34:17 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.local ([192.168.254.200]) (authenticated bits=0) by pooker.samsco.org (8.14.2/8.14.2) with ESMTP id n2I5YDHD082767; Tue, 17 Mar 2009 23:34:13 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <49C087D5.4060705@samsco.org> Date: Tue, 17 Mar 2009 23:34:13 -0600 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.13) Gecko/20080313 SeaMonkey/1.1.9 MIME-Version: 1.0 To: barney_cordoba@yahoo.com References: <914146.31317.qm@web63901.mail.re1.yahoo.com> In-Reply-To: <914146.31317.qm@web63901.mail.re1.yahoo.com> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-4.4 required=3.8 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org Cc: Sam Leffler , current@freebsd.org Subject: Re: Interrupt routine usage not shown by top in 8.0 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Mar 2009 05:34:18 -0000 Barney Cordoba wrote: > > > > --- On Tue, 3/17/09, Sam Leffler wrote: > >> From: Sam Leffler >> Subject: Re: Interrupt routine usage not shown by top in 8.0 >> To: barney_cordoba@yahoo.com >> Cc: current@freebsd.org >> Date: Tuesday, March 17, 2009, 4:41 PM >> Barney Cordoba wrote: >>> >>> --- On Tue, 3/17/09, Robert Watson >> wrote: >>> >>>> From: Robert Watson >>>> Subject: Re: Interrupt routine usage not shown by >> top in 8.0 >>>> To: "Paolo Pisati" >> >>>> Cc: "Barney Cordoba" >> , current@freebsd.org >>>> Date: Tuesday, March 17, 2009, 11:24 AM >>>> On Tue, 17 Mar 2009, Paolo Pisati wrote: >>>> >>>> >>>>> perhaps i misunderstood your question, but >> i'll >>>>> >>>> try to explain a bit: >>>> >>>>> before 7.0, bus_setup_intr() took just one >> function >>>>> >>>> thus you could have an INTR_FAST or an INTR_MPSAFE >> handler, >>>> and you choose the kind of handler via a flag >> (INTR_FAST in >>>> this case). >>>> >>>>> after 7.0, bus_setup_intr() took 2 functions, >> thus you >>>>> >>>> could have: a fast handler (aka filter), or an >> ithread >>>> handler (aka mpsafe), or a fast + ithread handler >> (available >>>> only with INTR_FILTER turned on). >>>> >>>>> in bus_setup_intr() the first function pointer >> is for >>>>> >>>> the filter side of the handler, while the second >> pointer is >>>> for the ithread part, and if you declare both you >> can filter >>>> events (interrupts) and call the rest of the >> device driver >>>> (the ithread part) after the filter has recognized >> and >>>> acknowledged&masked the interrupt. >>>> >>>> This clarifies my misunderstanding, thanks! >>>> >>>> >>> I'd still be interested in knowing the specific >> advantage/consequences >>> of a fast filter vs an MPSAFE ithread? >>> >>> In what circumstance would using a filter and then >> launching a task be advantageous over just using an ithread? >>> >> It mostly depends on the hardware (unless the fast handler >> does actual work). If ack'ing the interrupt improves >> latency (e.g. by allowing the device to do other things) >> then it's better to do that in the filter method even if >> the actual work is deferred to the ithread. It's also >> important when interrupts are not edge-triggered; you want >> to shut them up asap. >> >> So, what device are you doing a driver for? >> >> Sam > > I'm working with the igb and ixgbe drivers. Neither of them > use the filters, but em does. Since they are all maintained > by the same person I was curious as to why the em used filters > and the igb and ixgbe, which are supposedly higher performance > cards, use MPSAFE ithreads. > > Barney > Filters were introduced into the em driver to get around a problem in certain Intel chipsets that caused aliased interrupts. That's a different topic of discussion that you are welcome to search the mail archives on. The filter also solves performance and latency problems that are inherent to the ithread model when interrupts are shared between multiple devices. This is especially bad when a high speed device like em shares an interrupt with a low speed device like usb. In the course of testing and validating the filter work, I found that filters caused no degradation in performance or excess context switches, while cleanly solving the above two problems that were common on workstation and server class machines of only a few years ago. However, both of these problems stemmed from using legacy PCI interrupts. At the time, MSI was still very new and very unreliable. As the state of the art progressed and MSI became more reliable, its use has become more common and is the default in several drivers. The igb and ixgbe drivers and hardware both prefer MSI over legacy interrupts, while the em driver and hardware still has a lot of legacy hardware to deal with. So when MSI is the common/expected/default case, there is less of a need for the filter/taskqueue method. Filters rely on the driver being able to reliably control the interrupt enable state of the hardware. This is possible with em hardware, but not as reliable with bge hardware, so the stock driver code does not have it implemented. I am running a filter-enabled bge driver in large-scale production, but I also have precise control over the hardware being used. I also have filter patches for the bce driver, but bce also tends to prefer MSI, so there isn't a compelling reason to continue to develop the patches. Scott