From owner-freebsd-current@FreeBSD.ORG Wed Mar 18 20:44:50 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76E571065670 for ; Wed, 18 Mar 2009 20:44:50 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63903.mail.re1.yahoo.com (web63903.mail.re1.yahoo.com [69.147.97.118]) by mx1.freebsd.org (Postfix) with SMTP id 350668FC0C for ; Wed, 18 Mar 2009 20:44:50 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: (qmail 95552 invoked by uid 60001); 18 Mar 2009 20:44:49 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1237409089; bh=t9Qm2r6KPHmFW0TfJhteFFJPC/Vuqt8yqdNdefQvhXM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=kjRHm6SwmNDIXZjoOkKZYpy/meUu/JIMAfTmGDNCVMNBlZdmkfg3EB6nPQudHpURhI2Kfs1A3XmuDlL8ViViJHIqCc1v0iYzK7245wqyGI0PGaDI4agF6GE81Mms1n2FsNL7bHSgNdE0GvFMjx8N5wEMLKYLLIYVTQCibHPWpio= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=BJUYeXOWnbFPqPrWsjGQfj6qDqPBfjHwmF6LpjvEhndNkQTNk/h5SAtTi5K71guOp27aQtj3+DzSBCIoG4H38Sy/l/WdrkbTvUbbXzavquaykVdzI8H6Oewi+NLVWUVBkX9ZJmW7I1YHbFQVR6EzwfH3blA7g/lqUpPylzCGBXg=; Message-ID: <713528.91102.qm@web63903.mail.re1.yahoo.com> X-YMail-OSG: dwbrc28VM1kkuAmDQ8qNDyMa_hM55FK3i.RNG.o0XOBvSOjd4ozbl8XVMBfvFSU30x4V0tFSnVXVUUnmDc.eIaCFpSpVfm.FvVK3Rw5wZbaC8Gu2eiwae8pnW2G2MlA4qUj9YF2.oOJ_WRzg0mhhSTTTZGAsNFa5uEoZ3yQh6uOlfbhgEyr.r6UbaCBIKrwp7EyhbkfAP1Fl025fYWO6x5QJQBohNpkStjii Received: from [98.242.222.229] by web63903.mail.re1.yahoo.com via HTTP; Wed, 18 Mar 2009 13:44:49 PDT X-Mailer: YahooMailWebService/0.7.289.1 Date: Wed, 18 Mar 2009 13:44:49 -0700 (PDT) From: Barney Cordoba To: Scott Long In-Reply-To: <49C087D5.4060705@samsco.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Sam Leffler , current@freebsd.org Subject: Re: Interrupt routine usage not shown by top in 8.0 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: barney_cordoba@yahoo.com List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Mar 2009 20:44:50 -0000 --- On Wed, 3/18/09, Scott Long wrote: > From: Scott Long > Subject: Re: Interrupt routine usage not shown by top in 8.0 > To: barney_cordoba@yahoo.com > Cc: "Sam Leffler" , current@freebsd.org > Date: Wednesday, March 18, 2009, 1:34 AM > Barney Cordoba wrote: > > > > > > > > --- On Tue, 3/17/09, Sam Leffler > wrote: > > > >> From: Sam Leffler > >> Subject: Re: Interrupt routine usage not shown by > top in 8.0 > >> To: barney_cordoba@yahoo.com > >> Cc: current@freebsd.org > >> Date: Tuesday, March 17, 2009, 4:41 PM > >> Barney Cordoba wrote: > >>> > >>> --- On Tue, 3/17/09, Robert Watson > >> wrote: > >>> > >>>> From: Robert Watson > > >>>> Subject: Re: Interrupt routine usage not > shown by > >> top in 8.0 > >>>> To: "Paolo Pisati" > >> > >>>> Cc: "Barney Cordoba" > >> , > current@freebsd.org > >>>> Date: Tuesday, March 17, 2009, 11:24 AM > >>>> On Tue, 17 Mar 2009, Paolo Pisati wrote: > >>>> > >>>> > >>>>> perhaps i misunderstood your question, > but > >> i'll > >>>>> > >>>> try to explain a bit: > >>>> > >>>>> before 7.0, bus_setup_intr() took just > one > >> function > >>>>> > >>>> thus you could have an INTR_FAST or an > INTR_MPSAFE > >> handler, > >>>> and you choose the kind of handler via a > flag > >> (INTR_FAST in > >>>> this case). > >>>> > >>>>> after 7.0, bus_setup_intr() took 2 > functions, > >> thus you > >>>>> > >>>> could have: a fast handler (aka filter), > or an > >> ithread > >>>> handler (aka mpsafe), or a fast + ithread > handler > >> (available > >>>> only with INTR_FILTER turned on). > >>>> > >>>>> in bus_setup_intr() the first function > pointer > >> is for > >>>>> > >>>> the filter side of the handler, while the > second > >> pointer is > >>>> for the ithread part, and if you declare > both you > >> can filter > >>>> events (interrupts) and call the rest of > the > >> device driver > >>>> (the ithread part) after the filter has > recognized > >> and > >>>> acknowledged&masked the interrupt. > >>>> > >>>> This clarifies my misunderstanding, > thanks! > >>>> > >>>> > >>> I'd still be interested in knowing the > specific > >> advantage/consequences > >>> of a fast filter vs an MPSAFE ithread? > >>> > >>> In what circumstance would using a filter and > then > >> launching a task be advantageous over just using > an ithread? > >>> > >> It mostly depends on the hardware (unless the fast > handler > >> does actual work). If ack'ing the interrupt > improves > >> latency (e.g. by allowing the device to do other > things) > >> then it's better to do that in the filter > method even if > >> the actual work is deferred to the ithread. > It's also > >> important when interrupts are not edge-triggered; > you want > >> to shut them up asap. > >> > >> So, what device are you doing a driver for? > >> > >> Sam > > > > I'm working with the igb and ixgbe drivers. > Neither of them > > use the filters, but em does. Since they are all > maintained > > by the same person I was curious as to why the em used > filters > > and the igb and ixgbe, which are supposedly higher > performance > > cards, use MPSAFE ithreads. > > > > Barney > > > > Filters were introduced into the em driver to get around a > problem in > certain Intel chipsets that caused aliased interrupts. > That's a > different topic of discussion that you are welcome to > search the mail > archives on. The filter also solves performance and > latency problems > that are inherent to the ithread model when interrupts are > shared > between multiple devices. This is especially bad when a > high speed > device like em shares an interrupt with a low speed device > like usb. > In the course of testing and validating the filter work, I > found that > filters caused no degradation in performance or excess > context switches, > while cleanly solving the above two problems that were > common on > workstation and server class machines of only a few years > ago. > > However, both of these problems stemmed from using legacy > PCI > interrupts. At the time, MSI was still very new and very > unreliable. > As the state of the art progressed and MSI became more > reliable, its > use has become more common and is the default in several > drivers. The > igb and ixgbe drivers and hardware both prefer MSI over > legacy > interrupts, while the em driver and hardware still has a > lot of legacy > hardware to deal with. So when MSI is the > common/expected/default case, > there is less of a need for the filter/taskqueue method. > > Filters rely on the driver being able to reliably control > the interrupt > enable state of the hardware. This is possible with em > hardware, but > not as reliable with bge hardware, so the stock driver code > does not > have it implemented. I am running a filter-enabled bge > driver in > large-scale production, but I also have precise control > over the > hardware being used. I also have filter patches for the > bce driver, but > bce also tends to prefer MSI, so there isn't a > compelling reason to > continue to develop the patches. > > > Scott Assuming same technique is used within an ithread as with a fast interrupt, that is: filtered_foo(){ taskqueue_enqueue(); return FILTER_HANDLED; } ithread_foo(){ taskqueue_enqueue(); return; } Is there any additional overhead/locking in the ithread method? I'm looking to get better control over cpu distribution. Barney