Date: Wed, 18 Mar 2009 13:44:49 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Scott Long <scottl@samsco.org> Cc: Sam Leffler <sam@freebsd.org>, current@freebsd.org Subject: Re: Interrupt routine usage not shown by top in 8.0 Message-ID: <713528.91102.qm@web63903.mail.re1.yahoo.com> In-Reply-To: <49C087D5.4060705@samsco.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--- On Wed, 3/18/09, Scott Long <scottl@samsco.org> wrote:
> From: Scott Long <scottl@samsco.org>
> Subject: Re: Interrupt routine usage not shown by top in 8.0
> To: barney_cordoba@yahoo.com
> Cc: "Sam Leffler" <sam@freebsd.org>, current@freebsd.org
> Date: Wednesday, March 18, 2009, 1:34 AM
> Barney Cordoba wrote:
> >
> >
> >
> > --- On Tue, 3/17/09, Sam Leffler
> <sam@freebsd.org> wrote:
> >
> >> From: Sam Leffler <sam@freebsd.org>
> >> Subject: Re: Interrupt routine usage not shown by
> top in 8.0
> >> To: barney_cordoba@yahoo.com
> >> Cc: current@freebsd.org
> >> Date: Tuesday, March 17, 2009, 4:41 PM
> >> Barney Cordoba wrote:
> >>>
> >>> --- On Tue, 3/17/09, Robert Watson
> >> <rwatson@FreeBSD.org> wrote:
> >>>
> >>>> From: Robert Watson
> <rwatson@FreeBSD.org>
> >>>> Subject: Re: Interrupt routine usage not
> shown by
> >> top in 8.0
> >>>> To: "Paolo Pisati"
> >> <p.pisati@oltrelinux.com>
> >>>> Cc: "Barney Cordoba"
> >> <barney_cordoba@yahoo.com>,
> current@freebsd.org
> >>>> Date: Tuesday, March 17, 2009, 11:24 AM
> >>>> On Tue, 17 Mar 2009, Paolo Pisati wrote:
> >>>>
> >>>>
> >>>>> perhaps i misunderstood your question,
> but
> >> i'll
> >>>>>
> >>>> try to explain a bit:
> >>>>
> >>>>> before 7.0, bus_setup_intr() took just
> one
> >> function
> >>>>>
> >>>> thus you could have an INTR_FAST or an
> INTR_MPSAFE
> >> handler,
> >>>> and you choose the kind of handler via a
> flag
> >> (INTR_FAST in
> >>>> this case).
> >>>>
> >>>>> after 7.0, bus_setup_intr() took 2
> functions,
> >> thus you
> >>>>>
> >>>> could have: a fast handler (aka filter),
> or an
> >> ithread
> >>>> handler (aka mpsafe), or a fast + ithread
> handler
> >> (available
> >>>> only with INTR_FILTER turned on).
> >>>>
> >>>>> in bus_setup_intr() the first function
> pointer
> >> is for
> >>>>>
> >>>> the filter side of the handler, while the
> second
> >> pointer is
> >>>> for the ithread part, and if you declare
> both you
> >> can filter
> >>>> events (interrupts) and call the rest of
> the
> >> device driver
> >>>> (the ithread part) after the filter has
> recognized
> >> and
> >>>> acknowledged&masked the interrupt.
> >>>>
> >>>> This clarifies my misunderstanding,
> thanks!
> >>>>
> >>>>
> >>> I'd still be interested in knowing the
> specific
> >> advantage/consequences
> >>> of a fast filter vs an MPSAFE ithread?
> >>>
> >>> In what circumstance would using a filter and
> then
> >> launching a task be advantageous over just using
> an ithread?
> >>>
> >> It mostly depends on the hardware (unless the fast
> handler
> >> does actual work). If ack'ing the interrupt
> improves
> >> latency (e.g. by allowing the device to do other
> things)
> >> then it's better to do that in the filter
> method even if
> >> the actual work is deferred to the ithread.
> It's also
> >> important when interrupts are not edge-triggered;
> you want
> >> to shut them up asap.
> >>
> >> So, what device are you doing a driver for?
> >>
> >> Sam
> >
> > I'm working with the igb and ixgbe drivers.
> Neither of them
> > use the filters, but em does. Since they are all
> maintained
> > by the same person I was curious as to why the em used
> filters
> > and the igb and ixgbe, which are supposedly higher
> performance
> > cards, use MPSAFE ithreads.
> >
> > Barney
> >
>
> Filters were introduced into the em driver to get around a
> problem in
> certain Intel chipsets that caused aliased interrupts.
> That's a
> different topic of discussion that you are welcome to
> search the mail
> archives on. The filter also solves performance and
> latency problems
> that are inherent to the ithread model when interrupts are
> shared
> between multiple devices. This is especially bad when a
> high speed
> device like em shares an interrupt with a low speed device
> like usb.
> In the course of testing and validating the filter work, I
> found that
> filters caused no degradation in performance or excess
> context switches,
> while cleanly solving the above two problems that were
> common on
> workstation and server class machines of only a few years
> ago.
>
> However, both of these problems stemmed from using legacy
> PCI
> interrupts. At the time, MSI was still very new and very
> unreliable.
> As the state of the art progressed and MSI became more
> reliable, its
> use has become more common and is the default in several
> drivers. The
> igb and ixgbe drivers and hardware both prefer MSI over
> legacy
> interrupts, while the em driver and hardware still has a
> lot of legacy
> hardware to deal with. So when MSI is the
> common/expected/default case,
> there is less of a need for the filter/taskqueue method.
>
> Filters rely on the driver being able to reliably control
> the interrupt
> enable state of the hardware. This is possible with em
> hardware, but
> not as reliable with bge hardware, so the stock driver code
> does not
> have it implemented. I am running a filter-enabled bge
> driver in
> large-scale production, but I also have precise control
> over the
> hardware being used. I also have filter patches for the
> bce driver, but
> bce also tends to prefer MSI, so there isn't a
> compelling reason to
> continue to develop the patches.
>
>
> Scott
Assuming same technique is used within an ithread as with a fast
interrupt, that is:
filtered_foo(){
taskqueue_enqueue();
return FILTER_HANDLED;
}
ithread_foo(){
taskqueue_enqueue();
return;
}
Is there any additional overhead/locking in the ithread method? I'm
looking to get better control over cpu distribution.
Barney
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?713528.91102.qm>