From owner-freebsd-current@FreeBSD.ORG Wed Mar 18 21:25:53 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 589BD1065676; Wed, 18 Mar 2009 21:25:53 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id 011EB8FC0C; Wed, 18 Mar 2009 21:25:52 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.home (pooker.samsco.home [192.168.254.1]) by pooker.samsco.org (8.14.2/8.14.2) with ESMTP id n2ILPmJR086349; Wed, 18 Mar 2009 15:25:49 -0600 (MDT) (envelope-from scottl@samsco.org) Date: Wed, 18 Mar 2009 15:25:48 -0600 (MDT) From: Scott Long To: Barney Cordoba In-Reply-To: <713528.91102.qm@web63903.mail.re1.yahoo.com> Message-ID: <20090318150721.U22014@pooker.samsco.org> References: <713528.91102.qm@web63903.mail.re1.yahoo.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Spam-Status: No, score=-4.4 required=3.8 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org Cc: Sam Leffler , current@freebsd.org Subject: Re: Interrupt routine usage not shown by top in 8.0 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Mar 2009 21:25:55 -0000 On Wed, 18 Mar 2009, Barney Cordoba wrote: > --- On Wed, 3/18/09, Scott Long wrote: >> >> Filters were introduced into the em driver to get around a >> problem in >> certain Intel chipsets that caused aliased interrupts. >> That's a >> different topic of discussion that you are welcome to >> search the mail >> archives on. The filter also solves performance and >> latency problems >> that are inherent to the ithread model when interrupts are >> shared >> between multiple devices. This is especially bad when a >> high speed >> device like em shares an interrupt with a low speed device >> like usb. >> In the course of testing and validating the filter work, I >> found that >> filters caused no degradation in performance or excess >> context switches, >> while cleanly solving the above two problems that were >> common on >> workstation and server class machines of only a few years >> ago. >> >> However, both of these problems stemmed from using legacy >> PCI >> interrupts. At the time, MSI was still very new and very >> unreliable. >> As the state of the art progressed and MSI became more >> reliable, its >> use has become more common and is the default in several >> drivers. The >> igb and ixgbe drivers and hardware both prefer MSI over >> legacy >> interrupts, while the em driver and hardware still has a >> lot of legacy >> hardware to deal with. So when MSI is the >> common/expected/default case, >> there is less of a need for the filter/taskqueue method. >> >> Filters rely on the driver being able to reliably control >> the interrupt >> enable state of the hardware. This is possible with em >> hardware, but >> not as reliable with bge hardware, so the stock driver code >> does not >> have it implemented. I am running a filter-enabled bge >> driver in >> large-scale production, but I also have precise control >> over the >> hardware being used. I also have filter patches for the >> bce driver, but >> bce also tends to prefer MSI, so there isn't a >> compelling reason to >> continue to develop the patches. >> >> >> Scott > > Assuming same technique is used within an ithread as with a fast > interrupt, that is: > > filtered_foo(){ > taskqueue_enqueue(); > return FILTER_HANDLED; > } This will give you two context switches, one for the actual interrupt, and one for the taskqueue. It'll also encounter a spinlock in the taskqueue code, and a spinlock or two in the scheduler. > > ithread_foo(){ > taskqueue_enqueue(); > return; > } > > Is there any additional overhead/locking in the ithread method? I'm > looking to get better control over cpu distribution. > This will give you 3 context switches. First one will be for the actual interrupts. Second one will be for the ithread (recall that ithreads are full process contexts and are scheduled as such). Third one will be for the taskqueue. Along with the spinlocks for the scheduler and taskqueue code mentioned above, there will also be spinlocks to protect the APIC registers, as well as extra bus cycles to service the APIC. So, that's 2 trips through the scheduler, plus the associated spinlocks, plus the overhead of going through the APIC code, whereas the first method only goes through the scheduler once. Both will have a context switch to service the low-level interrupt. The second method will definitely have more context switches, and will almost certainly have higher overall service latency and CPU usage. Scott