From owner-cvs-all@FreeBSD.ORG Thu Jan 12 22:40:50 2006 Return-Path: X-Original-To: cvs-all@FreeBSD.org Delivered-To: cvs-all@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6266016A41F; Thu, 12 Jan 2006 22:40:50 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id DCD9A43D45; Thu, 12 Jan 2006 22:40:45 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.11] (junior.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.4/8.13.4) with ESMTP id k0CMegxm035854; Thu, 12 Jan 2006 15:40:42 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <43C6DAED.3040901@samsco.org> Date: Thu, 12 Jan 2006 15:40:45 -0700 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.12) Gecko/20051230 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Andrew Gallatin References: <200601110030.k0B0UPOx009098@repoman.freebsd.org> <20060112152119.A6776@grasshopper.cs.duke.edu> <43C6C4EA.20303@samsco.org> <17350.53992.494972.787933@grasshopper.cs.duke.edu> In-Reply-To: <17350.53992.494972.787933@grasshopper.cs.duke.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.1.0 X-Spam-Checker-Version: SpamAssassin 3.1.0 (2005-09-13) on pooker.samsco.org Cc: cvs-src@FreeBSD.org, src-committers@FreeBSD.org, cvs-all@FreeBSD.org Subject: Re: cvs commit: src/sys/dev/em if_em.c if_em.h X-BeenThere: cvs-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CVS commit messages for the entire tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2006 22:40:50 -0000 Andrew Gallatin wrote: > Scott Long writes: > > > > Touching the APIC is tricky. First, you have to pay the cost of a > > spinlock. Then you have to may the cost of at least one read and write > > across the FSB. Even though the APIC registers are memory mapped, they > > are still uncached. It's not terribly expensive, but it does add up. > > Bypassing this and using a fast interrupt means that you pay the cost of > > 1 PCI read, which you would have to do anyways with either method, and 1 > > PCI write, which will be posted at the host-pci bridge and thus only as > > expensive as an FSB write. Overall, I don't think that the cost > > difference is a whole lot, but when you are talking about thousands of > > interrupts per second, especially if multiple interfaces are running > > under load, it might be important. And the 750x and 752x chipsets are > > so common that it is worthwhile to deal with them (and there are reports > > that the aliasing problem is happening on more chipsets than just these > > now). > > I think you've sold me. > > I'm in the process of trying to justify time to write a FreeBSD driver > for our PCIe 10GbE cards, and I find what you're doing fascinating. > I'd like to use some of your techniques for the driver I'm writing. > > > As for latency, the taskqueue runs at the same PI_NET priority as an the > > ithread would. I thought that there was an optimization on some > > platforms to encourage quick preemption for ithreads when they are > > scheduled, but I can't find it now. So, the taskqueue shouldn't be all > > that different from an ithread, and it even means that there won't be > > any sharing between instances even if the interrupt vector is shared. > > OK that (a taskqueue not getting the same fast preemption an ithread > would) was just what I was afraid of. I'm glad that it is not a > problem. The only problems I saw here were early on when I had the taskqueue running at a normal thread priority. It was constantly being preempted by the netisr, resulting in really bad performance. Moving it up to ithread-equivalent priority fixed this. If you copy this, make sure you pay close attention to the sched_prio() call that is made in the driver. I don't like exposing the driver to the low-level scheduling interface, so the long term plan is to either wrap it into the taskqueue API, or implement the two-stage interrupt API. > > <...> > > > However, taskqueues are really just a proof of concept for what I really > > want, which is to allow drivers to register both a fast handler and an > > ithread handler. For drivers doing this, the ithread would be private > > Ah, the darwin / MacOSX model. That would be very cool. > Yep. Working in IOKit was very interesting, and this is one of the few things that transfers well to FreeBSD. C++ does have a certain elagence for drivers, but the cost of virtual methods in the fast path of the driver and stack is still far too high to justify using it. > Anyway, keep up the good work. It is inspiring! Thanks! Scott