Date: Thu, 12 Jan 2006 15:40:45 -0700 From: Scott Long <scottl@samsco.org> To: Andrew Gallatin <gallatin@cs.duke.edu> Cc: cvs-src@FreeBSD.org, src-committers@FreeBSD.org, cvs-all@FreeBSD.org Subject: Re: cvs commit: src/sys/dev/em if_em.c if_em.h Message-ID: <43C6DAED.3040901@samsco.org> In-Reply-To: <17350.53992.494972.787933@grasshopper.cs.duke.edu> References: <200601110030.k0B0UPOx009098@repoman.freebsd.org> <20060112152119.A6776@grasshopper.cs.duke.edu> <43C6C4EA.20303@samsco.org> <17350.53992.494972.787933@grasshopper.cs.duke.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Andrew Gallatin wrote: > Scott Long writes: > > > > Touching the APIC is tricky. First, you have to pay the cost of a > > spinlock. Then you have to may the cost of at least one read and write > > across the FSB. Even though the APIC registers are memory mapped, they > > are still uncached. It's not terribly expensive, but it does add up. > > Bypassing this and using a fast interrupt means that you pay the cost of > > 1 PCI read, which you would have to do anyways with either method, and 1 > > PCI write, which will be posted at the host-pci bridge and thus only as > > expensive as an FSB write. Overall, I don't think that the cost > > difference is a whole lot, but when you are talking about thousands of > > interrupts per second, especially if multiple interfaces are running > > under load, it might be important. And the 750x and 752x chipsets are > > so common that it is worthwhile to deal with them (and there are reports > > that the aliasing problem is happening on more chipsets than just these > > now). > > I think you've sold me. > > I'm in the process of trying to justify time to write a FreeBSD driver > for our PCIe 10GbE cards, and I find what you're doing fascinating. > I'd like to use some of your techniques for the driver I'm writing. > > > As for latency, the taskqueue runs at the same PI_NET priority as an the > > ithread would. I thought that there was an optimization on some > > platforms to encourage quick preemption for ithreads when they are > > scheduled, but I can't find it now. So, the taskqueue shouldn't be all > > that different from an ithread, and it even means that there won't be > > any sharing between instances even if the interrupt vector is shared. > > OK that (a taskqueue not getting the same fast preemption an ithread > would) was just what I was afraid of. I'm glad that it is not a > problem. The only problems I saw here were early on when I had the taskqueue running at a normal thread priority. It was constantly being preempted by the netisr, resulting in really bad performance. Moving it up to ithread-equivalent priority fixed this. If you copy this, make sure you pay close attention to the sched_prio() call that is made in the driver. I don't like exposing the driver to the low-level scheduling interface, so the long term plan is to either wrap it into the taskqueue API, or implement the two-stage interrupt API. > > <...> > > > However, taskqueues are really just a proof of concept for what I really > > want, which is to allow drivers to register both a fast handler and an > > ithread handler. For drivers doing this, the ithread would be private > > Ah, the darwin / MacOSX model. That would be very cool. > Yep. Working in IOKit was very interesting, and this is one of the few things that transfers well to FreeBSD. C++ does have a certain elagence for drivers, but the cost of virtual methods in the fast path of the driver and stack is still far too high to justify using it. > Anyway, keep up the good work. It is inspiring! Thanks! Scott
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?43C6DAED.3040901>