From owner-freebsd-current@FreeBSD.ORG Fri Nov 18 17:20:06 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2BC30106564A for ; Fri, 18 Nov 2011 17:20:06 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id F10F38FC1F for ; Fri, 18 Nov 2011 17:20:05 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id A8C6846B0D; Fri, 18 Nov 2011 12:20:05 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 3F1F88A050; Fri, 18 Nov 2011 12:20:05 -0500 (EST) From: John Baldwin To: Luigi Rizzo Date: Fri, 18 Nov 2011 12:20:04 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <201111180800.06593.jhb@freebsd.org> <20111118170615.GA9762@onelab2.iet.unipi.it> In-Reply-To: <20111118170615.GA9762@onelab2.iet.unipi.it> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201111181220.04846.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Fri, 18 Nov 2011 12:20:05 -0500 (EST) Cc: Matteo Landi , freebsd-current@freebsd.org Subject: Re: ixgbe and fast interrupts X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 17:20:06 -0000 On Friday, November 18, 2011 12:06:15 pm Luigi Rizzo wrote: > On Fri, Nov 18, 2011 at 08:00:06AM -0500, John Baldwin wrote: > > On Friday, November 18, 2011 3:46:02 am Matteo Landi wrote: > > > > you probably want to be using MSI-X for a 10G NIC instead of INTx anyway. > > > > > > Why do you say that? Is MSI-X faster than INTx in terms of interrupt > > > latency? When should I use MSI-X, instead of fast filters interrupts > > > (fast interrupt?), instead of ithread interrupts? Thanks in advace. > > > > With MSI-X you can have more than one interrupt and those interrupts can be > > distributed across CPUs. This means you can (somewhat) tie each queue on your > > NIC to a different CPU. > > > > MSI-X vs INTx is orthogonal to fast vs filter, but in general MSI and MSI-X > > interrupts are not shared, and require no interrupt masking in hardware (they > > are effectively edge-triggered), so using a filter for MSI is rather pointless > > and only adds needless complexity. For MSI I would just use a theraded > > interrupt handler. For INTx, I would only use a fast interrupt handler if > > there is a really good reason to do so (e.g. em(4) does so to work around > > broken Intel Host-PCI bridges). > > A bit more context: Matteo is looking at the latency of RPCs across > the network involving userspace processes, and possibly using the > netmap API. As we understand it: > > if you are not using a filter, the interrupt calls a "predefined" > filter (kern_intr.c::intr_event_schedule_thread() ? ) which wakes > up the handler thread which in turn wakes up the user process. This > means two scheduler invocations on each side. Yes, but if you use a filter you still have to do that as your filter would just be queueing a task to on a taskqueue which would then do the actual selwakeup() from a taskqueue thread. Filters are typically used to avoid masking the interrupt in the PIC, or to limit the handlers executed on a shared interrupt. > In the case of netmap, all the handler needs to do is a selwakeup() > of the user thread blocked on the file descriptor, so if this > can be done in the filter we can save an extra step through the > scheduler. You can't call selwakeup() from a filter, it is not safe since it uses mutexes, etc. There are only a few things you can do from a filter. You could do a plain wakeup() if you let userland use a custom ioctl to block on the filter, but not selwakeup(). -- John Baldwin