From owner-freebsd-net@FreeBSD.ORG Fri Sep 9 00:44:39 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3BA07106564A; Fri, 9 Sep 2011 00:44:39 +0000 (UTC) (envelope-from bhutchings@solarflare.com) Received: from exchange.solarflare.com (exchange.solarflare.com [216.237.3.220]) by mx1.freebsd.org (Postfix) with ESMTP id 171978FC17; Fri, 9 Sep 2011 00:44:38 +0000 (UTC) Received: from [10.17.20.137] ([10.17.20.137]) by exchange.solarflare.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Thu, 8 Sep 2011 17:44:37 -0700 From: Ben Hutchings To: George Neville-Neil In-Reply-To: <37419C45-4436-4738-851B-2B765BC2C60F@neville-neil.com> References: <1315221674.3092.282.camel@deadeye> <201109080834.11607.jhb@freebsd.org> <20110908184928.GA87872@hub.freebsd.org> <37419C45-4436-4738-851B-2B765BC2C60F@neville-neil.com> Content-Type: text/plain; charset="UTF-8" Organization: Solarflare Communications Date: Fri, 09 Sep 2011 01:44:34 +0100 Message-ID: <1315529074.2804.63.camel@bwh-desktop> Mime-Version: 1.0 X-Mailer: Evolution 2.32.2 (2.32.2-1.fc14) Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 09 Sep 2011 00:44:38.0366 (UTC) FILETIME=[A75237E0:01CC6E89] X-TM-AS-Product-Ver: SMEX-8.0.0.1181-6.500.1024-18372.005 X-TM-AS-Result: No--35.371100-0.000000-31 X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No Cc: jfv@freebsd.org, freebsd-net@freebsd.org, Navdeep Parhar , John Baldwin , Takuya ASADA Subject: Re: Adding Flow Director sysctls to ixgbe(4) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Sep 2011 00:44:39 -0000 On Thu, 2011-09-08 at 20:13 -0400, George Neville-Neil wrote: > On Sep 8, 2011, at 14:49 , Navdeep Parhar wrote: > > > On Thu, Sep 08, 2011 at 08:34:11AM -0400, John Baldwin wrote: > >> On Monday, September 05, 2011 7:21:12 am Ben Hutchings wrote: > >>> On Mon, 2011-09-05 at 15:51 +0900, Takuya ASADA wrote: > >>>> Hi, > >>>> > >>>> I implemented Ethernet Flow Director sysctls to ixgbe(4), here's a detail: > >>>> > >>>> - Adding removing signature filter > >>>> On linux version of ixgbe driver, it has ability to set/remove perfect > >>>> filter from userland using ethtool command. > >>>> I implemented similar feature, but on sysctl, and not perfect filter > >>>> but signature filter(which means hash collision may occurs). > >>> [...] > >>> > >>> Linux also has a generic interface to RX filtering and hashing > >>> (ethtool_rxnfc) which ixgbe supports; wouldn't it be better for FreeBSD > >>> to support something like that? > >> > >> Some sort of shared interface might be nice. The cxgb(4) and cxgbe(4) drivers > >> both provide their own tools to manipulate filters, though they do not > >> provide explicit steering IIRC. > > > > Both of them can filter as well as steer (and the tools let you do that). > > cxgbe(4) can do a lot more (rewrite + switch, replicate, etc.) but those > > features are perhaps too specialized to be configurable via a general > > purpose tool. > > > >> > >> We would need to come up with some sort of standard interface (ioctls?) for > >> adding filters however. > > > > +1 for a standard interface. > > > > imho the kernel needs to be aware of the rx and tx queues of a NIC, and > > not just for steering. But that's a separate discussion. > > > > Well I do think this is actually all of a part. Most of us realize by now that > high speed (e.g. 10G and higher) NICs only make sense if you can steer traffic and > pin queues to cores etc. Well, you can get way better than 1G performance without that. And for routers, flow hashing may be fine. But for a host, of course, steering packets properly can provide a major performance win. [...] > What this means is that we have > a failure of abstraction. Abstraction has a cost, and some of the people who want > access to low level queues are not interested in paying an extra abstraction cost. Abstraction has a cost, but it's not necessarily that high compared to rewriting a whole chunk of sockets code (especially if you don't actually have the source code). > I think that some of the abstractions we need are tied up in the work that Takuya did > for SoC and some of it is in the work done by Luigi on netmap. I'd go so far as to say > that what we should do is try to combine those two pieces of code into a set of > low level APIs for programs to interact with high speed NICs. The one thing most > people do not talk about is extending our socket API to do two things that I think would > be a win for 80% of our users. If a socket, and also a kqueue, could be pinned > to a CPU as well as a NIC queue that should improve overall bandwidth for a large > number of our users. The API there is definitely an ioctl() and the hard part is > doing the tying together. To do this we need to also work out our low level story. But it would be a lot nicer if this could be done automatically. Which I believe it can - see the RFS and XPS features in Linux. Ben. -- Ben Hutchings, Staff Engineer, Solarflare Not speaking for my employer; that's the marketing department's job. They asked us to note that Solarflare product names are trademarked.