From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 28 23:41:18 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B25A2196; Sat, 28 Mar 2015 23:41:18 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 63641C4A; Sat, 28 Mar 2015 23:41:18 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1Yc0Ls-0004LU-Ag; Sun, 29 Mar 2015 02:41:16 +0300 Date: Sun, 29 Mar 2015 02:41:16 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150328234116.GJ23643@zxy.spb.ru> References: <20150328194959.GE23643@zxy.spb.ru> <20150328201219.GF23643@zxy.spb.ru> <20150328221621.GG23643@zxy.spb.ru> <20150328224634.GH23643@zxy.spb.ru> <20150328230533.GI23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Mar 2015 23:41:18 -0000 On Sat, Mar 28, 2015 at 04:23:54PM -0700, Adrian Chadd wrote: > On 28 March 2015 at 16:05, Slawa Olhovchenkov wrote: > > On Sat, Mar 28, 2015 at 03:49:48PM -0700, Adrian Chadd wrote: > > > >> You should totally join #bsdcode on efnet and ask me about it. :) > > > > I am totaly don't use IRC (last 20 years). > > May be skype? > > Heh, IRC is better. There are more FreeBSD people in the channel. :) I don't understund IRC. I don't know how I can got history of chat in case of long disconnect. > >> on RSS, this is what would happen: > >> > >> * ALL NICs RSS BUCKET 0 -> core 0 > >> * ... > >> * ALL NICs RSS BUCKET 7 -> core 7 > > > > My expirens: this is worse vs dedicated core (one core handeled only > > one bucket of one NIC). > > The only reason(s) this becomes problematic is if things preempt other > things on that CPU. > Hopefully enough work gets done in each interrupt run - but, maybe the > scheduler is doing something odd and interleaving all the > supposedly-equivalent-ithreads based on what's blocking in locks and > what isn't. It's worth digging into. Sorry, I am missunderstund you there. Or, may be, I use unintelligible explanation. dedicated core != core only NIC and don't do other work. dedicated core :== for this NIC bucket only this core. on this core no other NIC buckets. Other task may be used this core. What you mean? Can you explain? > Not only that, but I also do handle the case of fragments going to the > "wrong" queue - then getting reassembled and reinjected back into the > right RSS CPU. That way things are correctly in-order. fragment reassembled done in-kernel and (for TCP) go to right queue automatic. > > > >> Now, that's not really 100% optimal for NUMA and multiple PCIe > >> controllers, but we're not there yet. > >> > >> Hopefully I can twist/cajole navdeep @ chelsio to continue doing a > >> little more RSS work so I can teach cxgbe/cxl about RSS configuration, > >> but ixgbe, igb and ixl all do the above when RSS is enabled. > > > > Most part of my setup use cxgbe. > > Ok. > > Well, that (and other stuff) will happen at the speed of "adrian's > doing this for fun as his home project", so if you/others would like > to help out then please do. I'd like to get this stuff very much done > and in -11 before it's released next year. I am understand that you home project. I can't request you. I only think that propose more light and simple solution (don't need drivers modification, don't break API/ABI/KBI).