Date: Fri, 27 Apr 2012 13:06:57 -0700 From: Jack Vogel <jfvogel@gmail.com> To: Juli Mallett <jmallett@freebsd.org> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Sean Bruno <seanbru@yahoo-inc.com> Subject: Re: igb(4) at peak in big purple Message-ID: <CAFOYbc=1P7-rT=L-g_wuK4RMoHTcGP-cNMERoSE3oaRJkdgBKQ@mail.gmail.com> In-Reply-To: <CACVs6=9RzaZAHx6RC4AGywTzpuc8hNrY4eD-e-AJoV32OEMVgg@mail.gmail.com> References: <1335463643.2727.10.camel@powernoodle-l7.corp.yahoo.com> <CACVs6=8B8fUX-csyZM%2BGbhQHPn70qxjO6va%2B%2BxtWDAiKWHywXQ@mail.gmail.com> <1335554950.9324.3.camel@powernoodle-l7.corp.yahoo.com> <CACVs6=9RzaZAHx6RC4AGywTzpuc8hNrY4eD-e-AJoV32OEMVgg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I suspect to do it right would involve having the stack/kernel have more interaction with the driver/interface data, and this IS the way RSS was envisioned to work. Its been talked about but hasn't happened so far. Jack On Fri, Apr 27, 2012 at 1:00 PM, Juli Mallett <jmallett@freebsd.org> wrote: > On Fri, Apr 27, 2012 at 12:29, Sean Bruno <seanbru@yahoo-inc.com> wrote: > > On Thu, 2012-04-26 at 11:13 -0700, Juli Mallett wrote: > >> Queue splitting in Intel cards is done using a hash of protocol > >> headers, so this is expected behavior. This also helps with TCP and > >> UDP performance, in terms of keeping packets for the same protocol > >> control block on the same core, but for other applications it's not > >> ideal. If your application does not require that kind of locality, > >> there are things that can be done in the driver to make it easier to > >> balance packets between all queues about-evenly. > > > > Oh? :-) > > > > What should I be looking at to balance more evenly? > > Dirty hacks are involved :) I've sent some code to Luigi that I think > would make sense in netmap (since for many tasks one's going to do > with netmap, you want to use as many cores as possible, and maybe > don't care about locality so much) but it could be useful in > conjunction with the network stack, too, for tasks that don't need a > lot of locality. > > Basically this is the deal: the Intel NICs hash of various header > fields. Then, some bits from that hash are used to index a table. > That table indicates what queue the received packet should go to. > Ideally you'd want to use some sort of counter to index that table and > get round-robin queue usage if you wanted to evenly saturate all > cores. Unfortunately there doesn't seem to be a way to do that. > > What you can do, though, is regularly update the table that is indexed > by hash. Very frequently, in fact, it's a pretty fast operation. So > what I've done, for example, is to go through an rotate all of the > entries every N packets, where N is something like the number of > receive descriptors per queue divided by the number of queues. So > bucket 0 goes to queue 0 and bucket 1 goes to queue 1 at first. Then > a few hundred packets are received, and the table is reprogrammed, so > now bucket 0 goes to queue 1 and bucket 1 goes to queue 0. > > I can provide code to do this, but I don't want to post it publicly > (unless it is actually going to become an option for netmap) for fear > that people will use it in scenarios where it's harmful and then > complain. It's potentially one more painful variation for the Intel > drivers that Intel can't support, and that just makes everyone > miserable. > > Thanks, > Juli. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbc=1P7-rT=L-g_wuK4RMoHTcGP-cNMERoSE3oaRJkdgBKQ>