Date: Fri, 27 Apr 2012 13:00:36 -0700 From: Juli Mallett <jmallett@FreeBSD.org> To: Sean Bruno <seanbru@yahoo-inc.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: igb(4) at peak in big purple Message-ID: <CACVs6=9RzaZAHx6RC4AGywTzpuc8hNrY4eD-e-AJoV32OEMVgg@mail.gmail.com> In-Reply-To: <1335554950.9324.3.camel@powernoodle-l7.corp.yahoo.com> References: <1335463643.2727.10.camel@powernoodle-l7.corp.yahoo.com> <CACVs6=8B8fUX-csyZM%2BGbhQHPn70qxjO6va%2B%2BxtWDAiKWHywXQ@mail.gmail.com> <1335554950.9324.3.camel@powernoodle-l7.corp.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Apr 27, 2012 at 12:29, Sean Bruno <seanbru@yahoo-inc.com> wrote: > On Thu, 2012-04-26 at 11:13 -0700, Juli Mallett wrote: >> Queue splitting in Intel cards is done using a hash of protocol >> headers, so this is expected behavior. =C2=A0This also helps with TCP an= d >> UDP performance, in terms of keeping packets for the same protocol >> control block on the same core, but for other applications it's not >> ideal. =C2=A0If your application does not require that kind of locality, >> there are things that can be done in the driver to make it easier to >> balance packets between all queues about-evenly. > > Oh? :-) > > What should I be looking at to balance more evenly? Dirty hacks are involved :) I've sent some code to Luigi that I think would make sense in netmap (since for many tasks one's going to do with netmap, you want to use as many cores as possible, and maybe don't care about locality so much) but it could be useful in conjunction with the network stack, too, for tasks that don't need a lot of locality. Basically this is the deal: the Intel NICs hash of various header fields. Then, some bits from that hash are used to index a table. That table indicates what queue the received packet should go to. Ideally you'd want to use some sort of counter to index that table and get round-robin queue usage if you wanted to evenly saturate all cores. Unfortunately there doesn't seem to be a way to do that. What you can do, though, is regularly update the table that is indexed by hash. Very frequently, in fact, it's a pretty fast operation. So what I've done, for example, is to go through an rotate all of the entries every N packets, where N is something like the number of receive descriptors per queue divided by the number of queues. So bucket 0 goes to queue 0 and bucket 1 goes to queue 1 at first. Then a few hundred packets are received, and the table is reprogrammed, so now bucket 0 goes to queue 1 and bucket 1 goes to queue 0. I can provide code to do this, but I don't want to post it publicly (unless it is actually going to become an option for netmap) for fear that people will use it in scenarios where it's harmful and then complain. It's potentially one more painful variation for the Intel drivers that Intel can't support, and that just makes everyone miserable. Thanks, Juli.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACVs6=9RzaZAHx6RC4AGywTzpuc8hNrY4eD-e-AJoV32OEMVgg>