Date: Thu, 26 Apr 2012 11:13:55 -0700 From: Juli Mallett <jmallett@FreeBSD.org> To: Sean Bruno <seanbru@yahoo-inc.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: igb(4) at peak in big purple Message-ID: <CACVs6=8B8fUX-csyZM%2BGbhQHPn70qxjO6va%2B%2BxtWDAiKWHywXQ@mail.gmail.com> In-Reply-To: <1335463643.2727.10.camel@powernoodle-l7.corp.yahoo.com> References: <1335463643.2727.10.camel@powernoodle-l7.corp.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Apr 26, 2012 at 11:07, Sean Bruno <seanbru@yahoo-inc.com> wrote: > I note form top that igb0 queue 0 is always "more busy" than any other > queue, there appears to be a second kernel igb0 "queue" process/thread > that appears to be moderately busy and 3 kernel igb0 "queue" > processes/threads that appear to be doing nothing in particular. Queue splitting in Intel cards is done using a hash of protocol headers, so this is expected behavior. This also helps with TCP and UDP performance, in terms of keeping packets for the same protocol control block on the same core, but for other applications it's not ideal. If your application does not require that kind of locality, there are things that can be done in the driver to make it easier to balance packets between all queues about-evenly. I've also seen it be the case that some cores will use more CPU time for reasons of unusually-slow memory access from that core, i.e. with non-uniform memory, or with some CPU threads, rather than because more packets are going to a queue. It would certainly be very nice at some point for netstat -I to grow awareness of multiple queues (which is a bit non-trivial because of how those statistics are gathered from struct ifnet) so that one could more easily watch in real time the packet rate to each queue. Probably someone with more patience than me could do this using the sysctl nodes the drivers export. Juli.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACVs6=8B8fUX-csyZM%2BGbhQHPn70qxjO6va%2B%2BxtWDAiKWHywXQ>