Date: Wed, 18 Apr 2012 09:46:47 -0700 From: Jack Vogel <jfvogel@gmail.com> To: "sbruno@freebsd.org" <sbruno@freebsd.org> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it> Subject: Re: igb(4) Raising IGB_MAX_TXD ?? Message-ID: <CAFOYbcmJcUtN5DdvDg4L9aJYY8g5bz9z5whQSKLMEA-dAPcvOQ@mail.gmail.com> In-Reply-To: <1334766438.3466.4.camel@powernoodle-l7.corp.yahoo.com> References: <1334705064.4486.23.camel@powernoodle-l7.corp.yahoo.com> <20120418072818.GA58850@onelab2.iet.unipi.it> <1334766438.3466.4.camel@powernoodle-l7.corp.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
The MAX value is something I set, not a hardware thing, it was based on reports I had from the various driver engineers in our org. If you increase the ring size you might run into other performance issues, however there's nothing stopping you from trying. Just be aware that its not something that's been tested. Let me know how it goes please :) Jack On Wed, Apr 18, 2012 at 9:27 AM, Sean Bruno <seanbru@yahoo-inc.com> wrote: > On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote: > > On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote: > > > We're running a service with a 82576 configured with 4 queues and a > > > maxed rxd/txd configuration: > > > > > > http://people.freebsd.org/~sbruno/igb_stats.txt > > > > these stats show that over half of your incoming traffic is > > made of small packets (65..127 bytes) but especially, that > > the "missed packets" count is very small (18k out of 40G packets) > > none of them is reported as "no_desc_avail", and only 76 are > > "recv_no_buffer". > > > > Are you dropping packets in the ip interrupt handler by chance ? > > what are your settings there ? > > > nope, doesn't look like it. > > http://people.freebsd.org/~sbruno/igb_ip_stats.txt > > > BTW it seems that there is only one global setting for the dispatch > > policy, but for instance there are two netisr_dispatch() calls > > in the incoming path, one for layer2 and one for layer3. > > The former has relatively little work to do and so it might > > make sense to have direct dispatch, the other can be expensive > > so i wonder if it wouldn't be better to use deferred dispatch. > > If not, perhaps you might try to reduce the rx_processing_limit > > to bring down the load on the intr thread. > > I don't really see any issue with horsepower on this host at the moment > with 4 queues. I mean it looks a little something like this under high > load: > > http://people.freebsd.org/~sbruno/igb_top.txt > > I guess my question still stands though, since the ethernet controller > is reporting that it doesn't have any more descriptors available is the > hardcoded 4k max descriptors a limit that an be raised? > > > With your numbers i doubt that raising the queue size helps. > > Indeed, you're probably right and this is more than likely an > application problem that will have to be resolved. However, I'm still > curious if the MAX_RXD/TXD is really 4k or if the documentation is > correct and we can raise it to 32k for testing? > > Sean >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFOYbcmJcUtN5DdvDg4L9aJYY8g5bz9z5whQSKLMEA-dAPcvOQ>