Date: Wed, 18 Apr 2012 09:49:06 -0700 From: Sean Bruno <seanbru@yahoo-inc.com> To: Jack Vogel <jfvogel@gmail.com> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Luigi Rizzo <rizzo@iet.unipi.it> Subject: Re: igb(4) Raising IGB_MAX_TXD ?? Message-ID: <1334767746.3466.6.camel@powernoodle-l7.corp.yahoo.com> In-Reply-To: <CAFOYbcmJcUtN5DdvDg4L9aJYY8g5bz9z5whQSKLMEA-dAPcvOQ@mail.gmail.com> References: <1334705064.4486.23.camel@powernoodle-l7.corp.yahoo.com> <20120418072818.GA58850@onelab2.iet.unipi.it> <1334766438.3466.4.camel@powernoodle-l7.corp.yahoo.com> <CAFOYbcmJcUtN5DdvDg4L9aJYY8g5bz9z5whQSKLMEA-dAPcvOQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
ok, good. that at least confirms that I correctly translated between the driver code and documented specification. I will try 8k as a test for now and see how that runs. sean On Wed, 2012-04-18 at 09:46 -0700, Jack Vogel wrote: > The MAX value is something I set, not a hardware thing, it was based > on reports > I had from the various driver engineers in our org. If you increase > the ring size > you might run into other performance issues, however there's nothing > stopping > you from trying. Just be aware that its not something that's been > tested. > > Let me know how it goes please :) > > Jack > > > On Wed, Apr 18, 2012 at 9:27 AM, Sean Bruno <seanbru@yahoo-inc.com> > wrote: > On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote: > > On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote: > > > We're running a service with a 82576 configured with 4 > queues and a > > > maxed rxd/txd configuration: > > > > > > http://people.freebsd.org/~sbruno/igb_stats.txt > > > > these stats show that over half of your incoming traffic is > > made of small packets (65..127 bytes) but especially, that > > the "missed packets" count is very small (18k out of 40G > packets) > > none of them is reported as "no_desc_avail", and only 76 are > > "recv_no_buffer". > > > > Are you dropping packets in the ip interrupt handler by > chance ? > > what are your settings there ? > > > > nope, doesn't look like it. > > http://people.freebsd.org/~sbruno/igb_ip_stats.txt > > > BTW it seems that there is only one global setting for the > dispatch > > policy, but for instance there are two netisr_dispatch() > calls > > in the incoming path, one for layer2 and one for layer3. > > The former has relatively little work to do and so it might > > make sense to have direct dispatch, the other can be > expensive > > so i wonder if it wouldn't be better to use deferred > dispatch. > > If not, perhaps you might try to reduce the > rx_processing_limit > > to bring down the load on the intr thread. > > > I don't really see any issue with horsepower on this host at > the moment > with 4 queues. I mean it looks a little something like this > under high > load: > > http://people.freebsd.org/~sbruno/igb_top.txt > > I guess my question still stands though, since the ethernet > controller > is reporting that it doesn't have any more descriptors > available is the > hardcoded 4k max descriptors a limit that an be raised? > > > With your numbers i doubt that raising the queue size helps. > > > Indeed, you're probably right and this is more than likely an > application problem that will have to be resolved. However, > I'm still > curious if the MAX_RXD/TXD is really 4k or if the > documentation is > correct and we can raise it to 32k for testing? > > Sean >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1334767746.3466.6.camel>