From owner-freebsd-net@FreeBSD.ORG Wed Apr 18 16:49:53 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1EAA5106566B; Wed, 18 Apr 2012 16:49:53 +0000 (UTC) (envelope-from seanbru@yahoo-inc.com) Received: from mrout1-b.corp.bf1.yahoo.com (mrout1-b.corp.bf1.yahoo.com [98.139.253.104]) by mx1.freebsd.org (Postfix) with ESMTP id DFEA18FC16; Wed, 18 Apr 2012 16:49:52 +0000 (UTC) Received: from [IPv6:::1] (rideseveral.corp.yahoo.com [10.73.160.231]) by mrout1-b.corp.bf1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id q3IGn6J0035093; Wed, 18 Apr 2012 09:49:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com; s=cobra; t=1334767747; bh=VdEEdvwR02rvWhH/wBm5l4JFDoHAjS/m9ePYMzFlCOM=; h=Subject:From:To:Cc:In-Reply-To:References:Content-Type:Date: Message-ID:Mime-Version:Content-Transfer-Encoding; b=epFwLr9t37OoTRXOwBTVHRlhcpYvImfNQEoy0M66PnrnoyZLMId0wo/OHWT1KrAhl iVrW2KRB1yMi/a7Gdxd+oC9ECcwPDdGhs9m1+LqGabu/3LP2GSIcSsJ+ICSUnoFc/l ZVYFDlcwbA6om/1AXAuNaoSWER7DFhkDK2tRJahQ= From: Sean Bruno To: Jack Vogel In-Reply-To: References: <1334705064.4486.23.camel@powernoodle-l7.corp.yahoo.com> <20120418072818.GA58850@onelab2.iet.unipi.it> <1334766438.3466.4.camel@powernoodle-l7.corp.yahoo.com> Content-Type: text/plain; charset="UTF-8" Date: Wed, 18 Apr 2012 09:49:06 -0700 Message-ID: <1334767746.3466.6.camel@powernoodle-l7.corp.yahoo.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit X-Milter-Version: master.31+4-gbc07cd5+ X-CLX-ID: 767746000 Cc: "freebsd-net@freebsd.org" , Luigi Rizzo Subject: Re: igb(4) Raising IGB_MAX_TXD ?? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Apr 2012 16:49:53 -0000 ok, good. that at least confirms that I correctly translated between the driver code and documented specification. I will try 8k as a test for now and see how that runs. sean On Wed, 2012-04-18 at 09:46 -0700, Jack Vogel wrote: > The MAX value is something I set, not a hardware thing, it was based > on reports > I had from the various driver engineers in our org. If you increase > the ring size > you might run into other performance issues, however there's nothing > stopping > you from trying. Just be aware that its not something that's been > tested. > > Let me know how it goes please :) > > Jack > > > On Wed, Apr 18, 2012 at 9:27 AM, Sean Bruno > wrote: > On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote: > > On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote: > > > We're running a service with a 82576 configured with 4 > queues and a > > > maxed rxd/txd configuration: > > > > > > http://people.freebsd.org/~sbruno/igb_stats.txt > > > > these stats show that over half of your incoming traffic is > > made of small packets (65..127 bytes) but especially, that > > the "missed packets" count is very small (18k out of 40G > packets) > > none of them is reported as "no_desc_avail", and only 76 are > > "recv_no_buffer". > > > > Are you dropping packets in the ip interrupt handler by > chance ? > > what are your settings there ? > > > > nope, doesn't look like it. > > http://people.freebsd.org/~sbruno/igb_ip_stats.txt > > > BTW it seems that there is only one global setting for the > dispatch > > policy, but for instance there are two netisr_dispatch() > calls > > in the incoming path, one for layer2 and one for layer3. > > The former has relatively little work to do and so it might > > make sense to have direct dispatch, the other can be > expensive > > so i wonder if it wouldn't be better to use deferred > dispatch. > > If not, perhaps you might try to reduce the > rx_processing_limit > > to bring down the load on the intr thread. > > > I don't really see any issue with horsepower on this host at > the moment > with 4 queues. I mean it looks a little something like this > under high > load: > > http://people.freebsd.org/~sbruno/igb_top.txt > > I guess my question still stands though, since the ethernet > controller > is reporting that it doesn't have any more descriptors > available is the > hardcoded 4k max descriptors a limit that an be raised? > > > With your numbers i doubt that raising the queue size helps. > > > Indeed, you're probably right and this is more than likely an > application problem that will have to be resolved. However, > I'm still > curious if the MAX_RXD/TXD is really 4k or if the > documentation is > correct and we can raise it to 32k for testing? > > Sean >