From owner-freebsd-net@FreeBSD.ORG Wed Apr 18 16:46:49 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DB930106566C; Wed, 18 Apr 2012 16:46:49 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-wg0-f42.google.com (mail-wg0-f42.google.com [74.125.82.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3698B8FC20; Wed, 18 Apr 2012 16:46:49 +0000 (UTC) Received: by wgbds11 with SMTP id ds11so773690wgb.1 for ; Wed, 18 Apr 2012 09:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=NSJeRcagKOd+eJX+EzmjISmnEvVbyoM+IwkMkMrml94=; b=pwEB/I88++fD8gQaGXLsvCJAyHyaXxUvsj9mOIyQPG2WGadMTGL9PXyne2Vw2YDnKh W5qt6pS1WQgquWZ6iOFUnuypyeufRDBPEc219ujPJbb8cNFcPjcpSZsqq8nMSdqZsJba KNhne8yvxujsNy24t5yutpKVEQCt72J6MUCn/Hzx7MetHJMgaZ0pPP51xVeciGksUGww xccAlqHQE+aCgcLrckB+nhAS5AfScHhakmo+gRq8q4C8KJgvTQASFKZJbUxdxZVQFFJP aXf36PE1o1a0lrED5NTsf0UWamgQt+OlVu4wHer7o96gwbsdb4S7u+E4W6BC8HawgV8C COHQ== MIME-Version: 1.0 Received: by 10.216.134.155 with SMTP id s27mr2101770wei.80.1334767607936; Wed, 18 Apr 2012 09:46:47 -0700 (PDT) Received: by 10.180.3.170 with HTTP; Wed, 18 Apr 2012 09:46:47 -0700 (PDT) In-Reply-To: <1334766438.3466.4.camel@powernoodle-l7.corp.yahoo.com> References: <1334705064.4486.23.camel@powernoodle-l7.corp.yahoo.com> <20120418072818.GA58850@onelab2.iet.unipi.it> <1334766438.3466.4.camel@powernoodle-l7.corp.yahoo.com> Date: Wed, 18 Apr 2012 09:46:47 -0700 Message-ID: From: Jack Vogel To: "sbruno@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "freebsd-net@freebsd.org" , Luigi Rizzo Subject: Re: igb(4) Raising IGB_MAX_TXD ?? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Apr 2012 16:46:49 -0000 The MAX value is something I set, not a hardware thing, it was based on reports I had from the various driver engineers in our org. If you increase the ring size you might run into other performance issues, however there's nothing stopping you from trying. Just be aware that its not something that's been tested. Let me know how it goes please :) Jack On Wed, Apr 18, 2012 at 9:27 AM, Sean Bruno wrote: > On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote: > > On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote: > > > We're running a service with a 82576 configured with 4 queues and a > > > maxed rxd/txd configuration: > > > > > > http://people.freebsd.org/~sbruno/igb_stats.txt > > > > these stats show that over half of your incoming traffic is > > made of small packets (65..127 bytes) but especially, that > > the "missed packets" count is very small (18k out of 40G packets) > > none of them is reported as "no_desc_avail", and only 76 are > > "recv_no_buffer". > > > > Are you dropping packets in the ip interrupt handler by chance ? > > what are your settings there ? > > > nope, doesn't look like it. > > http://people.freebsd.org/~sbruno/igb_ip_stats.txt > > > BTW it seems that there is only one global setting for the dispatch > > policy, but for instance there are two netisr_dispatch() calls > > in the incoming path, one for layer2 and one for layer3. > > The former has relatively little work to do and so it might > > make sense to have direct dispatch, the other can be expensive > > so i wonder if it wouldn't be better to use deferred dispatch. > > If not, perhaps you might try to reduce the rx_processing_limit > > to bring down the load on the intr thread. > > I don't really see any issue with horsepower on this host at the moment > with 4 queues. I mean it looks a little something like this under high > load: > > http://people.freebsd.org/~sbruno/igb_top.txt > > I guess my question still stands though, since the ethernet controller > is reporting that it doesn't have any more descriptors available is the > hardcoded 4k max descriptors a limit that an be raised? > > > With your numbers i doubt that raising the queue size helps. > > Indeed, you're probably right and this is more than likely an > application problem that will have to be resolved. However, I'm still > curious if the MAX_RXD/TXD is really 4k or if the documentation is > correct and we can raise it to 32k for testing? > > Sean >