From owner-freebsd-net@FreeBSD.ORG Wed Apr 18 16:38:08 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BB1BB106566B for ; Wed, 18 Apr 2012 16:38:08 +0000 (UTC) (envelope-from seanbru@yahoo-inc.com) Received: from mrout1-b.corp.bf1.yahoo.com (mrout1-b.corp.bf1.yahoo.com [98.139.253.104]) by mx1.freebsd.org (Postfix) with ESMTP id 714638FC0C for ; Wed, 18 Apr 2012 16:38:08 +0000 (UTC) Received: from [IPv6:::1] (rideseveral.corp.yahoo.com [10.73.160.231]) by mrout1-b.corp.bf1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id q3IGRJVJ024114; Wed, 18 Apr 2012 09:27:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com; s=cobra; t=1334766440; bh=g/IgYGod9Cwy0RuUlia7PRGxd9uh2Ij8iOwKfrYjjmI=; h=Subject:From:Reply-To:To:Cc:In-Reply-To:References:Content-Type: Date:Message-ID:Mime-Version; b=fJEdmM7z0kTc0zoQh+0Bb/kHuDEOFnSwWdj9ZHC5MhRksva6tN//INLlUzWjoYvlt tnjg6cIMmkKYBT9IojpiIK2wIyJeJj28AYGJhfNeBGDbUKv1i4aIgmKj7gYxHjgkk1 ubFPSoPyT4GPYqF9eJZxOG0NuPbf7dvk8f+/9GZg= From: Sean Bruno To: Luigi Rizzo In-Reply-To: <20120418072818.GA58850@onelab2.iet.unipi.it> References: <1334705064.4486.23.camel@powernoodle-l7.corp.yahoo.com> <20120418072818.GA58850@onelab2.iet.unipi.it> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-fZwHBsYnQWqPnYlo+ahK" Date: Wed, 18 Apr 2012 09:27:18 -0700 Message-ID: <1334766438.3466.4.camel@powernoodle-l7.corp.yahoo.com> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port X-Milter-Version: master.31+4-gbc07cd5+ X-CLX-ID: 766439004 Cc: "freebsd-net@freebsd.org" , Jack Vogel Subject: Re: igb(4) Raising IGB_MAX_TXD ?? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: "sbruno@freebsd.org" List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Apr 2012 16:38:08 -0000 --=-fZwHBsYnQWqPnYlo+ahK Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, 2012-04-18 at 00:28 -0700, Luigi Rizzo wrote: > On Tue, Apr 17, 2012 at 04:24:24PM -0700, Sean Bruno wrote: > > We're running a service with a 82576 configured with 4 queues and a > > maxed rxd/txd configuration: > >=20 > > http://people.freebsd.org/~sbruno/igb_stats.txt >=20 > these stats show that over half of your incoming traffic is > made of small packets (65..127 bytes) but especially, that > the "missed packets" count is very small (18k out of 40G packets) > none of them is reported as "no_desc_avail", and only 76 are > "recv_no_buffer". >=20 > Are you dropping packets in the ip interrupt handler by chance ? > what are your settings there ? >=20 nope, doesn't look like it. =20 http://people.freebsd.org/~sbruno/igb_ip_stats.txt > BTW it seems that there is only one global setting for the dispatch > policy, but for instance there are two netisr_dispatch() calls > in the incoming path, one for layer2 and one for layer3. > The former has relatively little work to do and so it might > make sense to have direct dispatch, the other can be expensive > so i wonder if it wouldn't be better to use deferred dispatch. > If not, perhaps you might try to reduce the rx_processing_limit > to bring down the load on the intr thread. I don't really see any issue with horsepower on this host at the moment with 4 queues. I mean it looks a little something like this under high load: http://people.freebsd.org/~sbruno/igb_top.txt I guess my question still stands though, since the ethernet controller is reporting that it doesn't have any more descriptors available is the hardcoded 4k max descriptors a limit that an be raised? > With your numbers i doubt that raising the queue size helps. Indeed, you're probably right and this is more than likely an application problem that will have to be resolved. However, I'm still curious if the MAX_RXD/TXD is really 4k or if the documentation is correct and we can raise it to 32k for testing? Sean --=-fZwHBsYnQWqPnYlo+ahK Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) iQEcBAABAgAGBQJPjutmAAoJEL2UHwafTLtOqVAH/Re6EjwbReJdEXq2O9ZYM3LE kgWQH4XIYfG+UvlIYRjaOxL/8vydd8iFYQIdCUh2AZseZ19uJJveUSD0bp42V/9H mFR7Z6OOYUxWwI15WjoVh2I/q2J+qP3dX0VGHt5kvJLUcBm+Ys9asqHu54RBZeNB UtacUALDgKOO4nasTBZCyEYOl2R3czmN9BVpI8TNuhbekksXcJIJxERsfHJ71KD4 EnUi24/TVJAjUcp4lU/ECEhmXJmB0XRJ45AigyiM1Iii+N09WLss+yfQLuq2c8bA QMA6SKEORK2ymyua7tN/n/BsVOdpnYscyItzXB6DvvH3MWrVRMij6bqpf7Bhop4= =G5jY -----END PGP SIGNATURE----- --=-fZwHBsYnQWqPnYlo+ahK--