From owner-freebsd-net@FreeBSD.ORG  Thu Mar 20 21:13:10 2014
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 4619413A
 for <freebsd-net@freebsd.org>; Thu, 20 Mar 2014 21:13:10 +0000 (UTC)
Received: from hergotha.csail.mit.edu
 (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id F41D7387
 for <freebsd-net@freebsd.org>; Thu, 20 Mar 2014 21:13:09 +0000 (UTC)
Received: from hergotha.csail.mit.edu (localhost [127.0.0.1])
 by hergotha.csail.mit.edu (8.14.7/8.14.7) with ESMTP id s2KLD7LA085086;
 Thu, 20 Mar 2014 17:13:07 -0400 (EDT)
 (envelope-from wollman@hergotha.csail.mit.edu)
Received: (from wollman@localhost)
 by hergotha.csail.mit.edu (8.14.7/8.14.4/Submit) id s2KLD7GB085085;
 Thu, 20 Mar 2014 17:13:07 -0400 (EDT) (envelope-from wollman)
Date: Thu, 20 Mar 2014 17:13:07 -0400 (EDT)
From: Garrett Wollman <wollman@hergotha.csail.mit.edu>
Message-Id: <201403202113.s2KLD7GB085085@hergotha.csail.mit.edu>
To: csforgeron@gmail.com
Subject: Re: 9.2 ixgbe tx queue hang
X-Newsgroups: mit.lcs.mail.freebsd-net
In-Reply-To: <CAB2_NwAOmPtZjB03pdDiTK2OvQgqk-tYf83Jq4Ukt9jnZA8CNA@mail.gmail.com>
References: <CAB2_NwDG=gB1WCJ7JKTHpkJCrvPuAhipkn+vPyT+xXzOBrTGkg@mail.gmail.com>
 <1159309884.25490921.1395282576806.JavaMail.root@uoguelph.ca>
Organization: none
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3
 (hergotha.csail.mit.edu [127.0.0.1]); Thu, 20 Mar 2014 17:13:07 -0400 (EDT)
X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED
 autolearn=disabled version=3.3.2
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on
 hergotha.csail.mit.edu
Cc: freebsd-net@freebsd.org
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 20 Mar 2014 21:13:10 -0000

In article
<CAB2_NwAOmPtZjB03pdDiTK2OvQgqk-tYf83Jq4Ukt9jnZA8CNA@mail.gmail.com>,
csforgeron@gmail.com writes:

>50/27433/0 requests for jumbo clusters denied (4k/9k/16k)

This is going to screw you.  You need to make sure that no NIC driver
ever allocates 9k jumbo pages -- unless you are using one of those
mythical drivers that can't do scatter/gather DMA on receive, which
you don't appear to be.

These failures occur when the driver is trying to replenish its
receive queue, but is unable to allocate three *physically* contiguous
pages of RAM to construct the 9k jumbo cluster (of which the remaining
3k is simply wasted).  This happens on any moderately active server,
once physical memory gets checkerboarded with active single pages,
particularly with ZFS where those pages are wired in kernel memory and
so can't be evicted.

-GAWollman