From owner-freebsd-net@FreeBSD.ORG Thu Mar 20 21:34:05 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4BC399B5 for ; Thu, 20 Mar 2014 21:34:05 +0000 (UTC) Received: from mail-qg0-x230.google.com (mail-qg0-x230.google.com [IPv6:2607:f8b0:400d:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 09F9E7F4 for ; Thu, 20 Mar 2014 21:34:04 +0000 (UTC) Received: by mail-qg0-f48.google.com with SMTP id j107so4426148qga.7 for ; Thu, 20 Mar 2014 14:34:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=yJ8Z9YpMhDk+5D4Pr9AnlL3TOvMIxeNPEBsAmvQclwA=; b=bifBoCkW/FFfqFJFY6dN6lA3ALS8oqnx/3UUdNd3RW70u8Ybs/bMa8lGRqqnqv4elm hJ7o005QHya+oeKuiL4xqcoN5AXzuhsXCmQirNqYSFSgOjcdCg/oJbTfFJLwS03vy9ul WyljwD5G701bdxU6y+vmunqPrQOgRD6hss1L2ud3zjEFlkgAXhypthT+FOWr854RFJys +3qe21hIQChqpOc3S2m1S7Bsxc2rlKpggx0QlCUVlCt2HZokDLHZB56v3+6U5VwUagyP 7XjMfB+7fWkOIhoss6pBo6rJWnz+1loSPiJARMOBTyl1H4t8T3RWgvAQVMYLDpNBUqiK vYiQ== MIME-Version: 1.0 X-Received: by 10.224.53.198 with SMTP id n6mr52909695qag.41.1395351244261; Thu, 20 Mar 2014 14:34:04 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Thu, 20 Mar 2014 14:34:04 -0700 (PDT) In-Reply-To: <201403202113.s2KLD7GB085085@hergotha.csail.mit.edu> References: <1159309884.25490921.1395282576806.JavaMail.root@uoguelph.ca> <201403202113.s2KLD7GB085085@hergotha.csail.mit.edu> Date: Thu, 20 Mar 2014 18:34:04 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Garrett Wollman Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Mar 2014 21:34:05 -0000 I have found this: http://lists.freebsd.org/pipermail/freebsd-net/2013-October/036955.html I think what you're saying is that; - a MTU of 9000 doesn't need to equal a 9k mbuf / jumbo cluster - modern NIC drivers can gather 9000 bytes of data from various memory locations - The fact that I'm seeing 9k jumbo clusters is showing me that my driver is trying to allocate 9k of contiguous space, and it's failing. Please correct me if I'm off here, I'd love to understand more. On Thu, Mar 20, 2014 at 6:13 PM, Garrett Wollman < wollman@hergotha.csail.mit.edu> wrote: > In article > , > csforgeron@gmail.com writes: > > >50/27433/0 requests for jumbo clusters denied (4k/9k/16k) > > This is going to screw you. You need to make sure that no NIC driver > ever allocates 9k jumbo pages -- unless you are using one of those > mythical drivers that can't do scatter/gather DMA on receive, which > you don't appear to be. > > These failures occur when the driver is trying to replenish its > receive queue, but is unable to allocate three *physically* contiguous > pages of RAM to construct the 9k jumbo cluster (of which the remaining > 3k is simply wasted). This happens on any moderately active server, > once physical memory gets checkerboarded with active single pages, > particularly with ZFS where those pages are wired in kernel memory and > so can't be evicted. > > -GAWollman >