From owner-freebsd-net@FreeBSD.ORG Thu Mar 20 22:00:08 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 71BDD57D for ; Thu, 20 Mar 2014 22:00:08 +0000 (UTC) Received: from mail-vc0-x22d.google.com (mail-vc0-x22d.google.com [IPv6:2607:f8b0:400c:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2CE7CA83 for ; Thu, 20 Mar 2014 22:00:08 +0000 (UTC) Received: by mail-vc0-f173.google.com with SMTP id il7so1731112vcb.18 for ; Thu, 20 Mar 2014 15:00:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=g33YzIHmHW1EB8nAbiYTV3jucIf6xJOPbs4vVpzPHzs=; b=ymIO2pMC1WHcAdEexGp/je3M79s68G8GIahYfD4nFi7U6PSJiGfYfIv1sCYj91si7/ b+S3PVJLm1a3ufzUVvW5M2+FC1uCnNU5HQVMrhXHbfsivGQJiMBjQPVOMqbhARraAIic CH9mQKTu0mm82dtlyPM2q/F89n4whvP4WqU2sbM3HhLnKLOqv3vLbvf0XqHWoZ0eSgqG UOQJbi1dROgVY8nkriaT89siov2rWi15+PkrCevjKhrBbf+vcCXrxsMjHxAhm31h0E79 AjThDVKS8YGzJsKAAvraCgNL8PqRm2GzJ44hBwtPFucG48AUDHCwO6APb1D8O0ZuqMQm 6Fig== MIME-Version: 1.0 X-Received: by 10.221.20.199 with SMTP id qp7mr17215035vcb.24.1395352807292; Thu, 20 Mar 2014 15:00:07 -0700 (PDT) Received: by 10.221.11.135 with HTTP; Thu, 20 Mar 2014 15:00:07 -0700 (PDT) In-Reply-To: References: <1159309884.25490921.1395282576806.JavaMail.root@uoguelph.ca> <201403202113.s2KLD7GB085085@hergotha.csail.mit.edu> Date: Thu, 20 Mar 2014 15:00:07 -0700 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Jack Vogel To: Christopher Forgeron Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Garrett Wollman X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Mar 2014 22:00:08 -0000 What he's saying is that the driver should not be using 9K mbuf clusters, I thought this had been changed but I see the code in HEAD is still using the larger clusters when you up the mtu. I will put it on my list to change with the next update to HEAD. What version of ixgbe are you using? Jack On Thu, Mar 20, 2014 at 2:34 PM, Christopher Forgeron wrote: > I have found this: > > http://lists.freebsd.org/pipermail/freebsd-net/2013-October/036955.html > > I think what you're saying is that; > - a MTU of 9000 doesn't need to equal a 9k mbuf / jumbo cluster > - modern NIC drivers can gather 9000 bytes of data from various memory > locations > - The fact that I'm seeing 9k jumbo clusters is showing me that my driver > is trying to allocate 9k of contiguous space, and it's failing. > > Please correct me if I'm off here, I'd love to understand more. > > > On Thu, Mar 20, 2014 at 6:13 PM, Garrett Wollman < > wollman@hergotha.csail.mit.edu> wrote: > > > In article > > , > > csforgeron@gmail.com writes: > > > > >50/27433/0 requests for jumbo clusters denied (4k/9k/16k) > > > > This is going to screw you. You need to make sure that no NIC driver > > ever allocates 9k jumbo pages -- unless you are using one of those > > mythical drivers that can't do scatter/gather DMA on receive, which > > you don't appear to be. > > > > These failures occur when the driver is trying to replenish its > > receive queue, but is unable to allocate three *physically* contiguous > > pages of RAM to construct the 9k jumbo cluster (of which the remaining > > 3k is simply wasted). This happens on any moderately active server, > > once physical memory gets checkerboarded with active single pages, > > particularly with ZFS where those pages are wired in kernel memory and > > so can't be evicted. > > > > -GAWollman > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >