From owner-freebsd-net@FreeBSD.ORG Thu Mar 20 22:32:18 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 811E2246 for ; Thu, 20 Mar 2014 22:32:18 +0000 (UTC) Received: from mail-qa0-x232.google.com (mail-qa0-x232.google.com [IPv6:2607:f8b0:400d:c00::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 39F68E30 for ; Thu, 20 Mar 2014 22:32:18 +0000 (UTC) Received: by mail-qa0-f50.google.com with SMTP id o15so1619339qap.37 for ; Thu, 20 Mar 2014 15:32:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=J8rd6wIdjmVReBrSAn7difcBlde/mlL6d37uqlSPzus=; b=w74w3Y83opq4q4cGBmH+0H/PNBrgh9fuDWlPaaXMRh6F7MVckjEzMOIsWnmWRSwdcv W6Aj/Kc6E8fVQXHLdnTzPcp7C2Z5JPdjP5jkyZxcRQFuVtxhgCyHmQObjBFx26CYeQhB v9xkH2ipq9azFKb2UftKQztRgKJFzb0Ki1Lx7V+vHmUPqUqv+xenYoEtF7/SFe4Row1J cuuhDwOY0OPj5iLJNaf8yf2ZdgQdrvaz8a2gbcNm7KtpW8ot3Bwo8rz3PjIFoWJxHM3W F1HgOyMjl9USz783JDPPZNCZKYTE9SRWW6fgnJraQSdONCqVe4JMsKJhOnTf39yLOACK X6QQ== MIME-Version: 1.0 X-Received: by 10.140.29.68 with SMTP id a62mr36662338qga.57.1395354737422; Thu, 20 Mar 2014 15:32:17 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Thu, 20 Mar 2014 15:32:17 -0700 (PDT) In-Reply-To: References: <1159309884.25490921.1395282576806.JavaMail.root@uoguelph.ca> <201403202113.s2KLD7GB085085@hergotha.csail.mit.edu> Date: Thu, 20 Mar 2014 19:32:17 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Jack Vogel Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Garrett Wollman X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Mar 2014 22:32:18 -0000 I agree, performance is noticeably worse with TSO off, but I thought it would be a good step in troubleshooting. I'm glad you're a regular reader of the list, so I don't have to settle for slow performance. :-) I'm applying your patch now, I think it will fix it - but I'll report in after it's run iometer for the night regardless. On another note: What's so different about memory allocation in 10 that is making this an issue? On Thu, Mar 20, 2014 at 7:24 PM, Jack Vogel wrote: > I strongly discourage anyone from disabling TSO on 10G, its necessary to > get the > performance one wants to see on the hardware. > > Here is a patch to do what i'm talking about: > > *** ixgbe.c Fri Jan 10 18:12:20 2014 > --- ixgbe.jfv.c Thu Mar 20 23:04:15 2014 > *************** ixgbe_init_locked(struct adapter *adapte > *** 1140,1151 **** > */ > if (adapter->max_frame_size <= 2048) > adapter->rx_mbuf_sz = MCLBYTES; > - else if (adapter->max_frame_size <= 4096) > - adapter->rx_mbuf_sz = MJUMPAGESIZE; > - else if (adapter->max_frame_size <= 9216) > - adapter->rx_mbuf_sz = MJUM9BYTES; > else > ! adapter->rx_mbuf_sz = MJUM16BYTES; > > /* Prepare receive descriptors and buffers */ > if (ixgbe_setup_receive_structures(adapter)) { > --- 1140,1147 ---- > */ > if (adapter->max_frame_size <= 2048) > adapter->rx_mbuf_sz = MCLBYTES; > else > ! adapter->rx_mbuf_sz = MJUMPAGESIZE; > > /* Prepare receive descriptors and buffers */ > if (ixgbe_setup_receive_structures(adapter)) { > > > > > > > On Thu, Mar 20, 2014 at 3:12 PM, Christopher Forgeron < > csforgeron@gmail.com> wrote: > >> Hi Jack, >> >> I'm on ixgbe 2.5.15 >> >> I see a few other threads about using MJUMPAGESIZE instead of MJUM9BYTES. >> >> If you have a patch you'd like me to test, I'll compile it in and let >> you know. I was just looking at Garrett's if_em.c patch and thinking about >> applying it to ixgbe.. >> >> As it stands I seem to not be having the problem now that I have >> disabled TSO on ix0, but I still need more test runs to confirm - Which is >> also in line (i think) with what you are all saying. >> >> >> >> >> On Thu, Mar 20, 2014 at 7:00 PM, Jack Vogel wrote: >> >>> What he's saying is that the driver should not be using 9K mbuf >>> clusters, I thought >>> this had been changed but I see the code in HEAD is still using the >>> larger clusters >>> when you up the mtu. I will put it on my list to change with the next >>> update to HEAD. >>> >>> >>> What version of ixgbe are you using? >>> >>> Jack >>> >>> >>> >>> On Thu, Mar 20, 2014 at 2:34 PM, Christopher Forgeron < >>> csforgeron@gmail.com> wrote: >>> >>>> I have found this: >>>> >>>> http://lists.freebsd.org/pipermail/freebsd-net/2013-October/036955.html >>>> >>>> I think what you're saying is that; >>>> - a MTU of 9000 doesn't need to equal a 9k mbuf / jumbo cluster >>>> - modern NIC drivers can gather 9000 bytes of data from various memory >>>> locations >>>> - The fact that I'm seeing 9k jumbo clusters is showing me that my >>>> driver >>>> is trying to allocate 9k of contiguous space, and it's failing. >>>> >>>> Please correct me if I'm off here, I'd love to understand more. >>>> >>>> >>>> On Thu, Mar 20, 2014 at 6:13 PM, Garrett Wollman < >>>> wollman@hergotha.csail.mit.edu> wrote: >>>> >>>> > In article >>>> > , >>>> > csforgeron@gmail.com writes: >>>> > >>>> > >50/27433/0 requests for jumbo clusters denied (4k/9k/16k) >>>> > >>>> > This is going to screw you. You need to make sure that no NIC driver >>>> > ever allocates 9k jumbo pages -- unless you are using one of those >>>> > mythical drivers that can't do scatter/gather DMA on receive, which >>>> > you don't appear to be. >>>> > >>>> > These failures occur when the driver is trying to replenish its >>>> > receive queue, but is unable to allocate three *physically* contiguous >>>> > pages of RAM to construct the 9k jumbo cluster (of which the remaining >>>> > 3k is simply wasted). This happens on any moderately active server, >>>> > once physical memory gets checkerboarded with active single pages, >>>> > particularly with ZFS where those pages are wired in kernel memory and >>>> > so can't be evicted. >>>> > >>>> > -GAWollman >>>> > >>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>>> >>> >>> >> >