From owner-freebsd-net@FreeBSD.ORG Sun Mar 23 15:19:26 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A53ABC1C; Sun, 23 Mar 2014 15:19:26 +0000 (UTC) Received: from mail-qa0-x22d.google.com (mail-qa0-x22d.google.com [IPv6:2607:f8b0:400d:c00::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 52A8C338; Sun, 23 Mar 2014 15:19:26 +0000 (UTC) Received: by mail-qa0-f45.google.com with SMTP id hw13so4288276qab.4 for ; Sun, 23 Mar 2014 08:19:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PhaSToECLAPC+3AH7DZil2toIj8LX1Ulh3x+r7e/034=; b=hAgZyg1vUrLE08Ahtxut4yCE6XMyXg6W46DqwL+4Mi2BsjUQ2C64TysnQ8UUjV50qj 3dWKDj2LvahX6Trhba1DBMZf6R/IWPFAW+nrXoa40cpoi5SyJeMq+BA2MnYUGyRv9C8u iPkRNXnHRXVZTLYAANInaMRaTH81J+dF7UUgjU1AZBlJtX5G9DKWhxxLzSLLEF4vVcih G4OhweNRcailAh3Icj2F0twtDLZTEApCozOzs9WOlhr/t0lXSaCxz/AT+y33qNUx1m4J G7yVQuMFJ86GAFDQTGSyyAzXimKnmpK4P2Ki+YPCuyibgQBHqouz+JtjPxHhJGHzGutJ fQDA== MIME-Version: 1.0 X-Received: by 10.140.48.199 with SMTP id o65mr31043448qga.16.1395587965557; Sun, 23 Mar 2014 08:19:25 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Sun, 23 Mar 2014 08:19:25 -0700 (PDT) In-Reply-To: <1752303953.1405506.1395524483238.JavaMail.root@uoguelph.ca> References: <1752303953.1405506.1395524483238.JavaMail.root@uoguelph.ca> Date: Sun, 23 Mar 2014 12:19:25 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Garrett Wollman , Jack Vogel , Markus Gebert X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2014 15:19:26 -0000 On Sat, Mar 22, 2014 at 6:41 PM, Rick Macklem wrote: > Christopher Forgeron wrote: > > #if defined(INET) || defined(INET6) > > /* Initialize to max value. */ > > if (ifp->if_hw_tsomax == 0) > > ifp->if_hw_tsomax = IP_MAXPACKET; > > KASSERT(ifp->if_hw_tsomax <= IP_MAXPACKET && > > ifp->if_hw_tsomax >= IP_MAXPACKET / 8, > > ("%s: tsomax outside of range", __func__)); > > #endif > > > > > > Should this be the location where it's being set rather than in > > ixgbe? I would assume that other drivers could fall prey to this > > issue. > > > All of this should be prepended with "I'm an NFS guy, not a networking > guy, so I might be wrong". > > Other drivers (and ixgbe for the 82598 chip) can handle a packet that > is in more than 32 mbufs. (I think the 82598 handles 100, grep for SCATTER > in *.h in sys/dev/ixgbe.) > [...] Yes, I agree we have to be careful about the limitations of other drivers, but I'm thinking setting tso to IP_MAXPACKET is a bad idea, unless all of the header subtractions are happening elsewhere. Then again, perhaps every other driver (and possibly ixgbe.. i need to look more) does a maxtso - various_headers to set a limit for data packets. I'm not familiar with the Freebsd network conventions/styles - I'm just asking questions, something I have a bad habit for, but I'm in charge of code stability issues at my work so it's hard to stop. > > Now, since several drivers do have this 32 mbufs limit, I can see an > argument > for making the default a little smaller to make these work, since the > driver can override the default. (About now someone usually jumps in and > says > something along the lines of "You can't do that until all the drivers that > can handle IP_MAXPACKET are fixed to set if_hw_tsomax" and since I can't > fix > drivers I can't test, that pretty much puts a stop on it.) > > Testing is a problem isn't it? I once again offer my stack of network cards and systems for some sort of testing.. I still have coax and token ring around. :-) > You see the problem isn't that IP_MAXPACKET is too big, but that the > hardware > has a limit of 32 non-contiguous chunks (mbufs)/packet and 32 * MCLBYTES = > 64K. > (Hardware/network drivers that can handle 35 or more chunks (they like to > call > them transmit segments, although ixgbe uses the term scatter) shouldn't > have > any problems.) > > I have an untested patch that adds a tsomaxseg count to use along with > tsomax > bytes so that a driver could inform tcp_output() it can only handle 32 > mbufs > and then tcp_output() would limit a TSO segment using both, but I can't > test > it, so who knows when/if that might happen. > > I think you give that to me in the next email - if not, please send. > I also have a patch that modifies NFS to use pagesize clusters (reducing > the > mbuf count in the list), but that one causes grief when testing on an i386 > (seems to run out of kernel memory to the point where it can't allocate > something > called "boundary tags" and pretty well wedges the machine at that point.) > Since I don't know how to fix this (I thought of making the patch "amd64 > only"), > I can't really commit this to head, either. > > Send me that one too. I love NFS patches. > As such, I think it's going to be "fix the drivers one at a time" and tell > folks to "disable TSO or limit rsize,wsize to 32K" when they run into > trouble. > (As you might have guessed, I'd rather just be "the NFS guy", but since NFS > "triggers the problem" I\m kinda stuck with it;-) > > I know in some circumstances disabling TSO can be a benefit, but in general you'd want it on a modern system with heavy data load. > Also should we not also subtract ETHER_VLAN_ENCAP_LEN from tsomax to > > make sure VLANs fit? > > > No idea. (I wouldn't know a VLAN if it jumped up and tried to > bite me on the nose.;-) So, I have no idea what does this, but > if it means the total ethernet header size can be > 14bytes, then I'd > agree. > > Yeah, you need another 4 bytes for VLAN header if you're not using hardware that strips it before the TCP stack gets it. I have a mix of hardware and software VLANs running on our backbone, mostly due to a mixed FreeBSD/OpenBSD/Windows environment. > > Perhaps there is something in the newer network code that is filling > > up the frames to the point where they are full - thus a TSO = > > IP_MAXPACKET is just now causing problems. > > > Yea, I have no idea why this didn't bite running 9.1. (Did 9.1 have > TSO enabled by default?) > I believe 9.0 has TSO on by default.. I seem to recall it always being there, but I can't easily confirm it now. My last 9.0-STABLE doesn't have an ixgbe card in it. > >