From owner-freebsd-net@FreeBSD.ORG Tue Mar 25 12:28:25 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 92D8359F for ; Tue, 25 Mar 2014 12:28:25 +0000 (UTC) Received: from mail-pa0-f53.google.com (mail-pa0-f53.google.com [209.85.220.53]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 63C5838A for ; Tue, 25 Mar 2014 12:28:25 +0000 (UTC) Received: by mail-pa0-f53.google.com with SMTP id ld10so369196pab.26 for ; Tue, 25 Mar 2014 05:28:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=PmRHdStjYdE4oW9iaJiHHelhc3okSnly6NFhq+7+lWs=; b=S1mz+oQMC3CdZF9Edh6mn6x8yLoJnfePIjMG+uJZV6gAFO2B3CDf80GMeF+TXDghTz XabId4LKlC7Bhv3QWS/Dc+DSztoCfLqXeeladrVZolUnqQa9dw86hLSlYBLYIRpxCvAG yxA8IE/5K+tOS30MOC4YP1tZOlpKdI8PnvUAmLi/1zFzk1diJet+F/E3ig1bgMsKavB6 kLGFcirApDQKqsCmGOKxLj91YRYcxy8oDTZ9o82cPawjnVsOvwP7IWneiT78Pjvxy1nv qTAia87FZDj9HQoBgLqcYofrAlTpYjIPZxTyDcyN9f5tocBR86cehDwdFQsMcoLsVECk 0MCA== X-Gm-Message-State: ALoCoQlAH5Hw6392Jcp0lUZcutpnBd1qDR+LeTjzAhQahFRWJsbdBZuel8ertoCUr/JUVIHnvfHQ MIME-Version: 1.0 X-Received: by 10.66.252.135 with SMTP id zs7mr79004679pac.13.1395750103933; Tue, 25 Mar 2014 05:21:43 -0700 (PDT) Received: by 10.68.111.37 with HTTP; Tue, 25 Mar 2014 05:21:43 -0700 (PDT) In-Reply-To: <906D7DF8-DD6E-4501-B3ED-42EF728241F4@hostpoint.ch> References: <1973302314.2516695.1395710288222.JavaMail.root@uoguelph.ca> <906D7DF8-DD6E-4501-B3ED-42EF728241F4@hostpoint.ch> Date: Tue, 25 Mar 2014 13:21:43 +0100 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Johan Kooijman To: Markus Gebert Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Rick Macklem , Garrett Wollman , Jack Vogel , Christopher Forgeron X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2014 12:28:25 -0000 Hey guys, I have nothing on your code level to add, but.. while investigating this issue I ran into the guy that originally created the bug ( http://www.freebsd.org/cgi/query-pr.cgi?pr=183390&cat=). In the email exchange that followed he told me that had found a workaround by running a specific -STABLE revision: "Yes, we found a workaround. We upgraded to the -STABLE branch of the 9.2, so we use this currently: [root@storagex ~]# uname -a FreeBSD storagex.lan.granaglia.com 9.2-STABLE FreeBSD 9.2-STABLE #0 r257712: Tue Nov 5 23:02:49 CET 2013 root@storagex.lan.granaglia.com:/usr/obj/usr/src/sys/GENERIC amd64" Maybe this could help you in your quest to hunt this bug down. On Tue, Mar 25, 2014 at 1:16 PM, Markus Gebert wrote: > > On 25.03.2014, at 02:18, Rick Macklem wrote: > > > Christopher Forgeron wrote: > >> > >> > >> > >> This is regarding the TSO patch that Rick suggested earlier. (With > >> many thanks for his time and suggestion) > >> > >> > >> As I mentioned earlier, it did not fix the issue on a 10.0 system. It > >> did make it less of a problem on 9.2, but either way, I think it's > >> not needed, and shouldn't be considered as a patch for testing/etc. > >> > >> > >> Patching TSO to anything other than a max value (and by default the > >> code gives it IP_MAXPACKET) is confusing the matter, as the packet > >> length ultimately needs to be adjusted for many things on the fly > >> like TCP Options, etc. Using static header sizes won't be a good > >> idea. > >> > > If you look at tcp_output(), you'll notice that it doesn't do TSO if > > there are any options. That way it knows that the TCP/IP header is > > just hdrlen. > > > > If you don't limit the TSO packet (including TCP/IP and ethernet headers) > > to 64K, then the "ix" driver can't send them, which is the problem > > you guys are seeing. > > > > There are other ways to fix this problem, but they all may introduce > > issues that reducing if_hw_tsomax by a small amount does not. > > For example, m_defrag() could be modified to use 4K pagesize clusters, > > but this might introduce memory fragmentation problems. (I observed > > what I think are memory fragmentation problems when I switched NFS > > to use 4K pagesize clusters for large I/O messages.) > > > > If setting IP_MAXPACKET to 65518 fixes the problem (no more EFBIG > > error replies), then that is the size that if_hw_tsomax can be set > > to (just can't change IP_MAXPACKET, but that is defined for other > > things). (It just happens that IP_MAXPACKET is what if_hw_tsomax > > defaults to. It has no other effect w.r.t. TSO.) > > > >> > >> Additionally, it seems that setting nic TSO will/may be ignored by > >> code like this in sys/netinet/tcp_output.c: > >> > > Is this confirmed or still a 'it seems'? Have you actually seen a > tp->t_tsomax value in tcp_output() bigger than if_hw_tsomax or was this > just speculation because the values are stored in different places? (Sorry, > if you already stated this in another email, it's currently hard to keep > track of all the information.) > > Anyway, this dtrace one-liner should be a good test if other values appear > in tp->t_tsomax: > > # dtrace -n 'fbt::tcp_output:entry / args[0]->t_tsomax != 0 && > args[0]->t_tsomax != 65518 / { printf("unexpected tp->t_tsomax: %i\n", > args[0]->t_tsomax); stack(); }' > > Remember to adjust the value in the condition to whatever you're currently > expecting. The value seems to be 0 for new connections, probably when > tcp_mss() has not been called yet. So that's seems normal and I have > excluded that case too. This will also print a kernel stack trace in case > it sees an unexpected value. > > > > Yes, but I don't know why. > > The only conjecture I can come up with is that another net driver is > > stacked above "ix" and the setting for if_hw_tsomax doesn't propagate > > up. (If you look at the commit log message for r251296, the intent > > of adding if_hw_tsomax was to allow device drivers to set a smaller > > tsomax than IP_MAXPACKET.) > > > > Are you using any of the "stacked" network device drivers like > > lagg? I don't even know what the others all are? > > Maybe someone else can list them? > > I guess the most obvious are lagg and vlan (and probably carp on FreeBSD > 9.x or older). > > On request from Jack, we've eliminated lagg and vlan from the picture, > which gives us plain ixgbe interfaces with no stacked interfaces on top of > it. And we can still reproduce the problem. > > > Markus > > > > > > rick > >> > >> 10.0 Code: > >> > >> 780 if (len > tp->t_tsomax - hdrlen) { !! > >> 781 len = tp->t_tsomax - hdrlen; !! > >> 782 sendalot = 1; > >> 783 } > >> > >> > >> > >> > >> I've put debugging here, set the nic's max TSO as per Rick's patch ( > >> set to say 32k), and have seen that tp->t_tsomax == IP_MAXPACKET. > >> It's being set someplace else, and thus our attempts to set TSO on > >> the nic may be in vain. > >> > >> > >> It may have mattered more in 9.2, as I see the code doesn't use > >> tp->t_tsomax in some locations, and may actually default to what the > >> nic is set to. > >> > >> The NIC may still win, I didn't walk through the code to confirm, it > >> was enough to suggest to me that setting TSO wouldn't fix this > >> issue. > >> > >> > >> However, this is still a TSO related issue, it's just not one related > >> to the setting of TSO's max size. > >> > >> A 10.0-STABLE system with tso disabled on ix0 doesn't have a single > >> packet over IP_MAXPACKET in 1 hour of runtime. I'll let it go a bit > >> longer to increase confidence in this assertion, but I don't want to > >> waste time on this when I could be logging problem packets on a > >> system with TSO enabled. > >> > >> > >> Comments are very welcome.. > >> > >> > >> > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > -- Met vriendelijke groeten / With kind regards, Johan Kooijman T +31(0) 6 43 44 45 27 F +31(0) 162 82 00 01 E mail@johankooijman.com