From owner-freebsd-net@FreeBSD.ORG Wed Mar 26 22:20:27 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 077B05D6; Wed, 26 Mar 2014 22:20:27 +0000 (UTC) Received: from mail-qc0-x22e.google.com (mail-qc0-x22e.google.com [IPv6:2607:f8b0:400d:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 958B9379; Wed, 26 Mar 2014 22:20:26 +0000 (UTC) Received: by mail-qc0-f174.google.com with SMTP id c9so3364779qcz.5 for ; Wed, 26 Mar 2014 15:20:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=96B24yRasPJDl7ovY7BkgeZLK2eiSfKtSoQLKJck3O0=; b=dJ3ZMoSlt1gWMWk2nYv7WRt06dDCxS2kqrt25DiN/bdw0dAcms47LZx1QIQSmxfMSa Wm3N3FMSqC/ytA7FqG27mZprfobByqDvBGgVLJ+sBi5j7apmgLp9naiSGrcP5g2XD9YE 1CBCKdO9eMzeptna9/97u7CfxpL32RbbAov5TIgXFblRRdJplO1gPu3Z9JcK1/fLNTJv nwB3SmngM4az/w5p2G8i3QOm8c88WldvXnSQp2Uk/pm/D1HDQRVAc83zVvocltQyQpna c672Y4zpRa3FPJgE1Z72lhw4RE4FZPQG3PCiSyKybrN6WgoZGtgUdfPeia1X/EyqLN/e iVwA== MIME-Version: 1.0 X-Received: by 10.224.22.147 with SMTP id n19mr5619227qab.93.1395872425576; Wed, 26 Mar 2014 15:20:25 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Wed, 26 Mar 2014 15:20:25 -0700 (PDT) In-Reply-To: References: <1985661701.540906.1395789389694.JavaMail.root@uoguelph.ca> Date: Wed, 26 Mar 2014 19:20:25 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Andre Oppermann , Garrett Wollman , Jack Vogel , Markus Gebert X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Mar 2014 22:20:27 -0000 Confirmed that adding this to sys/net/if.c fixes the issue for lagg as well as ixgbe. 660: if (ifp->if_hw_tsomax == 0) 661: ifp->if_hw_tsomax = IP_MAXPACKET - (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN); Code before (looks to be introduced in 9.2, r251296 as Rick mentions above) just sets ifp->if_hw_tsomax = IP_MAXPACKET , so I really don't see much downside in setting it to ~65518 if it enhances compatibility. This should also fix it for carp, vlan, and others. I'm going to do a few more tests, but in the meantime lets discuss the cons of doing this. On Tue, Mar 25, 2014 at 11:27 PM, Christopher Forgeron wrote: > That's interesting. I see here in the r251296 commit Andre says : > > Drivers can set ifp->if_hw_tsomax before calling ether_ifattach() to > change the limit. > > I wonder if we add your same TSO patch to if_lagg.c before line 356's > ether_ifattach() will fix it. > > Ultimately, it will need to load the if_hw_tsomax from the if below it - > but then again, if the calculation for ixgbe is good enough for that > driver, why wouldn't it be good enough for lagg? > > Unless people think I'm crazy, I'll compile that in at line 356 in > if_lagg.c and give it a test run tomorrow. > > This may need to go into vlan and carp as well, I'm not sure yet. > > > On Tue, Mar 25, 2014 at 8:16 PM, Rick Macklem wrote: > >> Christopher Forgeron wrote: >> > Update: >> > >> > I'm changing my mind, and I believe Rick's TSO patch is fixing >> > things >> > (sorry). In looking at my notes, it's possible I had lagg on for >> > those >> > tests. lagg does seem to negate the TSO patch in my case. >> > >> Ok, that's useful information. It implies that r251296 doesn't quite >> work and needs to be fixed for "stacked" network interface drivers >> before it can be used. I've cc'd Andre who is the author of that >> patch, in case he knows how to fix it. >> >> Thanks for checking this, rick >> >> > kernel.10stable_basicTSO_65535/ >> > >> > - IP_MAXPACKET = 65535; >> > - manually forced (no if statement) ifp->if_hw_tsomax = IP_MAXPACKET >> > - >> > (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN); >> > - Verified on boot via printf that ifp->if_hw_tsomax = 65517 >> > - Boot in a NON LAGG environment. ix0 only. >> > >> > ixgbe's printf is showing packets up to 65530. Haven't run long >> > enough yet >> > to see if anything will go over 65535 >> > >> > I have this tcpdump running to check packet size. >> > tcpdump -ennvvXS -i ix0 greater 65518 >> > >> > I do expect to get packets over 65518, but I was just curious to see >> > if any >> > of them would go over 65535. Time will tell. >> > >> > In a separate test, If I enable lagg, we have LOTS of oversized >> > packet >> > problems. It looks like tsomax is definitely not making it through in >> > if_lagg.c - Any recommendations there? I will eventually need lagg, >> > as I'm >> > sure will others. >> > >> > With dtrace, it's showing t_tsomax >= 65518. Shouldn't that not be >> > happening? >> > >> > >> > dtrace -n 'fbt::tcp_output:entry / args[0]->t_tsomax != 0 && >> > args[0]->t_tsomax >= 65518 / { printf("unexpected tp->t_tsomax: >> > %i\n", >> > args[0]->t_tsomax); stack(); }' >> > >> > >> > 6 31403 tcp_output:entry unexpected tp->t_tsomax: >> > 65535 >> > >> > kernel`tcp_do_segment+0x2c99 >> > kernel`tcp_input+0x11a2 >> > kernel`ip_input+0xa2 >> > kernel`netisr_dispatch_src+0x5e >> > kernel`ether_demux+0x12a >> > kernel`ether_nh_input+0x35f >> > kernel`netisr_dispatch_src+0x5e >> > kernel`bce_intr+0x765 >> > kernel`intr_event_execute_handlers+0xab >> > kernel`ithread_loop+0x96 >> > kernel`fork_exit+0x9a >> > kernel`0xffffffff80c75b2e >> > >> > 3 31403 tcp_output:entry unexpected tp->t_tsomax: >> > 65535 >> > >> > kernel`tcp_do_segment+0x2c99 >> > kernel`tcp_input+0x11a2 >> > kernel`ip_input+0xa2 >> > kernel`netisr_dispatch_src+0x5e >> > kernel`ether_demux+0x12a >> > kernel`ether_nh_input+0x35f >> > kernel`netisr_dispatch_src+0x5e >> > kernel`bce_intr+0x765 >> > kernel`intr_event_execute_handlers+0xab >> > kernel`ithread_loop+0x96 >> > kernel`fork_exit+0x9a >> > kernel`0xffffffff80c75b2e >> > >> > 6 31403 tcp_output:entry unexpected tp->t_tsomax: >> > 65535 >> > >> > kernel`tcp_do_segment+0x2c99 >> > kernel`tcp_input+0x11a2 >> > kernel`ip_input+0xa2 >> > kernel`netisr_dispatch_src+0x5e >> > kernel`ether_demux+0x12a >> > kernel`ether_nh_input+0x35f >> > kernel`netisr_dispatch_src+0x5e >> > kernel`bce_intr+0x765 >> > kernel`intr_event_execute_handlers+0xab >> > kernel`ithread_loop+0x96 >> > kernel`fork_exit+0x9a >> > kernel`0xffffffff80c75b2e >> > >> > 1 31403 tcp_output:entry unexpected tp->t_tsomax: >> > 65535 >> > >> > kernel`tcp_do_segment+0x2c99 >> > kernel`tcp_input+0x11a2 >> > kernel`ip_input+0xa2 >> > kernel`netisr_dispatch_src+0x5e >> > kernel`ether_demux+0x12a >> > kernel`ether_nh_input+0x35f >> > kernel`netisr_dispatch_src+0x5e >> > kernel`bce_intr+0x765 >> > kernel`intr_event_execute_handlers+0xab >> > kernel`ithread_loop+0x96 >> > kernel`fork_exit+0x9a >> > kernel`0xffffffff80c75b2e >> > _______________________________________________ >> > freebsd-net@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-net >> > To unsubscribe, send any mail to >> > "freebsd-net-unsubscribe@freebsd.org" >> > >> > >