From owner-freebsd-net@FreeBSD.ORG Wed Mar 26 02:27:34 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 72F25515; Wed, 26 Mar 2014 02:27:34 +0000 (UTC) Received: from mail-qc0-x22e.google.com (mail-qc0-x22e.google.com [IPv6:2607:f8b0:400d:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0ED4F8EE; Wed, 26 Mar 2014 02:27:33 +0000 (UTC) Received: by mail-qc0-f174.google.com with SMTP id c9so1833163qcz.19 for ; Tue, 25 Mar 2014 19:27:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Y418+WnnelDRCvvlIQ9egXwwWaO30n3S1FZFVBBiYQE=; b=O9sJRO0HnTJsKbnTAXTZIWRQ6mjMTQD2jDw8x4rv/mQyCaY7rU6QomGHbbuGuo8pl5 rpQ9LRnqlv2kkbaKQ5utig9kxdRCtpDeBDXWlXJPbcY7UbXyycL0K98b0uby4z3HJNB0 lzYp0BYC/IwG1cW7Ttx3uda0e+1isiTqnVdvV8Q+OWxuzNjx8i2eG9wU3JBIbOb+1KnU Dl6bmXDPANrRYTqlkvUBpcl09T38v37a7WotvekW7I/n1r6prCAhQLh+WsavUeMyXnA8 XpovWPTY/ClG1cVtiOEXjr7sADG+uRbVpJJNdbMy8V3oeDFSUFD7ffgXFFAp5MeYMfBa T8sA== MIME-Version: 1.0 X-Received: by 10.224.54.209 with SMTP id r17mr14802912qag.34.1395800853257; Tue, 25 Mar 2014 19:27:33 -0700 (PDT) Received: by 10.96.79.97 with HTTP; Tue, 25 Mar 2014 19:27:33 -0700 (PDT) In-Reply-To: <1985661701.540906.1395789389694.JavaMail.root@uoguelph.ca> References: <1985661701.540906.1395789389694.JavaMail.root@uoguelph.ca> Date: Tue, 25 Mar 2014 23:27:33 -0300 Message-ID: Subject: Re: 9.2 ixgbe tx queue hang From: Christopher Forgeron To: Rick Macklem Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: FreeBSD Net , Andre Oppermann , Garrett Wollman , Jack Vogel , Markus Gebert X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Mar 2014 02:27:34 -0000 That's interesting. I see here in the r251296 commit Andre says : Drivers can set ifp->if_hw_tsomax before calling ether_ifattach() to change the limit. I wonder if we add your same TSO patch to if_lagg.c before line 356's ether_ifattach() will fix it. Ultimately, it will need to load the if_hw_tsomax from the if below it - but then again, if the calculation for ixgbe is good enough for that driver, why wouldn't it be good enough for lagg? Unless people think I'm crazy, I'll compile that in at line 356 in if_lagg.c and give it a test run tomorrow. This may need to go into vlan and carp as well, I'm not sure yet. On Tue, Mar 25, 2014 at 8:16 PM, Rick Macklem wrote: > Christopher Forgeron wrote: > > Update: > > > > I'm changing my mind, and I believe Rick's TSO patch is fixing > > things > > (sorry). In looking at my notes, it's possible I had lagg on for > > those > > tests. lagg does seem to negate the TSO patch in my case. > > > Ok, that's useful information. It implies that r251296 doesn't quite > work and needs to be fixed for "stacked" network interface drivers > before it can be used. I've cc'd Andre who is the author of that > patch, in case he knows how to fix it. > > Thanks for checking this, rick > > > kernel.10stable_basicTSO_65535/ > > > > - IP_MAXPACKET = 65535; > > - manually forced (no if statement) ifp->if_hw_tsomax = IP_MAXPACKET > > - > > (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN); > > - Verified on boot via printf that ifp->if_hw_tsomax = 65517 > > - Boot in a NON LAGG environment. ix0 only. > > > > ixgbe's printf is showing packets up to 65530. Haven't run long > > enough yet > > to see if anything will go over 65535 > > > > I have this tcpdump running to check packet size. > > tcpdump -ennvvXS -i ix0 greater 65518 > > > > I do expect to get packets over 65518, but I was just curious to see > > if any > > of them would go over 65535. Time will tell. > > > > In a separate test, If I enable lagg, we have LOTS of oversized > > packet > > problems. It looks like tsomax is definitely not making it through in > > if_lagg.c - Any recommendations there? I will eventually need lagg, > > as I'm > > sure will others. > > > > With dtrace, it's showing t_tsomax >= 65518. Shouldn't that not be > > happening? > > > > > > dtrace -n 'fbt::tcp_output:entry / args[0]->t_tsomax != 0 && > > args[0]->t_tsomax >= 65518 / { printf("unexpected tp->t_tsomax: > > %i\n", > > args[0]->t_tsomax); stack(); }' > > > > > > 6 31403 tcp_output:entry unexpected tp->t_tsomax: > > 65535 > > > > kernel`tcp_do_segment+0x2c99 > > kernel`tcp_input+0x11a2 > > kernel`ip_input+0xa2 > > kernel`netisr_dispatch_src+0x5e > > kernel`ether_demux+0x12a > > kernel`ether_nh_input+0x35f > > kernel`netisr_dispatch_src+0x5e > > kernel`bce_intr+0x765 > > kernel`intr_event_execute_handlers+0xab > > kernel`ithread_loop+0x96 > > kernel`fork_exit+0x9a > > kernel`0xffffffff80c75b2e > > > > 3 31403 tcp_output:entry unexpected tp->t_tsomax: > > 65535 > > > > kernel`tcp_do_segment+0x2c99 > > kernel`tcp_input+0x11a2 > > kernel`ip_input+0xa2 > > kernel`netisr_dispatch_src+0x5e > > kernel`ether_demux+0x12a > > kernel`ether_nh_input+0x35f > > kernel`netisr_dispatch_src+0x5e > > kernel`bce_intr+0x765 > > kernel`intr_event_execute_handlers+0xab > > kernel`ithread_loop+0x96 > > kernel`fork_exit+0x9a > > kernel`0xffffffff80c75b2e > > > > 6 31403 tcp_output:entry unexpected tp->t_tsomax: > > 65535 > > > > kernel`tcp_do_segment+0x2c99 > > kernel`tcp_input+0x11a2 > > kernel`ip_input+0xa2 > > kernel`netisr_dispatch_src+0x5e > > kernel`ether_demux+0x12a > > kernel`ether_nh_input+0x35f > > kernel`netisr_dispatch_src+0x5e > > kernel`bce_intr+0x765 > > kernel`intr_event_execute_handlers+0xab > > kernel`ithread_loop+0x96 > > kernel`fork_exit+0x9a > > kernel`0xffffffff80c75b2e > > > > 1 31403 tcp_output:entry unexpected tp->t_tsomax: > > 65535 > > > > kernel`tcp_do_segment+0x2c99 > > kernel`tcp_input+0x11a2 > > kernel`ip_input+0xa2 > > kernel`netisr_dispatch_src+0x5e > > kernel`ether_demux+0x12a > > kernel`ether_nh_input+0x35f > > kernel`netisr_dispatch_src+0x5e > > kernel`bce_intr+0x765 > > kernel`intr_event_execute_handlers+0xab > > kernel`ithread_loop+0x96 > > kernel`fork_exit+0x9a > > kernel`0xffffffff80c75b2e > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to > > "freebsd-net-unsubscribe@freebsd.org" > > >