From owner-freebsd-net@FreeBSD.ORG Wed Mar 26 02:05:03 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 94073A33; Wed, 26 Mar 2014 02:05:03 +0000 (UTC) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 18C7F6CF; Wed, 26 Mar 2014 02:05:02 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqUEALU0MlODaFve/2dsb2JhbABZg0FXgwu4OoZkUYEwdIIlAQEBAwEBAQEgBCcgCxsYAgINGQIjBgEJJgYIBwQBHASHRAMJCA2tNpp1DYcdF4EpiymBOhACARsBMweCb4FJBJV2aoMfizaFSoNKITGBPQ X-IronPort-AV: E=Sophos;i="4.97,732,1389762000"; d="scan'208";a="109152802" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 25 Mar 2014 22:04:55 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 328E6B4052; Tue, 25 Mar 2014 22:04:55 -0400 (EDT) Date: Tue, 25 Mar 2014 22:04:55 -0400 (EDT) From: Rick Macklem To: Markus Gebert Message-ID: <1175880198.604234.1395799495187.JavaMail.root@uoguelph.ca> In-Reply-To: <1197F2E5-F20C-43E4-B8C8-8732F45457C2@hostpoint.ch> Subject: Re: 9.2 ixgbe tx queue hang MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.201] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: FreeBSD Net , Garrett Wollman , Jack Vogel , Christopher Forgeron X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Mar 2014 02:05:03 -0000 Markus Gebert wrote: >=20 > On 26.03.2014, at 00:06, Christopher Forgeron > wrote: >=20 > > Update: > >=20 > > I'm changing my mind, and I believe Rick's TSO patch is fixing > > things > > (sorry). In looking at my notes, it's possible I had lagg on for > > those > > tests. lagg does seem to negate the TSO patch in my case. >=20 > I=E2=80=99m glad to hear you could check that scenario again. In the othe= r > email I just sent, I just asked you to redo this test. Now it makes > perfect sense why you saw oversized packets despite Rick=E2=80=99s > if_hw_tsomax patch. >=20 >=20 > > kernel.10stable_basicTSO_65535/ > >=20 > > - IP_MAXPACKET =3D 65535; > > - manually forced (no if statement) ifp->if_hw_tsomax =3D > > IP_MAXPACKET - > > (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN); > > - Verified on boot via printf that ifp->if_hw_tsomax =3D 65517 >=20 > Is 65517 correct? With Ricks patch, I get this: >=20 > dev.ix.0.hw_tsomax: 65518 >=20 > Also the dtrace command you used excludes 65518... >=20 I am using 32 * MCLBYTES - (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN) which is 65518. Although IP_MAXPACKET (maximum IP len, not including ethernet hea= der) is 65535 (largest # that fits in 16bits), the maximum data length (including ethernet header) that will fit in 32 mbuf clusters is 65536. (In practice 65517 or anything <=3D 65518 should fix the problem.) rick > > - Boot in a NON LAGG environment. ix0 only. > >=20 > > ixgbe's printf is showing packets up to 65530. Haven't run long > > enough yet > > to see if anything will go over 65535 > >=20 With the ethernet header length, it can be <=3D 65536, because that is 32 * MCLBYTES. rick > > I have this tcpdump running to check packet size. > > tcpdump -ennvvXS -i ix0 greater 65518 > >=20 > > I do expect to get packets over 65518, but I was just curious to > > see if any > > of them would go over 65535. Time will tell. > >=20 > > In a separate test, If I enable lagg, we have LOTS of oversized > > packet > > problems. It looks like tsomax is definitely not making it through > > in > > if_lagg.c - Any recommendations there? I will eventually need lagg, > > as I'm > > sure will others. >=20 > I think somebody has to invent a way to propagate if_hw_maxtso to > interfaces on top of each other. >=20 >=20 > > With dtrace, it's showing t_tsomax >=3D 65518. Shouldn't that not be > > happening? >=20 > Looks like these all come from bce interfaces (bce_intr in the stack > trace), which probably have another value for if_hw_tsomax. >=20 >=20 > Markus >=20 >=20 > > dtrace -n 'fbt::tcp_output:entry / args[0]->t_tsomax !=3D 0 && > > args[0]->t_tsomax >=3D 65518 / { printf("unexpected tp->t_tsomax: > > %i\n", > > args[0]->t_tsomax); stack(); }' > >=20 > >=20 > > 6 31403 tcp_output:entry unexpected tp->t_tsomax: > > 65535 > >=20 > > kernel`tcp_do_segment+0x2c99 > > kernel`tcp_input+0x11a2 > > kernel`ip_input+0xa2 > > kernel`netisr_dispatch_src+0x5e > > kernel`ether_demux+0x12a > > kernel`ether_nh_input+0x35f > > kernel`netisr_dispatch_src+0x5e > > kernel`bce_intr+0x765 > > kernel`intr_event_execute_handlers+0xab > > kernel`ithread_loop+0x96 > > kernel`fork_exit+0x9a > > kernel`0xffffffff80c75b2e > >=20 > > 3 31403 tcp_output:entry unexpected tp->t_tsomax: > > 65535 > >=20 > > kernel`tcp_do_segment+0x2c99 > > kernel`tcp_input+0x11a2 > > kernel`ip_input+0xa2 > > kernel`netisr_dispatch_src+0x5e > > kernel`ether_demux+0x12a > > kernel`ether_nh_input+0x35f > > kernel`netisr_dispatch_src+0x5e > > kernel`bce_intr+0x765 > > kernel`intr_event_execute_handlers+0xab > > kernel`ithread_loop+0x96 > > kernel`fork_exit+0x9a > > kernel`0xffffffff80c75b2e > >=20 > > 6 31403 tcp_output:entry unexpected tp->t_tsomax: > > 65535 > >=20 > > kernel`tcp_do_segment+0x2c99 > > kernel`tcp_input+0x11a2 > > kernel`ip_input+0xa2 > > kernel`netisr_dispatch_src+0x5e > > kernel`ether_demux+0x12a > > kernel`ether_nh_input+0x35f > > kernel`netisr_dispatch_src+0x5e > > kernel`bce_intr+0x765 > > kernel`intr_event_execute_handlers+0xab > > kernel`ithread_loop+0x96 > > kernel`fork_exit+0x9a > > kernel`0xffffffff80c75b2e > >=20 > > 1 31403 tcp_output:entry unexpected tp->t_tsomax: > > 65535 > >=20 > > kernel`tcp_do_segment+0x2c99 > > kernel`tcp_input+0x11a2 > > kernel`ip_input+0xa2 > > kernel`netisr_dispatch_src+0x5e > > kernel`ether_demux+0x12a > > kernel`ether_nh_input+0x35f > > kernel`netisr_dispatch_src+0x5e > > kernel`bce_intr+0x765 > > kernel`intr_event_execute_handlers+0xab > > kernel`ithread_loop+0x96 > > kernel`fork_exit+0x9a > > kernel`0xffffffff80c75b2e > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to > > "freebsd-net-unsubscribe@freebsd.org" > >=20 >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >=20