Date: Tue, 25 Mar 2014 22:04:55 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Markus Gebert <markus.gebert@hostpoint.ch> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Garrett Wollman <wollman@freebsd.org>, Jack Vogel <jfvogel@gmail.com>, Christopher Forgeron <csforgeron@gmail.com> Subject: Re: 9.2 ixgbe tx queue hang Message-ID: <1175880198.604234.1395799495187.JavaMail.root@uoguelph.ca> In-Reply-To: <1197F2E5-F20C-43E4-B8C8-8732F45457C2@hostpoint.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
Markus Gebert wrote:
>=20
> On 26.03.2014, at 00:06, Christopher Forgeron <csforgeron@gmail.com>
> wrote:
>=20
> > Update:
> >=20
> > I'm changing my mind, and I believe Rick's TSO patch is fixing
> > things
> > (sorry). In looking at my notes, it's possible I had lagg on for
> > those
> > tests. lagg does seem to negate the TSO patch in my case.
>=20
> I=E2=80=99m glad to hear you could check that scenario again. In the othe=
r
> email I just sent, I just asked you to redo this test. Now it makes
> perfect sense why you saw oversized packets despite Rick=E2=80=99s
> if_hw_tsomax patch.
>=20
>=20
> > kernel.10stable_basicTSO_65535/
> >=20
> > - IP_MAXPACKET =3D 65535;
> > - manually forced (no if statement) ifp->if_hw_tsomax =3D
> > IP_MAXPACKET -
> > (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN);
> > - Verified on boot via printf that ifp->if_hw_tsomax =3D 65517
>=20
> Is 65517 correct? With Ricks patch, I get this:
>=20
> dev.ix.0.hw_tsomax: 65518
>=20
> Also the dtrace command you used excludes 65518...
>=20
I am using 32 * MCLBYTES - (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN) which
is 65518. Although IP_MAXPACKET (maximum IP len, not including ethernet hea=
der)
is 65535 (largest # that fits in 16bits), the maximum data length
(including ethernet header) that will fit in 32 mbuf clusters is 65536.
(In practice 65517 or anything <=3D 65518 should fix the problem.)
rick
> > - Boot in a NON LAGG environment. ix0 only.
> >=20
> > ixgbe's printf is showing packets up to 65530. Haven't run long
> > enough yet
> > to see if anything will go over 65535
> >=20
With the ethernet header length, it can be <=3D 65536, because that
is 32 * MCLBYTES.
rick
> > I have this tcpdump running to check packet size.
> > tcpdump -ennvvXS -i ix0 greater 65518
> >=20
> > I do expect to get packets over 65518, but I was just curious to
> > see if any
> > of them would go over 65535. Time will tell.
> >=20
> > In a separate test, If I enable lagg, we have LOTS of oversized
> > packet
> > problems. It looks like tsomax is definitely not making it through
> > in
> > if_lagg.c - Any recommendations there? I will eventually need lagg,
> > as I'm
> > sure will others.
>=20
> I think somebody has to invent a way to propagate if_hw_maxtso to
> interfaces on top of each other.
>=20
>=20
> > With dtrace, it's showing t_tsomax >=3D 65518. Shouldn't that not be
> > happening?
>=20
> Looks like these all come from bce interfaces (bce_intr in the stack
> trace), which probably have another value for if_hw_tsomax.
>=20
>=20
> Markus
>=20
>=20
> > dtrace -n 'fbt::tcp_output:entry / args[0]->t_tsomax !=3D 0 &&
> > args[0]->t_tsomax >=3D 65518 / { printf("unexpected tp->t_tsomax:
> > %i\n",
> > args[0]->t_tsomax); stack(); }'
> >=20
> >=20
> > 6 31403 tcp_output:entry unexpected tp->t_tsomax:
> > 65535
> >=20
> > kernel`tcp_do_segment+0x2c99
> > kernel`tcp_input+0x11a2
> > kernel`ip_input+0xa2
> > kernel`netisr_dispatch_src+0x5e
> > kernel`ether_demux+0x12a
> > kernel`ether_nh_input+0x35f
> > kernel`netisr_dispatch_src+0x5e
> > kernel`bce_intr+0x765
> > kernel`intr_event_execute_handlers+0xab
> > kernel`ithread_loop+0x96
> > kernel`fork_exit+0x9a
> > kernel`0xffffffff80c75b2e
> >=20
> > 3 31403 tcp_output:entry unexpected tp->t_tsomax:
> > 65535
> >=20
> > kernel`tcp_do_segment+0x2c99
> > kernel`tcp_input+0x11a2
> > kernel`ip_input+0xa2
> > kernel`netisr_dispatch_src+0x5e
> > kernel`ether_demux+0x12a
> > kernel`ether_nh_input+0x35f
> > kernel`netisr_dispatch_src+0x5e
> > kernel`bce_intr+0x765
> > kernel`intr_event_execute_handlers+0xab
> > kernel`ithread_loop+0x96
> > kernel`fork_exit+0x9a
> > kernel`0xffffffff80c75b2e
> >=20
> > 6 31403 tcp_output:entry unexpected tp->t_tsomax:
> > 65535
> >=20
> > kernel`tcp_do_segment+0x2c99
> > kernel`tcp_input+0x11a2
> > kernel`ip_input+0xa2
> > kernel`netisr_dispatch_src+0x5e
> > kernel`ether_demux+0x12a
> > kernel`ether_nh_input+0x35f
> > kernel`netisr_dispatch_src+0x5e
> > kernel`bce_intr+0x765
> > kernel`intr_event_execute_handlers+0xab
> > kernel`ithread_loop+0x96
> > kernel`fork_exit+0x9a
> > kernel`0xffffffff80c75b2e
> >=20
> > 1 31403 tcp_output:entry unexpected tp->t_tsomax:
> > 65535
> >=20
> > kernel`tcp_do_segment+0x2c99
> > kernel`tcp_input+0x11a2
> > kernel`ip_input+0xa2
> > kernel`netisr_dispatch_src+0x5e
> > kernel`ether_demux+0x12a
> > kernel`ether_nh_input+0x35f
> > kernel`netisr_dispatch_src+0x5e
> > kernel`bce_intr+0x765
> > kernel`intr_event_execute_handlers+0xab
> > kernel`ithread_loop+0x96
> > kernel`fork_exit+0x9a
> > kernel`0xffffffff80c75b2e
> > _______________________________________________
> > freebsd-net@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to
> > "freebsd-net-unsubscribe@freebsd.org"
> >=20
>=20
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to
> "freebsd-net-unsubscribe@freebsd.org"
>=20
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1175880198.604234.1395799495187.JavaMail.root>