Date: Wed, 26 Mar 2014 19:20:25 -0300 From: Christopher Forgeron <csforgeron@gmail.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Andre Oppermann <andre@freebsd.org>, Garrett Wollman <wollman@freebsd.org>, Jack Vogel <jfvogel@gmail.com>, Markus Gebert <markus.gebert@hostpoint.ch> Subject: Re: 9.2 ixgbe tx queue hang Message-ID: <CAB2_NwBfZ_QJL1KByYjBk6jwx-zjWCSE9JYPQBhPq8Aw5__Gyg@mail.gmail.com> In-Reply-To: <CAB2_NwBZjVYVoPMh4HgNtGoULBuy0gA2=K7K=Bn=zCkjAu=Paw@mail.gmail.com> References: <CAB2_NwA2zp9VaBm4ZRR1udghbSpSDs_x8bK6CPHCaBuiBKD=KA@mail.gmail.com> <1985661701.540906.1395789389694.JavaMail.root@uoguelph.ca> <CAB2_NwBZjVYVoPMh4HgNtGoULBuy0gA2=K7K=Bn=zCkjAu=Paw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Confirmed that adding this to sys/net/if.c fixes the issue for lagg as well
as ixgbe.
660: if (ifp->if_hw_tsomax == 0)
661: ifp->if_hw_tsomax = IP_MAXPACKET - (ETHER_HDR_LEN +
ETHER_VLAN_ENCAP_LEN);
Code before (looks to be introduced in 9.2, r251296 as Rick mentions above)
just sets ifp->if_hw_tsomax = IP_MAXPACKET , so I really don't see much
downside in setting it to ~65518 if it enhances compatibility.
This should also fix it for carp, vlan, and others.
I'm going to do a few more tests, but in the meantime lets discuss the cons
of doing this.
On Tue, Mar 25, 2014 at 11:27 PM, Christopher Forgeron <csforgeron@gmail.com
> wrote:
> That's interesting. I see here in the r251296 commit Andre says :
>
> Drivers can set ifp->if_hw_tsomax before calling ether_ifattach() to
> change the limit.
>
> I wonder if we add your same TSO patch to if_lagg.c before line 356's
> ether_ifattach() will fix it.
>
> Ultimately, it will need to load the if_hw_tsomax from the if below it -
> but then again, if the calculation for ixgbe is good enough for that
> driver, why wouldn't it be good enough for lagg?
>
> Unless people think I'm crazy, I'll compile that in at line 356 in
> if_lagg.c and give it a test run tomorrow.
>
> This may need to go into vlan and carp as well, I'm not sure yet.
>
>
> On Tue, Mar 25, 2014 at 8:16 PM, Rick Macklem <rmacklem@uoguelph.ca>wrote:
>
>> Christopher Forgeron wrote:
>> > Update:
>> >
>> > I'm changing my mind, and I believe Rick's TSO patch is fixing
>> > things
>> > (sorry). In looking at my notes, it's possible I had lagg on for
>> > those
>> > tests. lagg does seem to negate the TSO patch in my case.
>> >
>> Ok, that's useful information. It implies that r251296 doesn't quite
>> work and needs to be fixed for "stacked" network interface drivers
>> before it can be used. I've cc'd Andre who is the author of that
>> patch, in case he knows how to fix it.
>>
>> Thanks for checking this, rick
>>
>> > kernel.10stable_basicTSO_65535/
>> >
>> > - IP_MAXPACKET = 65535;
>> > - manually forced (no if statement) ifp->if_hw_tsomax = IP_MAXPACKET
>> > -
>> > (ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN);
>> > - Verified on boot via printf that ifp->if_hw_tsomax = 65517
>> > - Boot in a NON LAGG environment. ix0 only.
>> >
>> > ixgbe's printf is showing packets up to 65530. Haven't run long
>> > enough yet
>> > to see if anything will go over 65535
>> >
>> > I have this tcpdump running to check packet size.
>> > tcpdump -ennvvXS -i ix0 greater 65518
>> >
>> > I do expect to get packets over 65518, but I was just curious to see
>> > if any
>> > of them would go over 65535. Time will tell.
>> >
>> > In a separate test, If I enable lagg, we have LOTS of oversized
>> > packet
>> > problems. It looks like tsomax is definitely not making it through in
>> > if_lagg.c - Any recommendations there? I will eventually need lagg,
>> > as I'm
>> > sure will others.
>> >
>> > With dtrace, it's showing t_tsomax >= 65518. Shouldn't that not be
>> > happening?
>> >
>> >
>> > dtrace -n 'fbt::tcp_output:entry / args[0]->t_tsomax != 0 &&
>> > args[0]->t_tsomax >= 65518 / { printf("unexpected tp->t_tsomax:
>> > %i\n",
>> > args[0]->t_tsomax); stack(); }'
>> >
>> >
>> > 6 31403 tcp_output:entry unexpected tp->t_tsomax:
>> > 65535
>> >
>> > kernel`tcp_do_segment+0x2c99
>> > kernel`tcp_input+0x11a2
>> > kernel`ip_input+0xa2
>> > kernel`netisr_dispatch_src+0x5e
>> > kernel`ether_demux+0x12a
>> > kernel`ether_nh_input+0x35f
>> > kernel`netisr_dispatch_src+0x5e
>> > kernel`bce_intr+0x765
>> > kernel`intr_event_execute_handlers+0xab
>> > kernel`ithread_loop+0x96
>> > kernel`fork_exit+0x9a
>> > kernel`0xffffffff80c75b2e
>> >
>> > 3 31403 tcp_output:entry unexpected tp->t_tsomax:
>> > 65535
>> >
>> > kernel`tcp_do_segment+0x2c99
>> > kernel`tcp_input+0x11a2
>> > kernel`ip_input+0xa2
>> > kernel`netisr_dispatch_src+0x5e
>> > kernel`ether_demux+0x12a
>> > kernel`ether_nh_input+0x35f
>> > kernel`netisr_dispatch_src+0x5e
>> > kernel`bce_intr+0x765
>> > kernel`intr_event_execute_handlers+0xab
>> > kernel`ithread_loop+0x96
>> > kernel`fork_exit+0x9a
>> > kernel`0xffffffff80c75b2e
>> >
>> > 6 31403 tcp_output:entry unexpected tp->t_tsomax:
>> > 65535
>> >
>> > kernel`tcp_do_segment+0x2c99
>> > kernel`tcp_input+0x11a2
>> > kernel`ip_input+0xa2
>> > kernel`netisr_dispatch_src+0x5e
>> > kernel`ether_demux+0x12a
>> > kernel`ether_nh_input+0x35f
>> > kernel`netisr_dispatch_src+0x5e
>> > kernel`bce_intr+0x765
>> > kernel`intr_event_execute_handlers+0xab
>> > kernel`ithread_loop+0x96
>> > kernel`fork_exit+0x9a
>> > kernel`0xffffffff80c75b2e
>> >
>> > 1 31403 tcp_output:entry unexpected tp->t_tsomax:
>> > 65535
>> >
>> > kernel`tcp_do_segment+0x2c99
>> > kernel`tcp_input+0x11a2
>> > kernel`ip_input+0xa2
>> > kernel`netisr_dispatch_src+0x5e
>> > kernel`ether_demux+0x12a
>> > kernel`ether_nh_input+0x35f
>> > kernel`netisr_dispatch_src+0x5e
>> > kernel`bce_intr+0x765
>> > kernel`intr_event_execute_handlers+0xab
>> > kernel`ithread_loop+0x96
>> > kernel`fork_exit+0x9a
>> > kernel`0xffffffff80c75b2e
>> > _______________________________________________
>> > freebsd-net@freebsd.org mailing list
>> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> > To unsubscribe, send any mail to
>> > "freebsd-net-unsubscribe@freebsd.org"
>> >
>>
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAB2_NwBfZ_QJL1KByYjBk6jwx-zjWCSE9JYPQBhPq8Aw5__Gyg>