Date: Mon, 14 Jul 2014 11:34:53 -0400 From: John Jasem <jjasen@gmail.com> To: "Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net>, =?windows-1252?Q?Olivier_Cochard-Labb=E9?= <olivier@cochard.me> Cc: FreeBSD Net <freebsd-net@freebsd.org>, Navdeep Parhar <nparhar@gmail.com> Subject: Re: tuning routing using cxgbe and T580-CR cards? Message-ID: <53C3F89D.1020503@gmail.com> In-Reply-To: <68918930-4FEC-413B-AB5E-B544C936D54F@lists.zabbadoz.net> References: <53C01EB5.6090701@gmail.com> <01AABF44-4801-45B5-9509-1CA7BAA3CB30@lists.zabbadoz.net> <CA%2Bq%2BTcpv_UqWCT1z9yimqUYw_TqmQad=8Mirh8VWexMdQ4JGWA@mail.gmail.com> <68918930-4FEC-413B-AB5E-B544C936D54F@lists.zabbadoz.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Dropping lro on the interfaces decreased interrupt usage on the CPUs, as measured by top -CHIPSu, by 15-20%, at least from eyeballing it. It did not otherwise have an effect on packet rates. Thanks! On 07/12/2014 08:33 AM, Bjoern A. Zeeb wrote: > On 12 Jul 2014, at 12:17 , Olivier Cochard-Labbé <olivier@cochard.me> wrote: > >> On Fri, Jul 11, 2014 at 8:03 PM, Bjoern A. Zeeb < >> bzeeb-lists@lists.zabbadoz.net> wrote: >> >>> If you are primarily forwarding packets (you say "routing" multiple times) >>> the first thing you should do is turn off LRO and TSO on all ports. >>> >> Hi Bjoern, >> >> I was not aware of disabling LRO+TSO for forwarding packet. >> If I read correctly the wikipedia page of LRO[1]: Disabling LRO is not a >> concern of performance but only of not breaking the end-to-end principle, >> right ? >> But regarding TSO[2]: It should improve performance only between the TCP >> and IP layer. But paquet forwarded didn't have to cross TCP<->IP layer, >> then disabling TSO should not impact performance, right ? > For forwarding it means that you are re-assembling a packet on receive, buffering multiple, etc, then hand them up the stack, only to find that you are sending it out again, and thus you break them into multiple packets again. In other words: you do a lot more work and add latency than you need/want. > > I seem to remember that we added the knob to automatically disable our soft-LRO when forwarding is turned on (but I haven’t checked if I really did). If we did, at least for soft-LRO you won’t notice a difference indeed. > > >> - Multi-flows (different UDP ports) of small packet (60B) at about 10Mpps >> … >> No difference proven at 95.0% confidence >> >> => There is not difference: Then I can disable LRO for respecting the >> end-to-end principle. But why disabling TSO ? > Try TCP flows. > > — > Bjoern A. Zeeb "Come on. Learn, goddamn it.", WarGames, 1983 >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53C3F89D.1020503>