Date: Tue, 15 Sep 2015 10:13:02 -0700 From: Adrian Chadd <adrian.chadd@gmail.com> To: Hans Petter Selasky <hps@selasky.org> Cc: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= <royger@freebsd.org>, "src-committers@freebsd.org" <src-committers@freebsd.org>, "svn-src-all@freebsd.org" <svn-src-all@freebsd.org>, "svn-src-head@freebsd.org" <svn-src-head@freebsd.org> Subject: Re: svn commit: r271946 - in head/sys: dev/oce dev/vmware/vmxnet3 dev/xen/netfront kern net netinet ofed/drivers/net/mlx4 sys Message-ID: <CAJ-VmonsGdG9eddWLLPTG3wwW3FDHDkH-F2xD%2B2nt=yvwBRS-A@mail.gmail.com> In-Reply-To: <55F7FE1A.3050500@selasky.org> References: <201409220827.s8M8RRHB031526@svn.freebsd.org> <55F69093.5050807@FreeBSD.org> <55F6935C.9000000@selasky.org> <55F6A694.7020404@FreeBSD.org> <55F6A914.6050109@selasky.org> <55F6ED8F.5030402@FreeBSD.org> <CAJ-Vmo=vq6NbUNecTvkzDLQW3rQzpPToFGbR9TxdcyO_BL3s_g@mail.gmail.com> <55F7FE1A.3050500@selasky.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 15 September 2015 at 04:16, Hans Petter Selasky <hps@selasky.org> wrote: > On 09/14/15 20:35, Adrian Chadd wrote: >> >> Hi, >> >> So what's the actual behaviour of the new tso logic before and after >> the above change in tsomax? > > > Hi, > > The behaviour is the same, only the limits have changed a bit. > >> like, what are the actual packet sizes >> >> being sent up to the hardware? > > > It is not about the packet sizes, it is about the number of packets we > transmit per TSO block. > >> Is TSO or the TCP stack so fragile that >> a slight change in how packets are broken up results in ridiculously >> less throughput? It's only a few bytes. > > > Network adapters which support TSO typically has a hard limit on the number > of mbufs it can load. When we exceed this limit, m_defrag() or packet drop > is next. It is the responsibility of the TCP stack to generated mbuf chains > which are within the network adapter given limits. Previously only the > length was accounted for. Now we also account for the number of segments, > because there are many ways a 64K bytes long mbuf chain can be generated. I know all of this. What I'm asking is different - what about the change(s) that broke/fixed the Xen network performance actually caused the change in behaviour? I know things are sensitive to how the mbufs are broken up - I'd like to see exactly this. Whilst futzing around at Netflix on TCP behaviour, I found with everything lined up correctly I'd get 40gbit TCP over a lot of sockets on sandy bridge hardware - but only as long as the TCP send buffer size resulted in nicely 4k segments. Some (larger) send buffer sizes resulted in the stack settling on non-4k segments for whatever reason, which drastically changed TSO performance. So, subtle changes had much bigger effects. I'm asking what specifically is going on here. +1 on the "going to make things more correct" path, but -2 on the "subtle changes and big unintended, un-described effects" results :( -adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmonsGdG9eddWLLPTG3wwW3FDHDkH-F2xD%2B2nt=yvwBRS-A>