Date: Wed, 14 Aug 2013 01:37:14 +0800 From: Julian Elischer <julian@freebsd.org> To: FreeBSD Net <net@FreeBSD.org> Subject: Re: TSO and FreeBSD vs Linux Message-ID: <520A6ECA.5090307@freebsd.org> In-Reply-To: <520A6D07.5080106@freebsd.org> References: <520A6D07.5080106@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8/14/13 1:29 AM, Julian Elischer wrote: > I have been tracking down a performance embarrassment on AMAZON EC2 > and have found it I think. > Our OS cousins over at Linux land have implemented some interesting > behaviour when TSO is in use. > > They seem to aggregate ACKS when there is a lot of traffic so that > they can create the > largest possible TSO packet. We on the other hand respond to each > and every returning ACK, as it arrives and thus generally fall into > the behaviour of sending a bunch of small packets, the size of each > ack. > > for two examples look at: > > > http://www.freebsd.org/~julian/LvsF-tcp-start.tiff > and > http://www.freebsd.org/~julian/LvsF-tcp.tiff some people have troubel with tiff, so here they are a jpeg. http://www.freebsd.org/~julian/LvsF-tcp-start.jpg and http://www.freebsd.org/~julian/LvsF-tcp.jpg > > in each case, we can see FreeBSD on the left and Linux on the right. > > The first case shows the case as the sessions start, and the second > case shows > some distance later (when the sequence numbers wrap around.. no > particular > reason to use that, it was just fun to see). > In both cases you can see that each Linux packet (white)(once they > have got > going) is responding to multiple bumps in the send window sequence > number (green and yellow lines) (representing the arrival of several > ACKs) > while FreeBSD produces a whole bunch of smaller packets, slavishly > following > exactly the size of each incoming ack.. This gives us quite a > performance debt. > Notice that this behaviour in Linux seems to be modal.. it seems to > 'switch on' a little bit > into the 'starting' trace. > > In addition, you can see also that Linux gets going faster even in > the beginning where > TSO isn't in play, by sending a lot more packets up-front. (of > course the wisdom of this > can be argued). > > Has anyone done any work on aggregating ACKs, or delaying responding > to them? > > Julian > (Who's suspecting he's about to find out more about TSO and the send > path, than he ever wanted to). > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?520A6ECA.5090307>