From owner-freebsd-stable@FreeBSD.ORG Mon Jun 23 16:42:44 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 499151065676; Mon, 23 Jun 2008 16:42:44 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id D4B5C8FC23; Mon, 23 Jun 2008 16:42:43 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.1/8.14.1) with ESMTP id m5NGghCO089159; Mon, 23 Jun 2008 09:42:43 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.1/8.13.4/Submit) id m5NGggV8089156; Mon, 23 Jun 2008 09:42:42 -0700 (PDT) Date: Mon, 23 Jun 2008 09:42:42 -0700 (PDT) From: Matthew Dillon Message-Id: <200806231642.m5NGggV8089156@apollo.backplane.com> To: =?ISO-8859-15?Q?Stefan_E=DFer?= References: <0222EAC1-A278-41D2-9566-C9CF19811068@optusnet.com.au> <200806230827.m5N8RBlW085475@apollo.backplane.com> <485F7576.5070104@freebsd.org> Cc: Jerahmy Pocott , freebsd-stable@freebsd.org Subject: Re: Sysctl knob(s) to set TCP 'nagle' time-out? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jun 2008 16:42:44 -0000 :One possibility I see is a statistic about DelACKs per TCP connection, :counting those that were rightfully delayed (with hindsight). I.e., :if an ACK is delayed, but there was no chance to piggy-back it or to :combine it with another ACK, it could have been sent without delay. :Only those delayed ACKs that reduce load are "good", all others cause :additional state to be maintained and may increase latencies for no :good reason. : :... :consideration. And to me, automatic setting of TCP_NODELAY seems :more useful than automatic clearing (after delayed ACKs had been :found to be of no use for a window of say 8 or 16 ACKs). : :The implementation would be quite simple: Whenever a delayed ACK :is sent, check whether it is sent on its own (bad) or whether it :could be piggy-backed (good). If, say, 7 of 8 delayed ACKs had to :be sent as ACK-only packets, anyway, set TCP_NODELAY and do not :bother to keep on deciding whether delayed ACKs had become useful :in a different phase of the communication. If you want to be able :to automatically disable TCP_NODELAY, then just set a time-stamp :... :Regards, STefan That's an interesting approach. I think it would catch some of the cases, but not enough of them. If the round-trip in the server-relaying case is less then the delayed-ack, the acks will still wind up piggy-backed on return traffic but the latency will also still remain horrible. It should be noted that Nagle can cause high latencies even when delayed acks are turned off. Nagle's delay is not timed... in its simplest description it prevents packets from being transmitted for new data coming from userland if the data already in the sockbuf (and presumably already transmitted) has not yet been acknowledged. For interactive traffic this means that Nagle is putting the screws on the packet stream even if the acks aren't delayed, simply from the ack latency. With delayed acks turned off the latency is lower, but not 0, so interactive traffic is still being held up by Nagle. The effect is noticeable even on a LAN. Jerahmy brought up Samba... that is an excellent example. NFS-over-TCP would be another good example. Any protocol which multiplexes multiple commands from different sources over the same connection gets really messed up (slowed down) by Nagle. On the flip side, Nagle can't just be turned off by default because it would cause streaming connections from user programs which do tiny writes to generate a lot of unnecessarily tiny packets. This can become apparent when using SSH over a slow link. Numerous programs run from a shell generate fairly ineffcient packets which could have easily been batched when operating over SSH. The result can be sludgy performance for output which ought be batched up by TCP but isn't because SSH turns off Nagle unconditionally. -Matt Matthew Dillon