From owner-freebsd-stable@FreeBSD.ORG Mon Jun 23 08:27:12 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4FDB2106567D for ; Mon, 23 Jun 2008 08:27:12 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.freebsd.org (Postfix) with ESMTP id 1B4928FC22 for ; Mon, 23 Jun 2008 08:27:11 +0000 (UTC) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.14.1/8.14.1) with ESMTP id m5N8RBEO085476; Mon, 23 Jun 2008 01:27:11 -0700 (PDT) Received: (from dillon@localhost) by apollo.backplane.com (8.14.1/8.13.4/Submit) id m5N8RBlW085475; Mon, 23 Jun 2008 01:27:11 -0700 (PDT) Date: Mon, 23 Jun 2008 01:27:11 -0700 (PDT) From: Matthew Dillon Message-Id: <200806230827.m5N8RBlW085475@apollo.backplane.com> To: Jerahmy Pocott References: <0222EAC1-A278-41D2-9566-C9CF19811068@optusnet.com.au> Cc: freebsd-stable@freebsd.org Subject: Re: Sysctl knob(s) to set TCP 'nagle' time-out? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Jun 2008 08:27:12 -0000 :Hi, : :I'm wondering if anything exists to set this.. When you create an INET :socket :without the 'TCP_NODELAY' flag the network layer does 'naggling' on your :transmitted data. Sometimes with hosts that use Delayed_ACK :(net.inet.tcp. :delayed_ack) it creates a dead-lock where the host will not ACK until :it gets :another packet and the client will not send another packet until it :gets an ACK.. : :The dead-lock gets broken by a time-out, which I think is around 200ms? : :But I would like to change that time-out if possible to something :lower, yet :I can't really see any sysctl knobs that have a name that suggests :they do :that.. : :So does anyone know IF this can be tuned and if so by what? : :Cheers, :Jerahmy. : :(And yes you could solve it by setting the TCP_NODELAY flag on the :socket, :but not everything has programmed in options to set it and you don't :always :have access to the source, besides setting a sysctl value would be much :simpler than recompiling stuff) There is a sysctl which adjusts the delayed-ack timing, its called net.inet.tcp.delacktime. The default is 1/10 of a second (100 == 100 ms = 1/10 of a second). BUT, it shouldn't be possible for nagle to deadlock against delayed acks unless the TCP implementation is broken somehow. A delayed ack is simply that... the ack is delayed 100 ms in order to improve its chances of being piggy-backed on return data. The ack is not blocked completely, just delayed, and certain events (such as the receiving end turning around and sending data back, which is typical for an interactive connection)... certain events will cause the delayed ack to be aborted and for the ack to be immediately sent with the return data. Can it break down and cause excessive lag? Yes, it can. Interactive games almost universally have to disable Nagle because the lag is actually due to the data relay from client 1 -> server then relaying the interactive event to client 2. Without an immediate interactive response to client 1 the ack gets delayed and the next event from client 1 hits Nagle and stops dead in the water until the first event reaches client 2 and client 2 reacts to it (then client 2 -> server -> (abort delayed ack and send) -> client 1 (client 1's nagle now allows the second event to be transmitted). That isn't a deadlock, just really poor interactive performance in that particular situation. Delayed acks also have a safety valve. The spec says that an ack cannot be delayed more then two packets. In a batch link when the second (unacked) packet is received, the delayed ack is aborted and an ack is immediately returned to the sender. This is to prevent congestion control (which is based on acks) from getting completely out of whack and also to prevent the TCP window from getting exhausted. In anycase, the usual solution is to disable Nagle rather then mess with delayed acks. What we need is a new Nagle that understands the new reality for interactive connections... something that doesn't break performance in the 'server in the middle' data relaying case. -Matt