From owner-freebsd-transport@freebsd.org Mon Nov 23 16:38:20 2015 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8EE43A36A8E for ; Mon, 23 Nov 2015 16:38:20 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 7261611D3 for ; Mon, 23 Nov 2015 16:38:20 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 70CEFA36A8D; Mon, 23 Nov 2015 16:38:20 +0000 (UTC) Delivered-To: transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 568E0A36A8C for ; Mon, 23 Nov 2015 16:38:20 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2D0F311D1 for ; Mon, 23 Nov 2015 16:38:19 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 3FAA620502 for ; Mon, 23 Nov 2015 11:38:18 -0500 (EST) Received: from web3 ([10.202.2.213]) by compute4.internal (MEProxy); Mon, 23 Nov 2015 11:38:18 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-sasl-enc :x-sasl-enc; s=smtpout; bh=781GxRMxsFoOnXmL7f/OY4KzuUQ=; b=R3YW8 TfgptP0n2g1nSIEFz6SPoLVHB/45ESA9c6pwog/A1FRMG3zppQeub3O3assaw3np NE/oi0EcYC1Twhl0xWiJncZxv5U3PDuu+5RHhqS5rWehshAPIueS3QQ0IIPlmGO+ 9d8NCS0H9Fg7ysuzZIvdAmKAjazhMrTE6ncpqE= Received: by web3.nyi.internal (Postfix, from userid 99) id 1C12C108C06; Mon, 23 Nov 2015 11:38:18 -0500 (EST) Message-Id: <1448296698.1772293.447796761.5B2239BD@webmail.messagingengine.com> X-Sasl-Enc: +wa4D0tv+PBJzBAT/SD+L10E7kkaRGvM2SAAgd00qcoW 1448296698 From: Mark Felder To: transport@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-0a80c2aa Subject: Should delayed ACKs be enabled by default? Date: Mon, 23 Nov 2015 10:38:18 -0600 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Nov 2015 16:38:20 -0000 John Nagle recently commented on a HN post[1] regarding the issues delayed acks cause. Has this been studied in depth on FreeBSD? His post: > That still irks me. The real problem is not tinygram prevention. It's > ACK delays, and that stupid fixed timer. They both went into TCP around > the same time, but independently. I did tinygram prevention (the Nagle > algorithm) and Berkeley did delayed ACKs, both in the early 1980s. The > combination of the two is awful. Unfortunately by the time I found about > delayed ACKs, I had changed jobs, was out of networking, and doing a > product for Autodesk on non-networked PCs. > > Delayed ACKs are a win only in certain circumstances - mostly character > echo for Telnet. (When Berkeley installed delayed ACKs, they were doing > a lot of Telnet from terminal concentrators in student terminal rooms to > host VAX machines doing the work. For that particular situation, it made > sense.) The delayed ACK timer is scaled to expected human response time. > A delayed ACK is a bet that the other end will reply to what you just > sent almost immediately. Except for some RPC protocols, this is > unlikely. So the ACK delay mechanism loses the bet, over and over, > delaying the ACK, waiting for a packet on which the ACK can be > piggybacked, not getting it, and then sending the ACK, delayed. There's > nothing in TCP to automatically turn this off. However, Linux (and I > think Windows) now have a TCP_QUICKACK socket option. Turn that on > unless you have a very unusual application. > > Turning on TCP_NODELAY has similar effects, but can make throughput > worse for small writes. If you write a loop which sends just a few bytes > (worst case, one byte) to a socket with "write()", and the Nagle > algorithm is disabled with TCP_NODELAY, each write becomes one IP > packet. This increases traffic by a factor of 40, with IP and TCP > headers for each payload. Tinygram prevention won't let you send a > second packet if you have one in flight, unless you have enough data to > fill the maximum sized packet. It accumulates bytes for one round trip > time, then sends everything in the queue. That's almost always what you > want. If you have TCP_NODELAY set, you need to be much more aware of > buffering and flushing issues. > > None of this matters for bulk one-way transfers, which is most HTTP > today. (I've never looked at the impact of this on the SSL handshake, > where it might matter.) > > Short version: set TCP_QUICKACK. If you find a case where that makes > things worse, let me know. > > John Nagle [1] https://news.ycombinator.com/item?id=10608356 -- Mark Felder ports-secteam member feld@FreeBSD.org From owner-freebsd-transport@freebsd.org Mon Nov 23 19:23:13 2015 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB6A0A368F6 for ; Mon, 23 Nov 2015 19:23:13 +0000 (UTC) (envelope-from mmacy@nextbsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B44601D7C for ; Mon, 23 Nov 2015 19:23:13 +0000 (UTC) (envelope-from mmacy@nextbsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id B4036A368F5; Mon, 23 Nov 2015 19:23:13 +0000 (UTC) Delivered-To: transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 99C03A368F4 for ; Mon, 23 Nov 2015 19:23:13 +0000 (UTC) (envelope-from mmacy@nextbsd.org) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 891B31D7A for ; Mon, 23 Nov 2015 19:23:13 +0000 (UTC) (envelope-from mmacy@nextbsd.org) Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1448306585043188.95064392577183; Mon, 23 Nov 2015 11:23:05 -0800 (PST) Date: Mon, 23 Nov 2015 11:23:04 -0800 From: Matthew Macy To: "Mark Felder" Cc: "" Message-ID: <15135ccada3.10df21d5156685.7375508093788694995@nextbsd.org> In-Reply-To: <1448296698.1772293.447796761.5B2239BD@webmail.messagingengine.com> References: <1448296698.1772293.447796761.5B2239BD@webmail.messagingengine.com> Subject: Re: Should delayed ACKs be enabled by default? MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-Priority: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Nov 2015 19:23:13 -0000 The fixed delay is a known problem (in the CC community - not in FreeBSD) and is particularly excessive on FreeBSD (100ms vs 40ms on Linux). It hasn't discussed much because it is a minor annoyance vis a vis other shortfalls in TCP. Part of my incast mitigation changes is capping delack at 40ms and scaling it down dynamically at a rate of 1/2 RTO. With minRTO set to 3 ticks that would potentially bring it down to 1ms in the data center. Glenn Judd at Morgan Stanley wrote a good paper about their experiences called "Attaining the Promise and Avoiding the Pitfalls of TCP in the Datacenter" [https://www.usenix.org/system/files/conference/nsdi15/nsdi15-paper-judd.pdf]. MS found that eliminating the delack increased CPU overhead and that they were best served by setting delack to 1ms. I would wager that when the RTO (and thus delack) got well below 1ms that the extra interrupt overhead would outweigh any work coalescing benefits, but that is something we'll need to measure at that point. -M ---- On Mon, 23 Nov 2015 08:38:18 -0800 Mark Felder wrote ---- > John Nagle recently commented on a HN post[1] regarding the issues > delayed acks cause. Has this been studied in depth on FreeBSD? > > His post: > > > That still irks me. The real problem is not tinygram prevention. It's > > ACK delays, and that stupid fixed timer. They both went into TCP around > > the same time, but independently. I did tinygram prevention (the Nagle > > algorithm) and Berkeley did delayed ACKs, both in the early 1980s. The > > combination of the two is awful. Unfortunately by the time I found about > > delayed ACKs, I had changed jobs, was out of networking, and doing a > > product for Autodesk on non-networked PCs. > > > > Delayed ACKs are a win only in certain circumstances - mostly character > > echo for Telnet. (When Berkeley installed delayed ACKs, they were doing > > a lot of Telnet from terminal concentrators in student terminal rooms to > > host VAX machines doing the work. For that particular situation, it made > > sense.) The delayed ACK timer is scaled to expected human response time. > > A delayed ACK is a bet that the other end will reply to what you just > > sent almost immediately. Except for some RPC protocols, this is > > unlikely. So the ACK delay mechanism loses the bet, over and over, > > delaying the ACK, waiting for a packet on which the ACK can be > > piggybacked, not getting it, and then sending the ACK, delayed. There's > > nothing in TCP to automatically turn this off. However, Linux (and I > > think Windows) now have a TCP_QUICKACK socket option. Turn that on > > unless you have a very unusual application. > > > > Turning on TCP_NODELAY has similar effects, but can make throughput > > worse for small writes. If you write a loop which sends just a few bytes > > (worst case, one byte) to a socket with "write()", and the Nagle > > algorithm is disabled with TCP_NODELAY, each write becomes one IP > > packet. This increases traffic by a factor of 40, with IP and TCP > > headers for each payload. Tinygram prevention won't let you send a > > second packet if you have one in flight, unless you have enough data to > > fill the maximum sized packet. It accumulates bytes for one round trip > > time, then sends everything in the queue. That's almost always what you > > want. If you have TCP_NODELAY set, you need to be much more aware of > > buffering and flushing issues. > > > > None of this matters for bulk one-way transfers, which is most HTTP > > today. (I've never looked at the impact of this on the SSL handshake, > > where it might matter.) > > > > Short version: set TCP_QUICKACK. If you find a case where that makes > > things worse, let me know. > > > > John Nagle > > > [1] https://news.ycombinator.com/item?id=10608356 > > > > -- > Mark Felder > ports-secteam member > feld@FreeBSD.org > _______________________________________________ > freebsd-transport@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-transport > To unsubscribe, send any mail to "freebsd-transport-unsubscribe@freebsd.org" > From owner-freebsd-transport@freebsd.org Tue Nov 24 16:17:31 2015 Return-Path: Delivered-To: freebsd-transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F298DA371BD for ; Tue, 24 Nov 2015 16:17:31 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D977F14E5 for ; Tue, 24 Nov 2015 16:17:31 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: by mailman.ysv.freebsd.org (Postfix) id D606FA371BC; Tue, 24 Nov 2015 16:17:31 +0000 (UTC) Delivered-To: transport@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D59E5A371BB for ; Tue, 24 Nov 2015 16:17:31 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from smtp.hungerhost.com (smtp.hungerhost.com [216.38.51.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B2E4714E4 for ; Tue, 24 Nov 2015 16:17:31 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from pool-108-54-164-204.nycmny.fios.verizon.net ([108.54.164.204]:49743 helo=[192.168.1.6]) by vps.hungerhost.com with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.86) (envelope-from ) id 1a1GHb-0003C9-0O for transport@freebsd.org; Tue, 24 Nov 2015 11:17:31 -0500 From: "George Neville-Neil" To: transport@freebsd.org Subject: No hangout this week... Date: Tue, 24 Nov 2015 11:17:30 -0500 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Mailer: MailMate (1.9.3r5184) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com X-Authenticated-Sender: vps.hungerhost.com: gnn@neville-neil.com X-Source: X-Source-Args: X-Source-Dir: X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Nov 2015 16:17:32 -0000 Howdy, This Thursday, the 26th, is Thanksgiving in the US. As such we will not have a hangout this week. We'll resume on the following Thursday, the 3rd of December. Best, George