From owner-freebsd-net@FreeBSD.ORG Fri Jun 1 02:48:14 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9C230106564A for ; Fri, 1 Jun 2012 02:48:14 +0000 (UTC) (envelope-from lstewart@freebsd.org) Received: from lauren.room52.net (lauren.room52.net [210.50.193.198]) by mx1.freebsd.org (Postfix) with ESMTP id 58AE88FC12 for ; Fri, 1 Jun 2012 02:48:14 +0000 (UTC) Received: from lstewart.caia.swin.edu.au (lstewart.caia.swin.edu.au [136.186.229.95]) by lauren.room52.net (Postfix) with ESMTPSA id 095C77E891; Fri, 1 Jun 2012 12:48:13 +1000 (EST) Message-ID: <4FC82D6C.4050309@freebsd.org> Date: Fri, 01 Jun 2012 12:48:12 +1000 From: Lawrence Stewart User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:10.0.2) Gecko/20120311 Thunderbird/10.0.2 MIME-Version: 1.0 To: Kevin Oberman References: <4FBF88CE.20209@cs.duke.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=5.0 tests=UNPARSEABLE_RELAY autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lauren.room52.net Cc: freebsd-net@freebsd.org, Andrew Gallatin , Andrew Gallatin Subject: Re: Major performance hit with ToS setting X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2012 02:48:14 -0000 On 05/31/12 13:33, Kevin Oberman wrote: [snip] > I used SIFTR at the suggestion of Lawrence Stewart who headed the > project to bring plugable congestion algorithms to FreeBSD and found > really odd congestion behavior. First, I do see a triple ACK, but the > congestion window suddenly drops from 73K to 8K. If I understand > CUBIC, it should half the congestion window, not what is happening.. > It then increases slowly (in slow start) to 82K. while the slow-start > bytes are INCREASING, the congestion window again goes to 8K while the > SS size moves from 36K up to 52K. It just continues to bound wildly > between 8K (always the low point) and between 64k and 82K. The swings > start at 83K and, over the first few seconds the peaks drop to about > 64K. Oh, and a comment about this behaviour. Dropping back to 8k (1MSS) is only nasty if the TF_{CONG|FAST}RECOVERY flags are *not* set i.e. if you see cwnd grow, drop to 8k with those flags set, and then when the flags are unset, cwnd starts at the value of ssthresh, then that is perfectly normal recovery behaviour. What *is* nasty is if an RTO fires, which will reset cwnd to 8k, ssthresh to 2*MSS and make the connection effectively start from scratch again. There is evidence of RTOs in your siftr output, which is bad news e.g here's one example of 2 side-by-side log lines from your trace: # Direction,time,ssthresh,cwnd,flags i,1338319593.574706,27044,27044,1630544864 o,1338319593.831482,16384,8192,1092625377 Note the 300ms gap, and how cwnd resets to 1MSS and flags go from 1630544864 (TF_WASCRECOVERY|TF_CONGRECOVERY|TF_WASFRECOVERY|TF_FASTRECOVERY) to 1092625377 (TF_WASCRECOVERY|TF_WASFRECOVERY). Cheers, Lawrence