From owner-freebsd-net@FreeBSD.ORG Sun Jun 3 05:18:20 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 70C37106566C; Sun, 3 Jun 2012 05:18:20 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 3D1D88FC08; Sun, 3 Jun 2012 05:18:20 +0000 (UTC) Received: by dadv36 with SMTP id v36so4702225dad.13 for ; Sat, 02 Jun 2012 22:18:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=/04r3UWOzFi93uuR6/wPrkopWT8lfpAQ/8/RI+uzMzk=; b=iXrs9iBRTPZadbnTmfBWD6ZZmgdh4qdwZ4k3/ino9TlsZH3gBh8LTkZ8DB/C07nW+p wqJrkaFnBBRCFbbrG2zQuY4GjG6m/FXKTiUtFEc3XuX/RpdMHO/bmz4J0e38pRw/ReDU aeuQmeJAO/p6y5rV6zzYdRdPIqUyxfwwllpkvjN+7TCrwdDnG8xuFWxoqx6S7OPFXP9I d3kGCCV7OWOgGIOTAR67SMhEhf7SBcCZfKvTHYQOSEg6g9uD2OJz4pslnUYPQ5FVl7aa lcT9TiN5F75eYxGb8nSkRIHzlGcFeDycX59FSM7ubfzR/0ogomDRNeDlf22p0agU52MS DlFQ== MIME-Version: 1.0 Received: by 10.68.227.69 with SMTP id ry5mr26592953pbc.16.1338700694110; Sat, 02 Jun 2012 22:18:14 -0700 (PDT) Received: by 10.68.71.232 with HTTP; Sat, 2 Jun 2012 22:18:14 -0700 (PDT) In-Reply-To: <4FC82D6C.4050309@freebsd.org> References: <4FBF88CE.20209@cs.duke.edu> <4FC82D6C.4050309@freebsd.org> Date: Sun, 3 Jun 2012 05:18:14 +0000 Message-ID: From: Kevin Oberman To: Lawrence Stewart Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-net@freebsd.org, Andrew Gallatin , Andrew Gallatin Subject: Re: Major performance hit with ToS setting X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Jun 2012 05:18:20 -0000 On Fri, Jun 1, 2012 at 2:48 AM, Lawrence Stewart wrote: > On 05/31/12 13:33, Kevin Oberman wrote: > [snip] >> >> I used SIFTR at the suggestion of Lawrence Stewart who headed the >> >> project to bring plugable congestion algorithms to FreeBSD and found >> really odd congestion behavior. First, I do see a triple ACK, but the >> congestion window suddenly drops from 73K to 8K. If I understand >> CUBIC, it should half the congestion window, not what is happening.. >> It then increases slowly (in slow start) to 82K. while the slow-start >> bytes are INCREASING, the congestion window again goes to 8K while the >> SS size moves from 36K up to 52K. It just continues to bound wildly >> between 8K (always the low point) and between 64k and 82K. The swings >> start at 83K and, over the first few seconds the peaks drop to about >> 64K. > > > Oh, and a comment about this behaviour. Dropping back to 8k (1MSS) is only > nasty if the TF_{CONG|FAST}RECOVERY flags are *not* set i.e. if you see cwnd > grow, drop to 8k with those flags set, and then when the flags are unset, > cwnd starts at the value of ssthresh, then that is perfectly normal recovery > behaviour. What *is* nasty is if an RTO fires, which will reset cwnd to 8k, > ssthresh to 2*MSS and make the connection effectively start from scratch > again. > > There is evidence of RTOs in your siftr output, which is bad news e.g here's > one example of 2 side-by-side log lines from your trace: > > # Direction,time,ssthresh,cwnd,flags > i,1338319593.574706,27044,27044,1630544864 > o,1338319593.831482,16384,8192,1092625377 > > Note the 300ms gap, and how cwnd resets to 1MSS and flags go from 1630544864 > (TF_WASCRECOVERY|TF_CONGRECOVERY|TF_WASFRECOVERY|TF_FASTRECOVERY) to > 1092625377 (TF_WASCRECOVERY|TF_WASFRECOVERY). What can I say but that you are right. When I looked at the interface stats I found that the link overflow drops were through the roof! This confuses me a bit since the traffic is outbound and I woudl assume from the description on hte Myricom web page that these are input drops. A problem a problem with that card? On systems that are working "normally", I still see a sharp drop with the ToS bits set, but nothing nearly as drastic. Now it is a drop from 4.5G to 728M on a cross-country (US) circuit. I am now looking for issues on the route that might explain the performance, but the question of why the drop-of only shows up in FreeBSD 8 means something odd is still going on. It is even possible that the problem is with 7 and the losses are due to the policy for ToS 32 on the path. ToS 32 is less than best effort in our network. Maybe the marking was getting lost on 7. Not likely, but possible. I'll post further details after some very careful reviews of router statistics. Thanks, Lawrence! -- R. Kevin Oberman, Network Engineer E-mail: kob6558@gmail.com