From owner-freebsd-net@FreeBSD.ORG  Wed Jan 21 19:04:36 2015
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 36944B2E
 for <freebsd-net@freebsd.org>; Wed, 21 Jan 2015 19:04:36 +0000 (UTC)
Received: from mail-la0-x230.google.com (mail-la0-x230.google.com
 [IPv6:2a00:1450:4010:c03::230])
 (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
 (Client CN "smtp.gmail.com",
 Issuer "Google Internet Authority G2" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id AE0C0E0D
 for <freebsd-net@freebsd.org>; Wed, 21 Jan 2015 19:04:35 +0000 (UTC)
Received: by mail-la0-f48.google.com with SMTP id gf13so41924647lab.7
 for <freebsd-net@freebsd.org>; Wed, 21 Jan 2015 11:04:33 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:date:message-id:subject:from:to:content-type;
 bh=mRU2QlvTlZrLe1aPINziGCHcm+zTOHUB6KDutVnnxHM=;
 b=Wp3aJCHK5ktCl7DGdse5xRwphSRdCr9A1fOWBrUIAZuFmdPf4kuakE4eaJslm6Mmq1
 wSzxgGhT/n5klE3Dlrs4QzTinZ8WPGnXKLwYMXtmL73is7Wj3X7UC8y0axGjHlGJq1Ru
 8KX0zsDnxW2IK528Zph2sPTaqPMmoPU5MLM1Nye+mRZxRvCyPTpq6aNUg9Hx0k7qPLRA
 YrRLI5xTBL9vPZPdmwWB6U3tF41qLmNkTZh3sP5rz44jJXsoCDKc1Pwi6VL/CYBHavWi
 241tsvusNHr+S/ItxWcOnWxuxCtLca0or24qJ93/KFrSi8c/rlBX8T2M6TOvUN6MReBD
 L2oA==
MIME-Version: 1.0
X-Received: by 10.112.38.4 with SMTP id c4mr46322905lbk.46.1421867073137; Wed,
 21 Jan 2015 11:04:33 -0800 (PST)
Received: by 10.114.216.163 with HTTP; Wed, 21 Jan 2015 11:04:33 -0800 (PST)
Date: Wed, 21 Jan 2015 11:04:33 -0800
Message-ID: <CAP1HOmRbOX7RGp-WZBhFZ7P4Os791c3W4oaj2hr6iQQUH+YfTA@mail.gmail.com>
Subject: Tuning net.inet.tcp.sendbuf_max
From: javocado <javocado@gmail.com>
To: freebsd-net@freebsd.org
Content-Type: text/plain; charset=UTF-8
X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 21 Jan 2015 19:04:36 -0000

System: FreeBSD 8.3 amd64

I've been trying to tune my system to a long route (RTT 180ms) and I've
made 2 modifications and seeing some results which I cannot explain or
understand.

1. kern.ipc.maxsockbuf: 262144 -> 4194304

Speed improves from 85 Mbps -> 100 Mbps

I can run the test: iperf -c tserv20.hkg1.ipv6.he.net -t10 -P 10

all day long and consistently get 100Mbps


2. net.inet.tcp.sendbuf_max: 262144 -> 524288

1st run: 156Mbps

BUT, on subsequent runs, just moment later, I see the speed drop off
quickly with each successive run:

73 ... 50Mbps


Simply returning sendbuf_max to the original value of 262144 does not
return me to 100Mbps. Why is this?
Setting net.inet.tcp.sendbuf_max, net.inet.tcp.recvbuf_max
and kern.ipc.maxsockbuf to 4194304 also makes no difference.

I noticed that the speed returned to 100Mbps when I set it back to 262144
and re-ran the test ~12hrs later. Is there some kind of buffer that needs
to be cleared (or clears with enough time) so these changes have immediate
effect when returning these sysctls to original values?