From owner-freebsd-net@FreeBSD.ORG Thu Jan 22 23:04:38 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 60BCAED for ; Thu, 22 Jan 2015 23:04:38 +0000 (UTC) Received: from mail-yh0-x22d.google.com (mail-yh0-x22d.google.com [IPv6:2607:f8b0:4002:c01::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 18D2C2AE for ; Thu, 22 Jan 2015 23:04:38 +0000 (UTC) Received: by mail-yh0-f45.google.com with SMTP id f73so1782401yha.4 for ; Thu, 22 Jan 2015 15:04:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=QWQDdqCvcuTP1lDY/ZmzpND9mB6jPY2QfRrCixZm0T8=; b=HV/7L00obERCowRMluFRjyRymT65vGtl6nTVanpnatx3L5dUI1fICKTU7BeAr71zyL rg6akFRQdBJ6rFgwbZOwNGKt9iuBU2UtwA1Nc5vc6XzdtepGirU+i8XTWotCexNOZhfF DEOliRfKsl2Resnh0sVToVnJKZqaeFsL1jc8PMLlLCZe6TXnvDmg5wb6rEHICPqkZP6X lQ+NHSqje62+FqoJ9ConEfyZqJ6jkTO7EPneBZqY5OVDjGpAQqLVItXC+sTctDuuOfsU WE6wN/0J8zECaVfO0wQ9N+Mnlxd0ea4QTcdUam1sIy6WaT8TEgTs9IgWvmOhmhpvGMmR gLHQ== MIME-Version: 1.0 X-Received: by 10.170.44.4 with SMTP id 4mr2193586ykm.101.1421967877317; Thu, 22 Jan 2015 15:04:37 -0800 (PST) Sender: kmacybsd@gmail.com Received: by 10.170.70.132 with HTTP; Thu, 22 Jan 2015 15:04:37 -0800 (PST) In-Reply-To: References: Date: Thu, 22 Jan 2015 15:04:37 -0800 X-Google-Sender-Auth: eFVZFDKoTIwwSi8WUJ6l81-EvtA Message-ID: Subject: Re: Tuning net.inet.tcp.sendbuf_max From: "K. Macy" To: javocado Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-net@freebsd.org" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Jan 2015 23:04:38 -0000 TCP host cache? See netinet/tcp_hostcache.c for any fiddling that needs doing. Let me know if there are any values that should be exported as sysctls. -K SYSCTL_INT(_net_inet_tcp_hostcache, OID_AUTO, expire, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_hostcache.expire), 0, "Expire time of TCP hostcache entries"); SYSCTL_INT(_net_inet_tcp_hostcache, OID_AUTO, prune, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_hostcache.prune), 0, "Time between purge runs"); SYSCTL_INT(_net_inet_tcp_hostcache, OID_AUTO, purge, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_hostcache.purgeall), 0, "Expire all entires on next purge run"); /* * The tcp_hostcache moves the tcp-specific cached metrics from the routing * table to a dedicated structure indexed by the remote IP address. It keeps * information on the measured TCP parameters of past TCP sessions to allow * better initial start values to be used with later connections to/from the * same source. Depending on the network parameters (delay, bandwidth, max * MTU, congestion window) between local and remote sites, this can lead to * significant speed-ups for new TCP connections after the first one. * * Due to the tcp_hostcache, all TCP-specific metrics information in the * routing table have been removed. The inpcb no longer keeps a pointer to * the routing entry, and protocol-initiated route cloning has been removed * as well. With these changes, the routing table has gone back to being * more lightwight and only carries information related to packet forwarding. * * tcp_hostcache is designed for multiple concurrent access in SMP * environments and high contention. All bucket rows have their own lock and * thus multiple lookups and modifies can be done at the same time as long as * they are in different bucket rows. If a request for insertion of a new * record can't be satisfied, it simply returns an empty structure. Nobody * and nothing outside of tcp_hostcache.c will ever point directly to any * entry in the tcp_hostcache. All communication is done in an * object-oriented way and only functions of tcp_hostcache will manipulate * hostcache entries. Otherwise, we are unable to achieve good behaviour in * concurrent access situations. Since tcp_hostcache is only caching * information, there are no fatal consequences if we either can't satisfy * any particular request or have to drop/overwrite an existing entry because * of bucket limit memory constrains. */ On Wed, Jan 21, 2015 at 11:04 AM, javocado wrote: > System: FreeBSD 8.3 amd64 > > I've been trying to tune my system to a long route (RTT 180ms) and I've > made 2 modifications and seeing some results which I cannot explain or > understand. > > 1. kern.ipc.maxsockbuf: 262144 -> 4194304 > > Speed improves from 85 Mbps -> 100 Mbps > > I can run the test: iperf -c tserv20.hkg1.ipv6.he.net -t10 -P 10 > > all day long and consistently get 100Mbps > > > 2. net.inet.tcp.sendbuf_max: 262144 -> 524288 > > 1st run: 156Mbps > > BUT, on subsequent runs, just moment later, I see the speed drop off > quickly with each successive run: > > 73 ... 50Mbps > > > Simply returning sendbuf_max to the original value of 262144 does not > return me to 100Mbps. Why is this? > Setting net.inet.tcp.sendbuf_max, net.inet.tcp.recvbuf_max > and kern.ipc.maxsockbuf to 4194304 also makes no difference. > > I noticed that the speed returned to 100Mbps when I set it back to 262144 > and re-ran the test ~12hrs later. Is there some kind of buffer that needs > to be cleared (or clears with enough time) so these changes have immediate > effect when returning these sysctls to original values? > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"