From owner-freebsd-arch@FreeBSD.ORG Sun Mar 20 06:26:50 2011 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F550106564A for ; Sun, 20 Mar 2011 06:26:50 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx06.syd.optusnet.com.au (fallbackmx06.syd.optusnet.com.au [211.29.132.8]) by mx1.freebsd.org (Postfix) with ESMTP id 8FE7A8FC0C for ; Sun, 20 Mar 2011 06:26:48 +0000 (UTC) Received: from mail01.syd.optusnet.com.au (mail01.syd.optusnet.com.au [211.29.132.182]) by fallbackmx06.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p2K4MvBd019969 for ; Sun, 20 Mar 2011 15:22:57 +1100 Received: from c122-107-125-80.carlnfd1.nsw.optusnet.com.au (c122-107-125-80.carlnfd1.nsw.optusnet.com.au [122.107.125.80]) by mail01.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p2K4Mprj023625 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 20 Mar 2011 15:22:53 +1100 Date: Sun, 20 Mar 2011 15:22:51 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Jeff Roberson In-Reply-To: Message-ID: <20110320151003.A939@besplex.bde.org> References: <132388F1-44D9-45C9-AE05-1799A7A2DCD9@neville-neil.com> <20110319160400.000043f5@unknown> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Alexander Leidinger , George Neville-Neil , arch@FreeBSD.org Subject: Re: Updating our TCP and socket sysctl values... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Mar 2011 06:26:50 -0000 On Sat, 19 Mar 2011, Jeff Roberson wrote: > On Sat, 19 Mar 2011, Alexander Leidinger wrote: > >> On Sat, 19 Mar 2011 15:37:47 +0900 George Neville-Neil >> wrote: >>> >>> I believe it's time for us to upgrade our sysctl values for TCP >>> sockets so that they are more in line with the modern world. At the >>> moment we have these limits on our buffering: >>> >>> kern.ipc.maxsockbuf: 262144 >>> net.inet.tcp.recvbuf_max: 262144 >>> net.inet.tcp.sendbuf_max: 262144 >>> >>> I believe it's time to up these values to something that's in line >>> with higher speed local networks, such as 10G. Perhaps it's time to >>> move these to 2MB instead of 256K. All hard-coded limits are bogus. The same limit for a machine that has 8MB memory is nonense for a machine that has 8GB. In FreeBSD, AFAIK only the vm system has _very_ good auto-tuning of parameters and limits thanks to dyson's work 10-15 years ago. It has almost no user-settable parameters or limits like the above. >> I suggest to read >> http://www.bufferbloat.net/projects/bloat/wiki/Bufferbloat >> and do a before/after test to make sure we do not suffer from the >> described problem. Jim Getty has test descriptions: >> http://gettys.wordpress.com/category/bufferbloat/ > > Are they not talking about buffers in non-endpoint devices? Or perhaps even > overly large rx queues in endpoints, but not local socket receive buffers? > It seems that they are describing situations where excessive buffering masks > network conditions until it's too late. I don't know, but there is an mostly-unrelated bufferbloat problem that is purely local. If you have a buffer that is larger than an Ln cache (or about half than), then actually using just a single buffer of that size guarantees thrashing of the Ln cache, so that almost every memory access is an Ln cache miss. Even with current hardware, a buffer of size 256K will thrash most L1 caches and a buffer of size a few MB will thrash most L2 caches. Bruce