From owner-freebsd-net@FreeBSD.ORG Sat Mar 9 18:00:05 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C5033C08; Sat, 9 Mar 2013 18:00:05 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 66E9627C; Sat, 9 Mar 2013 18:00:05 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r29I04Sp062160; Sat, 9 Mar 2013 13:00:04 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r29I04gL062157; Sat, 9 Mar 2013 13:00:04 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20795.30884.330015.123616@hergotha.csail.mit.edu> Date: Sat, 9 Mar 2013 13:00:04 -0500 From: Garrett Wollman To: Rick Macklem Subject: Re: NFS DRC size In-Reply-To: <1639798917.3728142.1362846452693.JavaMail.root@erie.cs.uoguelph.ca> References: <20794.38381.221980.5038@hergotha.csail.mit.edu> <1639798917.3728142.1362846452693.JavaMail.root@erie.cs.uoguelph.ca> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Sat, 09 Mar 2013 13:00:04 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu Cc: freebsd-fs@freebsd.org, freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 18:00:05 -0000 < said: > around the highwater mark basically indicates this is working. If it wasn't > throwing away replies where the receipt has been ack'd at the TCP > level, the cache would grow very large, since they would only be > discarded after a loonnngg timeout (12hours unless you've changes > NFSRVCACHE_TCPTIMEOUT in sys/fs/nfs/nfs.h). That seems unreasonably large. > Well, the DRC will try to cache replies until the client's TCP layer > acknowledges receipt of the reply. It is hard to say how many replies > that is for a given TCP connection, since it is a function of the level > of concurrently (# of nfsiod threads in the FreeBSD client) > in the client. I'd guess it's somewhere between 1<->20? Nearly all our clients are Linux, so it's likely to be whatever Debian does by default. > Multiply that by the number of TCP connections from all clients and > you have about how big the server's DRC will be. (Some clients use > a single TCP connection for the client whereas others use a separate > TCP connection for each mount point.) The Debian client appears to use a single TCP connection for everything. So if I want to support 2,000 clients each with 20 requests in flight, that would suggest that I need a DRC size of 40,000, which my experience shows is not sufficient with even a much smaller number of clients. -GAWollman