From owner-freebsd-net@FreeBSD.ORG Sun Mar 10 02:42:47 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 209E187A; Sun, 10 Mar 2013 02:42:47 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id B9EF47B5; Sun, 10 Mar 2013 02:42:46 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqEEACryO1GDaFvO/2dsb2JhbABCiCi8JIF0dIIsAQEBAwEBAQEgBCcgCxsYAgINGQIpAQkmBggHBAEcBIdsBgyqI5FvgSOMOn00B4ItgRMDiHKLJYI+gR6PV4MoHjKBBTU X-IronPort-AV: E=Sophos;i="4.84,816,1355115600"; d="scan'208";a="20301161" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 09 Mar 2013 21:42:45 -0500 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id B9596B3F46; Sat, 9 Mar 2013 21:42:45 -0500 (EST) Date: Sat, 9 Mar 2013 21:42:45 -0500 (EST) From: Rick Macklem To: Garrett Wollman Message-ID: <663916089.3736429.1362883365710.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <20795.30884.330015.123616@hergotha.csail.mit.edu> Subject: Re: NFS DRC size MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org, freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2013 02:42:47 -0000 Garrett Wollman wrote: > < said: > > > around the highwater mark basically indicates this is working. If it > > wasn't > > throwing away replies where the receipt has been ack'd at the TCP > > level, the cache would grow very large, since they would only be > > discarded after a loonnngg timeout (12hours unless you've changes > > NFSRVCACHE_TCPTIMEOUT in sys/fs/nfs/nfs.h). > > That seems unreasonably large. > I suppose. How long a network partitioning do you want the cache to deal with? (My original design was trying to achieve a high level of correctness by default.) The only time cache entries normally hang around this long is when a client has dismounted the volume(s) using the TCP connection. The cached replies for the last few replies will then hang around until the timeout. For a few clients this isn't an issue. For 2,000 clients, I can see that it might be, if the clients choose to dismount volumes (using something like amd). Feel free to make it smaller, based on the longest network partitioning that you anticipate might occur. > > Well, the DRC will try to cache replies until the client's TCP layer > > acknowledges receipt of the reply. It is hard to say how many > > replies > > that is for a given TCP connection, since it is a function of the > > level > > of concurrently (# of nfsiod threads in the FreeBSD client) > > in the client. I'd guess it's somewhere between 1<->20? > > Nearly all our clients are Linux, so it's likely to be whatever Debian > does by default. > > > Multiply that by the number of TCP connections from all clients and > > you have about how big the server's DRC will be. (Some clients use > > a single TCP connection for the client whereas others use a separate > > TCP connection for each mount point.) > > The Debian client appears to use a single TCP connection for > everything. > > So if I want to support 2,000 clients each with 20 requests in flight, > that would suggest that I need a DRC size of 40,000, which my > experience shows is not sufficient with even a much smaller number of > clients. > Well, especially since Debian is using one TCP connection for everything from a client, the guess of 20 could be way low. rick > -GAWollman > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"