From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 01:52:46 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B2E9AABF; Sat, 9 Mar 2013 01:52:46 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) by mx1.freebsd.org (Postfix) with ESMTP id 5D8EB3E9; Sat, 9 Mar 2013 01:52:46 +0000 (UTC) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.5/8.14.5) with ESMTP id r291qj70010186; Fri, 8 Mar 2013 20:52:45 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.5/8.14.4/Submit) id r291qjvK010183; Fri, 8 Mar 2013 20:52:45 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <20794.38381.221980.5038@hergotha.csail.mit.edu> Date: Fri, 8 Mar 2013 20:52:45 -0500 From: Garrett Wollman To: Rick Macklem Subject: Re: NFS DRC size In-Reply-To: <2050712270.3721724.1362790033662.JavaMail.root@erie.cs.uoguelph.ca> References: <20794.7012.265887.99878@hergotha.csail.mit.edu> <2050712270.3721724.1362790033662.JavaMail.root@erie.cs.uoguelph.ca> X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (hergotha.csail.mit.edu [127.0.0.1]); Fri, 08 Mar 2013 20:52:45 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=disabled version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hergotha.csail.mit.edu Cc: freebsd-fs@freebsd.org, freebsd-net@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 01:52:46 -0000 < said: > The cached replies are copies of the mbuf list done via m_copym(). > As such, the clusters in these replies won't be free'd (ref cnt -> 0) > until the cache is trimmed (nfsrv_trimcache() gets called after the > TCP layer has received an ACK for receipt of the reply from the client). I wonder if this bit is even working at all. In my experience, the size of the DRC quickly grows under load up to the maximum (or actually, slightly beyond), and never drops much below that level. On my production server right now, "nfsstat -se" reports: Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 13036780 359901 1723623 3420 36397693 12385668 346590 109984 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 45173 16 116791 14192 1176 24 12876747 3398533 Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId SetClIdCf 0 2703 14992 7502 1329196 0 1 1 Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lock 263034 0 0 263019 0 0 545104 0 LockT LockU Close Verify NVerify PutFH PutPubFH PutRootFH 0 0 263012 0 0 23753375 0 1 Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create 2 263006 263033 0 0 0 Server: Retfailed Faults Clients 0 0 1 OpenOwner Opens LockOwner Locks Delegs 56 10 0 0 0 Server Cache Stats: Inprog Idem Non-idem Misses CacheSize TCPPeak 0 0 0 81714128 60997 61017 It's only been up for about the last 24 hours. Should I be setting the size limit to something truly outrageous, like 200,000? (I'd definitely need to deal with the mbuf cluster issue then!) The average request rate over this time is about 1000/s, but that includes several episodes of high-cpu spinning (which I resolved by increasing the DRC limit). Meanwhile, some relevant bits from sysctl: vfs.nfsd.udphighwater: 500 vfs.nfsd.tcphighwater: 61000 vfs.nfsd.minthreads: 16 vfs.nfsd.maxthreads: 64 vfs.nfsd.threads: 64 vfs.nfsd.request_space_used: 1416 vfs.nfsd.request_space_used_highest: 4284672 vfs.nfsd.request_space_high: 47185920 vfs.nfsd.request_space_low: 31457280 vfs.nfsd.request_space_throttled: 0 vfs.nfsd.request_space_throttle_count: 0 (I'd actually like to put maxthreads back up at 256, which is where I had it during testing, but I need to test that the jumbo-frames issue is fixed first. I did pre-production testing on a non-jumbo network.) -GAWollman