From owner-freebsd-fs@FreeBSD.ORG Sun Apr 14 03:00:36 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9F467264 for ; Sun, 14 Apr 2013 03:00:36 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 379C01DDC for ; Sun, 14 Apr 2013 03:00:35 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqIEAGkaalGDaFvO/2dsb2JhbABQgzyDML4dgRp0gh8BAQEDAQEBASArIAsbGAICDRkCKQEJGAENBggHBAEcBIdtBgyodJFugSOMQn40B4IugRMDkziBDIJBgSGPcIMnIDKBBTU X-IronPort-AV: E=Sophos;i="4.87,469,1363147200"; d="scan'208";a="23804207" Received: from erie.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.206]) by esa-annu.net.uoguelph.ca with ESMTP; 13 Apr 2013 23:00:34 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 4BC5AB3F19; Sat, 13 Apr 2013 23:00:34 -0400 (EDT) Date: Sat, 13 Apr 2013 23:00:34 -0400 (EDT) From: Rick Macklem To: Paul van der Zwan Message-ID: <678464111.812434.1365908434250.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <495AEA10-9B8F-4A03-B706-79BF43539482@vanderzwan.org> Subject: Re: FreeBSD 9.1 NFSv4 client attribute cache not caching ? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.202] X-Mailer: Zimbra 6.0.10_GA_2692 (ZimbraWebClient - FF3.0 (Win)/6.0.10_GA_2692) Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Apr 2013 03:00:36 -0000 Paul van der Zwan wrote: > On 12 Apr 2013, at 16:28 , Paul van der Zwan > wrote: > > > > > I am running a few VirtualBox VMs with 9.1 on my OpenIndiana server > > and I noticed that make buildworld seem to take much longer > > when the clients mount /usr/src and /usr/obj over NFS V4 than when > > they use V3. > > Unfortunately I have to use V4 as a buildworld on V3 hangs the > > server completely... > > I noticed the number of PUTFH/GETATTR/GETFH calls in in the order of > > a few thousand per second > > and if I snoop the traffic I see the same filenames appear over and > > over again. > > It looks like the client is not caching anything at all and using a > > server request everytime. > > I use the default mount options: > > 192.168.178.24:/data/ports on /usr/ports (nfs, nfsv4acls) > > 192.168.178.24:/data/src on /usr/src (nfs, nfsv4acls) > > 192.168.178.24:/data/obj on /usr/obj (nfs, nfsv4acls) > > > > > > I had a look with dtrace > $ sudo dtrace -n '::getattr:start { @[stack()]=count();}' > and it seems the vast majority of the calls to getattr are from open() > and close() system calls.: > kernel`newnfs_request+0x631 > kernel`nfscl_request+0x75 > kernel`nfsrpc_getattr+0xbe > kernel`nfs_getattr+0x280 > kernel`VOP_GETATTR_APV+0x74 > kernel`nfs_lookup+0x3cc > kernel`VOP_LOOKUP_APV+0x74 > kernel`lookup+0x69e > kernel`namei+0x6df > kernel`kern_execve+0x47a > kernel`sys_execve+0x43 > kernel`amd64_syscall+0x3bf > kernel`0xffffffff80784947 > 26 > > kernel`newnfs_request+0x631 > kernel`nfscl_request+0x75 > kernel`nfsrpc_getattr+0xbe > kernel`nfs_close+0x3e9 > kernel`VOP_CLOSE_APV+0x74 > kernel`kern_execve+0x15c5 > kernel`sys_execve+0x43 > kernel`amd64_syscall+0x3bf > kernel`0xffffffff80784947 > 26 > > kernel`newnfs_request+0x631 > kernel`nfscl_request+0x75 > kernel`nfsrpc_getattr+0xbe > kernel`nfs_getattr+0x280 > kernel`VOP_GETATTR_APV+0x74 > kernel`nfs_lookup+0x3cc > kernel`VOP_LOOKUP_APV+0x74 > kernel`lookup+0x69e > kernel`namei+0x6df > kernel`vn_open_cred+0x330 > kernel`vn_open+0x1c > kernel`kern_openat+0x207 > kernel`kern_open+0x19 > kernel`sys_open+0x18 > kernel`amd64_syscall+0x3bf > kernel`0xffffffff80784947 > 2512 > > kernel`newnfs_request+0x631 > kernel`nfscl_request+0x75 > kernel`nfsrpc_getattr+0xbe > kernel`nfs_close+0x3e9 > kernel`VOP_CLOSE_APV+0x74 > kernel`vn_close+0xee > kernel`vn_closefile+0xff > kernel`_fdrop+0x3a > kernel`closef+0x332 > kernel`kern_close+0x183 > kernel`sys_close+0xb > kernel`amd64_syscall+0x3bf > kernel`0xffffffff80784947 > 2530 > > I had a look at the source of nfs_close and could not find a call to > nfsrpc_getattr, and I am wondering why close would be calling getattr > anyway. > If the file is closed what do we care about it's attributes.... > Here are some random statements w.r.t. NFSv3 vs NFSv4 that might help with an understanding of what is going on. I do address the specific case of nfs_close() towards the end. (It is kinda long winded, but I threw out eveything I could think of..) NFSv3 doesn't have any open/close RPC, but NFSv4 does have Open and Close operations. In NFSv3, each RPC is defined and usually includes attributes for files before and after the operation (implicit getattrs not counted in the RPC counts reported by nfsstat). For NFSv4, every RPC is a compound built up of a list of Operations like Getattr. Since the NFSv4 server doesn't know what the compound is doing, nfsstat reports the counts of Operations for the NFSv4 server, so the counts will be much higher than with NFSv3, but do not reflect the number of RPCs being done. To get NFSv4 nfsstat output that can be compared to NFSv3, you need to do the command on the client(s) and it still is only roughly the same. (I just realized this should be documented in man nfsstat.) For the FreeBSD NFSv4 client, the compounds include Getattr operations similar to what NFSv3 does. It doesn't do a Getattr on the directory for Lookup, because that would have made the compound much more complex. I don't think this will have a significant performance impact, but will result in some additional Getattr RPCs. I suspect the slowness is caused by the extra overhead of doing the Open/Close operations against the server. The only way to avoid doing these against the server for NFSv4 is to enable delegations in both client and server. How to do this is documented in "man nfsv4". Basically starting up the nfscbd in the client and setting: vfs.nfsd.issue_delegations=1 in the server. Specifically for nfs_close(), the attributes (modify time) is used for what is called "close to open consistency". This can be disabled by the "nocto" mount option, if you don't need it for your build environment. (You only need it if one client is writing a file and then another client is reading the same file.) Both the attribute caching and close to open consistency algorithms in the client are essentially the same for NFSv3 vs NFSv4. The NFSv4 Close operation(s) are actually done when the v_usecount for the vnode goes to 0, since mmap'd files can do I/O on pages after the close syscall. As such, they are only loosely related to the close syscall. They are actually closing Windows style Openlock(s). You mention that you see the same file over and over in a packet trace. You don't give specifics, but I'd suggest that you look at both NFSv3 and NFSv4 for this (and file names are in lookups, not getattrs). I'd suggest you try enabling delegations in both client and server, plus trying the "nocto" mount option and see if that helps. rick > > Paul > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"