Date: Sun, 23 Oct 2016 22:21:06 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Marek Salwerowicz <marek.salwerowicz@misal.pl>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: ZFS - NFS server for VMware ESXi issues Message-ID: <YTXPR01MB0189EE88A51139BD8F31DA56DDD60@YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <a7bf8c97-34b1-9c8a-b056-983936ccc4f6@misal.pl> References: <930df17b-8db8-121a-a24b-b4909b8162dc@misal.pl> <YTXPR01MB018980B341213CA43277464ADDD40@YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM>, <a7bf8c97-34b1-9c8a-b056-983936ccc4f6@misal.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
Marek Salwerowicz wrote: >Hi Rick, > >W dniu 2016-10-21 o 23:47, Rick Macklem pisze: >> >> >> Btw, about the only area of the NFS server that might need tuning is >> the DRC and >> this doesn't suggest that. If you "nfsstat -e -s" on the server and >> see large #s for >> the last line under "Server Cache Stats:" there are tunables that can >> be used. >> I'd also suggest you capture the output of "ps axHl" on the server >> when it happens >> again, which tells you what all the nfsd threads are up to. > >I checked the >#ps axHL | grep nfs >now: >http://pastebin.com/x9LTN0nn What is there now is normal. "rpcsvc" just means the thread is waiting for = an RPC request from a client. This info might be useful when the server is hung/li= velocked. > >it looks like I have ~64 threads of nfs each cosuming ~one hour of CPU tim= e. If one hour of CPU seems excessive for you, you can disable the DRC. See below w.r.t. this. >That corresponds to: ># ps axl | grep nfs >UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMA= N > 0 1948 1 0 28 0 24632 5832 select Is - 0:00.10 >nfsd: master (nfsd) > 0 1949 1948 0 24 0 12344 4132 rpcsvc I - 66:56.42 >nfsd: server (nfsd) > >is it OK if threads are not being "recuperated" ? Not sure what you mean by this, but newer FreeBSD systems have minthreads a= nd maxthreads options on the nfsd to set lower/upper bounds on the # of thread= s. To be honest, having too many threads doesn't have much negative impact, so= I wouldn't worry about having too many. >The NFS statistics are as follows: ># nfsstat -e -s > >Server Info: > Getattr Setattr Lookup Readlink Read Write Create >Remove > 97818 311 107539 0 12018551 25266454 858 567 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus >Access > 296 0 0 0 0 0 427 7216 > Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId >SetClIdCf > 0 2232 0 0 0 0 0 0 > Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lo= ck > 0 0 0 0 0 0 0 0 > LockT LockU Close Verify NVerify PutFH PutPubFH >PutRootFH > 0 0 0 0 0 0 0 0 > Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create > 0 0 0 0 0 0 >Server: >Retfailed Faults Clients > 0 0 0 >OpenOwner Opens LockOwner Locks Delegs > 0 0 0 0 0 >Server Cache Stats: > Inprog Idem Non-idem Misses CacheSize TCPPeak > 0 0 0 37502946 94 592 > > >Is there any way I could decreas number of misses ? Break your network badly;-) You don't want hits for a Duplicate Request Cache (DRC). It doesn't improve= performance, but improves correctness by avoiding an RPC from being performed multiple t= imes on the server. (ie. Hits are BAD. Since the first 3 numbers are 0, there are 0= hits and that is good. A DRC is mainly for UDP mounts where the client retries the RPC too a= gressively. For TCP, RPCs are only retried when a client does a TCP reconnect.) Disabling the DRC will reduce the CPU overheads, but does put your data at = risk if/when a client does a TCP reconnect. You can disable the DRC for TCP via: sysctl vfs.nfsd.cachetcp=3D0 OR sysctl vfs.nfsd.tcphighwater=3D100000 allows the cache to grow larger, reducing the CPU overheads that occur when= it does housekeeping of it. (Trading CPU for kernel memory use.) Again, disabling the cache will reduce CPU overheads, but does put your dat= a at risk if/when a client does a TCP reconnect and resends outstanding RPCs to = the server. I doubt any of this DRC tuning will affect your hangs. Good luck with it, rick
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YTXPR01MB0189EE88A51139BD8F31DA56DDD60>