From owner-freebsd-fs@freebsd.org Sun Oct 23 18:43:23 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32836C1E411 for ; Sun, 23 Oct 2016 18:43:23 +0000 (UTC) (envelope-from marek.salwerowicz@misal.pl) Received: from mail3.misal.pl (mail3.misal.pl [83.19.131.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 97171268 for ; Sun, 23 Oct 2016 18:43:22 +0000 (UTC) (envelope-from marek.salwerowicz@misal.pl) Received: from localhost (mail3.misal.pl [127.0.0.1]) by mail3.misal.pl (Postfix) with ESMTP id 81A34317E; Sun, 23 Oct 2016 20:43:13 +0200 (CEST) X-Virus-Scanned: amavisd X-Spam-Flag: NO X-Spam-Score: -3 X-Spam-Level: X-Spam-Status: No, score=-3 tagged_above=-9999 required=9 tests=[ALL_TRUSTED=-1, BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1] autolearn=ham autolearn_force=no Authentication-Results: mail3.misal.pl (amavisd-new); dkim=pass (1024-bit key) header.d=misal.pl Received: from mail3.misal.pl ([127.0.0.1]) by localhost (mail3.misal.pl [127.0.0.1]) (amavisd-new, port 10024) with LMTP id Z_ohrn6wrvzd; Sun, 23 Oct 2016 20:43:12 +0200 (CEST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail3.misal.pl 18062315D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=misal.pl; s=misal.pl; t=1477248192; bh=2/ZvkbEHmitJ5idwdroRkB9zmA4gTNv++m1R4H8MF40=; h=Subject:To:References:From:Date:In-Reply-To:From; b=ymL8WcJl18c1XV1CEkRIsmCStlyfKFIvAMDehO8+fmDPQsG6a7I8XNfw8APtTmLC9 9PC/DYYkneagcoMhtLJjX9FgGQDAkKAyQiQaxJsQpclvafTRr6eZJRdougf88kqiC4 LH5ecQGDEC+xV1ZERXfY6IjCeS0DagVh1O2cgs0s= Subject: Re: ZFS - NFS server for VMware ESXi issues To: Rick Macklem , "freebsd-fs@freebsd.org" References: <930df17b-8db8-121a-a24b-b4909b8162dc@misal.pl> From: Marek Salwerowicz Message-ID: Date: Sun, 23 Oct 2016 20:43:02 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Oct 2016 18:43:23 -0000 Hi Rick, W dniu 2016-10-21 o 23:47, Rick Macklem pisze: > > > Btw, about the only area of the NFS server that might need tuning is > the DRC and > this doesn't suggest that. If you "nfsstat -e -s" on the server and > see large #s for > the last line under "Server Cache Stats:" there are tunables that can > be used. > I'd also suggest you capture the output of "ps axHl" on the server > when it happens > again, which tells you what all the nfsd threads are up to. I checked the #ps axHL | grep nfs now: http://pastebin.com/x9LTN0nn it looks like I have ~64 threads of nfs each cosuming ~one hour of CPU time. That corresponds to: # ps axl | grep nfs UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMAN 0 1948 1 0 28 0 24632 5832 select Is - 0:00.10 nfsd: master (nfsd) 0 1949 1948 0 24 0 12344 4132 rpcsvc I - 66:56.42 nfsd: server (nfsd) is it OK if threads are not being "recuperated" ? The NFS statistics are as follows: # nfsstat -e -s Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 97818 311 107539 0 12018551 25266454 858 567 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 296 0 0 0 0 0 427 7216 Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId SetClIdCf 0 2232 0 0 0 0 0 0 Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lock 0 0 0 0 0 0 0 0 LockT LockU Close Verify NVerify PutFH PutPubFH PutRootFH 0 0 0 0 0 0 0 0 Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create 0 0 0 0 0 0 Server: Retfailed Faults Clients 0 0 0 OpenOwner Opens LockOwner Locks Delegs 0 0 0 0 0 Server Cache Stats: Inprog Idem Non-idem Misses CacheSize TCPPeak 0 0 0 37502946 94 592 Is there any way I could decreas number of misses ? The server is up now for almost three days. Thanks for any hints ! Marek