From owner-freebsd-fs@freebsd.org Mon Oct 24 06:55:21 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32BF2C1FDAD for ; Mon, 24 Oct 2016 06:55:21 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0053.outbound.protection.outlook.com [157.56.110.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C65102C5 for ; Mon, 24 Oct 2016 06:55:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM (10.165.218.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.679.12; Sun, 23 Oct 2016 22:21:07 +0000 Received: from YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) by YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM ([10.165.218.133]) with mapi id 15.01.0679.015; Sun, 23 Oct 2016 22:21:07 +0000 From: Rick Macklem To: Marek Salwerowicz , "freebsd-fs@freebsd.org" Subject: Re: ZFS - NFS server for VMware ESXi issues Thread-Topic: ZFS - NFS server for VMware ESXi issues Thread-Index: AQHSK3xZ74x1v1Hl9kyKv24uLlFAmaCzbiXvgAL1cwCAADhVLg== Date: Sun, 23 Oct 2016 22:21:06 +0000 Message-ID: References: <930df17b-8db8-121a-a24b-b4909b8162dc@misal.pl> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rmacklem@uoguelph.ca; x-ms-office365-filtering-correlation-id: 12e136da-df1d-4e41-8b27-08d3fb92e255 x-microsoft-exchange-diagnostics: 1; YTXPR01MB0189; 7:P2O9FpKE3reGwg53jvHcwTid9iaZU3qOr6CGOC+9ROK1C3pnPCljX/tDEmDddotl0/RgtkO3jmw7xo/TDAG0Y9H6HcxB8NIcGk9xdlHjD2ReROc+QldX3az0j/EGIuREmmPMnJE8Nj6WIdfO3oHlUJUjKPD2JDVJ/BjQn4jiGdPTIgLK1hXXtkc24phUYhK6bfNgIFGLuTINu/OTGCP3jGPiY0+GHcx2ZCZCgtwq89Q6/uudQgXChKG+CIM8pfK7zD+0nJqN2Dr8Bo4l8XUcKT5jQCpnZXGMuqw1Rgjqss0fn7IBd90LrOuvKeLMC9asi8yrZ5Ps/Ki4f3kzBXp4RZLxAn4KWumXp89z7FkLZtE= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:YTXPR01MB0189; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(158342451672863); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(6040176)(2401047)(5005006)(8121501046)(10201501046)(3002001)(6043046)(6042046); SRVR:YTXPR01MB0189; BCL:0; PCL:0; RULEID:; SRVR:YTXPR01MB0189; x-forefront-prvs: 0104247462 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(7916002)(189002)(377424004)(199003)(24454002)(106116001)(2501003)(19580395003)(87936001)(81166006)(81156014)(77096005)(74316002)(305945005)(189998001)(1720100001)(8936002)(97736004)(107886002)(5001770100001)(5002640100001)(105586002)(106356001)(8676002)(15975445007)(122556002)(7846002)(92566002)(2906002)(4001150100001)(586003)(74482002)(86362001)(575784001)(68736007)(9686002)(2900100001)(102836003)(11100500001)(10400500002)(15395725005)(3280700002)(2950100002)(3660700001)(7696004)(54356999)(50986999)(76176999)(101416001)(33656002)(5660300001)(10126625002)(522954003); DIR:OUT; SFP:1101; SCL:1; SRVR:YTXPR01MB0189; H:YTXPR01MB0189.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Oct 2016 22:21:06.9675 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTXPR01MB0189 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Oct 2016 06:55:21 -0000 Marek Salwerowicz wrote: >Hi Rick, > >W dniu 2016-10-21 o 23:47, Rick Macklem pisze: >> >> >> Btw, about the only area of the NFS server that might need tuning is >> the DRC and >> this doesn't suggest that. If you "nfsstat -e -s" on the server and >> see large #s for >> the last line under "Server Cache Stats:" there are tunables that can >> be used. >> I'd also suggest you capture the output of "ps axHl" on the server >> when it happens >> again, which tells you what all the nfsd threads are up to. > >I checked the >#ps axHL | grep nfs >now: >http://pastebin.com/x9LTN0nn What is there now is normal. "rpcsvc" just means the thread is waiting for = an RPC request from a client. This info might be useful when the server is hung/li= velocked. > >it looks like I have ~64 threads of nfs each cosuming ~one hour of CPU tim= e. If one hour of CPU seems excessive for you, you can disable the DRC. See below w.r.t. this. >That corresponds to: ># ps axl | grep nfs >UID PID PPID CPU PRI NI VSZ RSS MWCHAN STAT TT TIME COMMA= N > 0 1948 1 0 28 0 24632 5832 select Is - 0:00.10 >nfsd: master (nfsd) > 0 1949 1948 0 24 0 12344 4132 rpcsvc I - 66:56.42 >nfsd: server (nfsd) > >is it OK if threads are not being "recuperated" ? Not sure what you mean by this, but newer FreeBSD systems have minthreads a= nd maxthreads options on the nfsd to set lower/upper bounds on the # of thread= s. To be honest, having too many threads doesn't have much negative impact, so= I wouldn't worry about having too many. >The NFS statistics are as follows: ># nfsstat -e -s > >Server Info: > Getattr Setattr Lookup Readlink Read Write Create >Remove > 97818 311 107539 0 12018551 25266454 858 567 > Rename Link Symlink Mkdir Rmdir Readdir RdirPlus >Access > 296 0 0 0 0 0 427 7216 > Mknod Fsstat Fsinfo PathConf Commit LookupP SetClId >SetClIdCf > 0 2232 0 0 0 0 0 0 > Open OpenAttr OpenDwnGr OpenCfrm DelePurge DeleRet GetFH Lo= ck > 0 0 0 0 0 0 0 0 > LockT LockU Close Verify NVerify PutFH PutPubFH >PutRootFH > 0 0 0 0 0 0 0 0 > Renew RestoreFH SaveFH Secinfo RelLckOwn V4Create > 0 0 0 0 0 0 >Server: >Retfailed Faults Clients > 0 0 0 >OpenOwner Opens LockOwner Locks Delegs > 0 0 0 0 0 >Server Cache Stats: > Inprog Idem Non-idem Misses CacheSize TCPPeak > 0 0 0 37502946 94 592 > > >Is there any way I could decreas number of misses ? Break your network badly;-) You don't want hits for a Duplicate Request Cache (DRC). It doesn't improve= performance, but improves correctness by avoiding an RPC from being performed multiple t= imes on the server. (ie. Hits are BAD. Since the first 3 numbers are 0, there are 0= hits and that is good. A DRC is mainly for UDP mounts where the client retries the RPC too a= gressively. For TCP, RPCs are only retried when a client does a TCP reconnect.) Disabling the DRC will reduce the CPU overheads, but does put your data at = risk if/when a client does a TCP reconnect. You can disable the DRC for TCP via: sysctl vfs.nfsd.cachetcp=3D0 OR sysctl vfs.nfsd.tcphighwater=3D100000 allows the cache to grow larger, reducing the CPU overheads that occur when= it does housekeeping of it. (Trading CPU for kernel memory use.) Again, disabling the cache will reduce CPU overheads, but does put your dat= a at risk if/when a client does a TCP reconnect and resends outstanding RPCs to = the server. I doubt any of this DRC tuning will affect your hangs. Good luck with it, rick