From owner-freebsd-stable@freebsd.org Thu Sep 24 12:32:14 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8275AA08059 for ; Thu, 24 Sep 2015 12:32:14 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 374ED14EC for ; Thu, 24 Sep 2015 12:32:13 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:3OCK/hzm8BQX91XXCy+O+j09IxM/srCxBDY+r6Qd0OsSIJqq85mqBkHD//Il1AaPBtWHra4ZwLGL+4nbGkU+or+5+EgYd5JNUxJXwe43pCcHRPC/NEvgMfTxZDY7FskRHHVs/nW8LFQHUJ2mPw6anHS+4HYoFwnlMkItf6KuStKU05n8jL360qaQSjsLrQL1Wal1IhSyoFeZnegtqqwmFJwMzADUqGBDYeVcyDAgD1uSmxHh+pX4p8Y7oGwD888n7NNKBKXmY7xqCvtcDS86KCY7/sDmvwLPCwyV6TwZW2QSlxNORAzE9w37WJn29SXgu+d3wyXfJtH/R7Q5CgiluolxQRnrwCsKfxQ+7CmDjs1rkLlzux+ovRd/0sjSbZ3DZ9RkeaaIR9IRRiJkV81SUyFEStemaoIEDO4MOM5FqIbgql8WrV21DF//V6vU1jZUiyqujuUB2OM7HFSe0Q== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DkAQAS7ANW/61jaINdg3hpBoMkuhQBDYFwCoUvSgKCBhQBAQEBAQEBAYEJgh2CCAEBBAEBASAEJyALEAIBCBgCAg0ZAgInAQkmAgQIBwQBGQMEiA0NtzSUOgEBAQEGAQEBAQEBARuBIoVRhH2EOwEBBRc0B4JpgUMFjH2IaoUShRWEM0aHEY4Xg2sCHwEBQoIRHIFwIjMHiCY6gQUBAQE X-IronPort-AV: E=Sophos;i="5.17,580,1437451200"; d="scan'208";a="240830023" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 24 Sep 2015 08:32:06 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 6B57215F56E; Thu, 24 Sep 2015 08:32:06 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id NBu1j36h7mnL; Thu, 24 Sep 2015 08:32:05 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id A7EC115F577; Thu, 24 Sep 2015 08:32:05 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id TXSaUU7roO_A; Thu, 24 Sep 2015 08:32:05 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 8D5F815F571; Thu, 24 Sep 2015 08:32:05 -0400 (EDT) Date: Thu, 24 Sep 2015 08:32:05 -0400 (EDT) From: Rick Macklem To: Frank de Bot Cc: freebsd-stable@FreeBSD.org Message-ID: <1887696626.8730412.1443097925392.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <5603AE3D.5090407@searchy.net> References: <1031959302.30289198.1430594914473.JavaMail.root@uoguelph.ca> <5603AE3D.5090407@searchy.net> Subject: Re: kernel process [nfscl] high cpu MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.10] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: kernel process [nfscl] high cpu Thread-Index: hQwXV3z/IkQwXxZDZPSkulSDDvKEXQ== X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Sep 2015 12:32:14 -0000 Frank de Bot wrote: > Rick Macklem wrote: > > Frank de Bot wrote: > >> Hi, > >> > >> On a 10.1-RELEASE-p9 server I have several NFS mounts used for a > >> jail. > >> Because it's a server only to test, there is a low load. But the > >> [nfscl] > >> process is hogging a CPU after a while. This happens pretty fast, > >> within > >> 1 or 2 days. I'm noticing the high CPU of the process when I want to > >> do > >> some test after a little while (those 1 or 2 days). > >> > >> My jail.conf look like: > >> > >> exec.start = "/bin/sh /etc/rc"; > >> exec.stop = "/bin/sh /etc/rc.shutdown"; > >> exec.clean; > >> mount.devfs; > >> exec.consolelog = "/var/log/jail.$name.log"; > >> #mount.fstab = "/usr/local/etc/jail.fstab.$name"; > >> > >> test01 { > >> host.hostname = "test01_hosting"; > >> ip4.addr = somepublicaddress; > >> ip4.addr += someprivateaddress; > >> > >> mount = "10.13.37.2:/tank/hostingbase /opt/jails/test01 > >> nfs nfsv4,minorversion=1,pnfs,ro,noatime 0 0"; > >> mount += "10.13.37.2:/tank/hosting/test > >> /opt/jails/test01/opt nfs nfsv4,minorversion=1,pnfs,noatime > >> 0 0"; > >> > >> path = "/opt/jails/test01"; > >> } > >> > >> Last test was with NFS 4.1, I also worked with NFS 4.(0) with the > >> same > >> result. In the readonly nfs share there are symbolic links point to > >> the > >> read-write share for logging, storing .run files, etc. When I monitor > >> my > >> network interface with tcpdump, there is little nfs traffic, only > >> when I > >> do try to access the shares there is activity. > >> > >> What is causing nfscl to run around in circles, hogging the CPU (it > >> makes the system slow to respond too) or how can I found out what's > >> the > >> cause? > >> > > Well, the nfscl does server->client RPCs referred to as callbacks. I > > have no idea what the implications of running it in a jail is, but I'd > > guess that these server->client RPCs get blocked somehow, etc... > > (The NFSv4.0 mechanism requires a separate IP address that the server > > can connect to on the client. For NFSv4.1, it should use the same > > TCP connection as is used for the client->server RPCs. The latter > > seems like it should work, but there is probably some glitch.) > > > > ** Just run without the nfscl daemon (it is only needed for delegations or > > pNFS). > > How can I disable the nfscl daemon? > Well, the daemon for the callbacks is called nfscbd. You should check via "ps ax", to see if you have it running. (For NFSv4.0 you probably don't want it running, but for NFSv4.1 you do need it. pNFS won't work at all without it, but unless you have a server that supports pNFS, it won't work anyhow. Unless your server is a clustered Netapp Filer, you should probably not have the "pnfs" option.) To run the "nfscbd" daemon you can set: nfscbd_enable="TRUE" in your /etc/rc.conf will start it on boot. Alternately, just type "nfscbd" as root. The "nfscl" thread is always started when an NFSv4 mount is done. It does an assortment of housekeeping things, including a Renew op to make sure the lease doesn't expire. If for some reason the jail blocks these Renew RPCs, it will try to do them over and over and ... because having the lease expire is bad news for NFSv4. How could you tell? Well, capturing packets between the client and server, then looking at them in wireshark is probably the only way. (Or maybe a large count for Renew in the output from "nfsstat -e".) "nfscbd" is optional for NFSv4.0. Without it, you simply don't do callbacks/delegations. For NFSv4.1 it is pretty much required, but doesn't need a separate server->client TCP connection. --> I'd enable it for NFSv4.1, but disable it for NFSv4.0 at least as a starting point. And as I said before, none of this is tested within jails, so I have no idea what effect the jails have. Someone who understands jails might have some insight w.r.t. this? rick > > > > > Since a big Netapp filer (the cluster ones) are about the only servers > > that currently support pNFS (no FreeBSD server support yet), you can > > probably forget about pNFS (I'd get rid of the "pnfs" mount option). > > It also won't work unless this callback path is working. > > > > As for delegations, they aren't required for NFSv4.[0-1] to work correctly > > and aren't enabled by default on the FreeBSD server. > > --> Running without the nfscl daemon will just ensure no delegations > > are issued, even if enabled on the server. > > > > rick > > > >> > >> Regards, > >> > >> Frank de Bot > >> _______________________________________________ > >> freebsd-stable@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable > >> To unsubscribe, send any mail to > >> "freebsd-stable-unsubscribe@freebsd.org" > >> > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > > > >