From owner-freebsd-stable@freebsd.org Thu Sep 24 21:24:58 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C0F6A0866F for ; Thu, 24 Sep 2015 21:24:58 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id 098611C2D for ; Thu, 24 Sep 2015 21:24:57 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) IronPort-PHdr: 9a23:kHJCgxHpz5fVur26sc/fLJ1GYnF86YWxBRYc798ds5kLTJ75os2wAkXT6L1XgUPTWs2DsrQf27aQ7/+rATVIyK3CmU5BWaQEbwUCh8QSkl5oK+++Imq/EsTXaTcnFt9JTl5v8iLzG0FUHMHjew+a+SXqvnYsExnyfTB4Ov7yUtaLyZ/ni6bupNaLOk1hv3mUX/BbFF2OtwLft80b08NJC50a7V/3mEZOYPlc3mhyJFiezF7W78a0+4N/oWwL46pyv50IbaKvUb4xS78QADluGWcprJnvtALfViOU63IGWWUH1BxFH16Wwgv9W8LLsyD5/s900yqeMMi+GaoxUD+h66puYALvhzoKMyY5tmre3J8jxJlHqQ6s8kQsi7XfZ5uYYaJz X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A2DkAQC+aQRW/61jaINdg3hpBoMkuhkBDYFwCoUvSgKCBRQBAQEBAQEBAYEJgh2CBwEBAQMBAQEBIAQnIAsFCwIBCBgCAg0ZAgInAQkmAgQIBwQBGQMEiAUIDbdflCoBAQEBAQEEAQEBAQEBAQEWBIEihVGEfYQ7AQEFFzQHgmmBQwWMfYhqhRKFFYQzh1eOF4NrAh8BAUKCERyBcCIzB4gmOoEFAQEB X-IronPort-AV: E=Sophos;i="5.17,583,1437451200"; d="scan'208";a="240944090" Received: from nipigon.cs.uoguelph.ca (HELO zcs1.mail.uoguelph.ca) ([131.104.99.173]) by esa-annu.net.uoguelph.ca with ESMTP; 24 Sep 2015 17:24:37 -0400 Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 5739415F56E; Thu, 24 Sep 2015 17:24:37 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id XJXGc25q0EyT; Thu, 24 Sep 2015 17:24:36 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id 1AF8115F571; Thu, 24 Sep 2015 17:24:36 -0400 (EDT) X-Virus-Scanned: amavisd-new at zcs1.mail.uoguelph.ca Received: from zcs1.mail.uoguelph.ca ([127.0.0.1]) by localhost (zcs1.mail.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id fEBKVK1TJTPx; Thu, 24 Sep 2015 17:24:35 -0400 (EDT) Received: from zcs1.mail.uoguelph.ca (zcs1.mail.uoguelph.ca [172.17.95.18]) by zcs1.mail.uoguelph.ca (Postfix) with ESMTP id CECA815F56E; Thu, 24 Sep 2015 17:24:35 -0400 (EDT) Date: Thu, 24 Sep 2015 17:24:35 -0400 (EDT) From: Rick Macklem To: Frank de Bot Cc: freebsd-stable@FreeBSD.org Message-ID: <486700591.10025468.1443129875818.JavaMail.zimbra@uoguelph.ca> In-Reply-To: <5603FC85.8070105@searchy.net> References: <1031959302.30289198.1430594914473.JavaMail.root@uoguelph.ca> <5603AE3D.5090407@searchy.net> <1887696626.8730412.1443097925392.JavaMail.zimbra@uoguelph.ca> <5603FC85.8070105@searchy.net> Subject: Re: kernel process [nfscl] high cpu MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.95.12] X-Mailer: Zimbra 8.0.9_GA_6191 (ZimbraWebClient - FF34 (Win)/8.0.9_GA_6191) Thread-Topic: kernel process [nfscl] high cpu Thread-Index: viQB5VmXHsp0yixVTW/D/BAmUeFFMQ== X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Sep 2015 21:24:58 -0000 Frank de Bot wrote: > Rick Macklem wrote: > > Frank de Bot wrote: > >> Rick Macklem wrote: > >>> Frank de Bot wrote: > >>>> Hi, > >>>> > >>>> On a 10.1-RELEASE-p9 server I have several NFS mounts used for a > >>>> jail. > >>>> Because it's a server only to test, there is a low load. But the > >>>> [nfscl] > >>>> process is hogging a CPU after a while. This happens pretty fast, > >>>> within > >>>> 1 or 2 days. I'm noticing the high CPU of the process when I want to > >>>> do > >>>> some test after a little while (those 1 or 2 days). > >>>> > >>>> My jail.conf look like: > >>>> > >>>> exec.start = "/bin/sh /etc/rc"; > >>>> exec.stop = "/bin/sh /etc/rc.shutdown"; > >>>> exec.clean; > >>>> mount.devfs; > >>>> exec.consolelog = "/var/log/jail.$name.log"; > >>>> #mount.fstab = "/usr/local/etc/jail.fstab.$name"; > >>>> > >>>> test01 { > >>>> host.hostname = "test01_hosting"; > >>>> ip4.addr = somepublicaddress; > >>>> ip4.addr += someprivateaddress; > >>>> > >>>> mount = "10.13.37.2:/tank/hostingbase /opt/jails/test01 > >>>> nfs nfsv4,minorversion=1,pnfs,ro,noatime 0 0"; > >>>> mount += "10.13.37.2:/tank/hosting/test > >>>> /opt/jails/test01/opt nfs nfsv4,minorversion=1,pnfs,noatime > >>>> 0 0"; > >>>> > >>>> path = "/opt/jails/test01"; > >>>> } > >>>> > >>>> Last test was with NFS 4.1, I also worked with NFS 4.(0) with the > >>>> same > >>>> result. In the readonly nfs share there are symbolic links point to > >>>> the > >>>> read-write share for logging, storing .run files, etc. When I monitor > >>>> my > >>>> network interface with tcpdump, there is little nfs traffic, only > >>>> when I > >>>> do try to access the shares there is activity. > >>>> > >>>> What is causing nfscl to run around in circles, hogging the CPU (it > >>>> makes the system slow to respond too) or how can I found out what's > >>>> the > >>>> cause? > >>>> > >>> Well, the nfscl does server->client RPCs referred to as callbacks. I > >>> have no idea what the implications of running it in a jail is, but I'd > >>> guess that these server->client RPCs get blocked somehow, etc... > >>> (The NFSv4.0 mechanism requires a separate IP address that the server > >>> can connect to on the client. For NFSv4.1, it should use the same > >>> TCP connection as is used for the client->server RPCs. The latter > >>> seems like it should work, but there is probably some glitch.) > >>> > >>> ** Just run without the nfscl daemon (it is only needed for delegations > >>> or > >>> pNFS). > >> > >> How can I disable the nfscl daemon? > >> > > Well, the daemon for the callbacks is called nfscbd. > > You should check via "ps ax", to see if you have it running. > > (For NFSv4.0 you probably don't want it running, but for NFSv4.1 you > > do need it. pNFS won't work at all without it, but unless you have a > > server that supports pNFS, it won't work anyhow. Unless your server is > > a clustered Netapp Filer, you should probably not have the "pnfs" option.) > > > > To run the "nfscbd" daemon you can set: > > nfscbd_enable="TRUE" > > in your /etc/rc.conf will start it on boot. > > Alternately, just type "nfscbd" as root. > > > > The "nfscl" thread is always started when an NFSv4 mount is done. It does > > an assortment of housekeeping things, including a Renew op to make sure the > > lease doesn't expire. If for some reason the jail blocks these Renew RPCs, > > it will try to do them over and over and ... because having the lease > > expire is bad news for NFSv4. How could you tell? > > Well, capturing packets between the client and server, then looking at them > > in wireshark is probably the only way. (Or maybe a large count for Renew > > in the output from "nfsstat -e".) > > > > "nfscbd" is optional for NFSv4.0. Without it, you simply don't do > > callbacks/delegations. > > For NFSv4.1 it is pretty much required, but doesn't need a separate > > server->client TCP > > connection. > > --> I'd enable it for NFSv4.1, but disable it for NFSv4.0 at least as a > > starting point. > > > > And as I said before, none of this is tested within jails, so I have no > > idea > > what effect the jails have. Someone who understands jails might have some > > insight > > w.r.t. this? > > > > rick > > > > Since last time I haven't tried to use pnfs and just sticked with > nfsv4.0. nfscbd is not running. The server is now running 10.2. The > number of renews is not very high (56k, getattr is for example 283M) > View with wireshark, renew calls look good ,the nfs status is ok. > > Is there a way to know what [nfscl] is active with? > Btw, I'm an old-school debugger, which means I'd add a bunch of "printf()s" to the function called nfscl_renewthread() in sys/fs/nfsclient/nfs_clstate.c. (That's the nfscl thread. It should only do the "for(;;)" loop once/sec, but if you get lots of loop iterations, you might be able to isolate why via printf()s.) You did say it was a test system. Good luck with it, rick > I do understand nfs + jails could have issues, but I like to understand > them. > > > Frank > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >