From owner-freebsd-stable@FreeBSD.ORG Thu Aug 12 02:00:29 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5DE01065670 for ; Thu, 12 Aug 2010 02:00:28 +0000 (UTC) (envelope-from alan.bryan@yahoo.com) Received: from web50506.mail.re2.yahoo.com (web50506.mail.re2.yahoo.com [206.190.38.82]) by mx1.freebsd.org (Postfix) with SMTP id 7EFC68FC0A for ; Thu, 12 Aug 2010 02:00:28 +0000 (UTC) Received: (qmail 89156 invoked by uid 60001); 12 Aug 2010 02:00:27 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1281578427; bh=wyXiJh/OXP2O7wlhNO6NoRcXlXRGBmDWkAW/FwsCxNw=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=fJQOlPGgz19fMMaJAevuMP/MOgqxPLP88NmoVVJyXPwwvSnJ+CDmgbBEViZTm0366KYLIxe5xrkYtE48ozBgHMrUUUm/HMPNrRqIhUfjL8o6+DWCIvBXwUowYPRsXBp1+Z57d0s1x39WM+0NDGIqDLyAJTYPXzVArbUYvov6Vyo= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=EPdsKvGN3EiTAv3LrkvR2/5HpcE/GQznazT7E1uOw5GBEzOH2jkT3PHhnM90EFh0yzFmR7uw0Z9nO/4FHyJVDsLWpyH8XDy42P2kFUuRNPT5o2PlkExC6FuAR8D+/NkJgSNuAoKkkvotBso8p2snvQ0WdjLiJM/DKbeqV9A1WzU=; Message-ID: <719967.84603.qm@web50506.mail.re2.yahoo.com> X-YMail-OSG: KkkzfGEVM1nx4YD72SmSPTtFiXFAzSAmAVwYqBSxAUxmJW. KuVq6_IIvFc1IxjcNsKDhDegJy98v_loGwUpevLF.wxctXvWGfmIj8cXBUr_ 2yn25RVlZDQGvKiric_smo3EQrkaU9mnNkFT_T_6.8e40g54cOf7UK4dzu7x V9CAs9_rIyvpd0eP6nKmhTkuuJxqwLlqy8dhqJ1EPgWMH.8gE.8RfKWVU1EE oH5n3_chLUJDobdijdTSXe34BalLHWzhMYA864XVBDRgfm7.RohQ_sPOxkGF UxVeS_3zAVZ2eytUIrHwHmxOwAplH17cg6gVWragTCgZFRsf0 Received: from [99.24.6.121] by web50506.mail.re2.yahoo.com via HTTP; Wed, 11 Aug 2010 19:00:27 PDT X-Mailer: YahooMailClassic/11.3.2 YahooMailWebService/0.8.105.279950 Date: Wed, 11 Aug 2010 19:00:27 -0700 (PDT) From: alan bryan To: Rick Macklem In-Reply-To: <1347461186.514707.1281494117995.JavaMail.root@erie.cs.uoguelph.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: FreeBSD 8.1-Release NFSD hang in rc_lo state X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Aug 2010 02:00:29 -0000 --- On Tue, 8/10/10, Rick Macklem wrote:=0A=0A> From= : Rick Macklem =0A> Subject: Re: FreeBSD 8.1-Release = NFSD hang in rc_lo state=0A> To: "alan bryan" =0A> Cc= : freebsd-stable@freebsd.org=0A> Date: Tuesday, August 10, 2010, 7:35 PM=0A= > > I'm doing some testing with=0A> loading up a new storage server. I have= =0A> > couple ZFS filesystems exported via NFS over UDP.=0A> > =0A> > I hav= e a client machine (8.1 also) that has mounted=0A> those filesystems=0A> > = along with some test PHP scripts that are doing a ton=0A> of=0A> > read/wri= te/fstat operations to load it up.=0A> > =0A> > If I ctrl-C to kill the scr= ipts on the client I found=0A> that I can end=0A> > up with nfsd on the ser= ver stuck at 100% CPU and in=0A> the rc_lo state.=0A> > /etc/rc.d/nfsd rest= art does nothing.=0A> > =0A> > # nfsstat -s -w 1 -W=0A> > GtAttr Lookup Rdl= ink Read Write Rename Access Rddir=0A> > 0 0 0 0 0 0 0 0=0A> > 0 0 0 0 0 0 = 0 0=0A> > 0 0 0 0 0 0 0 0=0A> > =0A> > =0A> > Client complaining with:=0A> = > kernel: nfs server 192.168.1.2:/tank/alantest2: not=0A> responding=0A> > = =0A> > =0A> > =0A> > Top on the server:=0A> > =0A> > last pid: 4776; load a= verages: 1.24, 1.15, 1.15 up=0A> 0+23:04:46 18:25:26=0A> > 53 processes: 2 = running, 35 sleeping, 16 lock=0A> > CPU: 0.0% user, 0.0% nice, 25.0% system= , 0.0%=0A> interrupt, 75.0% idle=0A> > Mem: 21M Active, 5812K Inact, 2029M = Wired, 88K Cache,=0A> 8592K Buf, 5857M=0A> > Free=0A> > Swap: 3942M Total, = 3942M Free=0A> > =0A> > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU=0A= > COMMAND=0A> > 922 root 44 0 5804K 1820K CPU2 2 63:52 100.00% {nfsd:=0A> s= ervice}=0A> > 922 root 45 0 5804K 1820K *rc_lo 1 30:23 0.00% {nfsd:=0A> ser= vice}=0A> > 922 root 48 0 5804K 1820K rpcsvc 1 30:20 0.00% {nfsd:=0A> servi= ce}=0A> > 922 root 46 0 5804K 1820K *rc_lo 1 30:17 0.00% {nfsd:=0A> service= }=0A> > 922 root 48 0 5804K 1820K rpcsvc 1 30:17 0.00% {nfsd:=0A> service}= =0A> > 922 root 46 0 5804K 1820K rpcsvc 1 30:13 0.00% {nfsd:=0A> service}= =0A> > 922 root 45 0 5804K 1820K *rc_lo 2 30:13 0.00% {nfsd:=0A> service}= =0A> > 922 root 45 0 5804K 1820K *rc_lo 3 30:12 0.00% {nfsd:=0A> service}= =0A> > 922 root 44 0 5804K 1820K *rc_lo 0 30:10 0.00% {nfsd:=0A> service}= =0A> > 922 root 46 0 5804K 1820K *rc_lo 2 30:07 0.00% {nfsd:=0A> service}= =0A> > 922 root 46 0 5804K 1820K *rc_lo 0 30:03 0.00% {nfsd:=0A> service}= =0A> > 922 root 45 0 5804K 1820K *rc_lo 1 30:03 0.00% {nfsd:=0A> service}= =0A> > 922 root 46 0 5804K 1820K *rc_lo 2 30:03 0.00% {nfsd:=0A> service}= =0A> > 922 root 45 0 5804K 1820K *rc_lo 2 30:02 0.00% {nfsd:=0A> service}= =0A> > 922 root 45 0 5804K 1820K *rc_lo 1 29:59 0.00% {nfsd:=0A> master}=0A= > > 922 root 45 0 5804K 1820K *rc_lo 2 29:54 0.00% {nfsd:=0A> service}=0A> = > 922 root 46 0 5804K 1820K *rc_lo 0 29:54 0.00% {nfsd:=0A> service}=0A> > = 922 root 46 0 5804K 1820K *rc_lo 0 29:52 0.00% {nfsd:=0A> service}=0A> > 92= 2 root 47 0 5804K 1820K *rc_lo 2 29:46 0.00% {nfsd:=0A> service}=0A> > 922 = root 46 0 5804K 1820K *rc_lo 1 29:44 0.00% {nfsd:=0A> service}=0A> > =0A> = =0A> I have a patch that I think might fix this. (Essentially=0A> the same= =0A> bug was posted on another list recently.) I'll post it to=0A> Alan=0A>= separately, but if anyone else wants to try it, it will be=0A> at=0A> =A0= =A0=A0http://people.freebsd.org/~rmacklem/freebsd8.1-patches/replay.patch= =0A> in a few minutes.=0A> =0A> rick=0A> =0A> =0A> ________________________= _______________________=0A> freebsd-stable@freebsd.org=0A> mailing list=0A>= http://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A> To unsubscrib= e, send any mail to "freebsd-stable-unsubscribe@freebsd.org"=0A> =0A=0A=0A= =0A=0A=0AI applied this patch and have been unable to reproduce the hang si= nce.=0A=0ARick, Thanks for an awesomely quick response!=0A=0AThis will go i= nto 8-Stable at some point?=0A=0A--Alan=0A=0A=0A=0A=0A=0A