Date: Tue, 10 Aug 2010 22:35:17 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: alan bryan <alan.bryan@yahoo.com> Cc: freebsd-stable@freebsd.org Subject: Re: FreeBSD 8.1-Release NFSD hang in rc_lo state Message-ID: <1347461186.514707.1281494117995.JavaMail.root@erie.cs.uoguelph.ca> In-Reply-To: <881208.18587.qm@web50506.mail.re2.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> I'm doing some testing with loading up a new storage server. I have
> couple ZFS filesystems exported via NFS over UDP.
>
> I have a client machine (8.1 also) that has mounted those filesystems
> along with some test PHP scripts that are doing a ton of
> read/write/fstat operations to load it up.
>
> If I ctrl-C to kill the scripts on the client I found that I can end
> up with nfsd on the server stuck at 100% CPU and in the rc_lo state.
> /etc/rc.d/nfsd restart does nothing.
>
> # nfsstat -s -w 1 -W
> GtAttr Lookup Rdlink Read Write Rename Access Rddir
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0
>
>
> Client complaining with:
> kernel: nfs server 192.168.1.2:/tank/alantest2: not responding
>
>
>
> Top on the server:
>
> last pid: 4776; load averages: 1.24, 1.15, 1.15 up 0+23:04:46 18:25:26
> 53 processes: 2 running, 35 sleeping, 16 lock
> CPU: 0.0% user, 0.0% nice, 25.0% system, 0.0% interrupt, 75.0% idle
> Mem: 21M Active, 5812K Inact, 2029M Wired, 88K Cache, 8592K Buf, 5857M
> Free
> Swap: 3942M Total, 3942M Free
>
> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
> 922 root 44 0 5804K 1820K CPU2 2 63:52 100.00% {nfsd: service}
> 922 root 45 0 5804K 1820K *rc_lo 1 30:23 0.00% {nfsd: service}
> 922 root 48 0 5804K 1820K rpcsvc 1 30:20 0.00% {nfsd: service}
> 922 root 46 0 5804K 1820K *rc_lo 1 30:17 0.00% {nfsd: service}
> 922 root 48 0 5804K 1820K rpcsvc 1 30:17 0.00% {nfsd: service}
> 922 root 46 0 5804K 1820K rpcsvc 1 30:13 0.00% {nfsd: service}
> 922 root 45 0 5804K 1820K *rc_lo 2 30:13 0.00% {nfsd: service}
> 922 root 45 0 5804K 1820K *rc_lo 3 30:12 0.00% {nfsd: service}
> 922 root 44 0 5804K 1820K *rc_lo 0 30:10 0.00% {nfsd: service}
> 922 root 46 0 5804K 1820K *rc_lo 2 30:07 0.00% {nfsd: service}
> 922 root 46 0 5804K 1820K *rc_lo 0 30:03 0.00% {nfsd: service}
> 922 root 45 0 5804K 1820K *rc_lo 1 30:03 0.00% {nfsd: service}
> 922 root 46 0 5804K 1820K *rc_lo 2 30:03 0.00% {nfsd: service}
> 922 root 45 0 5804K 1820K *rc_lo 2 30:02 0.00% {nfsd: service}
> 922 root 45 0 5804K 1820K *rc_lo 1 29:59 0.00% {nfsd: master}
> 922 root 45 0 5804K 1820K *rc_lo 2 29:54 0.00% {nfsd: service}
> 922 root 46 0 5804K 1820K *rc_lo 0 29:54 0.00% {nfsd: service}
> 922 root 46 0 5804K 1820K *rc_lo 0 29:52 0.00% {nfsd: service}
> 922 root 47 0 5804K 1820K *rc_lo 2 29:46 0.00% {nfsd: service}
> 922 root 46 0 5804K 1820K *rc_lo 1 29:44 0.00% {nfsd: service}
>
I have a patch that I think might fix this. (Essentially the same
bug was posted on another list recently.) I'll post it to Alan
separately, but if anyone else wants to try it, it will be at
http://people.freebsd.org/~rmacklem/freebsd8.1-patches/replay.patch
in a few minutes.
rick
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1347461186.514707.1281494117995.JavaMail.root>