Date: Wed, 11 Sep 2013 18:38:48 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Mark Saad <nonesuch@longcount.org> Cc: stable@freebsd.org, Lars Eggert <lars@netapp.com> Subject: Re: nfsd CPU usage? Message-ID: <852861204.22141010.1378939128966.JavaMail.root@uoguelph.ca> In-Reply-To: <F81C8CA1-F9EC-4769-8678-D76B5F9ECFE4@longcount.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Mark Saad wrote:
> Rick
> Would this affect 9.2-RCn ?
>
Yes (and 9.2 when it is released). If the CPU overheads
(same as 9.0, 9.1) are an issue for your server, you can
apply the patch or switch to the old nfs server or disable
the DRC for TCP by editting the sources and setting
nfsrc_tcpnonidempotent = 0; as explained below.
rick
> ---
> Mark saad | mark.saad@longcount.org
>
>
> On Sep 11, 2013, at 7:54 AM, Rick Macklem <rmacklem@uoguelph.ca>
> wrote:
>
> > Lars Eggert wrote:
> >> Hi,
> >>
> >> I'm seeing extremely high CPU usage withssh-st the new nfsd:
> >>
> >> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU
> >> COMMAND
> >> 2280 root 102 0 9932K 1376K *nfs_c 0 320:11 100.00%
> >> nfsd{nfsd: service}
> >> 2280 root 102 0 9932K 1376K CPU7 7 319:47 100.00%
> >> nfsd{nfsd: service}
> >> 2280 root 102 0 9932K 1376K CPU5 5 318:25 100.00%
> >> nfsd{nfsd: service}
> >> 2280 root 102 0 9932K 1376K CPU6 6 318:20 100.00%
> >> nfsd{nfsd: service}
> >> 2280 root 52 0 9932K 1376K CPU0 0 317:32 100.00%
> >> nfsd{nfsd: service}
> >> 2280 root 102 0 9932K 1376K *nfs_c 1 315:41 99.17%
> >> nfsd{nfsd: service}
> >> 2280 root 52 0 9932K 1376K *nfs_c 4 320:22 98.78%
> >> nfsd{nfsd: master}
> >> 2280 root 102 0 9932K 1376K *nfs_c 1 317:10 98.10%
> >> nfsd{nfsd: service}
> >>
> >> And this is at a few hundred KB/s with only a few clients:
> >>
> >> ifstat -i igb1 10
> >> igb1
> >> KB/s in KB/s out
> >> 796.56 208.66
> >> 431.19 232.36
> >> 316.11 280.31
> >> 1005.96 523.42
> >> 1077.74 342.25
> >> 340.63 217.73
> >> 1067.96 330.56
> >> 487.91 235.61
> >>
> >> Any ideas?
> >>
> >> FreeBSD stanley.muccbc.hq.netapp.com 9.2-PRERELEASE FreeBSD
> >> 9.2-PRERELEASE #7: Wed Sep 4 11:06:31 CEST 2013
> >> root@stanley.muccbc.hq.netapp.com:/usr/obj/usr/src/sys/STANLEY
> >> amd64
> >>
> >> Thanks,
> >> Lars
> > There is a patch in head (r254337) that I believe handles this.
> > It will be MFC'd to stable/9 in about a week, unless someone finds
> > problems with it before then.
> > If you want a semantically equivalent (but uglier code) patch,
> > you can find it here:
> > http://people.freebsd.org/~rmacklem/drc4-stable9.patch
> > After applying the patch, you need to set sysctl variable(s),
> > to avoid the aggressive trimming of stale DRC entries. Garrett
> > Wollman suggests the following for a large server:
> > vfs.nfsd.tcphighwater=100000
> > vfs.nfsd.tcpcachetimeout=300 (5 minutes instead of default of
> > several hrs)
> >
> > You can also use the sysctl
> > vfs.nfsd.cachetcp=0
> > to disable use of the DRC for TCP.
> >
> > The old nfs server did not use the DRC for TCP. The assumption
> > being that
> > TCP layer retransmits are good enough to maintain reliable RPC
> > transport.
> > Unfortuantely, you can get file corruption when the server reboots
> > or
> > there is a network partitioning, if the client chooses to redo the
> > RPC
> > over TCP (clients always do this after having to create a new TCP
> > connection).
> > In other words, vfs.nfsd.cachetcp=0 is roughly what the old nfsd
> > did.
> >
> > If you don't want to patch the 9.2 code, you can edit the sources
> > (sys/fs/nfsserver/nfs_nfsdcache.c) and change the line:
> > static int nfsrc_tcpnonidempotent = 1;
> > to
> > static int nfsrc_tcpnonidempotent = 0;
> > to do the same thing as vfs.nfsd.cachetcp=0
> >
> > rick
> > _______________________________________________
> > freebsd-stable@freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> > To unsubscribe, send any mail to
> > "freebsd-stable-unsubscribe@freebsd.org"
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
> "freebsd-stable-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?852861204.22141010.1378939128966.JavaMail.root>