Date: Wed, 17 Mar 2021 00:06:42 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Alan Somers <asomers@freebsd.org>, freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: Understanding the vfs.nfsd.request_space* sysctls Message-ID: <YQXPR0101MB0968CD521F2A19F44488367ADD6B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CAOtMX2geF7Tv7VYc9WtK23a%2BnDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com> References: <CAOtMX2geF7Tv7VYc9WtK23a%2BnDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Alan Somers wrote:=0A=
>Could somebody help me understand the meaning of the various=0A=
>vfs.nfsd.request_space* sysctls, and what implication they have for tuning=
=0A=
>NFS?=0A=
>=0A=
>I have several NFS servers. Most are performing well, but one with a=0A=
>metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, GetAtt=
r=0A=
>and Lookup ops) is not. Total throughput is a not-unreasonable several=0A=
>hundred MB/s, but some clients are limited to very slow speeds, sometimes=
=0A=
>writing at < 10 MB/s.=0A=
>=0A=
>The request_space sysctls show some pretty stark divergence between the=0A=
>well-performing and poor-performing servers:=0A=
>=0A=
>sysctl well-performing poor-performing=0A=
>request_space_used < 6 M varies, but=0A=
>currently 40 M=0A=
>request_space_used_highest < 39 M 24 G=0A=
>request_space_throttle_count 0 35=0A=
>=0A=
>So what does request_space_used measure, anyway?=0A=
It's the number of bytes of request message(s) received,=0A=
but not yet being processed by nfsd threads.=0A=
(They're actually in sys/rpc/svc.c, although named for the=0A=
nfs server.)=0A=
=0A=
Anytime used > high, it throttles, which means it leaves=0A=
the RPC messages on the socket receive queue.=0A=
This indicates the server is not keeping up with requests.=0A=
(ie Overloaded or ???)=0A=
=0A=
> And how can I either=0A=
>increase the available space or decrease the stuff that's using it?=0A=
Increasing vfs.nfsd.request_space_high avoids the throttling,=0A=
but it is hard to say that is a good idea, since there won't be=0A=
backpressure applied to the clients via TCP windows, etc.=0A=
=0A=
Fixing the server so that it isn't overloaded would be better,=0A=
I think?=0A=
--> I'm the last guy to take ZFS advice from, but I think there=0A=
is a tunable w.r.t. how much arc is used for metadata.=0A=
Getattrs will need cached (metadata) to reply quickly.=0A=
Lookups also depend on cached attributes for good=0A=
perf. as well.=0A=
--> Make sure you have lots of nfsd threads. They are=0A=
just kernel threads, so they don't use a lot of resources=0A=
and having too many is much better than not enough.=0A=
-->I can't remember what the upper limit is these days,=0A=
but I'd set it to that for a busy nfs server.=0A=
For NFSv4.1, each client can send a maximum of=0A=
session_slot_table_size concurrent RPCs. FreeBSD=0A=
uses a single 64slot table for each mount. I think=0A=
Linux uses a 32slot table, but supports trunking.=0A=
I don't know if the Linux client uses a separate=0A=
session table for each trunk or not?=0A=
--> Something like #clients * 32 (or 64) nfsd=0A=
threads running on the server.=0A=
=0A=
rick=0A=
=0A=
-Alan=0A=
_______________________________________________=0A=
freebsd-fs@freebsd.org mailing list=0A=
https://lists.freebsd.org/mailman/listinfo/freebsd-fs=0A=
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"=0A=
=0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB0968CD521F2A19F44488367ADD6B9>