Date: Wed, 17 Mar 2021 00:06:42 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Alan Somers <asomers@freebsd.org>, freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: Understanding the vfs.nfsd.request_space* sysctls Message-ID: <YQXPR0101MB0968CD521F2A19F44488367ADD6B9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CAOtMX2geF7Tv7VYc9WtK23a%2BnDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com> References: <CAOtMX2geF7Tv7VYc9WtK23a%2BnDzJzD2ehVKp94=xANjTA9S9Tg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Alan Somers wrote:
>Could somebody help me understand the meaning of the various
>vfs.nfsd.request_space* sysctls, and what implication they have for tuning
>NFS?
>
>I have several NFS servers. Most are performing well, but one with a
>metadata-heavy workload (NFS 4.1, no delegations, lots of Sequence, GetAttr
>and Lookup ops) is not. Total throughput is a not-unreasonable several
>hundred MB/s, but some clients are limited to very slow speeds, sometimes
>writing at < 10 MB/s.
>
>The request_space sysctls show some pretty stark divergence between the
>well-performing and poor-performing servers:
>
>sysctl well-performing poor-performing
>request_space_used < 6 M varies, but
>currently 40 M
>request_space_used_highest < 39 M 24 G
>request_space_throttle_count 0 35
>
>So what does request_space_used measure, anyway?
It's the number of bytes of request message(s) received,
but not yet being processed by nfsd threads.
(They're actually in sys/rpc/svc.c, although named for the
nfs server.)
Anytime used > high, it throttles, which means it leaves
the RPC messages on the socket receive queue.
This indicates the server is not keeping up with requests.
(ie Overloaded or ???)
> And how can I either
>increase the available space or decrease the stuff that's using it?
Increasing vfs.nfsd.request_space_high avoids the throttling,
but it is hard to say that is a good idea, since there won't be
backpressure applied to the clients via TCP windows, etc.
Fixing the server so that it isn't overloaded would be better,
I think?
--> I'm the last guy to take ZFS advice from, but I think there
is a tunable w.r.t. how much arc is used for metadata.
Getattrs will need cached (metadata) to reply quickly.
Lookups also depend on cached attributes for good
perf. as well.
--> Make sure you have lots of nfsd threads. They are
just kernel threads, so they don't use a lot of resources
and having too many is much better than not enough.
-->I can't remember what the upper limit is these days,
but I'd set it to that for a busy nfs server.
For NFSv4.1, each client can send a maximum of
session_slot_table_size concurrent RPCs. FreeBSD
uses a single 64slot table for each mount. I think
Linux uses a 32slot table, but supports trunking.
I don't know if the Linux client uses a separate
session table for each trunk or not?
--> Something like #clients * 32 (or 64) nfsd
threads running on the server.
rick
-Alan
_______________________________________________
freebsd-fs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB0968CD521F2A19F44488367ADD6B9>