Date: Mon, 14 Dec 2020 15:31:49 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: J David <j.david.lists@gmail.com>, Konstantin Belousov <kostikbel@gmail.com> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Major issues with nfsv4 Message-ID: <YQXPR0101MB09689A7BD3ADE2DB300E50C5DDC70@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CABXB=RTFSAEZvp%2BmoiF%2BrE9vpEjJVacLYa6G=yP641f9oHJ1zw@mail.gmail.com> References: <YQXPR0101MB096849ADF24051F7479E565CDDCA0@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <CABXB=RSyN%2Bo2yXcpmYw8sCSUUDhN-w28Vu9v_cCWa-2=pLZmHg@mail.gmail.com> <YQXPR0101MB09680D155B6D685442B5E25EDDCA0@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <CABXB=RSSE=yOwgOXsnbEYPqiWk5K5NfzLY=D%2BN9mXdVn%2B--qLQ@mail.gmail.com> <YQXPR0101MB0968B17010B3B36C8C41FDE1DDC90@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <X9Q9GAhNHbXGbKy7@kib.kiev.ua> <YQXPR0101MB0968C7629D57CA21319E50C2DDC90@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <X9UDArKjUqJVS035@kib.kiev.ua> <CABXB=RRNnW9nNhFCJS1evNUTEX9LNnzyf2gOmZHHGkzAoQxbPw@mail.gmail.com> <YQXPR0101MB0968B120A417AF69CEBB6A12DDC80@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <X9aGwshgh7Cwiv8p@kib.kiev.ua>, <CABXB=RTFSAEZvp%2BmoiF%2BrE9vpEjJVacLYa6G=yP641f9oHJ1zw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
J David wrote:
>On Sun, Dec 13, 2020 at 4:25 PM Konstantin Belousov <kostikbel@gmail.com> wrote:
>> Nullfs with -o nocache (default for NFS mounts) should not cache vnodes.
>> So it is more likely a local load that has 130k files open. Of course,
>> it is the OP who can answer the question.
>
>This I can rule out; there is no visible correlation between "Opens"
>and the number of files open on the system.
>
>Just finishing a test right now, and:
>
>$ sudo nfsstat -E -c | fgrep -A1 OpenOwner
> OpenOwner Opens LockOwner Locks Delegs LocalOwn
> 4678 36245 15 6 0 0
>$ sudo fstat | wc -l
> 2698
>$ ps Haxlww | wc -l
> 1012
>
>The value of Opens increases consistently over time.
>
>Killing the processes causing this behavior *did not* reduce the
>number of OpenOwner or Opens.
>
>Unmounting the nullfs mounts (after the processes were gone) *did*:
>
>$ sudo nfsstat -E -c | fgrep -A1 OpenOwner
> OpenOwner Opens LockOwner Locks Delegs LocalOwn
> 130 41 0 0 0 0
Iwill take a look at the nullfs code to see if I can spot anything that might
explain this.
>Mutex contention was observed this time, but once it was apparent that
>"Opens" was increasing over time, I didn't let the test get to the
>point of disrupting activities. This test ended at Opens = 36589,
>which is well short of the previous 130,000+. It is possible that
>mutex contention becomes an issue once system CPU resources are
>exhausted.
>
>More about the results of the latest test after the data is analyzed.
>
>After that's done, I'll attempt Rick's patch.
The patch should help w.r.t. lock contention, but if the Open
count is increasing, you'll "hit the wall eventually".
> In the long run, we
>would definitely like to get delegation to work. Baby steps!
Well, all delegations do is allow a file to be repeatedly opened/closed
without talking to the server. But they are per-file and issued on the
first open against the server. As such, they are only useful if processes
are opening/closing the same file over and over again.
-->They also add complexity and, as such, they are often a loss and
not a help. They are disabled on the FreeBSD NFS server by default
and disabled on the client side unless the nfscbd(8) is running.
rick
Thanks!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB09689A7BD3ADE2DB300E50C5DDC70>