Date: Mon, 14 Dec 2020 15:31:49 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: J David <j.david.lists@gmail.com>, Konstantin Belousov <kostikbel@gmail.com> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Major issues with nfsv4 Message-ID: <YQXPR0101MB09689A7BD3ADE2DB300E50C5DDC70@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CABXB=RTFSAEZvp%2BmoiF%2BrE9vpEjJVacLYa6G=yP641f9oHJ1zw@mail.gmail.com> References: <YQXPR0101MB096849ADF24051F7479E565CDDCA0@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <CABXB=RSyN%2Bo2yXcpmYw8sCSUUDhN-w28Vu9v_cCWa-2=pLZmHg@mail.gmail.com> <YQXPR0101MB09680D155B6D685442B5E25EDDCA0@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <CABXB=RSSE=yOwgOXsnbEYPqiWk5K5NfzLY=D%2BN9mXdVn%2B--qLQ@mail.gmail.com> <YQXPR0101MB0968B17010B3B36C8C41FDE1DDC90@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <X9Q9GAhNHbXGbKy7@kib.kiev.ua> <YQXPR0101MB0968C7629D57CA21319E50C2DDC90@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <X9UDArKjUqJVS035@kib.kiev.ua> <CABXB=RRNnW9nNhFCJS1evNUTEX9LNnzyf2gOmZHHGkzAoQxbPw@mail.gmail.com> <YQXPR0101MB0968B120A417AF69CEBB6A12DDC80@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> <X9aGwshgh7Cwiv8p@kib.kiev.ua>, <CABXB=RTFSAEZvp%2BmoiF%2BrE9vpEjJVacLYa6G=yP641f9oHJ1zw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
J David wrote:=0A=
>On Sun, Dec 13, 2020 at 4:25 PM Konstantin Belousov <kostikbel@gmail.com> =
wrote:=0A=
>> Nullfs with -o nocache (default for NFS mounts) should not cache vnodes.=
=0A=
>> So it is more likely a local load that has 130k files open. Of course,=
=0A=
>> it is the OP who can answer the question.=0A=
>=0A=
>This I can rule out; there is no visible correlation between "Opens"=0A=
>and the number of files open on the system.=0A=
>=0A=
>Just finishing a test right now, and:=0A=
>=0A=
>$ sudo nfsstat -E -c | fgrep -A1 OpenOwner=0A=
> OpenOwner Opens LockOwner Locks Delegs Loca=
lOwn=0A=
> 4678 36245 15 6 0 =
0=0A=
>$ sudo fstat | wc -l=0A=
> 2698=0A=
>$ ps Haxlww | wc -l=0A=
> 1012=0A=
>=0A=
>The value of Opens increases consistently over time.=0A=
>=0A=
>Killing the processes causing this behavior *did not* reduce the=0A=
>number of OpenOwner or Opens.=0A=
>=0A=
>Unmounting the nullfs mounts (after the processes were gone) *did*:=0A=
>=0A=
>$ sudo nfsstat -E -c | fgrep -A1 OpenOwner=0A=
> OpenOwner Opens LockOwner Locks Delegs Loca=
lOwn=0A=
> 130 41 0 0 0 =
0=0A=
Iwill take a look at the nullfs code to see if I can spot anything that mig=
ht=0A=
explain this.=0A=
=0A=
>Mutex contention was observed this time, but once it was apparent that=0A=
>"Opens" was increasing over time, I didn't let the test get to the=0A=
>point of disrupting activities. This test ended at Opens =3D 36589,=0A=
>which is well short of the previous 130,000+. It is possible that=0A=
>mutex contention becomes an issue once system CPU resources are=0A=
>exhausted.=0A=
>=0A=
>More about the results of the latest test after the data is analyzed.=0A=
>=0A=
>After that's done, I'll attempt Rick's patch.=0A=
The patch should help w.r.t. lock contention, but if the Open=0A=
count is increasing, you'll "hit the wall eventually".=0A=
=0A=
> In the long run, we=0A=
>would definitely like to get delegation to work. Baby steps!=0A=
Well, all delegations do is allow a file to be repeatedly opened/closed=0A=
without talking to the server. But they are per-file and issued on the=0A=
first open against the server. As such, they are only useful if processes=
=0A=
are opening/closing the same file over and over again.=0A=
-->They also add complexity and, as such, they are often a loss and=0A=
not a help. They are disabled on the FreeBSD NFS server by default=0A=
and disabled on the client side unless the nfscbd(8) is running.=0A=
=0A=
rick=0A=
=0A=
Thanks!=0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB09689A7BD3ADE2DB300E50C5DDC70>