Date: Wed, 29 Nov 2017 14:40:40 +0100 From: Emmanuel Vadot <manu@bidouilliste.com> To: Rick Macklem <rmacklem@uoguelph.ca> Cc: Konstantin Belousov <kostikbel@gmail.com>, FreeBSD Current <freebsd-current@freebsd.org>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Message-ID: <20171129144040.29a10234aa5fb1cfa2611bfc@bidouilliste.com> In-Reply-To: <YTOPR0101MB21721E82EF605249D38064A1DD3A0@YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM> References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> <YTOPR0101MB2172BE15A55BCA881F805AE6DD3A0@YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM> <20171128164132.290bf07218b16164db613242@bidouilliste.com> <YTOPR0101MB217255A72DD5E8555DF001E9DD3A0@YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM> <YTOPR0101MB217218369849273146801292DD3A0@YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM> <YTOPR0101MB21721E82EF605249D38064A1DD3A0@YTOPR0101MB2172.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello Rick, On Tue, 28 Nov 2017 23:18:57 +0000 Rick Macklem <rmacklem@uoguelph.ca> wrote: > Did my usual and forgot to attach it. Here's the patch, rick >=20 > ________________________________________ > From: Rick Macklem <rmacklem@uoguelph.ca> > Sent: Tuesday, November 28, 2017 6:17:13 PM > To: Emmanuel Vadot > Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Ma= cklem > Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? >=20 > I think the attached patch should fix your performance problem. > It simply checks for nfsrv_delegatecnt !=3D 0 before doing all the > locking stuff in nfsrv_checkgetattr(). >=20 > If this fixes your performance problem (with delegations disabled), > I'll probably add a separate counter for write delegations and > use atomics to increment/decrement it so that it is SMP safe without > acquiring any lock. >=20 > If you can test this, please let me know how it goes? rick I haven't test by I can say that it will work, I actually wondered at first doing that. The problem with this patch is what I tried to describe in my first and following mails, since you can turn on and off delegation you can still have delegation (so nfsrv_delegatecnt > 0) even if the sysctl is =3D=3D 0. That's why I went to the tunable way, seems cleaner to me as disabling delegation doesn't really disable them for current client. > ________________________________________ > From: Rick Macklem <rmacklem@uoguelph.ca> > Sent: Tuesday, November 28, 2017 2:09:51 PM > To: Emmanuel Vadot > Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Ma= cklem > Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? >=20 > Emmanuel Vadot wrote: > >I wrote: > >> Since it defaults to "disabled", I don't see why a tunable would be ne= cessary? > >> (Just do nothing and delegations don't happen. If you want the server > >> to issue delegations, then use the sysctl to turn them on. If you wan= t to turn > >> them off again at the server, reboot the server without setting the s= ysctl.) > > > >If you need to reboot to make things working again without delegation > >this shouldn't be a sysctl. > Turning them off without rebooting doesn't break anything. > I just wrote it that way since you seemed to want to use a tunable, which > implied rebooting was your preference. > > > > > The reason behind it is recent problem at work on some on our fi= ler > > > > > related to NFS. > > > > > We use NFSv4 without delegation as we never been able to have go= od > > > > > performance with FreeBSD server and Linux client (we need to do t= est > > > > > again but that's for later). > > Delegations are almost never useful, especially with Linux clients. > Emmanuel Vadot wrote: > >Can you elaborate ? Reading what delegation are I see that this is > >exactly what I'm looking for to have better performance with NFS for my > >workload (I only have one client per mount point). > Delegations allow the client to do Opens locally without contacting the > server. Unless, the delegation is a write delegation, this only applied > to read-only delegations. Also, since most implementors couldn`t agree > on how to check permissions via the delegation, most client implementatio= ns > still do an Access check at open, which is almost as much overhead as the > Open RPC. (For example, Solaris servers always specified an empty ACE in = the > delegation, forcing the client to do an Access. I have no idea what the > current Linux server=E9client does. If you capture packets when a Linux > client is mounted with delegations enabled, you could look for RPCs like = Access when > are Opened multiple times. If you see them, then delegations aren`t savin= g RPCs. > Also, they are `per file`, so are only useful if the client is Opening the > same file multiple times. > Further, if another client Opens the same file and the first client got a= Write > delegation, then the write delegation must be recalled, which is a lot of > overhead and one of the few cases where the FreeBSD server must exclusive= ly > lock the state lists, forcing all other RPCs related to Open, Close to wa= it. >=20 > They sounded good in theory and might have worked well if the implementors > had agreed to do them, but that didn=E8t happen. (Companies pay for serve= rs, but the > clients don=E8t get funded so delegation support in the clients are lacki= ng. I tried > to make them useful in the FreeBSD client, but I=E8m not sure I succeeded= .) >=20 > > [stuff snipped] > If I understood your original post, you have a performance problem caused > by lock contention, where the server grabs the state lock to check for de= legations > for every Getattr from a client. >=20 > As below, I think the fix is to add code to check for no delegations issu= ed that > does not require acquisition of the state lock. >=20 > Btw, large numbers of Getattrs will not perform well with delegations. > (Again, the client should be able to do Getattr operations locally in the > client when it has a delegation for the file, but if the client is not d= oing that...) >=20 > I wrote: > > > > Having a per-mount version of this would be overkill, I think. It would= have to > > disable callbacks on the mount point, since there is no way for a clien= t to say > > "don't give me delegations" except disabling callbacks, which the server > > requires for delegation issue. > > [stuff snipped] > > The case where there has never been delegations issued will result in an > > empty delegation queue. Maybe this case can be handled without acquiring > > the "global NFSv4 state lock", which is what sounds like is causing the > > performance issue. Maybe a global counter of how many delegations are > > issued that is handled by atomic ops. > > --> If it is 0, nfsrv_checkgetattr() can just return without acquiring = the lock. > > > > I'll take a look at this, rick > > > rick --=20 Emmanuel Vadot <manu@bidouilliste.com> <manu@freebsd.org>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20171129144040.29a10234aa5fb1cfa2611bfc>