From owner-freebsd-fs@freebsd.org Wed Nov 29 13:40:51 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 32D55E5453B; Wed, 29 Nov 2017 13:40:51 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.blih.net", Issuer "mail.blih.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8D9CB74AD0; Wed, 29 Nov 2017 13:40:50 +0000 (UTC) (envelope-from manu@bidouilliste.com) Received: from mail.blih.net (mail.blih.net [212.83.177.182]) by mail.blih.net (OpenSMTPD) with ESMTP id 579ada15; Wed, 29 Nov 2017 14:40:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; s=mail; bh=8XLQth92F4LHfNI9F26nkLXTVBs=; b=qi7wstgRGgxferEn32iV0FRs9YhR sTj/3n9wvtdRvpmEDVL9SZbFeDIAkvEIbFwGUr6O0TSPP77FnCUcPQ22lS8hnNEf kp5GckE0zSyWUmTiAP0Hkw1NAW/B1mstbKpdRtgAZd0IeZQTbIDX1cq5dqYDjQrl uPFEOZQOnKZAKWY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=bidouilliste.com; h=date :from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-type:content-transfer-encoding; q=dns; s= mail; b=ncUxWbBPUsLkXbkIds6zwfKGdBJ6aKtKfLFk71u3wXkKARVrSoSjNo7C Ou5py8cV06zElfLrkTKnVvtXlfaiEL6qyz3SmYK+NEgdeRAHtV/jm7GYRcJtDIzo lqxnTGUVhjTon5S2zLl5S6jmOfmk/U8rUfKF0hG2ledeZDbWnGs= Received: from arcadia (evadot.gandi.net [217.70.181.36]) by mail.blih.net (OpenSMTPD) with ESMTPSA id 4f704a56 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO; Wed, 29 Nov 2017 14:40:41 +0100 (CET) Date: Wed, 29 Nov 2017 14:40:40 +0100 From: Emmanuel Vadot To: Rick Macklem Cc: Konstantin Belousov , FreeBSD Current , "freebsd-fs@freebsd.org" Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? Message-Id: <20171129144040.29a10234aa5fb1cfa2611bfc@bidouilliste.com> In-Reply-To: References: <20171128114136.10e44d5bcfd563e9b4ab5453@bidouilliste.com> <20171128110428.GN2272@kib.kiev.ua> <20171128142610.6ab535b0b8c6b5d372cd3f27@bidouilliste.com> <20171128134009.GQ2272@kib.kiev.ua> <20171128164132.290bf07218b16164db613242@bidouilliste.com> X-Mailer: Sylpheed 3.6.0 (GTK+ 2.24.31; amd64-portbld-freebsd12.0) Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Nov 2017 13:40:51 -0000 Hello Rick, On Tue, 28 Nov 2017 23:18:57 +0000 Rick Macklem wrote: > Did my usual and forgot to attach it. Here's the patch, rick >=20 > ________________________________________ > From: Rick Macklem > Sent: Tuesday, November 28, 2017 6:17:13 PM > To: Emmanuel Vadot > Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Ma= cklem > Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? >=20 > I think the attached patch should fix your performance problem. > It simply checks for nfsrv_delegatecnt !=3D 0 before doing all the > locking stuff in nfsrv_checkgetattr(). >=20 > If this fixes your performance problem (with delegations disabled), > I'll probably add a separate counter for write delegations and > use atomics to increment/decrement it so that it is SMP safe without > acquiring any lock. >=20 > If you can test this, please let me know how it goes? rick I haven't test by I can say that it will work, I actually wondered at first doing that. The problem with this patch is what I tried to describe in my first and following mails, since you can turn on and off delegation you can still have delegation (so nfsrv_delegatecnt > 0) even if the sysctl is =3D=3D 0. That's why I went to the tunable way, seems cleaner to me as disabling delegation doesn't really disable them for current client. > ________________________________________ > From: Rick Macklem > Sent: Tuesday, November 28, 2017 2:09:51 PM > To: Emmanuel Vadot > Cc: Konstantin Belousov; FreeBSD Current; freebsd-fs@freebsd.org; Rick Ma= cklem > Subject: Re: Switch vfs.nfsd.issue_delegations to TUNABLE ? >=20 > Emmanuel Vadot wrote: > >I wrote: > >> Since it defaults to "disabled", I don't see why a tunable would be ne= cessary? > >> (Just do nothing and delegations don't happen. If you want the server > >> to issue delegations, then use the sysctl to turn them on. If you wan= t to turn > >> them off again at the server, reboot the server without setting the s= ysctl.) > > > >If you need to reboot to make things working again without delegation > >this shouldn't be a sysctl. > Turning them off without rebooting doesn't break anything. > I just wrote it that way since you seemed to want to use a tunable, which > implied rebooting was your preference. > > > > > The reason behind it is recent problem at work on some on our fi= ler > > > > > related to NFS. > > > > > We use NFSv4 without delegation as we never been able to have go= od > > > > > performance with FreeBSD server and Linux client (we need to do t= est > > > > > again but that's for later). > > Delegations are almost never useful, especially with Linux clients. > Emmanuel Vadot wrote: > >Can you elaborate ? Reading what delegation are I see that this is > >exactly what I'm looking for to have better performance with NFS for my > >workload (I only have one client per mount point). > Delegations allow the client to do Opens locally without contacting the > server. Unless, the delegation is a write delegation, this only applied > to read-only delegations. Also, since most implementors couldn`t agree > on how to check permissions via the delegation, most client implementatio= ns > still do an Access check at open, which is almost as much overhead as the > Open RPC. (For example, Solaris servers always specified an empty ACE in = the > delegation, forcing the client to do an Access. I have no idea what the > current Linux server=E9client does. If you capture packets when a Linux > client is mounted with delegations enabled, you could look for RPCs like = Access when > are Opened multiple times. If you see them, then delegations aren`t savin= g RPCs. > Also, they are `per file`, so are only useful if the client is Opening the > same file multiple times. > Further, if another client Opens the same file and the first client got a= Write > delegation, then the write delegation must be recalled, which is a lot of > overhead and one of the few cases where the FreeBSD server must exclusive= ly > lock the state lists, forcing all other RPCs related to Open, Close to wa= it. >=20 > They sounded good in theory and might have worked well if the implementors > had agreed to do them, but that didn=E8t happen. (Companies pay for serve= rs, but the > clients don=E8t get funded so delegation support in the clients are lacki= ng. I tried > to make them useful in the FreeBSD client, but I=E8m not sure I succeeded= .) >=20 > > [stuff snipped] > If I understood your original post, you have a performance problem caused > by lock contention, where the server grabs the state lock to check for de= legations > for every Getattr from a client. >=20 > As below, I think the fix is to add code to check for no delegations issu= ed that > does not require acquisition of the state lock. >=20 > Btw, large numbers of Getattrs will not perform well with delegations. > (Again, the client should be able to do Getattr operations locally in the > client when it has a delegation for the file, but if the client is not d= oing that...) >=20 > I wrote: > > > > Having a per-mount version of this would be overkill, I think. It would= have to > > disable callbacks on the mount point, since there is no way for a clien= t to say > > "don't give me delegations" except disabling callbacks, which the server > > requires for delegation issue. > > [stuff snipped] > > The case where there has never been delegations issued will result in an > > empty delegation queue. Maybe this case can be handled without acquiring > > the "global NFSv4 state lock", which is what sounds like is causing the > > performance issue. Maybe a global counter of how many delegations are > > issued that is handled by atomic ops. > > --> If it is 0, nfsrv_checkgetattr() can just return without acquiring = the lock. > > > > I'll take a look at this, rick > > > rick --=20 Emmanuel Vadot