From owner-freebsd-fs@freebsd.org Sun Dec 27 15:50:36 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4FC2D4BED06 for ; Sun, 27 Dec 2020 15:50:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4D3lWZ552Gz4lg5 for ; Sun, 27 Dec 2020 15:50:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.16.1/8.16.1) with ESMTPS id 0BRFoK4c042483 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 27 Dec 2020 17:50:24 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 0BRFoK4c042483 Received: (from kostik@localhost) by tom.home (8.16.1/8.16.1/Submit) id 0BRFoKTI042465; Sun, 27 Dec 2020 17:50:20 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 27 Dec 2020 17:50:20 +0200 From: Konstantin Belousov To: Rick Macklem Cc: J David , "freebsd-fs@freebsd.org" Subject: Re: Major issues with nfsv4 Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on tom.home X-Rspamd-Queue-Id: 4D3lWZ552Gz4lg5 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:470:d5e7:1::1:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.995]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; R_DKIM_NA(0.00)[]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[2001:470:d5e7:1::1:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-fs]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Dec 2020 15:50:36 -0000 On Sat, Dec 26, 2020 at 11:10:01PM +0000, Rick Macklem wrote: > Although you have not posted the value for > vfs.deferred_inact, if that value has become > relatively large when the problem occurs, > it might support this theory w.r.t. how this > could happen. > > Two processes in different jails do "stat()" or > similar on the same file in the NFS file system > at basically the same time. > --> They both get shared locked nullfs vnodes, > both of which hold shared locks on the > same lowervp (the NFS client one). > --> They both do vput() on these nullfs vnodes > concurrently. > > If both call vput_final() concurrently, I think both > could have the VOP_LOCK(vp, LK_UPGRADE | LK_INTERLOCK | > LK_NOWAIT) at line #3147 fail, since this will call null_lock() > for both nullfs vnodes and then both null_lock() calls will > do VOP_LOCK(lvp, flags); at line #705. > --> The call fails for both processes, since the other one still > holds the shared lock on the NFS client vnode. > > If I have this right, then both processes end up calling > vdefer_inactive() for the upper nullfs vnodes. > > If this is what is happening, then when does the VOP_INACTIVE() > get called for the lowervp? > > I see vfs_deferred_inactive() in sys/kern/vfs_subr.c, but I do not > know when/how it gets called? Right, vfs_deferred_inactive() is one way which tries to handle missed inactivations. If upon vput() the lock is only shared and upgrade failed, vnode is marked as VI_OWEINACT and put onto 'lazy' list, processed by vfs_sync(MNT_LAZY). It is typically called from syncer, which means each 60 secs. There, if the vnode is still unreferenced, it is inactivated. Another place where inactivation can occur is reclamation. There in vgonel(), we call VOP_INACTIVE() if VI_OWEINACT is set. In principle, this is redundand because correct filesystem must do the same cleanup (and more) at reclamation as at the inactivation. But we also call VOP_CLOSE(FNONBLOCK) before VOP_RECLAIM(). Looking at this from another angle, if inactivation for NFSv4 vnodes is not called longer than 2 minutes, perhaps there is a reference leak. It is not due to VFS forgetting about due VOP_INACTIVE() call.