From owner-svn-src-head@freebsd.org Thu Jul 30 15:44:10 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B42C83A8ADF; Thu, 30 Jul 2020 15:44:10 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BHZTQ4LcNz4WjS; Thu, 30 Jul 2020 15:44:10 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 77FE4132A6; Thu, 30 Jul 2020 15:44:10 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 06UFiARO099426; Thu, 30 Jul 2020 15:44:10 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 06UFiAIh099425; Thu, 30 Jul 2020 15:44:10 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <202007301544.06UFiAIh099425@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Thu, 30 Jul 2020 15:44:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r363704 - head/sys/kern X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: head/sys/kern X-SVN-Commit-Revision: 363704 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2020 15:44:10 -0000 Author: mjg Date: Thu Jul 30 15:44:10 2020 New Revision: 363704 URL: https://svnweb.freebsd.org/changeset/base/363704 Log: vfs: support negative entry promotion in lockless lookup Tested by: pho Modified: head/sys/kern/vfs_cache.c Modified: head/sys/kern/vfs_cache.c ============================================================================== --- head/sys/kern/vfs_cache.c Thu Jul 30 15:43:16 2020 (r363703) +++ head/sys/kern/vfs_cache.c Thu Jul 30 15:44:10 2020 (r363704) @@ -807,6 +807,15 @@ cache_negative_remove(struct namecache *ncp) } else { list_locked = true; mtx_lock(&neglist->nl_lock); + /* + * We may be racing against promotion in lockless lookup. + */ + if ((negstate->neg_flag & NEG_HOT) != 0) { + mtx_unlock(&neglist->nl_lock); + hot_locked = true; + mtx_lock(&ncneg_hot.nl_lock); + mtx_lock(&neglist->nl_lock); + } } if ((negstate->neg_flag & NEG_HOT) != 0) { mtx_assert(&ncneg_hot.nl_lock, MA_OWNED); @@ -3061,6 +3070,103 @@ cache_fplookup_vnode_supported(struct vnode *vp) } /* + * Move a negative entry to the hot list. + * + * We have to take locks, but they may be contended and in the worst + * case we may need to go off CPU. We don't want to spin within the + * smr section and we can't block with it. Instead we are going to + * look up the entry again. + */ +static int __noinline +cache_fplookup_negative_promote(struct cache_fpl *fpl, struct namecache *oncp, + uint32_t hash) +{ + struct componentname *cnp; + struct namecache *ncp; + struct neglist *neglist; + struct negstate *negstate; + struct vnode *dvp; + u_char nc_flag; + + cnp = fpl->cnp; + dvp = fpl->dvp; + + if (!vhold_smr(dvp)) + return (cache_fpl_aborted(fpl)); + + neglist = NCP2NEGLIST(oncp); + cache_fpl_smr_exit(fpl); + + mtx_lock(&ncneg_hot.nl_lock); + mtx_lock(&neglist->nl_lock); + /* + * For hash iteration. + */ + cache_fpl_smr_enter(fpl); + + /* + * Avoid all surprises by only succeeding if we got the same entry and + * bailing completely otherwise. + * + * In particular at this point there can be a new ncp which matches the + * search but hashes to a different neglist. + */ + CK_LIST_FOREACH(ncp, (NCHHASH(hash)), nc_hash) { + if (ncp == oncp) + break; + } + + /* + * No match to begin with. + */ + if (__predict_false(ncp == NULL)) { + goto out_abort; + } + + /* + * The newly found entry may be something different... + */ + if (!(ncp->nc_dvp == dvp && ncp->nc_nlen == cnp->cn_namelen && + !bcmp(ncp->nc_name, cnp->cn_nameptr, ncp->nc_nlen))) { + goto out_abort; + } + + /* + * ... and not even negative. + */ + nc_flag = atomic_load_char(&ncp->nc_flag); + if ((nc_flag & NCF_NEGATIVE) == 0) { + goto out_abort; + } + + if (__predict_false(cache_ncp_invalid(ncp))) { + goto out_abort; + } + + negstate = NCP2NEGSTATE(ncp); + if ((negstate->neg_flag & NEG_HOT) == 0) { + numhotneg++; + TAILQ_REMOVE(&neglist->nl_list, ncp, nc_dst); + TAILQ_INSERT_TAIL(&ncneg_hot.nl_list, ncp, nc_dst); + negstate->neg_flag |= NEG_HOT; + } + + SDT_PROBE2(vfs, namecache, lookup, hit__negative, dvp, ncp->nc_name); + counter_u64_add(numneghits, 1); + cache_fpl_smr_exit(fpl); + mtx_unlock(&neglist->nl_lock); + mtx_unlock(&ncneg_hot.nl_lock); + vdrop(dvp); + return (cache_fpl_handled(fpl, ENOENT)); +out_abort: + cache_fpl_smr_exit(fpl); + mtx_unlock(&neglist->nl_lock); + mtx_unlock(&ncneg_hot.nl_lock); + vdrop(dvp); + return (cache_fpl_aborted(fpl)); +} + +/* * The target vnode is not supported, prepare for the slow path to take over. */ static int @@ -3204,12 +3310,7 @@ cache_fplookup_next(struct cache_fpl *fpl) return (cache_fpl_partial(fpl)); } if (!neg_hot) { - /* - * TODO - * Promoting to hot negative requires locks, thus is - * left not yet supported for simplicity. - */ - return (cache_fpl_partial(fpl)); + return (cache_fplookup_negative_promote(fpl, ncp, hash)); } SDT_PROBE2(vfs, namecache, lookup, hit__negative, dvp, ncp->nc_name);