From nobody Fri Aug 25 22:55:18 2023 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXZzW0m54z4qvGt; Fri, 25 Aug 2023 22:55:19 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXZzW0JWXz4HdY; Fri, 25 Aug 2023 22:55:19 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693004119; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=V7WoK9BMPmoPWUq4EHVVLMaFgSz2Lj9XDAf3tzFGrBI=; b=MNnaW8lnlkF5zQqPwDhNpKYU6xKMz8QD3h1WVcd4WYdNMZopKer4uDNy2e9rlocoae+T+U R7KvW5Q/BPc1tRrG8BgJqqzB8ozHfZt0RY28zgrJIt7EToIOAucNhObt616RAFRBCE/8b2 GXmUByNyZR40zG0bfNL47HkPpr2whs17S2HU5+Q7p4ECn0I3N/oNY18OgbMpcl3FQF/9SB dTn6F1Ot5fC9L78nJNSB4Jpq1bMF49V3lXp62UBHyCS6pmSNxrnZ0jE0zn5L+bEjZQMm+O SAl6pVGg5qaaws/9KYiyVBwecG4Lh96b1Ntf/PzKMrPsvo5r9UVm16flKDCfHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693004119; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=V7WoK9BMPmoPWUq4EHVVLMaFgSz2Lj9XDAf3tzFGrBI=; b=Bg4VaaTpQSlF7boj4NhPKi9IRH9kyan+EPGkhJKvcW/w94S9xiNHEAdOE7nAl93sbixn6C wL0SKjwSn23LQFTwi1SyZEDeczHvFBSA9DCTpyiH3zAAgfMiM0I/U4q/YZM3rnrXn7fDNq AdzftZ1H81q18AYpavTrGw7AJGalr3rXP/uiTswNXPkuj4tkUqeK69ZiygYJdJ19iXfdC3 P8K6pJmYmu6wgFO+PpG+nhadYE++Gdz5nuiZomFFz+XrggEIEGM9DfJ5ogMCsJe/gNPUVB bvYbxpjt+J3nLFGFeEak8m9THnA8mNK0fcgApObDK2T9sVB0yWqMaoJC19702g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693004119; a=rsa-sha256; cv=none; b=HXj6eUMH69+BSvzF0Sx/XZKYyzRdJtKK51lIq4C8tr6C5Wc7vEs8s4K/RACNSQxpbC/ncE 052aYZzaQfFeaWt9jnb7of9AVIQ5/7tlAQdbQmmHZESONwCDoz1To0jjaixcMYYiOUmYXN IeyG0t0AELrmdZ8vynXx5TYcsX4a+k+eb9qktQ8V/evStJx/3rKxjhL1nTT3859EGxlHPT 8ev6xOol74jZWV+/PmDpIWLKrQxI8f0vqPJDAeKypR0wlL/g06bu6qHfvzpaULx0AHRVfl Y+3pYvqiuoOTVev1jbIZlaoObj9gQHL0u6ljNgHdvZ6Ta7ADsvlgn/9pPfwaNg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RXZzV6Sbpz1QQ1; Fri, 25 Aug 2023 22:55:18 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 37PMtIq3031685; Fri, 25 Aug 2023 22:55:18 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 37PMtIuf031682; Fri, 25 Aug 2023 22:55:18 GMT (envelope-from git) Date: Fri, 25 Aug 2023 22:55:18 GMT Message-Id: <202308252255.37PMtIuf031682@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mateusz Guzik Subject: git: b61645ef3c4e - stable/13 - vfs: decouple freevnodes from vnode batching List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mjg X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: b61645ef3c4e6b7f66e6e8accad61aab227a0dd9 Auto-Submitted: auto-generated The branch stable/13 has been updated by mjg: URL: https://cgit.FreeBSD.org/src/commit/?id=b61645ef3c4e6b7f66e6e8accad61aab227a0dd9 commit b61645ef3c4e6b7f66e6e8accad61aab227a0dd9 Author: Mateusz Guzik AuthorDate: 2023-03-21 07:27:25 +0000 Commit: Mateusz Guzik CommitDate: 2023-08-25 22:52:23 +0000 vfs: decouple freevnodes from vnode batching In principle one cpu can keep vholding vnodes, while another vdrops them. In this case it may be the local count will keep growing in an unbounded manner. Roll it up after a threshold instead. While here move it out of dpcpu into struct pcpu. Reviewed by: kib (previous version) Differential Revision: https://reviews.freebsd.org/D39195 (cherry picked from commit b5d43972e3941d6c03d935dc423385f3c2178d68) --- sys/kern/vfs_subr.c | 55 ++++++++++++++++++++++++++++++++--------------------- sys/sys/pcpu.h | 3 ++- 2 files changed, 35 insertions(+), 23 deletions(-) diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c index c85f0512b286..6befc5eead0e 100644 --- a/sys/kern/vfs_subr.c +++ b/sys/kern/vfs_subr.c @@ -284,7 +284,6 @@ SYSCTL_INT(_debug, OID_AUTO, rush_requests, CTLFLAG_RW, &stat_rush_requests, 0, #define VDBATCH_SIZE 8 struct vdbatch { u_int index; - long freevnodes; struct mtx lock; struct vnode *tab[VDBATCH_SIZE]; }; @@ -1431,48 +1430,62 @@ static int vnlruproc_sig; * at any given moment can still exceed slop, but it should not be by significant * margin in practice. */ -#define VNLRU_FREEVNODES_SLOP 128 +#define VNLRU_FREEVNODES_SLOP 126 + +static void __noinline +vfs_freevnodes_rollup(int8_t *lfreevnodes) +{ + + atomic_add_long(&freevnodes, *lfreevnodes); + *lfreevnodes = 0; + critical_exit(); +} static __inline void vfs_freevnodes_inc(void) { - struct vdbatch *vd; + int8_t *lfreevnodes; critical_enter(); - vd = DPCPU_PTR(vd); - vd->freevnodes++; - critical_exit(); + lfreevnodes = PCPU_PTR(vfs_freevnodes); + (*lfreevnodes)++; + if (__predict_false(*lfreevnodes == VNLRU_FREEVNODES_SLOP)) + vfs_freevnodes_rollup(lfreevnodes); + else + critical_exit(); } static __inline void vfs_freevnodes_dec(void) { - struct vdbatch *vd; + int8_t *lfreevnodes; critical_enter(); - vd = DPCPU_PTR(vd); - vd->freevnodes--; - critical_exit(); + lfreevnodes = PCPU_PTR(vfs_freevnodes); + (*lfreevnodes)--; + if (__predict_false(*lfreevnodes == -VNLRU_FREEVNODES_SLOP)) + vfs_freevnodes_rollup(lfreevnodes); + else + critical_exit(); } static u_long vnlru_read_freevnodes(void) { - struct vdbatch *vd; - long slop; + long slop, rfreevnodes; int cpu; - mtx_assert(&vnode_list_mtx, MA_OWNED); - if (freevnodes > freevnodes_old) - slop = freevnodes - freevnodes_old; + rfreevnodes = atomic_load_long(&freevnodes); + + if (rfreevnodes > freevnodes_old) + slop = rfreevnodes - freevnodes_old; else - slop = freevnodes_old - freevnodes; + slop = freevnodes_old - rfreevnodes; if (slop < VNLRU_FREEVNODES_SLOP) - return (freevnodes >= 0 ? freevnodes : 0); - freevnodes_old = freevnodes; + return (rfreevnodes >= 0 ? rfreevnodes : 0); + freevnodes_old = rfreevnodes; CPU_FOREACH(cpu) { - vd = DPCPU_ID_PTR((cpu), vd); - freevnodes_old += vd->freevnodes; + freevnodes_old += cpuid_to_pcpu[cpu]->pc_vfs_freevnodes; } return (freevnodes_old >= 0 ? freevnodes_old : 0); } @@ -3518,7 +3531,6 @@ vdbatch_process(struct vdbatch *vd) mtx_lock(&vnode_list_mtx); critical_enter(); - freevnodes += vd->freevnodes; for (i = 0; i < VDBATCH_SIZE; i++) { vp = vd->tab[i]; TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); @@ -3527,7 +3539,6 @@ vdbatch_process(struct vdbatch *vd) vp->v_dbatchcpu = NOCPU; } mtx_unlock(&vnode_list_mtx); - vd->freevnodes = 0; bzero(vd->tab, sizeof(vd->tab)); vd->index = 0; critical_exit(); diff --git a/sys/sys/pcpu.h b/sys/sys/pcpu.h index 15d74d90778a..58e47f11f917 100644 --- a/sys/sys/pcpu.h +++ b/sys/sys/pcpu.h @@ -187,7 +187,8 @@ struct pcpu { long pc_cp_time[CPUSTATES]; /* statclock ticks */ struct _device *pc_device; /* CPU device handle */ void *pc_netisr; /* netisr SWI cookie */ - int pc_unused1; /* unused field */ + int8_t pc_vfs_freevnodes; /* freevnodes counter */ + char pc_unused1[3]; /* unused pad */ int pc_domain; /* Memory domain. */ struct rm_queue pc_rm_queue; /* rmlock list of trackers */ uintptr_t pc_dynamic; /* Dynamic per-cpu data area */