From nobody Wed Mar 22 23:57:28 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhllF1dbxz41TGJ; Wed, 22 Mar 2023 23:57:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhllF13wrz4128; Wed, 22 Mar 2023 23:57:29 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679529449; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=IHUqQBZwzr0czMnbu/EyzvSSjnOrjSfmNGt5LvZEfXg=; b=tzS14aqOcfKiwH1mE/EdBVTSpJUfUD9CP42fnnpLe4dIm9cMYJHsSa04IcFwpIf56xucuq glcalsrK9qa13x5FaAF1IaohgcDpYSXAUYS5Yg0HYKxnQ4ElvMbvpLgm1wxuVOczgTaenA KoRgBUc8ZUOfjM9ZiAb9oT8CoVcnpTvUGgnwd07SvIS/gcwInO0vEGIbuRDMkJC9dgWK+G nc1CSmEAvAW1XCgqDetmu1U4jC+loPJ5IbMucK+MvhL4+Y15Y2l04scbz1XtvGxswB084g F7S5YzT4FL2AGBibft+El4QKvKNRGNw9rZYvBRECrpGRQLCIp5+UYAVF8iI9lg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679529449; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=IHUqQBZwzr0czMnbu/EyzvSSjnOrjSfmNGt5LvZEfXg=; b=JuS68mDiU2dTc0ykE+L/Lv7Vmse5KIUi6mbb/Yf1LbsnvU1UTip1eB6F8EODoPvALwhhnj Fd/fnA6hZ+6494HSYPIf3y4m3E5b43lzljvjgg2Bz4VzyhsS69AiKRBorgxWSLyKCAuXb+ zLQ8KhSEunSheS18wx4n9E4merdMFp5YdfXXOnlByTlMMCz2I36DsNQWuyS6Bj+byf914f 5f+yI7jARSqfqlTF4lDlFGCw1r4jlxPWQtC26BX1dza5FjKrQcsck99wmnVq6Yda35ZwsG u3ZMMh7nvcbAlXSziXsb+BGbxKepteK6UsfwFtCk7u2U1bgiK/eCJegCeheQWQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679529449; a=rsa-sha256; cv=none; b=pAduJJJKGqjY1of1gctZAVyRXkbXkQxLpWtrC6UBzAfrCQ435njrIYhyyu4g+B/iNjtdLg IfF9ekrT53zaw00iL4/dshJ3JFskEslv8/zM3DMiL1R92aAf0DoD/nQkApq2TLvakLVKm6 JMNQYOkBqh6THk4wWnZ0DWkY8cBeMh3L0U48QBMlF0zH8BPqgHvVP7sYk8Fr9ibp3m4U/f IIPwAGEljkonpLg91RRR8KIIURxA4ntxGuOgaa4vNVu/eGQdR6AfV65dGGKuSUEZlofQn+ Lbs54uqKDznESyCkSF7WYIy54RLm0C/PsLYyhteItzcgU6wxuqSryoDL7oRaKg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PhllF069HzF7G; Wed, 22 Mar 2023 23:57:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 32MNvSDE048635; Wed, 22 Mar 2023 23:57:28 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 32MNvSbm048634; Wed, 22 Mar 2023 23:57:28 GMT (envelope-from git) Date: Wed, 22 Mar 2023 23:57:28 GMT Message-Id: <202303222357.32MNvSbm048634@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mateusz Guzik Subject: git: b5d43972e394 - main - vfs: decouple freevnodes from vnode batching List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mjg X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: b5d43972e3941d6c03d935dc423385f3c2178d68 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by mjg: URL: https://cgit.FreeBSD.org/src/commit/?id=b5d43972e3941d6c03d935dc423385f3c2178d68 commit b5d43972e3941d6c03d935dc423385f3c2178d68 Author: Mateusz Guzik AuthorDate: 2023-03-21 07:27:25 +0000 Commit: Mateusz Guzik CommitDate: 2023-03-22 23:57:25 +0000 vfs: decouple freevnodes from vnode batching In principle one cpu can keep vholding vnodes, while another vdrops them. In this case it may be the local count will keep growing in an unbounded manner. Roll it up after a threshold instead. While here move it out of dpcpu into struct pcpu. Reviewed by: kib (previous version) Differential Revision: https://reviews.freebsd.org/D39195 --- sys/kern/vfs_subr.c | 55 ++++++++++++++++++++++++++++++++--------------------- sys/sys/pcpu.h | 3 ++- 2 files changed, 35 insertions(+), 23 deletions(-) diff --git a/sys/kern/vfs_subr.c b/sys/kern/vfs_subr.c index c117ada017c9..d89e0ed13dd5 100644 --- a/sys/kern/vfs_subr.c +++ b/sys/kern/vfs_subr.c @@ -287,7 +287,6 @@ SYSCTL_INT(_debug, OID_AUTO, rush_requests, CTLFLAG_RW, &stat_rush_requests, 0, #define VDBATCH_SIZE 8 struct vdbatch { u_int index; - long freevnodes; struct mtx lock; struct vnode *tab[VDBATCH_SIZE]; }; @@ -1418,48 +1417,62 @@ static int vnlruproc_sig; * at any given moment can still exceed slop, but it should not be by significant * margin in practice. */ -#define VNLRU_FREEVNODES_SLOP 128 +#define VNLRU_FREEVNODES_SLOP 126 + +static void __noinline +vfs_freevnodes_rollup(int8_t *lfreevnodes) +{ + + atomic_add_long(&freevnodes, *lfreevnodes); + *lfreevnodes = 0; + critical_exit(); +} static __inline void vfs_freevnodes_inc(void) { - struct vdbatch *vd; + int8_t *lfreevnodes; critical_enter(); - vd = DPCPU_PTR(vd); - vd->freevnodes++; - critical_exit(); + lfreevnodes = PCPU_PTR(vfs_freevnodes); + (*lfreevnodes)++; + if (__predict_false(*lfreevnodes == VNLRU_FREEVNODES_SLOP)) + vfs_freevnodes_rollup(lfreevnodes); + else + critical_exit(); } static __inline void vfs_freevnodes_dec(void) { - struct vdbatch *vd; + int8_t *lfreevnodes; critical_enter(); - vd = DPCPU_PTR(vd); - vd->freevnodes--; - critical_exit(); + lfreevnodes = PCPU_PTR(vfs_freevnodes); + (*lfreevnodes)--; + if (__predict_false(*lfreevnodes == -VNLRU_FREEVNODES_SLOP)) + vfs_freevnodes_rollup(lfreevnodes); + else + critical_exit(); } static u_long vnlru_read_freevnodes(void) { - struct vdbatch *vd; - long slop; + long slop, rfreevnodes; int cpu; - mtx_assert(&vnode_list_mtx, MA_OWNED); - if (freevnodes > freevnodes_old) - slop = freevnodes - freevnodes_old; + rfreevnodes = atomic_load_long(&freevnodes); + + if (rfreevnodes > freevnodes_old) + slop = rfreevnodes - freevnodes_old; else - slop = freevnodes_old - freevnodes; + slop = freevnodes_old - rfreevnodes; if (slop < VNLRU_FREEVNODES_SLOP) - return (freevnodes >= 0 ? freevnodes : 0); - freevnodes_old = freevnodes; + return (rfreevnodes >= 0 ? rfreevnodes : 0); + freevnodes_old = rfreevnodes; CPU_FOREACH(cpu) { - vd = DPCPU_ID_PTR((cpu), vd); - freevnodes_old += vd->freevnodes; + freevnodes_old += cpuid_to_pcpu[cpu]->pc_vfs_freevnodes; } return (freevnodes_old >= 0 ? freevnodes_old : 0); } @@ -3513,7 +3526,6 @@ vdbatch_process(struct vdbatch *vd) mtx_lock(&vnode_list_mtx); critical_enter(); - freevnodes += vd->freevnodes; for (i = 0; i < VDBATCH_SIZE; i++) { vp = vd->tab[i]; TAILQ_REMOVE(&vnode_list, vp, v_vnodelist); @@ -3522,7 +3534,6 @@ vdbatch_process(struct vdbatch *vd) vp->v_dbatchcpu = NOCPU; } mtx_unlock(&vnode_list_mtx); - vd->freevnodes = 0; bzero(vd->tab, sizeof(vd->tab)); vd->index = 0; critical_exit(); diff --git a/sys/sys/pcpu.h b/sys/sys/pcpu.h index 14bef5cf690e..9ac4144d3267 100644 --- a/sys/sys/pcpu.h +++ b/sys/sys/pcpu.h @@ -189,7 +189,8 @@ struct pcpu { long pc_cp_time[CPUSTATES]; /* statclock ticks */ struct _device *pc_device; /* CPU device handle */ void *pc_netisr; /* netisr SWI cookie */ - int pc_unused1; /* unused field */ + int8_t pc_vfs_freevnodes; /* freevnodes counter */ + char pc_unused1[3]; /* unused pad */ int pc_domain; /* Memory domain. */ struct rm_queue pc_rm_queue; /* rmlock list of trackers */ uintptr_t pc_dynamic; /* Dynamic per-cpu data area */