From owner-dev-commits-src-all@freebsd.org Fri Jul 9 01:38:37 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9C9436573A8; Fri, 9 Jul 2021 01:38:37 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GLbR12dXcz4bLT; Fri, 9 Jul 2021 01:38:37 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 418FF3ABE; Fri, 9 Jul 2021 01:38:37 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1691cbRm014749; Fri, 9 Jul 2021 01:38:37 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1691cbYk014748; Fri, 9 Jul 2021 01:38:37 GMT (envelope-from git) Date: Fri, 9 Jul 2021 01:38:37 GMT Message-Id: <202107090138.1691cbYk014748@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Alexander Motin Subject: git: 4a58aca40df9 - stable/13 - Allow sleepq_signal() to drop the lock. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mav X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 4a58aca40df9a5a5e5db430f3744a2712f955ed7 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jul 2021 01:38:37 -0000 The branch stable/13 has been updated by mav: URL: https://cgit.FreeBSD.org/src/commit/?id=4a58aca40df9a5a5e5db430f3744a2712f955ed7 commit 4a58aca40df9a5a5e5db430f3744a2712f955ed7 Author: Alexander Motin AuthorDate: 2021-06-25 17:52:58 +0000 Commit: Alexander Motin CommitDate: 2021-07-09 01:38:32 +0000 Allow sleepq_signal() to drop the lock. Introduce SLEEPQ_DROP sleepq_signal() flag, allowing one to drop the sleep queue chain lock before returning. Reduced lock scope allows significantly reduce lock contention inside taskqueue_enqueue() for ZFS worker threads doing ~350K disk reads/s on 40-thread system. MFC after: 2 weeks Sponsored by: iXsystems, Inc. (cherry picked from commit 6df35af4d85c6311d8e762521580e7176b69394e) --- sys/kern/kern_synch.c | 8 +++----- sys/kern/subr_sleepqueue.c | 8 ++++++-- sys/sys/sleepqueue.h | 1 + 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/sys/kern/kern_synch.c b/sys/kern/kern_synch.c index 4c0491ab6e85..b63877e26b68 100644 --- a/sys/kern/kern_synch.c +++ b/sys/kern/kern_synch.c @@ -366,8 +366,7 @@ wakeup_one(const void *ident) int wakeup_swapper; sleepq_lock(ident); - wakeup_swapper = sleepq_signal(ident, SLEEPQ_SLEEP, 0, 0); - sleepq_release(ident); + wakeup_swapper = sleepq_signal(ident, SLEEPQ_SLEEP | SLEEPQ_DROP, 0, 0); if (wakeup_swapper) kick_proc0(); } @@ -378,9 +377,8 @@ wakeup_any(const void *ident) int wakeup_swapper; sleepq_lock(ident); - wakeup_swapper = sleepq_signal(ident, SLEEPQ_SLEEP | SLEEPQ_UNFAIR, - 0, 0); - sleepq_release(ident); + wakeup_swapper = sleepq_signal(ident, SLEEPQ_SLEEP | SLEEPQ_UNFAIR | + SLEEPQ_DROP, 0, 0); if (wakeup_swapper) kick_proc0(); } diff --git a/sys/kern/subr_sleepqueue.c b/sys/kern/subr_sleepqueue.c index 0718f01fa48a..b146a978a60c 100644 --- a/sys/kern/subr_sleepqueue.c +++ b/sys/kern/subr_sleepqueue.c @@ -927,8 +927,11 @@ sleepq_signal(const void *wchan, int flags, int pri, int queue) KASSERT(wchan != NULL, ("%s: invalid NULL wait channel", __func__)); MPASS((queue >= 0) && (queue < NR_SLEEPQS)); sq = sleepq_lookup(wchan); - if (sq == NULL) + if (sq == NULL) { + if (flags & SLEEPQ_DROP) + sleepq_release(wchan); return (0); + } KASSERT(sq->sq_type == (flags & SLEEPQ_TYPE), ("%s: mismatch between sleep/wakeup and cv_*", __func__)); @@ -961,7 +964,8 @@ sleepq_signal(const void *wchan, int flags, int pri, int queue) } } MPASS(besttd != NULL); - wakeup_swapper = sleepq_resume_thread(sq, besttd, pri, SRQ_HOLD); + wakeup_swapper = sleepq_resume_thread(sq, besttd, pri, + (flags & SLEEPQ_DROP) ? 0 : SRQ_HOLD); return (wakeup_swapper); } diff --git a/sys/sys/sleepqueue.h b/sys/sys/sleepqueue.h index 18c7568777b6..6715894ed1f3 100644 --- a/sys/sys/sleepqueue.h +++ b/sys/sys/sleepqueue.h @@ -85,6 +85,7 @@ struct thread; #define SLEEPQ_LK 0x04 /* Used by a lockmgr. */ #define SLEEPQ_INTERRUPTIBLE 0x100 /* Sleep is interruptible. */ #define SLEEPQ_UNFAIR 0x200 /* Unfair wakeup order. */ +#define SLEEPQ_DROP 0x400 /* Return without lock held. */ void init_sleepqueues(void); int sleepq_abort(struct thread *td, int intrval);