From owner-dev-commits-src-all@freebsd.org Fri Jun 25 18:12:30 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E31C16405E3; Fri, 25 Jun 2021 18:12:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GBQ8G635Jz4tZW; Fri, 25 Jun 2021 18:12:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id ADF0F7E77; Fri, 25 Jun 2021 18:12:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 15PICUMD053127; Fri, 25 Jun 2021 18:12:30 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 15PICUbF053117; Fri, 25 Jun 2021 18:12:30 GMT (envelope-from git) Date: Fri, 25 Jun 2021 18:12:30 GMT Message-Id: <202106251812.15PICUbF053117@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alexander Motin Subject: git: 6df35af4d85c - main - Allow sleepq_signal() to drop the lock. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: mav X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6df35af4d85c6311d8e762521580e7176b69394e Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jun 2021 18:12:31 -0000 The branch main has been updated by mav: URL: https://cgit.FreeBSD.org/src/commit/?id=6df35af4d85c6311d8e762521580e7176b69394e commit 6df35af4d85c6311d8e762521580e7176b69394e Author: Alexander Motin AuthorDate: 2021-06-25 17:52:58 +0000 Commit: Alexander Motin CommitDate: 2021-06-25 18:12:21 +0000 Allow sleepq_signal() to drop the lock. Introduce SLEEPQ_DROP sleepq_signal() flag, allowing one to drop the sleep queue chain lock before returning. Reduced lock scope allows significantly reduce lock contention inside taskqueue_enqueue() for ZFS worker threads doing ~350K disk reads/s on 40-thread system. MFC after: 2 weeks Sponsored by: iXsystems, Inc. --- sys/kern/kern_synch.c | 8 +++----- sys/kern/subr_sleepqueue.c | 8 ++++++-- sys/sys/sleepqueue.h | 1 + 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/sys/kern/kern_synch.c b/sys/kern/kern_synch.c index dcca67326264..793c5309a30b 100644 --- a/sys/kern/kern_synch.c +++ b/sys/kern/kern_synch.c @@ -366,8 +366,7 @@ wakeup_one(const void *ident) int wakeup_swapper; sleepq_lock(ident); - wakeup_swapper = sleepq_signal(ident, SLEEPQ_SLEEP, 0, 0); - sleepq_release(ident); + wakeup_swapper = sleepq_signal(ident, SLEEPQ_SLEEP | SLEEPQ_DROP, 0, 0); if (wakeup_swapper) kick_proc0(); } @@ -378,9 +377,8 @@ wakeup_any(const void *ident) int wakeup_swapper; sleepq_lock(ident); - wakeup_swapper = sleepq_signal(ident, SLEEPQ_SLEEP | SLEEPQ_UNFAIR, - 0, 0); - sleepq_release(ident); + wakeup_swapper = sleepq_signal(ident, SLEEPQ_SLEEP | SLEEPQ_UNFAIR | + SLEEPQ_DROP, 0, 0); if (wakeup_swapper) kick_proc0(); } diff --git a/sys/kern/subr_sleepqueue.c b/sys/kern/subr_sleepqueue.c index 0718f01fa48a..b146a978a60c 100644 --- a/sys/kern/subr_sleepqueue.c +++ b/sys/kern/subr_sleepqueue.c @@ -927,8 +927,11 @@ sleepq_signal(const void *wchan, int flags, int pri, int queue) KASSERT(wchan != NULL, ("%s: invalid NULL wait channel", __func__)); MPASS((queue >= 0) && (queue < NR_SLEEPQS)); sq = sleepq_lookup(wchan); - if (sq == NULL) + if (sq == NULL) { + if (flags & SLEEPQ_DROP) + sleepq_release(wchan); return (0); + } KASSERT(sq->sq_type == (flags & SLEEPQ_TYPE), ("%s: mismatch between sleep/wakeup and cv_*", __func__)); @@ -961,7 +964,8 @@ sleepq_signal(const void *wchan, int flags, int pri, int queue) } } MPASS(besttd != NULL); - wakeup_swapper = sleepq_resume_thread(sq, besttd, pri, SRQ_HOLD); + wakeup_swapper = sleepq_resume_thread(sq, besttd, pri, + (flags & SLEEPQ_DROP) ? 0 : SRQ_HOLD); return (wakeup_swapper); } diff --git a/sys/sys/sleepqueue.h b/sys/sys/sleepqueue.h index 18c7568777b6..6715894ed1f3 100644 --- a/sys/sys/sleepqueue.h +++ b/sys/sys/sleepqueue.h @@ -85,6 +85,7 @@ struct thread; #define SLEEPQ_LK 0x04 /* Used by a lockmgr. */ #define SLEEPQ_INTERRUPTIBLE 0x100 /* Sleep is interruptible. */ #define SLEEPQ_UNFAIR 0x200 /* Unfair wakeup order. */ +#define SLEEPQ_DROP 0x400 /* Return without lock held. */ void init_sleepqueues(void); int sleepq_abort(struct thread *td, int intrval);