From owner-svn-src-head@freebsd.org Tue Jul 21 14:42:23 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1683735EB97; Tue, 21 Jul 2020 14:42:23 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BB1XG6tPfz3V7j; Tue, 21 Jul 2020 14:42:22 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CFBAD1FBFD; Tue, 21 Jul 2020 14:42:22 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 06LEgMLx024704; Tue, 21 Jul 2020 14:42:22 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 06LEgM0R024701; Tue, 21 Jul 2020 14:42:22 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <202007211442.06LEgM0R024701@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Tue, 21 Jul 2020 14:42:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r363394 - in head/sys: kern sys X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: in head/sys: kern sys X-SVN-Commit-Revision: 363394 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jul 2020 14:42:23 -0000 Author: mjg Date: Tue Jul 21 14:42:22 2020 New Revision: 363394 URL: https://svnweb.freebsd.org/changeset/base/363394 Log: lockmgr: denote recursion with a bit in lock value This reduces excessive reads from the lock. Tested by: pho Modified: head/sys/kern/kern_lock.c head/sys/sys/lockmgr.h Modified: head/sys/kern/kern_lock.c ============================================================================== --- head/sys/kern/kern_lock.c Tue Jul 21 14:41:25 2020 (r363393) +++ head/sys/kern/kern_lock.c Tue Jul 21 14:42:22 2020 (r363394) @@ -736,6 +736,7 @@ lockmgr_xlock_hard(struct lock *lk, u_int flags, struc panic("%s: recursing on non recursive lockmgr %p " "@ %s:%d\n", __func__, lk, file, line); } + atomic_set_ptr(&lk->lk_lock, LK_WRITER_RECURSED); lk->lk_recurse++; LOCK_LOG2(lk, "%s: %p recursing", __func__, lk); LOCK_LOG_LOCK("XLOCK", &lk->lock_object, 0, @@ -1039,9 +1040,11 @@ lockmgr_xunlock_hard(struct lock *lk, uintptr_t x, u_i * The lock is held in exclusive mode. * If the lock is recursed also, then unrecurse it. */ - if (lockmgr_xlocked_v(x) && lockmgr_recursed(lk)) { + if (lockmgr_recursed_v(x)) { LOCK_LOG2(lk, "%s: %p unrecursing", __func__, lk); lk->lk_recurse--; + if (lk->lk_recurse == 0) + atomic_clear_ptr(&lk->lk_lock, LK_WRITER_RECURSED); goto out; } if (tid != LK_KERNPROC) @@ -1187,9 +1190,8 @@ lockmgr_unlock(struct lock *lk) } else { tid = (uintptr_t)curthread; lockmgr_note_exclusive_release(lk, file, line); - if (!lockmgr_recursed(lk) && - atomic_cmpset_rel_ptr(&lk->lk_lock, tid, LK_UNLOCKED)) { - LOCKSTAT_PROFILE_RELEASE_RWLOCK(lockmgr__release, lk, LOCKSTAT_WRITER); + if (x == tid && atomic_cmpset_rel_ptr(&lk->lk_lock, tid, LK_UNLOCKED)) { + LOCKSTAT_PROFILE_RELEASE_RWLOCK(lockmgr__release, lk,LOCKSTAT_WRITER); } else { return (lockmgr_xunlock_hard(lk, x, LK_RELEASE, NULL, file, line)); } Modified: head/sys/sys/lockmgr.h ============================================================================== --- head/sys/sys/lockmgr.h Tue Jul 21 14:41:25 2020 (r363393) +++ head/sys/sys/lockmgr.h Tue Jul 21 14:42:22 2020 (r363394) @@ -42,13 +42,14 @@ #define LK_SHARED_WAITERS 0x02 #define LK_EXCLUSIVE_WAITERS 0x04 #define LK_EXCLUSIVE_SPINNERS 0x08 +#define LK_WRITER_RECURSED 0x10 #define LK_ALL_WAITERS \ (LK_SHARED_WAITERS | LK_EXCLUSIVE_WAITERS) #define LK_FLAGMASK \ - (LK_SHARE | LK_ALL_WAITERS | LK_EXCLUSIVE_SPINNERS) + (LK_SHARE | LK_ALL_WAITERS | LK_EXCLUSIVE_SPINNERS | LK_WRITER_RECURSED) #define LK_HOLDER(x) ((x) & ~LK_FLAGMASK) -#define LK_SHARERS_SHIFT 4 +#define LK_SHARERS_SHIFT 5 #define LK_SHARERS(x) (LK_HOLDER(x) >> LK_SHARERS_SHIFT) #define LK_SHARERS_LOCK(x) ((x) << LK_SHARERS_SHIFT | LK_SHARE) #define LK_ONE_SHARER (1 << LK_SHARERS_SHIFT) @@ -131,8 +132,10 @@ _lockmgr_args_rw(struct lock *lk, u_int flags, struct LOCK_FILE, LOCK_LINE) #define lockmgr_disown(lk) \ _lockmgr_disown((lk), LOCK_FILE, LOCK_LINE) +#define lockmgr_recursed_v(v) \ + (v & LK_WRITER_RECURSED) #define lockmgr_recursed(lk) \ - ((lk)->lk_recurse != 0) + lockmgr_recursed_v((lk)->lk_lock) #define lockmgr_rw(lk, flags, ilk) \ _lockmgr_args_rw((lk), (flags), (ilk), LK_WMESG_DEFAULT, \ LK_PRIO_DEFAULT, LK_TIMO_DEFAULT, LOCK_FILE, LOCK_LINE)