From owner-svn-src-head@freebsd.org Tue Jul 21 14:41:25 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EAC8035E44F; Tue, 21 Jul 2020 14:41:25 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BB1W95tDQz3TcD; Tue, 21 Jul 2020 14:41:25 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AE2271FD98; Tue, 21 Jul 2020 14:41:25 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 06LEfPYI021999; Tue, 21 Jul 2020 14:41:25 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 06LEfPA9021998; Tue, 21 Jul 2020 14:41:25 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <202007211441.06LEfPA9021998@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Tue, 21 Jul 2020 14:41:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r363393 - head/sys/kern X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: head/sys/kern X-SVN-Commit-Revision: 363393 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jul 2020 14:41:26 -0000 Author: mjg Date: Tue Jul 21 14:41:25 2020 New Revision: 363393 URL: https://svnweb.freebsd.org/changeset/base/363393 Log: lockmgr: rewrite upgrade to stop always dropping the lock This matches rw and sx locks. Modified: head/sys/kern/kern_lock.c Modified: head/sys/kern/kern_lock.c ============================================================================== --- head/sys/kern/kern_lock.c Tue Jul 21 14:39:20 2020 (r363392) +++ head/sys/kern/kern_lock.c Tue Jul 21 14:41:25 2020 (r363393) @@ -878,9 +878,8 @@ static __noinline int lockmgr_upgrade(struct lock *lk, u_int flags, struct lock_object *ilk, const char *file, int line, struct lockmgr_wait *lwa) { - uintptr_t tid, x, v; + uintptr_t tid, v, setv; int error = 0; - int wakeup_swapper = 0; int op; if (KERNEL_PANICKED()) @@ -889,48 +888,47 @@ lockmgr_upgrade(struct lock *lk, u_int flags, struct l tid = (uintptr_t)curthread; _lockmgr_assert(lk, KA_SLOCKED, file, line); - v = lockmgr_read_value(lk); - x = v & LK_ALL_WAITERS; - v &= LK_EXCLUSIVE_SPINNERS; - /* - * Try to switch from one shared lock to an exclusive one. - * We need to preserve waiters flags during the operation. - */ - if (atomic_cmpset_ptr(&lk->lk_lock, LK_SHARERS_LOCK(1) | x | v, - tid | x)) { - LOCK_LOG_LOCK("XUPGRADE", &lk->lock_object, 0, 0, file, - line); - WITNESS_UPGRADE(&lk->lock_object, LOP_EXCLUSIVE | - LK_TRYWIT(flags), file, line); - LOCKSTAT_RECORD0(lockmgr__upgrade, lk); - TD_SLOCKS_DEC(curthread); - goto out; - } - op = flags & LK_TYPE_MASK; + v = lockmgr_read_value(lk); + for (;;) { + if (LK_SHARERS_LOCK(v) > 1) { + if (op == LK_TRYUPGRADE) { + LOCK_LOG2(lk, "%s: %p failed the nowait upgrade", + __func__, lk); + error = EBUSY; + goto out; + } + if (lockmgr_sunlock_try(lk, &v)) { + lockmgr_note_shared_release(lk, file, line); + goto out_xlock; + } + } + MPASS((v & ~LK_ALL_WAITERS) == LK_SHARERS_LOCK(1)); - /* - * In LK_TRYUPGRADE mode, do not drop the lock, - * returning EBUSY instead. - */ - if (op == LK_TRYUPGRADE) { - LOCK_LOG2(lk, "%s: %p failed the nowait upgrade", - __func__, lk); - error = EBUSY; - goto out; + setv = tid; + setv |= (v & LK_ALL_WAITERS); + + /* + * Try to switch from one shared lock to an exclusive one. + * We need to preserve waiters flags during the operation. + */ + if (atomic_fcmpset_ptr(&lk->lk_lock, &v, setv)) { + LOCK_LOG_LOCK("XUPGRADE", &lk->lock_object, 0, 0, file, + line); + WITNESS_UPGRADE(&lk->lock_object, LOP_EXCLUSIVE | + LK_TRYWIT(flags), file, line); + LOCKSTAT_RECORD0(lockmgr__upgrade, lk); + TD_SLOCKS_DEC(curthread); + goto out; + } } - /* - * We have been unable to succeed in upgrading, so just - * give up the shared lock. - */ - lockmgr_note_shared_release(lk, file, line); - wakeup_swapper |= wakeupshlk(lk, file, line); +out_xlock: error = lockmgr_xlock_hard(lk, flags, ilk, file, line, lwa); flags &= ~LK_INTERLOCK; out: - lockmgr_exit(flags, ilk, wakeup_swapper); + lockmgr_exit(flags, ilk, 0); return (error); }