From owner-svn-src-stable@freebsd.org Thu Mar 16 07:06:08 2017 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8457FD0E85D; Thu, 16 Mar 2017 07:06:08 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3B67D1A65; Thu, 16 Mar 2017 07:06:08 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v2G767nu094825; Thu, 16 Mar 2017 07:06:07 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v2G767c4094822; Thu, 16 Mar 2017 07:06:07 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <201703160706.v2G767c4094822@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Thu, 16 Mar 2017 07:06:07 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r315382 - stable/11/sys/kern X-SVN-Group: stable-11 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Mar 2017 07:06:08 -0000 Author: mjg Date: Thu Mar 16 07:06:06 2017 New Revision: 315382 URL: https://svnweb.freebsd.org/changeset/base/315382 Log: MFC r313467: locks: tidy up unlock fallback paths Update comments to note these functions are reachable if lockstat is enabled. Check if the lock has any bits set before attempting unlock, which saves an unnecessary atomic operation. Modified: stable/11/sys/kern/kern_mutex.c stable/11/sys/kern/kern_rwlock.c stable/11/sys/kern/kern_sx.c Directory Properties: stable/11/ (props changed) Modified: stable/11/sys/kern/kern_mutex.c ============================================================================== --- stable/11/sys/kern/kern_mutex.c Thu Mar 16 06:56:23 2017 (r315381) +++ stable/11/sys/kern/kern_mutex.c Thu Mar 16 07:06:06 2017 (r315382) @@ -858,25 +858,24 @@ thread_lock_set(struct thread *td, struc /* * __mtx_unlock_sleep: the tougher part of releasing an MTX_DEF lock. * - * We are only called here if the lock is recursed or contested (i.e. we - * need to wake up a blocked thread). + * We are only called here if the lock is recursed, contested (i.e. we + * need to wake up a blocked thread) or lockstat probe is active. */ void __mtx_unlock_sleep(volatile uintptr_t *c, int opts, const char *file, int line) { struct mtx *m; struct turnstile *ts; + uintptr_t tid, v; if (SCHEDULER_STOPPED()) return; + tid = (uintptr_t)curthread; m = mtxlock2mtx(c); + v = MTX_READ_VALUE(m); - if (!mtx_recursed(m)) { - LOCKSTAT_PROFILE_RELEASE_LOCK(adaptive__release, m); - if (_mtx_release_lock(m, (uintptr_t)curthread)) - return; - } else { + if (v & MTX_RECURSED) { if (--(m->mtx_recurse) == 0) atomic_clear_ptr(&m->mtx_lock, MTX_RECURSED); if (LOCK_LOG_TEST(&m->lock_object, opts)) @@ -884,6 +883,10 @@ __mtx_unlock_sleep(volatile uintptr_t *c return; } + LOCKSTAT_PROFILE_RELEASE_LOCK(adaptive__release, m); + if (v == tid && _mtx_release_lock(m, tid)) + return; + /* * We have to lock the chain before the turnstile so this turnstile * can be removed from the hash list if it is empty. Modified: stable/11/sys/kern/kern_rwlock.c ============================================================================== --- stable/11/sys/kern/kern_rwlock.c Thu Mar 16 06:56:23 2017 (r315381) +++ stable/11/sys/kern/kern_rwlock.c Thu Mar 16 07:06:06 2017 (r315382) @@ -1030,9 +1030,10 @@ __rw_wlock_hard(volatile uintptr_t *c, u } /* - * This function is called if the first try at releasing a write lock failed. - * This means that one of the 2 waiter bits must be set indicating that at - * least one thread is waiting on this lock. + * This function is called if lockstat is active or the first try at releasing + * a write lock failed. The latter means that the lock is recursed or one of + * the 2 waiter bits must be set indicating that at least one thread is waiting + * on this lock. */ void __rw_wunlock_hard(volatile uintptr_t *c, uintptr_t tid, const char *file, @@ -1047,18 +1048,19 @@ __rw_wunlock_hard(volatile uintptr_t *c, return; rw = rwlock2rw(c); - - if (!rw_recursed(rw)) { - LOCKSTAT_PROFILE_RELEASE_RWLOCK(rw__release, rw, - LOCKSTAT_WRITER); - if (_rw_write_unlock(rw, tid)) - return; - } else { + v = RW_READ_VALUE(rw); + if (v & RW_LOCK_WRITER_RECURSED) { if (--(rw->rw_recurse) == 0) atomic_clear_ptr(&rw->rw_lock, RW_LOCK_WRITER_RECURSED); + if (LOCK_LOG_TEST(&rw->lock_object, 0)) + CTR2(KTR_LOCK, "%s: %p unrecursing", __func__, rw); return; } + LOCKSTAT_PROFILE_RELEASE_RWLOCK(rw__release, rw, LOCKSTAT_WRITER); + if (v == tid && _rw_write_unlock(rw, tid)) + return; + KASSERT(rw->rw_lock & (RW_LOCK_READ_WAITERS | RW_LOCK_WRITE_WAITERS), ("%s: neither of the waiter flags are set", __func__)); Modified: stable/11/sys/kern/kern_sx.c ============================================================================== --- stable/11/sys/kern/kern_sx.c Thu Mar 16 06:56:23 2017 (r315381) +++ stable/11/sys/kern/kern_sx.c Thu Mar 16 07:06:06 2017 (r315382) @@ -747,12 +747,8 @@ _sx_xunlock_hard(struct sx *sx, uintptr_ MPASS(!(sx->sx_lock & SX_LOCK_SHARED)); - if (!sx_recursed(sx)) { - LOCKSTAT_PROFILE_RELEASE_RWLOCK(sx__release, sx, - LOCKSTAT_WRITER); - if (atomic_cmpset_rel_ptr(&sx->sx_lock, tid, SX_LOCK_UNLOCKED)) - return; - } else { + x = SX_READ_VALUE(sx); + if (x & SX_LOCK_RECURSED) { /* The lock is recursed, unrecurse one level. */ if ((--sx->sx_recurse) == 0) atomic_clear_ptr(&sx->sx_lock, SX_LOCK_RECURSED); @@ -760,6 +756,12 @@ _sx_xunlock_hard(struct sx *sx, uintptr_ CTR2(KTR_LOCK, "%s: %p unrecursing", __func__, sx); return; } + + LOCKSTAT_PROFILE_RELEASE_RWLOCK(sx__release, sx, LOCKSTAT_WRITER); + if (x == tid && + atomic_cmpset_rel_ptr(&sx->sx_lock, tid, SX_LOCK_UNLOCKED)) + return; + MPASS(sx->sx_lock & (SX_LOCK_SHARED_WAITERS | SX_LOCK_EXCLUSIVE_WAITERS)); if (LOCK_LOG_TEST(&sx->lock_object, 0))