From owner-svn-src-stable@freebsd.org Thu Mar 16 06:51:02 2017
Return-Path: <owner-svn-src-stable@freebsd.org>
Delivered-To: svn-src-stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
[IPv6:2001:1900:2254:206a::19:1])
by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27C9AD0E171;
Thu, 16 Mar 2017 06:51:02 +0000 (UTC) (envelope-from mjg@FreeBSD.org)
Received: from repo.freebsd.org (repo.freebsd.org
[IPv6:2610:1c1:1:6068::e6a:0])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client did not present a certificate)
by mx1.freebsd.org (Postfix) with ESMTPS id EB6B11025;
Thu, 16 Mar 2017 06:51:01 +0000 (UTC) (envelope-from mjg@FreeBSD.org)
Received: from repo.freebsd.org ([127.0.1.37])
by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v2G6p1P2087520;
Thu, 16 Mar 2017 06:51:01 GMT (envelope-from mjg@FreeBSD.org)
Received: (from mjg@localhost)
by repo.freebsd.org (8.15.2/8.15.2/Submit) id v2G6p0Jl087518;
Thu, 16 Mar 2017 06:51:00 GMT (envelope-from mjg@FreeBSD.org)
Message-Id: <201703160651.v2G6p0Jl087518@repo.freebsd.org>
X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org
using -f
From: Mateusz Guzik <mjg@FreeBSD.org>
Date: Thu, 16 Mar 2017 06:51:00 +0000 (UTC)
To: src-committers@freebsd.org, svn-src-all@freebsd.org,
svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org
Subject: svn commit: r315379 - in stable/11/sys: kern sys
X-SVN-Group: stable-11
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-BeenThere: svn-src-stable@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: SVN commit messages for all the -stable branches of the src tree
<svn-src-stable.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/svn-src-stable>,
<mailto:svn-src-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/svn-src-stable/>
List-Post: <mailto:svn-src-stable@freebsd.org>
List-Help: <mailto:svn-src-stable-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/svn-src-stable>,
<mailto:svn-src-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 16 Mar 2017 06:51:02 -0000
Author: mjg
Date: Thu Mar 16 06:51:00 2017
New Revision: 315379
URL: https://svnweb.freebsd.org/changeset/base/315379
Log:
MFC r313392,r313784:
rwlock: implement RW_LOCK_WRITER_RECURSED bit
This moves recursion handling out of the inlined wunlock path and in
particular saves a read and a branch.
==
rwlock: tidy up r313392
While a new bit was added and thread alignment got shifted to accomodate it,
RW_READERS_SHIFT was not modified accordingly and clashed with the new flag.
This was surprisingly harmless. If the lock was taken for writing, other flags
were tested. If the lock was taken for reading, it would correctly work for
readers > 1 and this was the only relevant test performed.
Modified:
stable/11/sys/kern/kern_rwlock.c
stable/11/sys/sys/rwlock.h
Directory Properties:
stable/11/ (props changed)
Modified: stable/11/sys/kern/kern_rwlock.c
==============================================================================
--- stable/11/sys/kern/kern_rwlock.c Thu Mar 16 06:45:36 2017 (r315378)
+++ stable/11/sys/kern/kern_rwlock.c Thu Mar 16 06:51:00 2017 (r315379)
@@ -312,6 +312,7 @@ __rw_try_wlock(volatile uintptr_t *c, co
if (rw_wlocked(rw) &&
(rw->lock_object.lo_flags & LO_RECURSABLE) != 0) {
rw->rw_recurse++;
+ atomic_set_ptr(&rw->rw_lock, RW_LOCK_WRITER_RECURSED);
rval = 1;
} else
rval = atomic_cmpset_acq_ptr(&rw->rw_lock, RW_UNLOCKED,
@@ -345,10 +346,8 @@ _rw_wunlock_cookie(volatile uintptr_t *c
WITNESS_UNLOCK(&rw->lock_object, LOP_EXCLUSIVE, file, line);
LOCK_LOG_LOCK("WUNLOCK", &rw->lock_object, 0, rw->rw_recurse, file,
line);
- if (rw->rw_recurse)
- rw->rw_recurse--;
- else
- _rw_wunlock_hard(rw, (uintptr_t)curthread, file, line);
+
+ _rw_wunlock_hard(rw, (uintptr_t)curthread, file, line);
TD_LOCKS_DEC(curthread);
}
@@ -802,6 +801,7 @@ __rw_wlock_hard(volatile uintptr_t *c, u
("%s: recursing but non-recursive rw %s @ %s:%d\n",
__func__, rw->lock_object.lo_name, file, line));
rw->rw_recurse++;
+ atomic_set_ptr(&rw->rw_lock, RW_LOCK_WRITER_RECURSED);
if (LOCK_LOG_TEST(&rw->lock_object, 0))
CTR2(KTR_LOCK, "%s: %p recursing", __func__, rw);
return;
@@ -994,12 +994,17 @@ __rw_wunlock_hard(volatile uintptr_t *c,
return;
rw = rwlock2rw(c);
- MPASS(!rw_recursed(rw));
- LOCKSTAT_PROFILE_RELEASE_RWLOCK(rw__release, rw,
- LOCKSTAT_WRITER);
- if (_rw_write_unlock(rw, tid))
+ if (!rw_recursed(rw)) {
+ LOCKSTAT_PROFILE_RELEASE_RWLOCK(rw__release, rw,
+ LOCKSTAT_WRITER);
+ if (_rw_write_unlock(rw, tid))
+ return;
+ } else {
+ if (--(rw->rw_recurse) == 0)
+ atomic_clear_ptr(&rw->rw_lock, RW_LOCK_WRITER_RECURSED);
return;
+ }
KASSERT(rw->rw_lock & (RW_LOCK_READ_WAITERS | RW_LOCK_WRITE_WAITERS),
("%s: neither of the waiter flags are set", __func__));
Modified: stable/11/sys/sys/rwlock.h
==============================================================================
--- stable/11/sys/sys/rwlock.h Thu Mar 16 06:45:36 2017 (r315378)
+++ stable/11/sys/sys/rwlock.h Thu Mar 16 06:51:00 2017 (r315379)
@@ -58,13 +58,14 @@
#define RW_LOCK_READ_WAITERS 0x02
#define RW_LOCK_WRITE_WAITERS 0x04
#define RW_LOCK_WRITE_SPINNER 0x08
+#define RW_LOCK_WRITER_RECURSED 0x10
#define RW_LOCK_FLAGMASK \
(RW_LOCK_READ | RW_LOCK_READ_WAITERS | RW_LOCK_WRITE_WAITERS | \
- RW_LOCK_WRITE_SPINNER)
+ RW_LOCK_WRITE_SPINNER | RW_LOCK_WRITER_RECURSED)
#define RW_LOCK_WAITERS (RW_LOCK_READ_WAITERS | RW_LOCK_WRITE_WAITERS)
#define RW_OWNER(x) ((x) & ~RW_LOCK_FLAGMASK)
-#define RW_READERS_SHIFT 4
+#define RW_READERS_SHIFT 5
#define RW_READERS(x) (RW_OWNER((x)) >> RW_READERS_SHIFT)
#define RW_READERS_LOCK(x) ((x) << RW_READERS_SHIFT | RW_LOCK_READ)
#define RW_ONE_READER (1 << RW_READERS_SHIFT)
@@ -111,13 +112,9 @@
#define __rw_wunlock(rw, tid, file, line) do { \
uintptr_t _tid = (uintptr_t)(tid); \
\
- if ((rw)->rw_recurse) \
- (rw)->rw_recurse--; \
- else { \
- if (__predict_false(LOCKSTAT_PROFILE_ENABLED(rw__release) ||\
- !_rw_write_unlock((rw), _tid))) \
- _rw_wunlock_hard((rw), _tid, (file), (line)); \
- } \
+ if (__predict_false(LOCKSTAT_PROFILE_ENABLED(rw__release) || \
+ !_rw_write_unlock((rw), _tid))) \
+ _rw_wunlock_hard((rw), _tid, (file), (line)); \
} while (0)
/*