From owner-svn-src-head@freebsd.org Tue Apr 10 22:32:32 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 96A7BF9F356; Tue, 10 Apr 2018 22:32:32 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBD47089B; Tue, 10 Apr 2018 22:32:32 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 45D5113C47; Tue, 10 Apr 2018 22:32:32 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w3AMWWeU051027; Tue, 10 Apr 2018 22:32:32 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w3AMWWDl051026; Tue, 10 Apr 2018 22:32:32 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <201804102232.w3AMWWDl051026@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Tue, 10 Apr 2018 22:32:32 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r332394 - head/sys/kern X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: head/sys/kern X-SVN-Commit-Revision: 332394 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Apr 2018 22:32:32 -0000 Author: mjg Date: Tue Apr 10 22:32:31 2018 New Revision: 332394 URL: https://svnweb.freebsd.org/changeset/base/332394 Log: rw: whack avoidable re-reads in try_upgrade Modified: head/sys/kern/kern_rwlock.c Modified: head/sys/kern/kern_rwlock.c ============================================================================== --- head/sys/kern/kern_rwlock.c Tue Apr 10 21:14:54 2018 (r332393) +++ head/sys/kern/kern_rwlock.c Tue Apr 10 22:32:31 2018 (r332394) @@ -1204,7 +1204,7 @@ __rw_wunlock_hard(volatile uintptr_t *c, uintptr_t v L int __rw_try_upgrade_int(struct rwlock *rw LOCK_FILE_LINE_ARG_DEF) { - uintptr_t v, x, tid; + uintptr_t v, setv, tid; struct turnstile *ts; int success; @@ -1224,12 +1224,12 @@ __rw_try_upgrade_int(struct rwlock *rw LOCK_FILE_LINE_ */ tid = (uintptr_t)curthread; success = 0; + v = RW_READ_VALUE(rw); for (;;) { - v = rw->rw_lock; if (RW_READERS(v) > 1) break; if (!(v & RW_LOCK_WAITERS)) { - success = atomic_cmpset_acq_ptr(&rw->rw_lock, v, tid); + success = atomic_fcmpset_acq_ptr(&rw->rw_lock, &v, tid); if (!success) continue; break; @@ -1239,7 +1239,8 @@ __rw_try_upgrade_int(struct rwlock *rw LOCK_FILE_LINE_ * Ok, we think we have waiters, so lock the turnstile. */ ts = turnstile_trywait(&rw->lock_object); - v = rw->rw_lock; + v = RW_READ_VALUE(rw); +retry_ts: if (RW_READERS(v) > 1) { turnstile_cancel(ts); break; @@ -1250,16 +1251,16 @@ __rw_try_upgrade_int(struct rwlock *rw LOCK_FILE_LINE_ * If we obtain the lock with the flags set, then claim * ownership of the turnstile. */ - x = rw->rw_lock & RW_LOCK_WAITERS; - success = atomic_cmpset_ptr(&rw->rw_lock, v, tid | x); + setv = tid | (v & RW_LOCK_WAITERS); + success = atomic_fcmpset_ptr(&rw->rw_lock, &v, setv); if (success) { - if (x) + if (v & RW_LOCK_WAITERS) turnstile_claim(ts); else turnstile_cancel(ts); break; } - turnstile_cancel(ts); + goto retry_ts; } LOCK_LOG_TRY("WUPGRADE", &rw->lock_object, 0, success, file, line); if (success) {