From owner-svn-src-head@freebsd.org Thu Oct 5 19:18:04 2017 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1694DE401D9; Thu, 5 Oct 2017 19:18:04 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E102A82C82; Thu, 5 Oct 2017 19:18:03 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v95JI262016037; Thu, 5 Oct 2017 19:18:02 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v95JI2aA016034; Thu, 5 Oct 2017 19:18:02 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <201710051918.v95JI2aA016034@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Thu, 5 Oct 2017 19:18:02 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r324335 - in head/sys: kern sys X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: in head/sys: kern sys X-SVN-Commit-Revision: 324335 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Oct 2017 19:18:04 -0000 Author: mjg Date: Thu Oct 5 19:18:02 2017 New Revision: 324335 URL: https://svnweb.freebsd.org/changeset/base/324335 Log: locks: take the number of readers into account when waiting Previous code would always spin once before checking the lock. But a lock with e.g. 6 readers is not going to become free in the duration of once spin even if they start draining immediately. Conservatively perform one for each reader. Note that the total number of allowed spins is still extremely small and is subject to change later. MFC after: 1 week Modified: head/sys/kern/kern_rwlock.c head/sys/kern/kern_sx.c head/sys/sys/lock.h Modified: head/sys/kern/kern_rwlock.c ============================================================================== --- head/sys/kern/kern_rwlock.c Thu Oct 5 19:11:25 2017 (r324334) +++ head/sys/kern/kern_rwlock.c Thu Oct 5 19:18:02 2017 (r324335) @@ -414,7 +414,7 @@ __rw_rlock_hard(volatile uintptr_t *c, struct thread * #ifdef ADAPTIVE_RWLOCKS volatile struct thread *owner; int spintries = 0; - int i; + int i, n; #endif #ifdef LOCK_PROFILING uint64_t waittime = 0; @@ -488,8 +488,9 @@ __rw_rlock_hard(volatile uintptr_t *c, struct thread * KTR_STATE1(KTR_SCHED, "thread", sched_tdname(curthread), "spinning", "lockname:\"%s\"", rw->lock_object.lo_name); - for (i = 0; i < rowner_loops; i++) { - cpu_spinwait(); + for (i = 0; i < rowner_loops; i += n) { + n = RW_READERS(v); + lock_delay_spin(n); v = RW_READ_VALUE(rw); if ((v & RW_LOCK_READ) == 0 || RW_CAN_READ(td, v)) break; @@ -830,7 +831,7 @@ __rw_wlock_hard(volatile uintptr_t *c, uintptr_t v, ui #ifdef ADAPTIVE_RWLOCKS volatile struct thread *owner; int spintries = 0; - int i; + int i, n; #endif uintptr_t x; #ifdef LOCK_PROFILING @@ -928,8 +929,9 @@ __rw_wlock_hard(volatile uintptr_t *c, uintptr_t v, ui KTR_STATE1(KTR_SCHED, "thread", sched_tdname(curthread), "spinning", "lockname:\"%s\"", rw->lock_object.lo_name); - for (i = 0; i < rowner_loops; i++) { - cpu_spinwait(); + for (i = 0; i < rowner_loops; i += n) { + n = RW_READERS(v); + lock_delay_spin(n); v = RW_READ_VALUE(rw); if ((v & RW_LOCK_WRITE_SPINNER) == 0) break; Modified: head/sys/kern/kern_sx.c ============================================================================== --- head/sys/kern/kern_sx.c Thu Oct 5 19:11:25 2017 (r324334) +++ head/sys/kern/kern_sx.c Thu Oct 5 19:18:02 2017 (r324335) @@ -502,7 +502,7 @@ _sx_xlock_hard(struct sx *sx, uintptr_t x, uintptr_t t GIANT_DECLARE; #ifdef ADAPTIVE_SX volatile struct thread *owner; - u_int i, spintries = 0; + u_int i, n, spintries = 0; #endif #ifdef LOCK_PROFILING uint64_t waittime = 0; @@ -600,12 +600,13 @@ _sx_xlock_hard(struct sx *sx, uintptr_t x, uintptr_t t "lockname:\"%s\"", sx->lock_object.lo_name); GIANT_SAVE(); spintries++; - for (i = 0; i < asx_loops; i++) { + for (i = 0; i < asx_loops; i += n) { if (LOCK_LOG_TEST(&sx->lock_object, 0)) CTR4(KTR_LOCK, "%s: shared spinning on %p with %u and %u", __func__, sx, spintries, i); - cpu_spinwait(); + n = SX_SHARERS(x); + lock_delay_spin(n); x = SX_READ_VALUE(sx); if ((x & SX_LOCK_SHARED) == 0 || SX_SHARERS(x) == 0) Modified: head/sys/sys/lock.h ============================================================================== --- head/sys/sys/lock.h Thu Oct 5 19:11:25 2017 (r324334) +++ head/sys/sys/lock.h Thu Oct 5 19:18:02 2017 (r324335) @@ -226,6 +226,13 @@ lock_delay_arg_init(struct lock_delay_arg *la, struct la->spin_cnt = 0; } +#define lock_delay_spin(n) do { \ + u_int _i; \ + \ + for (_i = (n); _i > 0; _i--) \ + cpu_spinwait(); \ +} while (0) + #define LOCK_DELAY_SYSINIT(func) \ SYSINIT(func##_ld, SI_SUB_LOCK, SI_ORDER_ANY, func, NULL)