From owner-svn-src-all@freebsd.org Fri May 25 01:15:52 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 44A94F6F766; Fri, 25 May 2018 01:15:52 +0000 (UTC) (envelope-from mjg@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC4F26CB21; Fri, 25 May 2018 01:15:51 +0000 (UTC) (envelope-from mjg@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1296) id E47881718D; Fri, 25 May 2018 01:15:51 +0000 (UTC) Date: Fri, 25 May 2018 01:15:51 +0000 From: Mateusz Guzik To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r334197 - head/sys/kern Message-ID: <20180525011551.GB19943@freefall.freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-joke: no User-Agent: Mutt/1.9.5 (2018-04-13) X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 May 2018 01:15:52 -0000 Author: mjg Date: Thu May 25 23:58:57 2018 New Revision: 334197 URL: https://svnweb.freebsd.org/changeset/base/334197 Log: Implement Mostly Exclusive locks. High lock contention is one of the biggest scalability bottlenecks on multicore systems. Although the real fix consists of making lock consumers better suited for multicore operation, locking primitive performance is still a crucial factor in real-world systems. It turns out that a lot of the locking is overzealous - the lock is held longer than it needs to be and sometimes even completely unnecessarily in the first place. Even when lock is needed in principle, all relevant consumers may be only *reading* the state protected by the lock or modifying disjoint parts of protected data. As such, a lot of the locking can be elided. The idea boils down to trying to take the lock for a limited amount of time and in case of failure pretending we got it anyway. The approach along with practical analysis of performance win/new crash ratio is described in "Towards reliability-oblivious low latency locking" by Leland Oller. Modified: head/sys/kern/kern_mutex.c Modified: head/sys/kern/kern_mutex.c =================================================================== --- sys/kern/kern_mutex.c (revision 334196) +++ sys/kern/kern_mutex.c (working copy) @@ -557,6 +557,9 @@ __mtx_lock_sleep(volatile uintptr_t *c, uintptr_t lda.spin_cnt++; #endif #ifdef ADAPTIVE_MUTEXES + if (lda.spin_cnt > 16384) + break; + /* * If the owner is running on another CPU, spin until the * owner stops running or the state of the lock changes. @@ -1020,16 +1023,22 @@ __mtx_unlock_sleep(volatile uintptr_t *c, uintptr_ turnstile_chain_lock(&m->lock_object); _mtx_release_lock_quick(m); ts = turnstile_lookup(&m->lock_object); - MPASS(ts != NULL); - if (LOCK_LOG_TEST(&m->lock_object, opts)) - CTR1(KTR_LOCK, "_mtx_unlock_sleep: %p contested", m); - turnstile_broadcast(ts, TS_EXCLUSIVE_QUEUE); - /* - * This turnstile is now no longer associated with the mutex. We can - * unlock the chain lock so a new turnstile may take it's place. + * We failed to previously grab the lock. Unlock fast path brings + * us here thinking there are blocked threads, but there may be + * none */ - turnstile_unpend(ts, TS_EXCLUSIVE_LOCK); + if (ts != NULL) { + if (LOCK_LOG_TEST(&m->lock_object, opts)) + CTR1(KTR_LOCK, "_mtx_unlock_sleep: %p contested", m); + turnstile_broadcast(ts, TS_EXCLUSIVE_QUEUE); + + /* + * This turnstile is now no longer associated with the mutex. We can + * unlock the chain lock so a new turnstile may take it's place. + */ + turnstile_unpend(ts, TS_EXCLUSIVE_LOCK); + } turnstile_chain_unlock(&m->lock_object); }