From owner-svn-src-stable@freebsd.org Mon Jan 1 23:45:10 2018 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95D8FEA2B0D; Mon, 1 Jan 2018 23:45:10 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A437633C6; Mon, 1 Jan 2018 23:45:10 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w01Nj9Jg016512; Mon, 1 Jan 2018 23:45:09 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w01Nj917016511; Mon, 1 Jan 2018 23:45:09 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <201801012345.w01Nj917016511@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Mon, 1 Jan 2018 23:45:09 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r327477 - stable/11/sys/kern X-SVN-Group: stable-11 X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: stable/11/sys/kern X-SVN-Commit-Revision: 327477 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jan 2018 23:45:10 -0000 Author: mjg Date: Mon Jan 1 23:45:09 2018 New Revision: 327477 URL: https://svnweb.freebsd.org/changeset/base/327477 Log: MFC r327394,r327395: mtx: pre-read the lock value in thread_lock_flags_ Since this function is effectively slow path, if we get here the lock is most likely already taken in which case it is cheaper to not blindly attempt the atomic op. While here move hwpmc probe out of the loop to match other primitives. ============= mtx: deduplicate indefinite wait check in spinlocks and thread lock Modified: stable/11/sys/kern/kern_mutex.c Directory Properties: stable/11/ (props changed) Modified: stable/11/sys/kern/kern_mutex.c ============================================================================== --- stable/11/sys/kern/kern_mutex.c Mon Jan 1 22:33:57 2018 (r327476) +++ stable/11/sys/kern/kern_mutex.c Mon Jan 1 23:45:09 2018 (r327477) @@ -168,6 +168,8 @@ LOCK_DELAY_SYSINIT_DEFAULT(mtx_spin_delay); struct mtx blocked_lock; struct mtx Giant; +static void _mtx_lock_indefinite_check(struct mtx *, struct lock_delay_arg *); + void assert_mtx(const struct lock_object *lock, int what) { @@ -672,25 +674,6 @@ __mtx_lock_sleep(volatile uintptr_t *c, uintptr_t v) #endif } -static void -_mtx_lock_spin_failed(struct mtx *m) -{ - struct thread *td; - - td = mtx_owner(m); - - /* If the mutex is unlocked, try again. */ - if (td == NULL) - return; - - printf( "spin lock %p (%s) held by %p (tid %d) too long\n", - m, m->lock_object.lo_name, td, td->td_tid); -#ifdef WITNESS - witness_display_spinlock(&m->lock_object, td, printf); -#endif - panic("spin lock held too long"); -} - #ifdef SMP /* * _mtx_lock_spin_cookie: the tougher part of acquiring an MTX_SPIN lock. @@ -762,16 +745,10 @@ _mtx_lock_spin_cookie(volatile uintptr_t *c, uintptr_t /* Give interrupts a chance while we spin. */ spinlock_exit(); do { - if (lda.spin_cnt < 10000000) { + if (__predict_true(lda.spin_cnt < 10000000)) { lock_delay(&lda); } else { - lda.spin_cnt++; - if (lda.spin_cnt < 60000000 || kdb_active || - panicstr != NULL) - DELAY(1); - else - _mtx_lock_spin_failed(m); - cpu_spinwait(); + _mtx_lock_indefinite_check(m, &lda); } v = MTX_READ_VALUE(m); } while (v != MTX_UNOWNED); @@ -897,6 +874,10 @@ thread_lock_flags_(struct thread *td, int opts, const lock_delay_arg_init(&lda, &mtx_spin_delay); +#ifdef HWPMC_HOOKS + PMC_SOFT_CALL( , , lock, failed); +#endif + #ifdef LOCK_PROFILING doing_lockprof = 1; #elif defined(KDTRACE_HOOKS) @@ -906,37 +887,29 @@ thread_lock_flags_(struct thread *td, int opts, const #endif for (;;) { retry: - v = MTX_UNOWNED; spinlock_enter(); m = td->td_lock; thread_lock_validate(m, opts, file, line); + v = MTX_READ_VALUE(m); for (;;) { - if (_mtx_obtain_lock_fetch(m, &v, tid)) - break; - if (v == MTX_UNOWNED) + if (v == MTX_UNOWNED) { + if (_mtx_obtain_lock_fetch(m, &v, tid)) + break; continue; + } if (v == tid) { m->mtx_recurse++; break; } -#ifdef HWPMC_HOOKS - PMC_SOFT_CALL( , , lock, failed); -#endif lock_profile_obtain_lock_failed(&m->lock_object, &contested, &waittime); /* Give interrupts a chance while we spin. */ spinlock_exit(); do { - if (lda.spin_cnt < 10000000) { + if (__predict_true(lda.spin_cnt < 10000000)) { lock_delay(&lda); } else { - lda.spin_cnt++; - if (lda.spin_cnt < 60000000 || - kdb_active || panicstr != NULL) - DELAY(1); - else - _mtx_lock_spin_failed(m); - cpu_spinwait(); + _mtx_lock_indefinite_check(m, &lda); } if (m != td->td_lock) goto retry; @@ -1225,6 +1198,31 @@ mutex_init(void) mtx_init(&proc0.p_profmtx, "pprofl", NULL, MTX_SPIN); mtx_init(&devmtx, "cdev", NULL, MTX_DEF); mtx_lock(&Giant); +} + +static void __noinline +_mtx_lock_indefinite_check(struct mtx *m, struct lock_delay_arg *ldap) +{ + struct thread *td; + + ldap->spin_cnt++; + if (ldap->spin_cnt < 60000000 || kdb_active || panicstr != NULL) + DELAY(1); + else { + td = mtx_owner(m); + + /* If the mutex is unlocked, try again. */ + if (td == NULL) + return; + + printf( "spin lock %p (%s) held by %p (tid %d) too long\n", + m, m->lock_object.lo_name, td, td->td_tid); +#ifdef WITNESS + witness_display_spinlock(&m->lock_object, td, printf); +#endif + panic("spin lock held too long"); + } + cpu_spinwait(); } #ifdef DDB