From owner-svn-src-all@freebsd.org Sat Nov 30 17:24:43 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 224E01B1687; Sat, 30 Nov 2019 17:24:43 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47QJCb03LMz4G9j; Sat, 30 Nov 2019 17:24:43 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DA6841A4E; Sat, 30 Nov 2019 17:24:42 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id xAUHOgFO054318; Sat, 30 Nov 2019 17:24:42 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id xAUHOg42054317; Sat, 30 Nov 2019 17:24:42 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <201911301724.xAUHOg42054317@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Sat, 30 Nov 2019 17:24:42 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r355231 - head/sys/kern X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: head/sys/kern X-SVN-Commit-Revision: 355231 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Nov 2019 17:24:43 -0000 Author: mjg Date: Sat Nov 30 17:24:42 2019 New Revision: 355231 URL: https://svnweb.freebsd.org/changeset/base/355231 Log: lockprof: use IPI-injecetd fences to fix hangs on stat dump and reset The previously used quiesce_all_cpus walks all CPUs and waits until curthread can run on them. Even on contemporary machines this becomes a significant problem under load when it can literally take minutes for the operation to complete. With the patch the stall is normally less than 1 second. Reviewed by: kib, jeff (previous version) Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21740 Modified: head/sys/kern/subr_lock.c Modified: head/sys/kern/subr_lock.c ============================================================================== --- head/sys/kern/subr_lock.c Sat Nov 30 17:22:10 2019 (r355230) +++ head/sys/kern/subr_lock.c Sat Nov 30 17:24:42 2019 (r355231) @@ -324,8 +324,14 @@ lock_prof_reset(void) atomic_store_rel_int(&lock_prof_resetting, 1); enabled = lock_prof_enable; lock_prof_enable = 0; - quiesce_all_cpus("profreset", 0); /* + * This both publishes lock_prof_enable as disabled and makes sure + * everyone else reads it if they are not far enough. We wait for the + * rest down below. + */ + cpus_fence_seq_cst(); + quiesce_all_critical(); + /* * Some objects may have migrated between CPUs. Clear all links * before we zero the structures. Some items may still be linked * into per-thread lists as well. @@ -343,6 +349,9 @@ lock_prof_reset(void) lock_prof_init_type(&lpc->lpc_types[0]); lock_prof_init_type(&lpc->lpc_types[1]); } + /* + * Paired with the fence from cpus_fence_seq_cst() + */ atomic_store_rel_int(&lock_prof_resetting, 0); lock_prof_enable = enabled; } @@ -433,12 +442,17 @@ dump_lock_prof_stats(SYSCTL_HANDLER_ARGS) "max", "wait_max", "total", "wait_total", "count", "avg", "wait_avg", "cnt_hold", "cnt_lock", "name"); enabled = lock_prof_enable; lock_prof_enable = 0; - quiesce_all_cpus("profstat", 0); + /* + * See the comment in lock_prof_reset + */ + cpus_fence_seq_cst(); + quiesce_all_critical(); t = ticks; CPU_FOREACH(cpu) { lock_prof_type_stats(&LP_CPU(cpu)->lpc_types[0], sb, 0, t); lock_prof_type_stats(&LP_CPU(cpu)->lpc_types[1], sb, 1, t); } + atomic_thread_fence_rel(); lock_prof_enable = enabled; error = sbuf_finish(sb); @@ -591,6 +605,10 @@ lock_profile_obtain_lock_success(struct lock_object *l else l->lpo_waittime = 0; out: + /* + * Paired with cpus_fence_seq_cst(). + */ + atomic_thread_fence_rel(); critical_exit(); } @@ -677,6 +695,10 @@ release: type = &LP_CPU_SELF->lpc_types[spin]; LIST_INSERT_HEAD(&type->lpt_lpoalloc, l, lpo_link); out: + /* + * Paired with cpus_fence_seq_cst(). + */ + atomic_thread_fence_rel(); critical_exit(); }