From owner-svn-src-stable@freebsd.org Wed Jan 23 01:23:21 2019 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6DA814B5FF7; Wed, 23 Jan 2019 01:23:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 532D083089; Wed, 23 Jan 2019 01:23:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 440319A69; Wed, 23 Jan 2019 01:23:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x0N1NKhm097903; Wed, 23 Jan 2019 01:23:20 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x0N1NKnQ097902; Wed, 23 Jan 2019 01:23:20 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201901230123.x0N1NKnQ097902@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Wed, 23 Jan 2019 01:23:20 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r343333 - stable/12/sys/geom X-SVN-Group: stable-12 X-SVN-Commit-Author: mav X-SVN-Commit-Paths: stable/12/sys/geom X-SVN-Commit-Revision: 343333 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 532D083089 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_SHORT(-0.98)[-0.980,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-0.999,0] X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jan 2019 01:23:21 -0000 Author: mav Date: Wed Jan 23 01:23:19 2019 New Revision: 343333 URL: https://svnweb.freebsd.org/changeset/base/343333 Log: MFC r342558: Switch from mutexes to atomics in GEOM_DEV I/O path. Mutexes in I/O path there were used twice per I/O to atomically access several variables to close and/or destroy the device on last request completion. I found the way to fit all required info into one integer, suitable for atomic operations. It opened race window on device close, but addition of timeout to the msleep() there should cover it. Profiling shows removal of significant spinning time on those mutexes and IOPS increase from ~600K to >800K to NVMe on 72-core systems. Modified: stable/12/sys/geom/geom_dev.c Directory Properties: stable/12/ (props changed) Modified: stable/12/sys/geom/geom_dev.c ============================================================================== --- stable/12/sys/geom/geom_dev.c Wed Jan 23 01:16:59 2019 (r343332) +++ stable/12/sys/geom/geom_dev.c Wed Jan 23 01:23:19 2019 (r343333) @@ -64,7 +64,10 @@ struct g_dev_softc { struct cdev *sc_dev; struct cdev *sc_alias; int sc_open; - int sc_active; + u_int sc_active; +#define SC_A_DESTROY (1 << 31) +#define SC_A_OPEN (1 << 30) +#define SC_A_ACTIVE (SC_A_OPEN - 1) }; static d_open_t g_dev_open; @@ -420,9 +423,13 @@ g_dev_open(struct cdev *dev, int flags, int fmt, struc if (error == 0) { sc = cp->private; mtx_lock(&sc->sc_mtx); - if (sc->sc_open == 0 && sc->sc_active != 0) + if (sc->sc_open == 0 && (sc->sc_active & SC_A_ACTIVE) != 0) wakeup(&sc->sc_active); sc->sc_open += r + w + e; + if (sc->sc_open == 0) + atomic_clear_int(&sc->sc_active, SC_A_OPEN); + else + atomic_set_int(&sc->sc_active, SC_A_OPEN); mtx_unlock(&sc->sc_mtx); } return (error); @@ -465,8 +472,12 @@ g_dev_close(struct cdev *dev, int flags, int fmt, stru sc = cp->private; mtx_lock(&sc->sc_mtx); sc->sc_open += r + w + e; - while (sc->sc_open == 0 && sc->sc_active != 0) - msleep(&sc->sc_active, &sc->sc_mtx, 0, "PRIBIO", 0); + if (sc->sc_open == 0) + atomic_clear_int(&sc->sc_active, SC_A_OPEN); + else + atomic_set_int(&sc->sc_active, SC_A_OPEN); + while (sc->sc_open == 0 && (sc->sc_active & SC_A_ACTIVE) != 0) + msleep(&sc->sc_active, &sc->sc_mtx, 0, "g_dev_close", hz / 10); mtx_unlock(&sc->sc_mtx); g_topology_lock(); error = g_access(cp, r, w, e); @@ -693,7 +704,7 @@ g_dev_done(struct bio *bp2) struct g_consumer *cp; struct g_dev_softc *sc; struct bio *bp; - int destroy; + int active; cp = bp2->bio_from; sc = cp->private; @@ -713,17 +724,13 @@ g_dev_done(struct bio *bp2) bp2, bp, bp2->bio_resid, (intmax_t)bp2->bio_completed); } g_destroy_bio(bp2); - destroy = 0; - mtx_lock(&sc->sc_mtx); - if ((--sc->sc_active) == 0) { - if (sc->sc_open == 0) + active = atomic_fetchadd_int(&sc->sc_active, -1) - 1; + if ((active & SC_A_ACTIVE) == 0) { + if ((active & SC_A_OPEN) == 0) wakeup(&sc->sc_active); - if (sc->sc_dev == NULL) - destroy = 1; + if (active & SC_A_DESTROY) + g_post_event(g_dev_destroy, cp, M_NOWAIT, NULL); } - mtx_unlock(&sc->sc_mtx); - if (destroy) - g_post_event(g_dev_destroy, cp, M_NOWAIT, NULL); biodone(bp); } @@ -755,10 +762,8 @@ g_dev_strategy(struct bio *bp) return; } #endif - mtx_lock(&sc->sc_mtx); KASSERT(sc->sc_open > 0, ("Closed device in g_dev_strategy")); - sc->sc_active++; - mtx_unlock(&sc->sc_mtx); + atomic_add_int(&sc->sc_active, 1); for (;;) { /* @@ -796,18 +801,16 @@ g_dev_callback(void *arg) { struct g_consumer *cp; struct g_dev_softc *sc; - int destroy; + int active; cp = arg; sc = cp->private; g_trace(G_T_TOPOLOGY, "g_dev_callback(%p(%s))", cp, cp->geom->name); - mtx_lock(&sc->sc_mtx); sc->sc_dev = NULL; sc->sc_alias = NULL; - destroy = (sc->sc_active == 0); - mtx_unlock(&sc->sc_mtx); - if (destroy) + active = atomic_fetchadd_int(&sc->sc_active, SC_A_DESTROY); + if ((active & SC_A_ACTIVE) == 0) g_post_event(g_dev_destroy, cp, M_WAITOK, NULL); }