From owner-svn-src-all@freebsd.org Sun Mar 31 21:34:59 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95828156DA00; Sun, 31 Mar 2019 21:34:59 +0000 (UTC) (envelope-from mckusick@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 39D1C92864; Sun, 31 Mar 2019 21:34:59 +0000 (UTC) (envelope-from mckusick@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 131B74C9A; Sun, 31 Mar 2019 21:34:59 +0000 (UTC) (envelope-from mckusick@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x2VLYwo7018670; Sun, 31 Mar 2019 21:34:58 GMT (envelope-from mckusick@FreeBSD.org) Received: (from mckusick@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x2VLYwO9018669; Sun, 31 Mar 2019 21:34:58 GMT (envelope-from mckusick@FreeBSD.org) Message-Id: <201903312134.x2VLYwO9018669@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mckusick set sender to mckusick@FreeBSD.org using -f From: Kirk McKusick Date: Sun, 31 Mar 2019 21:34:58 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r345758 - head/sys/dev/md X-SVN-Group: head X-SVN-Commit-Author: mckusick X-SVN-Commit-Paths: head/sys/dev/md X-SVN-Commit-Revision: 345758 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 39D1C92864 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.95 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.95)[-0.948,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Mar 2019 21:34:59 -0000 Author: mckusick Date: Sun Mar 31 21:34:58 2019 New Revision: 345758 URL: https://svnweb.freebsd.org/changeset/base/345758 Log: When using the force option to shut down a memory-disk device, I/O operations already in its queue were not being properly drained. The GEOM framework does the queue draining, but the device driver needs to wait for the draining to happen. The waiting is done by adding a g_md_providergone() function to wait for the I/O operations to finish up. It is likely that every GEOM provider that implements orphaning attached GEOM consumers needs to use the "providergone" mechanism for this same reason, but some of them do not do so. Apparently Kenneth Merry (ken@) added the drain for just such races, but he missed adding it to some of the device drivers that needed it. Submitted by: Chuck Silvers Reviewed by: imp Tested by: Chuck Silvers MFC after: 1 week Sponsored by: Netflix Modified: head/sys/dev/md/md.c Modified: head/sys/dev/md/md.c ============================================================================== --- head/sys/dev/md/md.c Sun Mar 31 19:27:44 2019 (r345757) +++ head/sys/dev/md/md.c Sun Mar 31 21:34:58 2019 (r345758) @@ -110,6 +110,7 @@ #define MD_SHUTDOWN 0x10000 /* Tell worker thread to terminate. */ #define MD_EXITING 0x20000 /* Worker thread is exiting. */ +#define MD_PROVIDERGONE 0x40000 /* Safe to free the softc */ #ifndef MD_NSECT #define MD_NSECT (10000 * 2) @@ -199,6 +200,7 @@ static g_start_t g_md_start; static g_access_t g_md_access; static void g_md_dumpconf(struct sbuf *sb, const char *indent, struct g_geom *gp, struct g_consumer *cp __unused, struct g_provider *pp); +static g_provgone_t g_md_providergone; static struct cdev *status_dev = NULL; static struct sx md_sx; @@ -220,6 +222,7 @@ struct g_class g_md_class = { .start = g_md_start, .access = g_md_access, .dumpconf = g_md_dumpconf, + .providergone = g_md_providergone, }; DECLARE_GEOM_CLASS(g_md_class, g_md); @@ -481,8 +484,8 @@ g_md_start(struct bio *bp) } mtx_lock(&sc->queue_mtx); bioq_disksort(&sc->bio_queue, bp); - mtx_unlock(&sc->queue_mtx); wakeup(sc); + mtx_unlock(&sc->queue_mtx); } #define MD_MALLOC_MOVE_ZERO 1 @@ -1496,17 +1499,30 @@ bad: return (error); } +static void +g_md_providergone(struct g_provider *pp) +{ + struct md_s *sc = pp->geom->softc; + + mtx_lock(&sc->queue_mtx); + sc->flags |= MD_PROVIDERGONE; + wakeup(&sc->flags); + mtx_unlock(&sc->queue_mtx); +} + static int mddestroy(struct md_s *sc, struct thread *td) { if (sc->gp) { - sc->gp->softc = NULL; g_topology_lock(); g_wither_geom(sc->gp, ENXIO); g_topology_unlock(); - sc->gp = NULL; - sc->pp = NULL; + + mtx_lock(&sc->queue_mtx); + while (!(sc->flags & MD_PROVIDERGONE)) + msleep(&sc->flags, &sc->queue_mtx, PRIBIO, "mddestroy", 0); + mtx_unlock(&sc->queue_mtx); } if (sc->devstat) { devstat_remove_entry(sc->devstat);