From owner-svn-src-all@freebsd.org Fri Jan 10 00:42:07 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 850561FDE9C; Fri, 10 Jan 2020 00:42:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47v41q2ZGDz3whZ; Fri, 10 Jan 2020 00:42:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 533E09CA2; Fri, 10 Jan 2020 00:42:07 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 00A0g7v7016546; Fri, 10 Jan 2020 00:42:07 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 00A0g6qj016540; Fri, 10 Jan 2020 00:42:06 GMT (envelope-from mav@FreeBSD.org) Message-Id: <202001100042.00A0g6qj016540@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 10 Jan 2020 00:42:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r356577 - stable/11/sys/geom/vinum X-SVN-Group: stable-11 X-SVN-Commit-Author: mav X-SVN-Commit-Paths: stable/11/sys/geom/vinum X-SVN-Commit-Revision: 356577 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Jan 2020 00:42:07 -0000 Author: mav Date: Fri Jan 10 00:42:05 2020 New Revision: 356577 URL: https://svnweb.freebsd.org/changeset/base/356577 Log: MFC r356138: Reimplement gvinum orphanization. gvinum was the only GEOM class, using consumer nstart/nend fields. Making it do its own accounting for orphanization purposes allows in perspective to remove burden of that expensive for SMP accounting from GEOM. Also the previous implementation spinned in a tight event loop, waiting for all active BIOs to complete, while the new one knows exactly when it is possible to close the consumer. Modified: stable/11/sys/geom/vinum/geom_vinum.h stable/11/sys/geom/vinum/geom_vinum_events.c stable/11/sys/geom/vinum/geom_vinum_plex.c stable/11/sys/geom/vinum/geom_vinum_raid5.c stable/11/sys/geom/vinum/geom_vinum_var.h stable/11/sys/geom/vinum/geom_vinum_volume.c Directory Properties: stable/11/ (props changed) Modified: stable/11/sys/geom/vinum/geom_vinum.h ============================================================================== --- stable/11/sys/geom/vinum/geom_vinum.h Fri Jan 10 00:41:15 2020 (r356576) +++ stable/11/sys/geom/vinum/geom_vinum.h Fri Jan 10 00:42:05 2020 (r356577) @@ -125,6 +125,7 @@ void gv_post_event(struct gv_softc *, int, void *, voi void gv_worker_exit(struct gv_softc *); struct gv_event *gv_get_event(struct gv_softc *); void gv_remove_event(struct gv_softc *, struct gv_event *); +void gv_drive_done(struct gv_drive *); void gv_drive_tasted(struct gv_softc *, struct g_provider *); void gv_drive_lost(struct gv_softc *, struct gv_drive *); void gv_setup_objects(struct gv_softc *); Modified: stable/11/sys/geom/vinum/geom_vinum_events.c ============================================================================== --- stable/11/sys/geom/vinum/geom_vinum_events.c Fri Jan 10 00:41:15 2020 (r356576) +++ stable/11/sys/geom/vinum/geom_vinum_events.c Fri Jan 10 00:42:05 2020 (r356577) @@ -192,6 +192,20 @@ failed: } /* + * Count completed BIOs and handle orphanization when all are done. + */ +void +gv_drive_done(struct gv_drive *d) +{ + + KASSERT(d->active >= 0, ("Negative number of BIOs (%d)", d->active)); + if (--d->active == 0 && (d->flags & GV_DRIVE_ORPHANED)) { + d->flags &= ~GV_DRIVE_ORPHANED; + gv_post_event(d->vinumconf, GV_EVENT_DRIVE_LOST, d, NULL, 0, 0); + } +} + +/* * When losing a drive (e.g. hardware failure), we cut down the consumer * attached to the underlying device and bring the drive itself to a * "referenced" state so that normal tasting could bring it up cleanly if it @@ -211,10 +225,10 @@ gv_drive_lost(struct gv_softc *sc, struct gv_drive *d) cp = d->consumer; if (cp != NULL) { - if (cp->nstart != cp->nend) { - G_VINUM_DEBUG(0, "dead drive '%s' has still active " + if (d->active > 0) { + G_VINUM_DEBUG(2, "dead drive '%s' has still active " "requests, unable to detach consumer", d->name); - gv_post_event(sc, GV_EVENT_DRIVE_LOST, d, NULL, 0, 0); + d->flags |= GV_DRIVE_ORPHANED; return; } g_topology_lock(); Modified: stable/11/sys/geom/vinum/geom_vinum_plex.c ============================================================================== --- stable/11/sys/geom/vinum/geom_vinum_plex.c Fri Jan 10 00:41:15 2020 (r356576) +++ stable/11/sys/geom/vinum/geom_vinum_plex.c Fri Jan 10 00:42:05 2020 (r356577) @@ -275,6 +275,7 @@ gv_plex_normal_request(struct gv_plex *p, struct bio * cbp->bio_data = addr; cbp->bio_done = gv_done; cbp->bio_caller1 = s; + s->drive_sc->active++; /* Store the sub-requests now and let others issue them. */ bioq_insert_tail(p->bqueue, cbp); @@ -577,10 +578,10 @@ gv_sync_request(struct gv_plex *from, struct gv_plex * return (ENOMEM); } bp->bio_length = length; - bp->bio_done = gv_done; + bp->bio_done = NULL; bp->bio_pflags |= GV_BIO_SYNCREQ; bp->bio_offset = offset; - bp->bio_caller1 = from; + bp->bio_caller1 = from; bp->bio_caller2 = to; bp->bio_cmd = type; if (data == NULL) @@ -691,7 +692,7 @@ gv_grow_request(struct gv_plex *p, off_t offset, off_t } bp->bio_cmd = type; - bp->bio_done = gv_done; + bp->bio_done = NULL; bp->bio_error = 0; bp->bio_caller1 = p; bp->bio_offset = offset; @@ -799,7 +800,7 @@ gv_init_request(struct gv_sd *s, off_t start, caddr_t } bp->bio_cmd = BIO_WRITE; bp->bio_data = data; - bp->bio_done = gv_done; + bp->bio_done = NULL; bp->bio_error = 0; bp->bio_length = length; bp->bio_pflags |= GV_BIO_INIT; @@ -816,6 +817,7 @@ gv_init_request(struct gv_sd *s, off_t start, caddr_t } cbp->bio_done = gv_done; cbp->bio_caller1 = s; + d->active++; /* Send it off to the consumer. */ g_io_request(cbp, cp); } @@ -902,7 +904,7 @@ gv_parity_request(struct gv_plex *p, int flags, off_t } bp->bio_cmd = BIO_WRITE; - bp->bio_done = gv_done; + bp->bio_done = NULL; bp->bio_error = 0; bp->bio_length = p->stripesize; bp->bio_caller1 = p; Modified: stable/11/sys/geom/vinum/geom_vinum_raid5.c ============================================================================== --- stable/11/sys/geom/vinum/geom_vinum_raid5.c Fri Jan 10 00:41:15 2020 (r356576) +++ stable/11/sys/geom/vinum/geom_vinum_raid5.c Fri Jan 10 00:42:05 2020 (r356577) @@ -90,11 +90,13 @@ gv_raid5_start(struct gv_plex *p, struct bio *bp, cadd if (wp->waiting != NULL) { if (wp->waiting->bio_cflags & GV_BIO_MALLOC) g_free(wp->waiting->bio_data); + gv_drive_done(wp->waiting->bio_caller1); g_destroy_bio(wp->waiting); } if (wp->parity != NULL) { if (wp->parity->bio_cflags & GV_BIO_MALLOC) g_free(wp->parity->bio_data); + gv_drive_done(wp->parity->bio_caller1); g_destroy_bio(wp->parity); } g_free(wp); @@ -115,6 +117,7 @@ gv_raid5_start(struct gv_plex *p, struct bio *bp, cadd while (cbp != NULL) { if (cbp->bio_cflags & GV_BIO_MALLOC) g_free(cbp->bio_data); + gv_drive_done(cbp->bio_caller1); g_destroy_bio(cbp); cbp = bioq_takefirst(p->bqueue); } @@ -654,6 +657,7 @@ gv_raid5_clone_bio(struct bio *bp, struct gv_sd *s, st cbp->bio_length = wp->length; cbp->bio_done = gv_done; cbp->bio_caller1 = s; + s->drive_sc->active++; if (use_wp) cbp->bio_caller2 = wp; Modified: stable/11/sys/geom/vinum/geom_vinum_var.h ============================================================================== --- stable/11/sys/geom/vinum/geom_vinum_var.h Fri Jan 10 00:41:15 2020 (r356576) +++ stable/11/sys/geom/vinum/geom_vinum_var.h Fri Jan 10 00:42:05 2020 (r356577) @@ -258,10 +258,12 @@ struct gv_drive { #define GV_DRIVE_REFERENCED 0x01 /* The drive isn't really existing, but was referenced by a subdisk during taste. */ +#define GV_DRIVE_ORPHANED 0x02 /* The drive was orphaned. */ struct gv_hdr *hdr; /* The drive header. */ struct g_consumer *consumer; /* Consumer attached to this drive. */ + int active; /* Number of active requests. */ int freelist_entries; /* Count of freelist entries. */ LIST_HEAD(,gv_freelist) freelist; /* List of freelist entries. */ Modified: stable/11/sys/geom/vinum/geom_vinum_volume.c ============================================================================== --- stable/11/sys/geom/vinum/geom_vinum_volume.c Fri Jan 10 00:41:15 2020 (r356576) +++ stable/11/sys/geom/vinum/geom_vinum_volume.c Fri Jan 10 00:42:05 2020 (r356577) @@ -161,4 +161,6 @@ gv_bio_done(struct gv_softc *sc, struct bio *bp) gv_plex_raid5_done(p, bp); break; } + + gv_drive_done(s->drive_sc); }