From owner-svn-src-stable@FreeBSD.ORG Wed May 2 07:17:54 2012 Return-Path: Delivered-To: svn-src-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B244106566C; Wed, 2 May 2012 07:17:54 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id DF62E8FC16; Wed, 2 May 2012 07:17:53 +0000 (UTC) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.4/8.14.4) with ESMTP id q427Hr5U070964; Wed, 2 May 2012 07:17:53 GMT (envelope-from mav@svn.freebsd.org) Received: (from mav@localhost) by svn.freebsd.org (8.14.4/8.14.4/Submit) id q427HrYY070960; Wed, 2 May 2012 07:17:53 GMT (envelope-from mav@svn.freebsd.org) Message-Id: <201205020717.q427HrYY070960@svn.freebsd.org> From: Alexander Motin Date: Wed, 2 May 2012 07:17:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r234916 - in stable/9: sbin/geom/class/multipath sys/geom/multipath X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 May 2012 07:17:54 -0000 Author: mav Date: Wed May 2 07:17:53 2012 New Revision: 234916 URL: http://svn.freebsd.org/changeset/base/234916 Log: MFC r234415: Some improvements to GEOM MULTIPATH: - Implement "configure" command to allow switching operation mode of running device on-fly without destroying and recreation. - Implement Active/Read mode as hybrid of Active/Active and Active/Passive. In this mode all paths not marked FAIL may handle reads same time, but unlike Active/Active only one path handles write requests at any point in time. It allows to closer follow original write request order if above layers need it for data consistency (not waiting for requisite write completion before sending dependent write). - Hide duplicate messages about device status change. - Remove periodic thread wake up with 10Hz rate. Sponsored by: iXsystems, Inc. Modified: stable/9/sbin/geom/class/multipath/geom_multipath.c stable/9/sbin/geom/class/multipath/gmultipath.8 stable/9/sys/geom/multipath/g_multipath.c Directory Properties: stable/9/sbin/geom/ (props changed) stable/9/sys/ (props changed) Modified: stable/9/sbin/geom/class/multipath/geom_multipath.c ============================================================================== --- stable/9/sbin/geom/class/multipath/geom_multipath.c Wed May 2 07:08:04 2012 (r234915) +++ stable/9/sbin/geom/class/multipath/geom_multipath.c Wed May 2 07:17:53 2012 (r234916) @@ -55,17 +55,28 @@ struct g_command class_commands[] = { "create", G_FLAG_VERBOSE | G_FLAG_LOADKLD, NULL, { { 'A', "active_active", NULL, G_TYPE_BOOL }, + { 'R', "active_read", NULL, G_TYPE_BOOL }, G_OPT_SENTINEL }, - "[-vA] name prov ..." + "[-vAR] name prov ..." }, { "label", G_FLAG_VERBOSE | G_FLAG_LOADKLD, mp_main, { { 'A', "active_active", NULL, G_TYPE_BOOL }, + { 'R', "active_read", NULL, G_TYPE_BOOL }, G_OPT_SENTINEL }, - "[-vA] name prov ..." + "[-vAR] name prov ..." + }, + { "configure", G_FLAG_VERBOSE, NULL, + { + { 'A', "active_active", NULL, G_TYPE_BOOL }, + { 'P', "active_passive", NULL, G_TYPE_BOOL }, + { 'R', "active_read", NULL, G_TYPE_BOOL }, + G_OPT_SENTINEL + }, + "[-vAPR] name" }, { "add", G_FLAG_VERBOSE, NULL, G_NULL_OPTS, @@ -195,6 +206,8 @@ mp_label(struct gctl_req *req) } strlcpy(md.md_uuid, ptr, sizeof (md.md_uuid)); md.md_active_active = gctl_get_int(req, "active_active"); + if (gctl_get_int(req, "active_read")) + md.md_active_active = 2; free(ptr); /* Modified: stable/9/sbin/geom/class/multipath/gmultipath.8 ============================================================================== --- stable/9/sbin/geom/class/multipath/gmultipath.8 Wed May 2 07:08:04 2012 (r234915) +++ stable/9/sbin/geom/class/multipath/gmultipath.8 Wed May 2 07:17:53 2012 (r234916) @@ -24,7 +24,7 @@ .\" .\" $FreeBSD$ .\" -.Dd October 31, 2011 +.Dd April 18, 2012 .Dt GMULTIPATH 8 .Os .Sh NAME @@ -33,15 +33,19 @@ .Sh SYNOPSIS .Nm .Cm create -.Op Fl Av +.Op Fl ARv .Ar name .Ar prov ... .Nm .Cm label -.Op Fl Av +.Op Fl ARv .Ar name .Ar prov ... .Nm +.Cm configure +.Op Fl APRv +.Ar name +.Nm .Cm add .Op Fl v .Ar name prov @@ -121,7 +125,9 @@ Kernel will only check that all given pr sector sizes. .Pp .Fl A -option enables Active/Active mode, otherwise Active/Passive mode is used +option enables Active/Active mode, +.Fl R +option enables Active/Read mode, otherwise Active/Passive mode is used by default. .It Cm label Create multipath device with @@ -134,8 +140,19 @@ It reliably protects against specifying Providers with no matching metadata detected will not be added to the device. .Pp .Fl A -option enables Active/Active mode, otherwise Active/Passive mode is used +option enables Active/Active mode, +.Fl R +option enables Active/Read mode, otherwise Active/Passive mode is used by default. +.It Cm configure +Configure the given multipath device. +.Pp +.Fl A +option enables Active/Active mode, +.Fl P +option enables Active/Passive mode, +.Fl R +option enables Active/Read mode. .It Cm add Add the given provider as a path to the given multipath device. Should normally be used only for devices created with @@ -223,7 +240,8 @@ of multiple pathnames refer to the same system operator who will use tools and knowledge of their own storage subsystem to make the correct configuration selection. .Pp -There are Active/Passive and Active/Active operation modes supported. +There are Active/Passive, Active/Read and Active/Active operation modes +supported. In Active/Passive mode only one path has I/O moving on it at any point in time. This I/O continues until an I/O is returned with @@ -233,6 +251,12 @@ in a list is selected as active and the In Active/Active mode all paths not marked FAIL may handle I/O same time. Requests are distributed between paths to equalize load. For capable devices it allows to utilize bandwidth of all paths. +In Active/Read mode all paths not marked FAIL may handle reads same time, +but unlike Active/Active only one path handles write requests at any +point in time. +It allows to closer follow original write request order if above layer +needs it for data consistency (not waiting for requisite write completion +before sending dependent write). .Pp When new devices are added to the system the .Nm MULTIPATH Modified: stable/9/sys/geom/multipath/g_multipath.c ============================================================================== --- stable/9/sys/geom/multipath/g_multipath.c Wed May 2 07:08:04 2012 (r234915) +++ stable/9/sys/geom/multipath/g_multipath.c Wed May 2 07:17:53 2012 (r234916) @@ -151,20 +151,21 @@ g_multipath_fault(struct g_consumer *cp, if (sc->sc_active == NULL) { printf("GEOM_MULTIPATH: out of providers for %s\n", sc->sc_name); - } else if (!sc->sc_active_active) { + } else if (sc->sc_active_active != 1) { printf("GEOM_MULTIPATH: %s is now active path in %s\n", sc->sc_active->provider->name, sc->sc_name); } } static struct g_consumer * -g_multipath_choose(struct g_geom *gp) +g_multipath_choose(struct g_geom *gp, struct bio *bp) { struct g_multipath_softc *sc; struct g_consumer *best, *cp; sc = gp->softc; - if (!sc->sc_active_active) + if (sc->sc_active_active == 0 || + (sc->sc_active_active == 2 && bp->bio_cmd != BIO_READ)) return (sc->sc_active); best = NULL; LIST_FOREACH(cp, &gp->consumer, consumer) { @@ -253,7 +254,7 @@ g_multipath_start(struct bio *bp) return; } mtx_lock(&sc->sc_mtx); - cp = g_multipath_choose(gp); + cp = g_multipath_choose(gp, bp); if (cp == NULL) { mtx_unlock(&sc->sc_mtx); g_destroy_bio(cbp); @@ -323,9 +324,11 @@ g_multipath_done_error(struct bio *bp) cnt = (uintptr_t *)&cp->private; mtx_lock(&sc->sc_mtx); - printf("GEOM_MULTIPATH: Error %d, %s in %s marked FAIL\n", - bp->bio_error, pp->name, sc->sc_name); - g_multipath_fault(cp, MP_FAIL); + if ((cp->index & MP_FAIL) == 0) { + printf("GEOM_MULTIPATH: Error %d, %s in %s marked FAIL\n", + bp->bio_error, pp->name, sc->sc_name); + g_multipath_fault(cp, MP_FAIL); + } (*cnt)--; if (*cnt == 0 && (cp->index & (MP_LOST | MP_POSTED)) == MP_LOST) { cp->index |= MP_POSTED; @@ -363,8 +366,10 @@ g_multipath_kt(void *arg) g_multipath_done_error(bp); mtx_lock(&gmtbq_mtx); } + if (g_multipath_kt_state != GKT_RUN) + break; msleep(&g_multipath_kt_state, &gmtbq_mtx, PRIBIO, - "gkt:wait", hz / 10); + "gkt:wait", 0); } mtx_unlock(&gmtbq_mtx); wakeup(&g_multipath_kt_state); @@ -525,7 +530,7 @@ g_multipath_add_disk(struct g_geom *gp, pp->name, sc->sc_name); if (sc->sc_active == NULL) { sc->sc_active = cp; - if (!sc->sc_active_active) + if (sc->sc_active_active != 1) printf("GEOM_MULTIPATH: %s is now active path in %s\n", pp->name, sc->sc_name); } @@ -599,7 +604,7 @@ g_multipath_rotate(struct g_geom *gp) } if (lcp) { sc->sc_active = lcp; - if (!sc->sc_active_active) + if (sc->sc_active_active != 1) printf("GEOM_MULTIPATH: %s is now active path in %s\n", lcp->provider->name, sc->sc_name); } @@ -611,8 +616,7 @@ g_multipath_init(struct g_class *mp) { bioq_init(&gmtbq); mtx_init(&gmtbq_mtx, "gmtbq", NULL, MTX_DEF); - if (kproc_create(g_multipath_kt, mp, NULL, 0, 0, "g_mp_kt") == 0) - g_multipath_kt_state = GKT_RUN; + kproc_create(g_multipath_kt, mp, NULL, 0, 0, "g_mp_kt"); } static void @@ -879,7 +883,7 @@ g_multipath_ctl_create(struct gctl_req * struct g_geom *gp; const char *mpname, *name; char param[16]; - int *nargs, i, *active_active; + int *nargs, i, *val; g_topology_assert(); @@ -908,10 +912,13 @@ g_multipath_ctl_create(struct gctl_req * md.md_size = 0; md.md_sectorsize = 0; md.md_uuid[0] = 0; - active_active = gctl_get_paraml(req, "active_active", - sizeof(*active_active)); - md.md_active_active = - (active_active == NULL || *active_active == 0) ? 0 : 1; + md.md_active_active = 0; + val = gctl_get_paraml(req, "active_active", sizeof(*val)); + if (val != NULL && *val != 0) + md.md_active_active = 1; + val = gctl_get_paraml(req, "active_read", sizeof(*val)); + if (val != NULL && *val != 0) + md.md_active_active = 2; gp = g_multipath_create(mp, &md); if (gp == NULL) { gctl_error(req, "GEOM_MULTIPATH: cannot create geom %s/%s\n", @@ -931,6 +938,67 @@ g_multipath_ctl_create(struct gctl_req * } static void +g_multipath_ctl_configure(struct gctl_req *req, struct g_class *mp) +{ + struct g_multipath_softc *sc; + struct g_geom *gp; + struct g_consumer *cp; + struct g_provider *pp; + struct g_multipath_metadata *md; + const char *name; + int error, *val; + void *buf; + + g_topology_assert(); + + name = gctl_get_asciiparam(req, "arg0"); + if (name == NULL) { + gctl_error(req, "No 'arg0' argument"); + return; + } + gp = g_multipath_find_geom(mp, name); + if (gp == NULL) { + gctl_error(req, "Device %s is invalid", name); + return; + } + sc = gp->softc; + val = gctl_get_paraml(req, "active_active", sizeof(*val)); + if (val != NULL && *val != 0) + sc->sc_active_active = 1; + val = gctl_get_paraml(req, "active_read", sizeof(*val)); + if (val != NULL && *val != 0) + sc->sc_active_active = 2; + val = gctl_get_paraml(req, "active_passive", sizeof(*val)); + if (val != NULL && *val != 0) + sc->sc_active_active = 0; + if (sc->sc_uuid[0] != 0 && sc->sc_active != NULL) { + cp = sc->sc_active; + pp = cp->provider; + error = g_access(cp, 1, 1, 1); + if (error != 0) { + gctl_error(req, "Can't open %s (%d)", pp->name, error); + return; + } + g_topology_unlock(); + md = buf = g_malloc(pp->sectorsize, M_WAITOK | M_ZERO); + strlcpy(md->md_magic, G_MULTIPATH_MAGIC, sizeof(md->md_magic)); + memcpy(md->md_uuid, sc->sc_uuid, sizeof (sc->sc_uuid)); + strlcpy(md->md_name, name, sizeof(md->md_name)); + md->md_version = G_MULTIPATH_VERSION; + md->md_size = pp->mediasize; + md->md_sectorsize = pp->sectorsize; + md->md_active_active = sc->sc_active_active; + error = g_write_data(cp, pp->mediasize - pp->sectorsize, + buf, pp->sectorsize); + g_topology_lock(); + g_access(cp, -1, -1, -1); + if (error != 0) + gctl_error(req, "Can't update metadata on %s (%d)", + pp->name, error); + } +} + +static void g_multipath_ctl_fail(struct gctl_req *req, struct g_class *mp, int fail) { struct g_multipath_softc *sc; @@ -964,6 +1032,8 @@ g_multipath_ctl_fail(struct gctl_req *re strcmp(cp->provider->name, name) == 0 && (cp->index & MP_LOST) == 0) { found = 1; + if (!fail == !(cp->index & MP_FAIL)) + continue; printf("GEOM_MULTIPATH: %s in %s is marked %s.\n", name, sc->sc_name, fail ? "FAIL" : "OK"); if (fail) { @@ -1172,7 +1242,7 @@ g_multipath_ctl_getactive(struct gctl_re return; } sc = gp->softc; - if (sc->sc_active_active) { + if (sc->sc_active_active == 1) { empty = 1; LIST_FOREACH(cp, &gp->consumer, consumer) { if (cp->index & MP_BAD) @@ -1209,6 +1279,8 @@ g_multipath_config(struct gctl_req *req, g_multipath_ctl_add(req, mp); } else if (strcmp(verb, "create") == 0) { g_multipath_ctl_create(req, mp); + } else if (strcmp(verb, "configure") == 0) { + g_multipath_ctl_configure(req, mp); } else if (strcmp(verb, "stop") == 0) { g_multipath_ctl_stop(req, mp); } else if (strcmp(verb, "destroy") == 0) { @@ -1245,8 +1317,9 @@ g_multipath_dumpconf(struct sbuf *sb, co (cp->index & MP_NEW) ? "NEW" : (cp->index & MP_LOST) ? "LOST" : (cp->index & MP_FAIL) ? "FAIL" : - (sc->sc_active_active || sc->sc_active == cp) ? - "ACTIVE" : "PASSIVE"); + (sc->sc_active_active == 1 || sc->sc_active == cp) ? + "ACTIVE" : + sc->sc_active_active == 2 ? "READ" : "PASSIVE"); } else { good = g_multipath_good(gp); sbuf_printf(sb, "%s%s", indent, @@ -1257,7 +1330,8 @@ g_multipath_dumpconf(struct sbuf *sb, co if (cp == NULL && pp == NULL) { sbuf_printf(sb, "%s%s", indent, sc->sc_uuid); sbuf_printf(sb, "%sActive/%s", indent, - sc->sc_active_active ? "Active" : "Passive"); + sc->sc_active_active == 2 ? "Read" : + sc->sc_active_active == 1 ? "Active" : "Passive"); sbuf_printf(sb, "%s%s", indent, sc->sc_uuid[0] == 0 ? "MANUAL" : "AUTOMATIC"); }