Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 20 Jan 2012 10:09:56 +0200
From:      Nikolay Denev <ndenev@gmail.com>
To:        Gary Palmer <gpalmer@freebsd.org>
Cc:        Alexander Motin <mav@FreeBSD.org>, FreeBSD-Current <freebsd-current@freebsd.org>, Dennis K?gel <dk@neveragain.de>, freebsd-geom@freebsd.org
Subject:   Re: RFC: GEOM MULTIPATH rewrite
Message-ID:  <059C17DB-3A7B-41AA-BF91-2F8EBAF17D01@gmail.com>
In-Reply-To: <20111114210957.GA68559@in-addr.com>
References:  <4EAF00A6.5060903@FreeBSD.org> <05E0E64F-5EC4-425A-81E4-B6C35320608B@neveragain.de> <4EB05566.3060700@FreeBSD.org> <20111114210957.GA68559@in-addr.com>

next in thread | previous in thread | raw e-mail | index | archive | help

On Nov 14, 2011, at 11:09 PM, Gary Palmer wrote:

> On Tue, Nov 01, 2011 at 10:24:06PM +0200, Alexander Motin wrote:
>> On 01.11.2011 19:50, Dennis K?gel wrote:
>>> Not sure if replying on-list or off-list makes more sense...
>>=20
>> Replying on-list could share experience to other users.
>>=20
>>> Anyway, some first impressions, on stable/9:
>>>=20
>>> The lab environment here is a EMC VNX / Clariion SAN, which has two =
Storage Processors, connected to different switches, connected to two =
isp(4)s on the test machine. So at any time, the machine sees four =
paths, but only two are available (depending on which SP owns the LUN).
>>>=20
>>> 580# camcontrol devlist
>>> <DGC VRAID 0531>                   at scbus0 target 0 lun 0 =
(da0,pass0)
>>> <DGC VRAID 0531>                   at scbus0 target 1 lun 0 =
(da1,pass1)
>>> <DGC VRAID 0531>                   at scbus1 target 0 lun 0 =
(da2,pass2)
>>> <DGC VRAID 0531>                   at scbus1 target 1 lun 0 =
(da3,pass3)
>>> <COMPAQ RAID 1(1VOLUME OK>         at scbus2 target 0 lun 0 =
(da4,pass4)
>>> <COMPAQ RAID 0  VOLUME OK>         at scbus2 target 1 lun 0 =
(da5,pass5)
>>> <hp DVD D  DS8D3SH HHE7>           at scbus4 target 0 lun 0 =
(cd0,pass6)
>>>=20
>>> I miss the ability to "add" disks to automatic mode multipaths, but =
I (just now) realized this only makes sense when gmultipath has some =
kind of path checking facility (like periodically trying to read sector =
0 of each configured device, this is was Linux' devicemapper-multipathd =
does).
>>=20
>> In automatic mode other paths supposed to be detected via metadata
>> reading. If in your case some paths are not readable, automatic mode
>> can't work as expected. By the way, could you describe how your
>> configuration supposed to work, like when other paths will start
>> working?=20
>=20
> Without knowledge of the particular Clariion SAN Dennis is working =
with,
> I've seen some so-called active/active RAID controllers force a LUN=20
> fail over from one controller to another (taking it offline for 3 =
seconds
> in the process) because the LUN received an I/O down a path to the =
controller
> that was formerly taking the standby role for that LUN (and it was =
per-LUN,
> so some would be owned by one controller and some by the other).  =
During
> the controller switch, all I/O to the LUN would fail.  Thankfully that
> particular RAID model where I observed this behaviour hasn't been sold =
in
> several years, but I would tend to expect such behaviour at the lower
> end of the storage market with the higher end units doing true =
active/active
> configurations. (and no, I won't name the manufacturer on a public =
list)
>=20
> This is exactly why Linux ships with a multipath configuration file, =
so
> it can describe exactly what form of brain damage the controller in
> question implements so it can work around it, and maybe even=20
> document some vendor-specific extensions so that the host can detect
> which controller is taking which role for a particular path.
>=20
> Even some controllers that don't have pathological behaviour when
> they receive I/O down the wrong path have sub-optimal behaviour unless
> you choose the right path.  NetApp SANs in particular typically have =
two
> independant controllers with a high-speed internal interconnect, =
however
> there is a measurable and not-insignificant penalty for sending the =
I/O
> to the "partner" controller for a LUN, across the internal =
interconnect
> (called a "VTIC" I believe) to the "owner" controller.  I've been =
told,
> although I have not measured this myself, that it can add several ms =
to
> a transaction, which when talking about SAN storage is potentially =
several
> times what it takes to do the same I/O directly to the controller that
> owns it.  There's probably a way to make the "partner" controller not
> advertise the LUN until it takes over in a failover scenario, but =
every
> NetApp I've worked with is set (by default I believe) to advertise the
> LUN out both controllers.
>=20
> Gary
> _______________________________________________
> freebsd-current@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to =
"freebsd-current-unsubscribe@freebsd.org"

Another thing I've observed is that active/active probably only makes =
sense if you are accessing single LUN.
In my tests where I have 24 LUNS that form 4 vdevs in a single zpool, =
the highest performance was achieved
when I split the active paths among the controllers installed in the =
server importing the pool. (basically "gmultipath rotate $LUN" in =
rc.local for half of the paths)
Using active/active in this situation resulted in fluctuating =
performance.





Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?059C17DB-3A7B-41AA-BF91-2F8EBAF17D01>