Date: Wed, 28 Jul 2010 10:55:47 +0100 From: Vincent Hoffman <vince@unsane.co.uk> To: freebsd-questions@freebsd.org, geom@freebsd.org Subject: Re: gmultipath woes Message-ID: <4C4FFEA3.7060307@unsane.co.uk> In-Reply-To: <AANLkTin-u62Uq9nWOC9hTBRkNw=OHMntT7_HM=1uY060@mail.gmail.com> References: <AANLkTin-u62Uq9nWOC9hTBRkNw=OHMntT7_HM=1uY060@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 28/07/2010 08:17, Corey Smith wrote:
> Has anyone tried gmultipath under 8.1-RELEASE/amd64 yet? This is my
> first attempt at multipathing so it is possible I'm doing something
> wrong but anyways:
>
> # uname -a
> FreeBSD host 8.1-RELEASE FreeBSD 8.1-RELEASE #0: Mon Jul 19 02:36:49
> UTC 2010 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC
> amd64
>
> # dmesg | egrep '^(isp|da1|da2)'
> isp0: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xd800-0xd8ff mem
> 0xdf4f8000-0xdf4fbfff irq 38 at device 0.0 on pci6
> isp1: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xdc00-0xdcff mem
> 0xdf4fc000-0xdf4fffff irq 45 at device 0.1 on pci6
> da1 at isp0 bus 0 scbus1 target 0 lun 0
> da1: <HITACHI DF600F 0000> Fixed Direct Access SCSI-4 device
> da1: 400.000MB/s transfers
> da1: Command Queueing enabled
> da1: 2097152MB (4294967296 512 byte sectors: 255H 63S/T 267349C)
> da2 at isp1 bus 0 scbus2 target 0 lun 0
> da2: <HITACHI DF600F 0000> Fixed Direct Access SCSI-4 device
> da2: 400.000MB/s transfers
> da2: Command Queueing enabled
> da2: 2097152MB (4294967296 512 byte sectors: 255H 63S/T 267349C)
>
> # camcontrol devlist
> <Dell VIRTUAL DISK 1028> at scbus0 target 0 lun 0 (pass0,da0)
> <DP BACKPLANE 1.07> at scbus0 target 8 lun 0 (pass1,ses0)
> <HITACHI DF600F 0000> at scbus1 target 0 lun 0 (pass2,da1)
> <HITACHI DF600F 0000> at scbus2 target 0 lun 0 (pass3,da2)
>
> # camcontrol inquiry da1 -S
> 870405180007
> # camcontrol inquiry da2 -S
> 870405180007
>
> # gmultipath label -v FRED /dev/da1 /dev/da2
> gmultipath: Unknown verb add
>
> # gmultipath status
> Name Status Components
> multipath/FRED N/A da1
>
> # gmultipath list
> Geom name: FRED
> Providers:
> 1. Name: multipath/FRED
> Mediasize: 2199023255040 (2.0T)
> Sectorsize: 512
> Mode: r0w0e0
> Consumers:
> 1. Name: da1
> Mediasize: 2199023255552 (2.0T)
> Sectorsize: 512
> Mode: r0w0e0
>
> ------ EOT ------
>
> I'm wondering what is going on with the "Unknown verb add" and why is
> it only listing da1 in the components? Does the disk need to be in a
> particular state (unpartitioned, glabel, etc...) or does it not
>
I dont have any storage with multiple paths handy other than creating md
devices, however doing that as you have (and as the manpage says) gives
the same results.
[root@ostracod /mnt]# mdconfig -a -t vnode -f /scratch/test.dsk -u 12
[root@ostracod /mnt]# mdconfig -a -t vnode -f /scratch/test.dsk -u 13
[root@ostracod /mnt]# sysctl kern.geom.multipath.debug=1
kern.geom.multipath.debug: 1 -> 1
[root@ostracod /mnt]# gmultipath label -v TEST /dev/md12 /dev/md13
gmultipath: Unknown verb add
[root@ostracod /mnt]# gmultipath list
Geom name: TEST
Providers:
1. Name: multipath/TEST
Mediasize: 52428288 (50M)
Sectorsize: 512
Mode: r0w0e0
Consumers:
1. Name: md12
Mediasize: 52428800 (50M)
Sectorsize: 512
Mode: r0w0e0
[root@ostracod /mnt]# gmultipath status
Name Status Components
multipath/TEST N/A md12
However doing
echo 1 > /dev/md13 made geom re-taste the device so I get
[root@ostracod /mnt]# gmultipath list
Geom name: TEST
Providers:
1. Name: multipath/TEST
Mediasize: 52428288 (50M)
Sectorsize: 512
Mode: r0w0e0
Consumers:
1. Name: md12
Mediasize: 52428800 (50M)
Sectorsize: 512
Mode: r0w0e0
2. Name: md13
Mediasize: 52428800 (50M)
Sectorsize: 512
Mode: r0w0e0
Also doing
mdconfig -a -t vnode -f /scratch/test.dsk -u 12
gmultipath load
gmultipath label -v FRED /dev/md12
mdconfig -a -t vnode -f /scratch/test.dsk -u 13
gave me
Jul 28 10:32:43 ostracod kernel: md12 is not MULTIPATH
Jul 28 10:32:43 ostracod kernel: MULTIPATH:
FRED/12a41222-9a2b-11df-aa80-00012e273257
Jul 28 10:32:43 ostracod kernel: GEOM_MULTIPATH: adding md12 to
FRED/12a41222-9a2b-11df-aa80-00012e273257
Jul 28 10:32:43 ostracod kernel: GEOM_MULTIPATH: md12 now active path in
FRED
Jul 28 10:32:43 ostracod kernel: multipath/FRED is not MULTIPATH
Jul 28 10:33:37 ostracod kernel: MULTIPATH:
FRED/12a41222-9a2b-11df-aa80-00012e273257
Jul 28 10:33:37 ostracod kernel: GEOM_MULTIPATH: adding md13 to
FRED/12a41222-9a2b-11df-aa80-00012e273257
[root@ostracod /mnt]# gmultipath list
Geom name: FRED
Providers:
1. Name: multipath/FRED
Mediasize: 52428288 (50M)
Sectorsize: 512
Mode: r0w0e0
Consumers:
1. Name: md12
Mediasize: 52428800 (50M)
Sectorsize: 512
Mode: r0w0e0
2. Name: md13
Mediasize: 52428800 (50M)
Sectorsize: 512
Mode: r0w0e0
[root@ostracod /mnt]# gpart create -s gpt multipath/FRED
multipath/FRED created
[root@ostracod /mnt]# gpart add -s 10M -t freebsd-ufs multipath/FRED
multipath/FREDp1 added
[root@ostracod /mnt]# newfs /dev/multipath/FREDp1
/dev/multipath/FREDp1: 10.0MB (20480 sectors) block size 16384, fragment
size 2048
using 4 cylinder groups of 2.52MB, 161 blks, 384 inodes.
super-block backups (for fsck -b #) at:
160, 5312, 10464, 15616
[root@ostracod /mnt]# mount /dev/multipath/FREDp1 /mnt/foo/
[root@ostracod /mnt]
so it all seems to works but not as expected/documented.
Also I get these various errors in /var/log/messages
Jul 28 10:41:17 ostracod kernel: GEOM: md13: the secondary GPT table is
corrupt or invalid.
Jul 28 10:41:17 ostracod kernel: GEOM: md13: using the primary only --
recovery suggested.
Jul 28 10:41:17 ostracod kernel: GEOM: md12: the secondary GPT table is
corrupt or invalid.
Jul 28 10:41:17 ostracod kernel: GEOM: md12: using the primary only --
recovery suggested.
Jul 28 10:41:17 ostracod kernel: multipath/FRED is not MULTIPATH
Jul 28 10:42:00 ostracod kernel: multipath/FREDp1 is not MULTIPATH
Jul 28 10:42:00 ostracod kernel: multipath/FRED is not MULTIPATH
Jul 28 10:42:00 ostracod kernel:
gptid/5ed86f82-9a2c-11df-aa80-00012e273257 is not MULTIPATH
Jul 28 10:42:14 ostracod kernel: GEOM: md13: the secondary GPT table is
corrupt or invalid.
Jul 28 10:42:14 ostracod kernel: GEOM: md13: using the primary only --
recovery suggested.
Jul 28 10:42:14 ostracod kernel: GEOM: md12: the secondary GPT table is
corrupt or invalid.
Jul 28 10:42:14 ostracod kernel: GEOM: md12: using the primary only --
recovery suggested.
Jul 28 10:42:14 ostracod kernel: multipath/FRED is not MULTIPATH
Jul 28 10:42:14 ostracod kernel: multipath/FREDp1 is not MULTIPATH
Jul 28 10:42:14 ostracod kernel: md13p1 is not MULTIPATH
Jul 28 10:42:14 ostracod kernel: md12p1 is not MULTIPATH
Jul 28 10:42:14 ostracod kernel: ufsid/4c4ffb754e3cdb5c is not MULTIPATH
Jul 28 10:42:14 ostracod kernel:
gptid/5ed86f82-9a2c-11df-aa80-00012e273257 is not MULTIPATH
so I'd say that gmultipath needs some attention, and would guess using
gpt with geom_multipath isnt a great idea
I'd file a pr if I were you. I'm cross posting to geom@ in case I'm
being a muppet somewhere though.
Vince
> matter?
>
> -Corey Smith
> _______________________________________________
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C4FFEA3.7060307>