Date: Mon, 29 Oct 2007 10:01:41 +0100 From: Pawel Jakub Dawidek <pjd@FreeBSD.org> To: Peter Losher <Peter_Losher@isc.org> Cc: freebsd-current@freebsd.org Subject: Re: weird ZFS and mpt interactions (WAS Re: Clearing out vdevs (can't create new zpool)) Message-ID: <20071029090141.GA4407@garage.freebsd.pl> In-Reply-To: <4725895A.8030905@isc.org> References: <47246FED.6000103@isc.org> <4725895A.8030905@isc.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--envbJBWh7q8WU6mo
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Mon, Oct 29, 2007 at 12:18:50AM -0700, Peter Losher wrote:
> Peter Losher wrote:
>=20
> > I have been poking around with our 16-port SATA RAID chassis (in JBOD
> > mode) with a fresh RELENG_7 internal build from Saturday. I only had
> > half the disks {da0-7) at first, when I brought over the second set
> > (da8-15), I destroyed the existing pool and then tried to create a new
> > one with all 16 disks, and encountered this error:
> >=20
> > -=3D-
> > # zpool create tank raidz da{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
> > cannot create 'tank': one or more vdevs refer to the same device
> > -=3D-
> >=20
> > I can create one with the second set of disks (and destroy/create again)
> > but somehow I have some old vdevs sitting around which is preventing me
> > from using the first set of disks. How can I clear them out?
>=20
> O.k. poked at this some more this evening and I see what the problem
> might be. The SATA RAID chassis (in JBOD mode) is presenting the 16
> drives as two sets of 8 drives each (it has two FC connections, one for
> each set - incidentally in RAID mode, it just needs one to cover all 16
> disks as it's able to bridge the two controllers together). Those two
> FC connections are plugged into a dual port LSI Logic FC HBA:
>=20
> -=3D-
> mpt0: <Dual LSILogic FC7X04X 4Gb/s FC PCI-X Adapter> port 0x2400-0x24ff
> mem 0xb8a20000-0xb8a23fff,0xb8a00000-0xb8a0ffff irq 24 at device 3.0 on p=
ci8
> mpt0: [ITHREAD]
> mpt0: MPI Version=3D1.5.10.0
> mpt1: <Dual LSILogic FC7X04X 4Gb/s FC PCI-X Adapter> port 0x2000-0x20ff
> mem 0xb8a24000-0xb8a27fff,0xb8a10000-0xb8a1ffff irq 27 at device 3.1 on p=
ci8
> mpt1: [ITHREAD]
> mpt1: MPI Version=3D1.5.10.0
> -=3D-
>=20
> so this disks show up as such:
>=20
> -=3D-
> da0 at mpt0 bus 0 target 0 lun 0
> da0: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da0: 100.000MB/s transfers
> da0: Command Queueing Enabled
> da0: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da1 at mpt0 bus 0 target 0 lun 1
> da1: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da1: 100.000MB/s transfers
> da1: Command Queueing Enabled
> da1: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da2 at mpt0 bus 0 target 0 lun 2
> da2: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da2: 100.000MB/s transfers
> da2: Command Queueing Enabled
> da2: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da3 at mpt0 bus 0 target 0 lun 3
> da3: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da3: 100.000MB/s transfers
> da3: Command Queueing Enabled
> da3: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da4 at mpt0 bus 0 target 0 lun 4
> da4: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da4: 100.000MB/s transfers
> da4: Command Queueing Enabled
> da4: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da5 at mpt0 bus 0 target 0 lun 5
> da5: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da5: 100.000MB/s transfers
> da5: Command Queueing Enabled
> da5: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da6 at mpt0 bus 0 target 0 lun 6
> da6: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da6: 100.000MB/s transfers
> da6: Command Queueing Enabled
> da6: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da7 at mpt0 bus 0 target 0 lun 7
> da7: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da7: 100.000MB/s transfers
> da7: Command Queueing Enabled
> da7: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da8 at mpt1 bus 0 target 0 lun 0
> da8: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da8: 100.000MB/s transfers
> da8: Command Queueing Enabled
> da8: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da9 at mpt1 bus 0 target 0 lun 1
> da9: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da9: 100.000MB/s transfers
> da9: Command Queueing Enabled
> da9: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da10 at mpt1 bus 0 target 0 lun 2
> da10: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da10: 100.000MB/s transfers
> da10: Command Queueing Enabled
> da10: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da11 at mpt1 bus 0 target 0 lun 3
> da11: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da11: 100.000MB/s transfers
> da11: Command Queueing Enabled
> da11: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da12 at mpt1 bus 0 target 0 lun 4
> da12: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da12: 100.000MB/s transfers
> da12: Command Queueing Enabled
> da12: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da13 at mpt1 bus 0 target 0 lun 5
> da13: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da13: 100.000MB/s transfers
> da13: Command Queueing Enabled
> da13: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da14 at mpt1 bus 0 target 0 lun 6
> da14: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da14: 100.000MB/s transfers
> da14: Command Queueing Enabled
> da14: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> da15 at mpt1 bus 0 target 0 lun 7
> da15: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
> da15: 100.000MB/s transfers
> da15: Command Queueing Enabled
> da15: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
> -=3D-
>=20
> Two identical sets of 8 disks, the only difference is that one set is
> presented over mpt0, the other over mpt1. I suspect ZFS is treating
> this as just one set of eight vdevs as I can create a pool with either
> the first set of eight or the second set of eight, just not with BOTH
> sets of disks.
>=20
> Hope this helps explain things a bit better.
Let me get this right. Those are 16 physically different disks? Not 8
disks visible through two FC path? Can you show:
# apply "camcontrol inquiry da%1 -S" `jot 16 0`
--=20
Pawel Jakub Dawidek http://www.wheel.pl
pjd@FreeBSD.org http://www.FreeBSD.org
FreeBSD committer Am I Evil? Yes, I Am!
--envbJBWh7q8WU6mo
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)
iD8DBQFHJaF1ForvXbEpPzQRAvTgAJ9lPws2KzW6BuRQkF2x7pm8m4FUgwCfRXXA
L0qJ84X1Z+F7ixz92riO8/M=
=GX+c
-----END PGP SIGNATURE-----
--envbJBWh7q8WU6mo--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20071029090141.GA4407>