Date: Mon, 29 Oct 2007 00:18:50 -0700 From: Peter Losher <Peter_Losher@isc.org> To: freebsd-current@freebsd.org Subject: weird ZFS and mpt interactions (WAS Re: Clearing out vdevs (can't create new zpool)) Message-ID: <4725895A.8030905@isc.org> In-Reply-To: <47246FED.6000103@isc.org> References: <47246FED.6000103@isc.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig41E5856CD630E50EADA71368
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Peter Losher wrote:
> I have been poking around with our 16-port SATA RAID chassis (in JBOD
> mode) with a fresh RELENG_7 internal build from Saturday. I only had
> half the disks {da0-7) at first, when I brought over the second set
> (da8-15), I destroyed the existing pool and then tried to create a new
> one with all 16 disks, and encountered this error:
>=20
> -=3D-
> # zpool create tank raidz da{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}
> cannot create 'tank': one or more vdevs refer to the same device
> -=3D-
>=20
> I can create one with the second set of disks (and destroy/create again=
)
> but somehow I have some old vdevs sitting around which is preventing me=
> from using the first set of disks. How can I clear them out?
O.k. poked at this some more this evening and I see what the problem
might be. The SATA RAID chassis (in JBOD mode) is presenting the 16
drives as two sets of 8 drives each (it has two FC connections, one for
each set - incidentally in RAID mode, it just needs one to cover all 16
disks as it's able to bridge the two controllers together). Those two
FC connections are plugged into a dual port LSI Logic FC HBA:
-=3D-
mpt0: <Dual LSILogic FC7X04X 4Gb/s FC PCI-X Adapter> port 0x2400-0x24ff
mem 0xb8a20000-0xb8a23fff,0xb8a00000-0xb8a0ffff irq 24 at device 3.0 on p=
ci8
mpt0: [ITHREAD]
mpt0: MPI Version=3D1.5.10.0
mpt1: <Dual LSILogic FC7X04X 4Gb/s FC PCI-X Adapter> port 0x2000-0x20ff
mem 0xb8a24000-0xb8a27fff,0xb8a10000-0xb8a1ffff irq 27 at device 3.1 on p=
ci8
mpt1: [ITHREAD]
mpt1: MPI Version=3D1.5.10.0
-=3D-
so this disks show up as such:
-=3D-
da0 at mpt0 bus 0 target 0 lun 0
da0: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da0: 100.000MB/s transfers
da0: Command Queueing Enabled
da0: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da1 at mpt0 bus 0 target 0 lun 1
da1: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da1: 100.000MB/s transfers
da1: Command Queueing Enabled
da1: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da2 at mpt0 bus 0 target 0 lun 2
da2: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da2: 100.000MB/s transfers
da2: Command Queueing Enabled
da2: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da3 at mpt0 bus 0 target 0 lun 3
da3: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da3: 100.000MB/s transfers
da3: Command Queueing Enabled
da3: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da4 at mpt0 bus 0 target 0 lun 4
da4: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da4: 100.000MB/s transfers
da4: Command Queueing Enabled
da4: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da5 at mpt0 bus 0 target 0 lun 5
da5: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da5: 100.000MB/s transfers
da5: Command Queueing Enabled
da5: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da6 at mpt0 bus 0 target 0 lun 6
da6: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da6: 100.000MB/s transfers
da6: Command Queueing Enabled
da6: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da7 at mpt0 bus 0 target 0 lun 7
da7: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da7: 100.000MB/s transfers
da7: Command Queueing Enabled
da7: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da8 at mpt1 bus 0 target 0 lun 0
da8: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da8: 100.000MB/s transfers
da8: Command Queueing Enabled
da8: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da9 at mpt1 bus 0 target 0 lun 1
da9: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da9: 100.000MB/s transfers
da9: Command Queueing Enabled
da9: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da10 at mpt1 bus 0 target 0 lun 2
da10: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da10: 100.000MB/s transfers
da10: Command Queueing Enabled
da10: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da11 at mpt1 bus 0 target 0 lun 3
da11: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da11: 100.000MB/s transfers
da11: Command Queueing Enabled
da11: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da12 at mpt1 bus 0 target 0 lun 4
da12: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da12: 100.000MB/s transfers
da12: Command Queueing Enabled
da12: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da13 at mpt1 bus 0 target 0 lun 5
da13: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da13: 100.000MB/s transfers
da13: Command Queueing Enabled
da13: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da14 at mpt1 bus 0 target 0 lun 6
da14: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da14: 100.000MB/s transfers
da14: Command Queueing Enabled
da14: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
da15 at mpt1 bus 0 target 0 lun 7
da15: <Seagate ST3400633NS R001> Fixed Direct Access SCSI-3 device
da15: 100.000MB/s transfers
da15: Command Queueing Enabled
da15: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C)
-=3D-
Two identical sets of 8 disks, the only difference is that one set is
presented over mpt0, the other over mpt1. I suspect ZFS is treating
this as just one set of eight vdevs as I can create a pool with either
the first set of eight or the second set of eight, just not with BOTH
sets of disks.
Hope this helps explain things a bit better.
Best Wishes - Peter
--=20
Peter_Losher@isc.org | ISC | OpenPGP 0xE8048D08 | "The bits must flow"
--------------enig41E5856CD630E50EADA71368
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iD8DBQFHJYlgPtVx9OgEjQgRAnQQAJ9OoPuAr0qCaVwqK+yaYyqEmtZr2ACeO38Z
pY2u7oNVOS2N6RTDyD2xGTM=
=sRYs
-----END PGP SIGNATURE-----
--------------enig41E5856CD630E50EADA71368--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4725895A.8030905>