From owner-freebsd-current@FreeBSD.ORG Mon Oct 29 07:18:55 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 43CD916A418 for ; Mon, 29 Oct 2007 07:18:55 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Received: from mx.isc.org (mx.isc.org [IPv6:2001:4f8:0:2::1c]) by mx1.freebsd.org (Postfix) with ESMTP id 11E2E13C49D for ; Mon, 29 Oct 2007 07:18:55 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Received: from farside.isc.org (farside.isc.org [IPv6:2001:4f8:3:bb::5]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "farside.isc.org", Issuer "ISC CA" (verified OK)) by mx.isc.org (Postfix) with ESMTP id 03876114027 for ; Mon, 29 Oct 2007 07:18:54 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Received: from manx.plosh.net (manx.vpn.isc.org [149.20.66.7]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by farside.isc.org (Postfix) with ESMTP id 17304E6059; Mon, 29 Oct 2007 07:18:54 +0000 (UTC) (envelope-from Peter_Losher@isc.org) Message-ID: <4725895A.8030905@isc.org> Date: Mon, 29 Oct 2007 00:18:50 -0700 From: Peter Losher Organization: ISC User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: freebsd-current@freebsd.org References: <47246FED.6000103@isc.org> In-Reply-To: <47246FED.6000103@isc.org> X-Enigmail-Version: 0.95.5 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig41E5856CD630E50EADA71368" Subject: weird ZFS and mpt interactions (WAS Re: Clearing out vdevs (can't create new zpool)) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2007 07:18:55 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig41E5856CD630E50EADA71368 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Peter Losher wrote: > I have been poking around with our 16-port SATA RAID chassis (in JBOD > mode) with a fresh RELENG_7 internal build from Saturday. I only had > half the disks {da0-7) at first, when I brought over the second set > (da8-15), I destroyed the existing pool and then tried to create a new > one with all 16 disks, and encountered this error: >=20 > -=3D- > # zpool create tank raidz da{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15} > cannot create 'tank': one or more vdevs refer to the same device > -=3D- >=20 > I can create one with the second set of disks (and destroy/create again= ) > but somehow I have some old vdevs sitting around which is preventing me= > from using the first set of disks. How can I clear them out? O.k. poked at this some more this evening and I see what the problem might be. The SATA RAID chassis (in JBOD mode) is presenting the 16 drives as two sets of 8 drives each (it has two FC connections, one for each set - incidentally in RAID mode, it just needs one to cover all 16 disks as it's able to bridge the two controllers together). Those two FC connections are plugged into a dual port LSI Logic FC HBA: -=3D- mpt0: port 0x2400-0x24ff mem 0xb8a20000-0xb8a23fff,0xb8a00000-0xb8a0ffff irq 24 at device 3.0 on p= ci8 mpt0: [ITHREAD] mpt0: MPI Version=3D1.5.10.0 mpt1: port 0x2000-0x20ff mem 0xb8a24000-0xb8a27fff,0xb8a10000-0xb8a1ffff irq 27 at device 3.1 on p= ci8 mpt1: [ITHREAD] mpt1: MPI Version=3D1.5.10.0 -=3D- so this disks show up as such: -=3D- da0 at mpt0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-3 device da0: 100.000MB/s transfers da0: Command Queueing Enabled da0: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da1 at mpt0 bus 0 target 0 lun 1 da1: Fixed Direct Access SCSI-3 device da1: 100.000MB/s transfers da1: Command Queueing Enabled da1: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da2 at mpt0 bus 0 target 0 lun 2 da2: Fixed Direct Access SCSI-3 device da2: 100.000MB/s transfers da2: Command Queueing Enabled da2: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da3 at mpt0 bus 0 target 0 lun 3 da3: Fixed Direct Access SCSI-3 device da3: 100.000MB/s transfers da3: Command Queueing Enabled da3: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da4 at mpt0 bus 0 target 0 lun 4 da4: Fixed Direct Access SCSI-3 device da4: 100.000MB/s transfers da4: Command Queueing Enabled da4: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da5 at mpt0 bus 0 target 0 lun 5 da5: Fixed Direct Access SCSI-3 device da5: 100.000MB/s transfers da5: Command Queueing Enabled da5: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da6 at mpt0 bus 0 target 0 lun 6 da6: Fixed Direct Access SCSI-3 device da6: 100.000MB/s transfers da6: Command Queueing Enabled da6: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da7 at mpt0 bus 0 target 0 lun 7 da7: Fixed Direct Access SCSI-3 device da7: 100.000MB/s transfers da7: Command Queueing Enabled da7: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da8 at mpt1 bus 0 target 0 lun 0 da8: Fixed Direct Access SCSI-3 device da8: 100.000MB/s transfers da8: Command Queueing Enabled da8: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da9 at mpt1 bus 0 target 0 lun 1 da9: Fixed Direct Access SCSI-3 device da9: 100.000MB/s transfers da9: Command Queueing Enabled da9: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da10 at mpt1 bus 0 target 0 lun 2 da10: Fixed Direct Access SCSI-3 device da10: 100.000MB/s transfers da10: Command Queueing Enabled da10: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da11 at mpt1 bus 0 target 0 lun 3 da11: Fixed Direct Access SCSI-3 device da11: 100.000MB/s transfers da11: Command Queueing Enabled da11: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da12 at mpt1 bus 0 target 0 lun 4 da12: Fixed Direct Access SCSI-3 device da12: 100.000MB/s transfers da12: Command Queueing Enabled da12: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da13 at mpt1 bus 0 target 0 lun 5 da13: Fixed Direct Access SCSI-3 device da13: 100.000MB/s transfers da13: Command Queueing Enabled da13: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da14 at mpt1 bus 0 target 0 lun 6 da14: Fixed Direct Access SCSI-3 device da14: 100.000MB/s transfers da14: Command Queueing Enabled da14: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) da15 at mpt1 bus 0 target 0 lun 7 da15: Fixed Direct Access SCSI-3 device da15: 100.000MB/s transfers da15: Command Queueing Enabled da15: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) -=3D- Two identical sets of 8 disks, the only difference is that one set is presented over mpt0, the other over mpt1. I suspect ZFS is treating this as just one set of eight vdevs as I can create a pool with either the first set of eight or the second set of eight, just not with BOTH sets of disks. Hope this helps explain things a bit better. Best Wishes - Peter --=20 Peter_Losher@isc.org | ISC | OpenPGP 0xE8048D08 | "The bits must flow" --------------enig41E5856CD630E50EADA71368 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iD8DBQFHJYlgPtVx9OgEjQgRAnQQAJ9OoPuAr0qCaVwqK+yaYyqEmtZr2ACeO38Z pY2u7oNVOS2N6RTDyD2xGTM= =sRYs -----END PGP SIGNATURE----- --------------enig41E5856CD630E50EADA71368--