From owner-freebsd-current@FreeBSD.ORG Mon Oct 29 20:31:25 2007 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7BF3216A41A for ; Mon, 29 Oct 2007 20:31:25 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id B866913C4BD for ; Mon, 29 Oct 2007 20:31:22 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 1849845E93; Mon, 29 Oct 2007 09:01:49 +0100 (CET) Received: from localhost (pjd.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 8BA9E45CD9; Mon, 29 Oct 2007 09:01:43 +0100 (CET) Date: Mon, 29 Oct 2007 10:01:41 +0100 From: Pawel Jakub Dawidek To: Peter Losher Message-ID: <20071029090141.GA4407@garage.freebsd.pl> References: <47246FED.6000103@isc.org> <4725895A.8030905@isc.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="envbJBWh7q8WU6mo" Content-Disposition: inline In-Reply-To: <4725895A.8030905@isc.org> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=3.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-current@freebsd.org Subject: Re: weird ZFS and mpt interactions (WAS Re: Clearing out vdevs (can't create new zpool)) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2007 20:31:25 -0000 --envbJBWh7q8WU6mo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Oct 29, 2007 at 12:18:50AM -0700, Peter Losher wrote: > Peter Losher wrote: >=20 > > I have been poking around with our 16-port SATA RAID chassis (in JBOD > > mode) with a fresh RELENG_7 internal build from Saturday. I only had > > half the disks {da0-7) at first, when I brought over the second set > > (da8-15), I destroyed the existing pool and then tried to create a new > > one with all 16 disks, and encountered this error: > >=20 > > -=3D- > > # zpool create tank raidz da{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15} > > cannot create 'tank': one or more vdevs refer to the same device > > -=3D- > >=20 > > I can create one with the second set of disks (and destroy/create again) > > but somehow I have some old vdevs sitting around which is preventing me > > from using the first set of disks. How can I clear them out? >=20 > O.k. poked at this some more this evening and I see what the problem > might be. The SATA RAID chassis (in JBOD mode) is presenting the 16 > drives as two sets of 8 drives each (it has two FC connections, one for > each set - incidentally in RAID mode, it just needs one to cover all 16 > disks as it's able to bridge the two controllers together). Those two > FC connections are plugged into a dual port LSI Logic FC HBA: >=20 > -=3D- > mpt0: port 0x2400-0x24ff > mem 0xb8a20000-0xb8a23fff,0xb8a00000-0xb8a0ffff irq 24 at device 3.0 on p= ci8 > mpt0: [ITHREAD] > mpt0: MPI Version=3D1.5.10.0 > mpt1: port 0x2000-0x20ff > mem 0xb8a24000-0xb8a27fff,0xb8a10000-0xb8a1ffff irq 27 at device 3.1 on p= ci8 > mpt1: [ITHREAD] > mpt1: MPI Version=3D1.5.10.0 > -=3D- >=20 > so this disks show up as such: >=20 > -=3D- > da0 at mpt0 bus 0 target 0 lun 0 > da0: Fixed Direct Access SCSI-3 device > da0: 100.000MB/s transfers > da0: Command Queueing Enabled > da0: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da1 at mpt0 bus 0 target 0 lun 1 > da1: Fixed Direct Access SCSI-3 device > da1: 100.000MB/s transfers > da1: Command Queueing Enabled > da1: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da2 at mpt0 bus 0 target 0 lun 2 > da2: Fixed Direct Access SCSI-3 device > da2: 100.000MB/s transfers > da2: Command Queueing Enabled > da2: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da3 at mpt0 bus 0 target 0 lun 3 > da3: Fixed Direct Access SCSI-3 device > da3: 100.000MB/s transfers > da3: Command Queueing Enabled > da3: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da4 at mpt0 bus 0 target 0 lun 4 > da4: Fixed Direct Access SCSI-3 device > da4: 100.000MB/s transfers > da4: Command Queueing Enabled > da4: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da5 at mpt0 bus 0 target 0 lun 5 > da5: Fixed Direct Access SCSI-3 device > da5: 100.000MB/s transfers > da5: Command Queueing Enabled > da5: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da6 at mpt0 bus 0 target 0 lun 6 > da6: Fixed Direct Access SCSI-3 device > da6: 100.000MB/s transfers > da6: Command Queueing Enabled > da6: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da7 at mpt0 bus 0 target 0 lun 7 > da7: Fixed Direct Access SCSI-3 device > da7: 100.000MB/s transfers > da7: Command Queueing Enabled > da7: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da8 at mpt1 bus 0 target 0 lun 0 > da8: Fixed Direct Access SCSI-3 device > da8: 100.000MB/s transfers > da8: Command Queueing Enabled > da8: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da9 at mpt1 bus 0 target 0 lun 1 > da9: Fixed Direct Access SCSI-3 device > da9: 100.000MB/s transfers > da9: Command Queueing Enabled > da9: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da10 at mpt1 bus 0 target 0 lun 2 > da10: Fixed Direct Access SCSI-3 device > da10: 100.000MB/s transfers > da10: Command Queueing Enabled > da10: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da11 at mpt1 bus 0 target 0 lun 3 > da11: Fixed Direct Access SCSI-3 device > da11: 100.000MB/s transfers > da11: Command Queueing Enabled > da11: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da12 at mpt1 bus 0 target 0 lun 4 > da12: Fixed Direct Access SCSI-3 device > da12: 100.000MB/s transfers > da12: Command Queueing Enabled > da12: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da13 at mpt1 bus 0 target 0 lun 5 > da13: Fixed Direct Access SCSI-3 device > da13: 100.000MB/s transfers > da13: Command Queueing Enabled > da13: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da14 at mpt1 bus 0 target 0 lun 6 > da14: Fixed Direct Access SCSI-3 device > da14: 100.000MB/s transfers > da14: Command Queueing Enabled > da14: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > da15 at mpt1 bus 0 target 0 lun 7 > da15: Fixed Direct Access SCSI-3 device > da15: 100.000MB/s transfers > da15: Command Queueing Enabled > da15: 381554MB (781422768 512 byte sectors: 255H 63S/T 48641C) > -=3D- >=20 > Two identical sets of 8 disks, the only difference is that one set is > presented over mpt0, the other over mpt1. I suspect ZFS is treating > this as just one set of eight vdevs as I can create a pool with either > the first set of eight or the second set of eight, just not with BOTH > sets of disks. >=20 > Hope this helps explain things a bit better. Let me get this right. Those are 16 physically different disks? Not 8 disks visible through two FC path? Can you show: # apply "camcontrol inquiry da%1 -S" `jot 16 0` --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --envbJBWh7q8WU6mo Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFHJaF1ForvXbEpPzQRAvTgAJ9lPws2KzW6BuRQkF2x7pm8m4FUgwCfRXXA L0qJ84X1Z+F7ixz92riO8/M= =GX+c -----END PGP SIGNATURE----- --envbJBWh7q8WU6mo--