From owner-freebsd-scsi@FreeBSD.ORG Thu Sep 23 13:56:04 2010 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7D2FA106564A for ; Thu, 23 Sep 2010 13:56:04 +0000 (UTC) (envelope-from darksoul@darkbsd.org) Received: from zelretch.yomi.darkbsd.org (fnttkyo035000.tkyo.fnt.ftth.ppp.infoweb.ne.jp [58.1.242.192]) by mx1.freebsd.org (Postfix) with ESMTP id 8ED5F8FC25 for ; Thu, 23 Sep 2010 13:56:03 +0000 (UTC) Received: from zelretch.yomi.darkbsd.org (localhost [127.0.0.1]) by zelretch.yomi.darkbsd.org (Postfix) with ESMTP id 0DBE22840C7; Thu, 23 Sep 2010 22:36:11 +0900 (JST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=darkbsd.org; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=selector1; bh=andHIrl bifbbrThP7xuPamKtvEE=; b=UQaXm9x5bImvis3hx/2Gm6OrilaQp6eY/+DAZgo 79Wp9pIhKcvUeL+NIycrKsJD8orOWaPaiDW/XJM/KGCDBKENCOm7GGZVXkDOkUQ2 F6CXZkCcvM0L3BG/cwy5MMBC41Z/Ve9LFmMcJR+T7jGxBN6MjZ4oj0aVBnEbqZfs flYA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=darkbsd.org; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=selector1; b=Y dn/OhszlOk3fYmGRCtCCDpEBapq1P5jzF32xwtZJbUDA7UJXP+UIaa3+Y8rbJowp s/wkNuyvJWczZLSDHwTh+n2GX3empEq8x+EdSzLUK1njUBipvSDmUquYDXuA8Vq4 hGDDo0aAYn4AuXbsLCThv52az3Y0MQcfJ40UFDL688= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=darkbsd.org; h= content-transfer-encoding:content-type:content-type:in-reply-to :references:subject:subject:mime-version:user-agent:from:from :date:date:message-id:received:received; s=selector1; t= 1285248960; bh=xENIcoOsSaxrFNUCjH05G9v/Oe0MmuuQ8IrO+8IN2nw=; b=C aHcTxQGp32U2GLNdX1nQBEwQk32OdYMro4sHOkGtRs6BsZa9KzEHRQmzFsgHEdbO pPwvyKH1XmPIbwbdc0QqBnZQd070y0Ir/V9F6WH7mCOjX2aKnD/eWMpsIjTixkzd RG4iwhnZZBOoEKyk1w3F+ltEaO51A+9V49nEr2JiHE= Received: from zelretch.yomi.darkbsd.org ([127.0.0.1]) by zelretch.yomi.darkbsd.org (zelretch.yomi.darkbsd.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id H1948hLH4BcZ; Thu, 23 Sep 2010 22:36:00 +0900 (JST) Received: from [192.168.3.42] (archer.yomi.darkbsd.org [192.168.3.42]) (Authenticated sender: darksoul) by zelretch.yomi.darkbsd.org (Postfix) with ESMTPSA id 060882840C4; Thu, 23 Sep 2010 22:35:59 +0900 (JST) Message-ID: <4C9B57DF.4090905@darkbsd.org> Date: Thu, 23 Sep 2010 22:36:31 +0900 From: DarkSoul User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.9pre) Gecko/20100821 Lanikai/3.1.3pre MIME-Version: 1.0 To: Niklas Saers References: <2EA9CBBC-3F97-4AF2-BFB5-96DF39FDE376@saers.com> In-Reply-To: <2EA9CBBC-3F97-4AF2-BFB5-96DF39FDE376@saers.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-scsi@freebsd.org Subject: Re: mfi - setting up disks X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Sep 2010 13:56:04 -0000 Hello, I already tinkered with that kind of controller too (it seems we faced the same problems in the same order too...). For this kind of purposes, you really want a controller that does real JBOD and tells you about what is going on with the disk. :/ I bumped in exactly the same problem as you, meaning that removing the drive will destroy the RAID-0 volume, and frankly, even if the tools were working for recreating the volumes, having to do that to see the disk again is a "dirty hack"(tm) that is bound to blow up in your face at some point, or to be really hard to maintain for any upcoming upgrade. Sorry for not providing any additional help with your problem. :/ On 09/23/2010 09:55 PM, Niklas Saers wrote: > Hi guys, > In the SuperMicro system where I had problems with the mpt controller, I switched it for a mfi-based controller. I had it set up with 36x RAID0 volumes with each their own disk (no way to access the disk otherwise I found), and added them to a ZFS system. The numbering became a bit weird, so I pulled the disks out one by one and put them back to figure out and note down what disk number was in what slot. Only test data on my ZFS volume, so I didn't mind that crashing. > > Now that all disks have been taken out and put back in one by one, I do: > > # mfiutil show volumes > mfi0 Volumes: > Id Size Level Stripe State Cache Name > > Whoops, no volumes? That can't be good. I check up on the disks, they're all there: > > mfiutil show drives > mfi0 Physical Drives: > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 2 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 3 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 4 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 14 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 15 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 17 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 23 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 1 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 4 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 5 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 6 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 7 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 8 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 10 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 11 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 0 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 5 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 6 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 7 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 8 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 9 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 10 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 11 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 12 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 13 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 16 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 19 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 21 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 22 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 2 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 3 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 9 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 1 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 20 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 2, slot 0 > ( 1863G) UNCONFIGURED GOOD SATA enclosure 1, slot 18 > > Well, that's an excelent way of adding the volumes in the sequence they appear physically, so I start with the first one from top left: > # mfiutil create raid0 -v E01:S05 > Adding drive 26 to array 0 > Adding array 0 to volume 0 > mfiutil: Command failed: Wrong firmware or drive state > mfiutil: Failed to add volume: Input/output error > > Firmware error? Invalid drive state? > > # mfiutil good E01:S05 > mfiutil: Drive 26 is already in the desired state > > Seems to be good... do I have a firmware issue? Check dmesg: > > mfi0: port 0xc000-0xc0ff mem 0xfad7c000-0xfad7ffff,0xfadc0000-0xfadfffff irq 16 at device 0.0 on pci5 > mfi0: Megaraid SAS driver Ver 3.00 > mfi0: 1966 (338565595s/0x0020/info) - Shutdown command received from host > mfi0: 1967 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0079/1000/9261/1000) > mfi0: 1968 (boot + 3s/0x0020/info) - Firmware version 2.0.03-0673 > mfi0: 1969 (boot + 4s/0x0020/info) - Board Revision > mfi0: 1970 (boot + 24s/0x0004/info) - Enclosure (SES) discovered on PD 08(c Port 0 - 3/p1) > mfi0: 1971 (boot + 24s/0x0004/info) - Enclosure (SES) discovered on PD 09(c Port 0 - 3/p2) > mfi0: 1972 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) communication restored > mfi0: 1973 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) fan 1 speed changed > mfi0: 1974 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) fan 2 speed changed > mfi0: 1975 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) fan 3 speed changed > mfi0: 1976 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) communication restored > mfi0: 1977 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) fan 1 speed changed > mfi0: 1978 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) fan 2 speed changed > mfi0: 1979 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) fan 3 speed changed > mfi0: 1980 (boot + 24s/0x0002/info) - Inserted: Encl PD 08 > mfi0: 1981 (boot + 24s/0x0002/info) - Inserted: PD 08(c Port 0 - 3/p1) Info: enclPd=08, scsiType=d, portMap=00, sasAddr=50030480008fb0fd,0000000000000000 > mfi0: 1982 (boot + 24s/0x0002/info) - Inserted: Encl PD 09 > mfi0: 1983 (boot + 24s/0x0002/info) - Inserted: PD 09(c Port 0 - 3/p2) Info: enclPd=09, scsiType=d, portMap=00, sasAddr=50030480008e7b7d,0000000000000000 > mfi0: 1984 (boot + 24s/0x0002/info) - Inserted: PD 0a(e0x08/s2) > > and then lots of disks.... looks fine, right? > > What am I missing? I badly want mfid0-mfid35 back so that I can recreate my ZFS and get to work :-) > > On a side note, the ZFS I'll make is this, any comments on the configuration? > > zpool create tank \ > raidz2 /dev/mfid0 /dev/mfid1 /dev/mfid2 /dev/mfid3 /dev/mfid4 /dev/mfid5 /dev/mfid30 \ > raidz2 /dev/mfid6 /dev/mfid7 /dev/mfid8 /dev/mfid9 /dev/mfid10 /dev/mfid11 /dev/mfid31 \ > raidz2 /dev/mfid12 /dev/mfid13 /dev/mfid14 /dev/mfid15 /dev/mfid16 /dev/mfid17 /dev/mfid32 \ > raidz2 /dev/mfid18 /dev/mfid19 /dev/mfid20 /dev/mfid21 /dev/mfid22 /dev/mfid23 /dev/mfid33 \ > raidz2 /dev/mfid24 /dev/mfid25 /dev/mfid26 /dev/mfid27 /dev/mfid28 /dev/mfid29 /dev/mfid34 \ > spare /dev/mfid35 > > > Cheers > > Nik_______________________________________________ > freebsd-scsi@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-scsi > To unsubscribe, send any mail to "freebsd-scsi-unsubscribe@freebsd.org" -- Stephane LAPIE, EPITA SRS, Promo 2005 "Even when they have digital readouts, I can't understand them." --MegaTokyo