From owner-freebsd-scsi@FreeBSD.ORG Thu Sep 23 12:55:07 2010 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBAF11065695 for ; Thu, 23 Sep 2010 12:55:07 +0000 (UTC) (envelope-from niklas@saers.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 73D718FC1C for ; Thu, 23 Sep 2010 12:55:07 +0000 (UTC) Received: by ewy22 with SMTP id 22so509758ewy.13 for ; Thu, 23 Sep 2010 05:55:06 -0700 (PDT) Received: by 10.213.22.200 with SMTP id o8mr1859451ebb.62.1285246506362; Thu, 23 Sep 2010 05:55:06 -0700 (PDT) Received: from [10.32.100.120] (webmail.danskscanning.dk [89.184.151.254]) by mx.google.com with ESMTPS id u9sm1167619eeh.17.2010.09.23.05.55.04 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 23 Sep 2010 05:55:04 -0700 (PDT) From: Niklas Saers Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Thu, 23 Sep 2010 14:55:02 +0200 Message-Id: <2EA9CBBC-3F97-4AF2-BFB5-96DF39FDE376@saers.com> To: freebsd-scsi@freebsd.org Mime-Version: 1.0 (Apple Message framework v1081) X-Mailer: Apple Mail (2.1081) Subject: mfi - setting up disks X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Sep 2010 12:55:08 -0000 Hi guys, In the SuperMicro system where I had problems with the mpt controller, I = switched it for a mfi-based controller. I had it set up with 36x RAID0 = volumes with each their own disk (no way to access the disk otherwise I = found), and added them to a ZFS system. The numbering became a bit = weird, so I pulled the disks out one by one and put them back to figure = out and note down what disk number was in what slot. Only test data on = my ZFS volume, so I didn't mind that crashing. Now that all disks have been taken out and put back in one by one, I do: # mfiutil show volumes mfi0 Volumes: Id Size Level Stripe State Cache Name Whoops, no volumes? That can't be good. I check up on the disks, they're = all there: mfiutil show drives mfi0 Physical Drives: ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 2 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 3 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 4 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 14 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 15 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 17 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 23 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 1 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 4 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 5 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 6 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 7 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 8 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 10 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 11 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 0 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 5 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 6 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 7 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 8 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 9 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 10 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 11 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 12 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 13 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 16 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 19 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 21 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 22 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 2 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 3 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 9 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 1 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 20 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 2, slot 0 ( 1863G) UNCONFIGURED GOOD = SATA enclosure 1, slot 18 Well, that's an excelent way of adding the volumes in the sequence they = appear physically, so I start with the first one from top left: # mfiutil create raid0 -v E01:S05 Adding drive 26 to array 0 Adding array 0 to volume 0 mfiutil: Command failed: Wrong firmware or drive state mfiutil: Failed to add volume: Input/output error Firmware error? Invalid drive state? # mfiutil good E01:S05 mfiutil: Drive 26 is already in the desired state Seems to be good... do I have a firmware issue? Check dmesg: mfi0: port 0xc000-0xc0ff mem = 0xfad7c000-0xfad7ffff,0xfadc0000-0xfadfffff irq 16 at device 0.0 on pci5 mfi0: Megaraid SAS driver Ver 3.00=20 mfi0: 1966 (338565595s/0x0020/info) - Shutdown command received from = host mfi0: 1967 (boot + 3s/0x0020/info) - Firmware initialization started = (PCI ID 0079/1000/9261/1000) mfi0: 1968 (boot + 3s/0x0020/info) - Firmware version 2.0.03-0673 mfi0: 1969 (boot + 4s/0x0020/info) - Board Revision=20 mfi0: 1970 (boot + 24s/0x0004/info) - Enclosure (SES) discovered on PD = 08(c Port 0 - 3/p1) mfi0: 1971 (boot + 24s/0x0004/info) - Enclosure (SES) discovered on PD = 09(c Port 0 - 3/p2) mfi0: 1972 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) = communication restored mfi0: 1973 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) = fan 1 speed changed mfi0: 1974 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) = fan 2 speed changed mfi0: 1975 (boot + 24s/0x0004/info) - Enclosure PD 08(c Port 0 - 3/p1) = fan 3 speed changed mfi0: 1976 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) = communication restored mfi0: 1977 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) = fan 1 speed changed mfi0: 1978 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) = fan 2 speed changed mfi0: 1979 (boot + 24s/0x0004/info) - Enclosure PD 09(c Port 0 - 3/p2) = fan 3 speed changed mfi0: 1980 (boot + 24s/0x0002/info) - Inserted: Encl PD 08 mfi0: 1981 (boot + 24s/0x0002/info) - Inserted: PD 08(c Port 0 - 3/p1) = Info: enclPd=3D08, scsiType=3Dd, portMap=3D00, = sasAddr=3D50030480008fb0fd,0000000000000000 mfi0: 1982 (boot + 24s/0x0002/info) - Inserted: Encl PD 09 mfi0: 1983 (boot + 24s/0x0002/info) - Inserted: PD 09(c Port 0 - 3/p2) = Info: enclPd=3D09, scsiType=3Dd, portMap=3D00, = sasAddr=3D50030480008e7b7d,0000000000000000 mfi0: 1984 (boot + 24s/0x0002/info) - Inserted: PD 0a(e0x08/s2) and then lots of disks.... looks fine, right? What am I missing? I badly want mfid0-mfid35 back so that I can recreate = my ZFS and get to work :-) On a side note, the ZFS I'll make is this, any comments on the = configuration? zpool create tank \ raidz2 /dev/mfid0 /dev/mfid1 /dev/mfid2 /dev/mfid3 /dev/mfid4 /dev/mfid5 = /dev/mfid30 \ raidz2 /dev/mfid6 /dev/mfid7 /dev/mfid8 /dev/mfid9 /dev/mfid10 = /dev/mfid11 /dev/mfid31 \ raidz2 /dev/mfid12 /dev/mfid13 /dev/mfid14 /dev/mfid15 /dev/mfid16 = /dev/mfid17 /dev/mfid32 \ raidz2 /dev/mfid18 /dev/mfid19 /dev/mfid20 /dev/mfid21 /dev/mfid22 = /dev/mfid23 /dev/mfid33 \ raidz2 /dev/mfid24 /dev/mfid25 /dev/mfid26 /dev/mfid27 /dev/mfid28 = /dev/mfid29 /dev/mfid34 \ spare /dev/mfid35 Cheers Nik=