From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 15 14:56:48 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3AB7C1065673 for ; Thu, 15 Dec 2011 14:56:48 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 0E6918FC18 for ; Thu, 15 Dec 2011 14:56:48 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [96.47.65.170]) by cyrus.watson.org (Postfix) with ESMTPSA id B918B46B3B; Thu, 15 Dec 2011 09:56:47 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 1D61AB95C; Thu, 15 Dec 2011 09:56:47 -0500 (EST) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Thu, 15 Dec 2011 09:56:45 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p8; KDE/4.5.5; amd64; ; ) References: <4EE8A005.5030607@barafranca.com> <9317551F-CBE0-4368-B798-498E58E240B2@sarenet.es> <2EA3FFF4-E6A2-4371-8891-26E99C551C67@transactionware.com> In-Reply-To: <2EA3FFF4-E6A2-4371-8891-26E99C551C67@transactionware.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201112150956.45214.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 15 Dec 2011 09:56:47 -0500 (EST) Cc: Borja Marcos , Jan Mikkelsen , Hugo Silva Subject: Re: mfi (Dell H700) + hot swapping doesn't appear to work with RC1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2011 14:56:48 -0000 On Thursday, December 15, 2011 4:19:58 am Jan Mikkelsen wrote: > On 15/12/2011, at 2:16 AM, Borja Marcos wrote: > > > > > On Dec 14, 2011, at 2:09 PM, Hugo Silva wrote: > > > >> Hello, > >> > >> First of all apologies if this has been fixed in RC3. I set this server > >> up with mfsbsd, which is RC1, and didn't get to update the system yet. > >> > >> This box has 6 hdds, a 2-mirror zpool was set up as the root pool, with > >> 2 spares. > >> > >> While testing hot swapping I noticed that while the controller detects > >> disk removal/insertion, the zpool will never recover. The problem seems > >> to be deeper than ZFS, as disklabel/fdisk/etc also fail on the > >> removed-and-reinserted disk. > >> > >> At the ZFS level, doing a zpool clear yields more errors on the removed > >> disk; rebooting becomes the only option to make the pool healthy again. > >> > >> > >> Is this normal? Did I miss any step? > > > > I assume that you have tried to use the H700 as a "JBOD" card, defining logical volume for each hard disk. > > > > The problem is: that gorgeous, fantastic, masterful, Nobel award candidate card, has a wonderful behavior in that case. If you extract one of the disks, the logical volume associated to it is invalidated. So, you insert a replacement disk, and the card refuses to recognize the volume. What is even worse, in order to recover it's mandatory to reboot the complete system *AND* go through the RAID configuration utility. > > > > That's the problem. The card refuses to work as a simple disk controller without frills, and the frills get in the way. > > > > To summarize: it isn't FreeBSD's fault, no matter which version you use. It's a "feature" coming directly from the geniuses who designed the card. > > (Sending again to avoid moderation.) > > Hugo: You missed a step. Borja: No reboot required. > > For the mfi controllers I have been testing recently (MegaRAID 9261-8i), you need to install the sysutils/megacli port, and use that to clear the "foreignness" of the disk you just added. Something like: > > MegaCli -CfgForeign -Clear -a0 > > You should be able to then recreate it as a JBOD device, and progress through whatever higher level recovery you need to do. Can you do this by marking it as 'good' via mfiutil and then using mfiutil to create a volume? -- John Baldwin