From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 12 18:22:37 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 869371065674 for ; Fri, 12 Sep 2008 18:22:37 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from QMTA04.emeryville.ca.mail.comcast.net (qmta04.emeryville.ca.mail.comcast.net [76.96.30.40]) by mx1.freebsd.org (Postfix) with ESMTP id 69E728FC18 for ; Fri, 12 Sep 2008 18:22:37 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from OMTA01.emeryville.ca.mail.comcast.net ([76.96.30.11]) by QMTA04.emeryville.ca.mail.comcast.net with comcast id DtWY1a0040EPchoA4uNdbN; Fri, 12 Sep 2008 18:22:37 +0000 Received: from koitsu.dyndns.org ([67.180.253.227]) by OMTA01.emeryville.ca.mail.comcast.net with comcast id DuNb1a00Y4v8bD78MuNc6n; Fri, 12 Sep 2008 18:22:36 +0000 X-Authority-Analysis: v=1.0 c=1 a=FEcCtSrf6_wA:10 a=SSZ9KyxJ8eYA:10 a=QycZ5dHgAAAA:8 a=D_HWRfpMJYFIkFXSiXsA:9 a=R9ANlix2jvog273o3ewA:7 a=fxdJgh-zXqg-5lFS96MCL21dCW8A:4 a=EoioJ0NPDVgA:10 a=SV7veod9ZcQA:10 a=LY0hPdMaydYA:10 Received: by icarus.home.lan (Postfix, from userid 1000) id C0B1917B81A; Fri, 12 Sep 2008 11:22:35 -0700 (PDT) Date: Fri, 12 Sep 2008 11:22:35 -0700 From: Jeremy Chadwick To: Freddie Cash Message-ID: <20080912182235.GA62771@icarus.home.lan> References: <5f67a8c40809120904o49b6e410l5b65a20f5216202@mail.gmail.com> <20080912163207.GE60094@icarus.home.lan> <200809121012.10195.fjwcash@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200809121012.10195.fjwcash@gmail.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-hackers@freebsd.org Subject: Re: ZFS w/failing drives - any equivalent of Solaris FMA? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Sep 2008 18:22:37 -0000 On Fri, Sep 12, 2008 at 10:12:09AM -0700, Freddie Cash wrote: > On September 12, 2008 09:32 am Jeremy Chadwick wrote: > > For home use, sure. Since most home/consumer systems do not include > > hot-swappable drive bays, rebooting is required. Although more and > > more consumer motherboards are offering AHCI -- which is the only > > reliable way you'll get that capability with SATA. > > > > In my case with servers in a co-lo, it's not acceptable. Our systems > > contain SATA backplanes that support hot-swapping, and it works how it > > should (yank the disk, replace with a new one) on Linux -- there is no > > need to do a bunch of hoopla like on FreeBSD. On FreeBSD, with that > > hoopla, also take the risk of inducing a kernel panic. That risk does > > not sit well with me, but thankfully I've only been in that situation > > (replacing a bad disk + using hot-swapping) once -- and it did work. > > Hrm, is this with software RAID or hardware RAID? I do not use either, but have tried software RAID (Intel MatrixRAID) in the past (and major, MAJOR bugs are why I do not any longer). Speaking (mostly) strictly of FreeBSD, let me list off the problems with both: Software RAID: 1) Buggy as hell. Using Intel MatrixRAID as an example, even with RAID 1, due to ata(4) driver bugs, you are practically guaranteed to lose your data, 3) Limited userland interface to RAID BIOS; many operations do not work with atacontrol, requiring a system reboot + entering BIOS to do things like add/remove disks or rebuild an array 3) SMART monitoring lost; if the card or BIOS supports passthrough (basically ATA version of pass(4)), FreeBSD will see the disks natively (e.g. arX for the RAID, ad4 and ad8 for the disks), and you can use smartmontools. Otherwise, you're screwed 4) Support is questionable; numerous mainstream chips unsupported, including Adaptec HostRAID Hardware RAID: 1) You are "locked in" to that controller. Your data is at the mercy of the company who makes the HBA; if your controller dies and is no longer made, your data is dead in the water. Chances are a newer model/revision of controller will not understand the the disk metadata from the previous controller 2) Performance problems as a result of excessive caching levels; onboard hardware cache vs. system memory cache vs. disk layer cache in OS vs. other kernel caching mechanisms 3) Controller firmware upgrades are risky -- 3Ware has a very nasty history of this, for sake of example. I've heard of some upgrades changing the metadata format, requiring complete array re-creation I can pull Ade Lovett into this conversation if you think any of the above is exaggerated. :-) The only hardware RAID controller I'd trust at this point would be Areca -- but hardware RAID is not what I want. On the other hand, I really want Areca to make a standard 4 or 8-port SATA controller -- no RAID, but full driver support under arcmsr(4) (which uses CAM and da(4)). This would be perfect. > With our hardware RAID systems, the process has always been the same, > regardless of which OS (Windows 2003 Servers, Debian Linux, FreeBSD) is > on the system: > - go into RAID management GUI, remove drive > - pull dead drive from system > - insert new drive into system > - go into RAID management GUI, make sure it picked up new drive and > started the rebuild The simplicity there is correct -- that's really how simple it should be. But a GUI? What card is this that requires a GUI? Does it require a reboot? No command-line support? > We've been lucky so far, and not had to do any drive replacements on our > non-ZFS software RAID systems (md on Debian, gmirror on FreeBSD). I'm > not looking forward to a drive failing, as these systems have > non-hot-pluggable SATA setups. I'm hearing you loud and clear. :-) > On the ZFS systems, we just "zpool offline" the drive, physically replace > the drive, and "zpool replace" the drive. On one system, this was done > via hot-pluggable SATA backplane, on another, it required a reboot. If this was done on the hardware RAID controller (presuming it uses CAM and da(4)), I'm not surprised it worked perfectly. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |