From owner-freebsd-current@FreeBSD.ORG Tue May 26 09:30:54 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E60AB106566B for ; Tue, 26 May 2009 09:30:54 +0000 (UTC) (envelope-from hk@alogis.com) Received: from alogis.com (firewall.solit-ag.de [212.184.102.1]) by mx1.freebsd.org (Postfix) with ESMTP id 3827D8FC0C for ; Tue, 26 May 2009 09:30:53 +0000 (UTC) (envelope-from hk@alogis.com) Received: from alogis.com (localhost [127.0.0.1]) by alogis.com (8.13.4/8.13.1) with ESMTP id n4Q9Hw0S009749; Tue, 26 May 2009 11:17:58 +0200 (CEST) (envelope-from hk@alogis.com) Received: (from hk@localhost) by alogis.com (8.13.4/8.13.1/Submit) id n4Q9HvY0009748; Tue, 26 May 2009 11:17:57 +0200 (CEST) (envelope-from hk) Date: Tue, 26 May 2009 11:17:57 +0200 From: Holger Kipp To: Freddie Cash Message-ID: <20090526091757.GA9405@intserv.int1.b.intern> References: <4E6E325D-BB18-4478-BCFD-633D6F4CFD88@exscape.org> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.4.2.1i X-Mailman-Approved-At: Tue, 26 May 2009 11:20:55 +0000 Cc: freebsd-current@freebsd.org Subject: Re: ZFS panic under extreme circumstances (2/3 disks corrupted) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 May 2009 09:30:55 -0000 On Mon, May 25, 2009 at 09:19:21AM -0700, Freddie Cash wrote: > On Mon, May 25, 2009 at 9:12 AM, Thomas Backman wrote: > > On May 25, 2009, at 05:39 PM, Freddie Cash wrote: > >> On Mon, May 25, 2009 at 2:13 AM, Thomas Backman > >> wrote: > >>> On May 24, 2009, at 09:02 PM, Thomas Backman wrote: > >>> > >>>> So, I was playing around with RAID-Z and self-healing... > >>> > >> On our storage server that was initially configured using 1 large > >> 24-drive raidz2 vdev (don't do that, by the way), we had 1 drive go > >> south.  "zpool status" was full of errors.  And the error counts > >> survived reboots.  Either that, or the drive was so bad that the error > >> counts started increasing right away after a boot.  After a week of > >> fighting with it to get the new drive to resilver and get added to the > >> vdev, we nuked it and re-created it using 3 raidz2 vdevs each > >> comprised of 8 drives. > >> > >> (Un)fortunately, that was the only failure we've had so far, so can't > >> really confirm/deny the "error counts reset after reboot". > > > > Was this on FreeBSD? > > 64-bit FreeBSD 7.1 using ZFS v6. SATA drives connected to 3Ware RAID > controllers, but configured as "Single Drive" arrays not using > hardware RAID in any way. Not sure if this is related, but we have a 16-disk raid with fibrechannel and have the disks configured as single-disk raid0 (it seems explicit jbod without at least a logical raid0 container for the controller is not possible on most or even all of these raid systems) on the controller. ZFS is of course raidz2. We made a check to just remove one disk while system is up, reinserting the disk after some time - boom. We couldn't get it to resilver. Reason is quite simple. The raid system recognized the disk and because it is a raid0 it can't be repaired by the underlying raid system, so status sent to computer was always of type 'drive broken'. Inserting a new disk would not help either because the raid system could also not repair the raid0 with a new disk. Obvious if you think about it.. What we had to do was to remove the raid0-container from raid configuration and create a new one with the same name on the raid device. As soon as the controller was happy, thinking the disk was ok, ZFS was able to access the drive again, starting resilver without problems (that was with ZFS version 6). Regards, Holger