From owner-freebsd-current@FreeBSD.ORG Sun May 24 19:34:00 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 54260106566B for ; Sun, 24 May 2009 19:34:00 +0000 (UTC) (envelope-from serenity@exscape.org) Received: from ch-smtp01.sth.basefarm.net (ch-smtp01.sth.basefarm.net [80.76.149.212]) by mx1.freebsd.org (Postfix) with ESMTP id D6E118FC17 for ; Sun, 24 May 2009 19:33:59 +0000 (UTC) (envelope-from serenity@exscape.org) Received: from c83-253-252-234.bredband.comhem.se ([83.253.252.234]:35347 helo=mx.exscape.org) by ch-smtp01.sth.basefarm.net with esmtp (Exim 4.69) (envelope-from ) id 1M8JSF-0002eW-4j for freebsd-current@freebsd.org; Sun, 24 May 2009 21:33:57 +0200 Received: from [192.168.1.5] (macbookpro [192.168.1.5]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx.exscape.org (Postfix) with ESMTPSA id A4B8B38955 for ; Sun, 24 May 2009 21:33:53 +0200 (CEST) Message-Id: <4FE794E9-075D-4563-B395-BD5E459937DF@exscape.org> From: Thomas Backman To: freebsd-current@freebsd.org In-Reply-To: <4E6E325D-BB18-4478-BCFD-633D6F4CFD88@exscape.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v935.3) Date: Sun, 24 May 2009 21:33:53 +0200 References: <4E6E325D-BB18-4478-BCFD-633D6F4CFD88@exscape.org> X-Mailer: Apple Mail (2.935.3) X-Originating-IP: 83.253.252.234 X-Scan-Result: No virus found in message 1M8JSF-0002eW-4j. X-Scan-Signature: ch-smtp01.sth.basefarm.net 1M8JSF-0002eW-4j a5125941f92b4cbe1d363a50fea869e4 Subject: Re: ZFS panic under extreme circumstances (2/3 disks corrupted) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 May 2009 19:34:00 -0000 On May 24, 2009, at 09:02 PM, Thomas Backman wrote: > So, I was playing around with RAID-Z and self-healing, when I > decided to take it another step and corrupt the data on *two* disks > (well, files via ggate) and see what happened. I obviously expected > the pool to go offline, but I didn't expect a kernel panic to follow! > > What I did was something resembling: > 1) create three 100MB files, ggatel create to create GEOM providers > from them > 2) zpool create test raidz ggate{1..3} > 3) create a 100MB file inside the pool, md5 the file > 4) overwrite 10~20MB (IIRC) of disk2 with /dev/random, with dd if=/ > dev/random of=./disk2 bs=1000k count=20 skip=40, or so (I now know > that I wanted *seek*, not *skip*, but it still shouldn't panic!) > 5) Check if the md5 of file: everything OK, zpool status shows a > degraded pool. > 6) Repeat step #4, but with disk 3. > 7) zpool scrub test > 8) Panic! > [...] FWIW, I couldn't replicate this when using seek (i.e. corrupt the middle of the "disk" rather than the beginning): [root@clone ~/zfscrash]# zpool status test pool: test state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: scrub in progress for 0h0m, 7.72% done, 0h6m to go config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 18 raidz1 ONLINE 0 0 161 ggate0 ONLINE 0 0 0 512 repaired ## note that I did *not* touch this "disk" at all, so why "512 repaired"? ggate1 ONLINE 0 0 702 73K repaired ggate2 ONLINE 0 0 62 64.5K repaired errors: 9 data errors, use '-v' for a list After overwriting the *beginning* of disk2 and disk3 as well, "zpool scrub" appears to hang. Two vdev failures on the console, and zpool status hangs as well. No panic this time around (I've waited 5 minutes and nothing appears to happen, but the computer is usable on other ttys). The failmode property was set to the default, i.e. wait, in both cases. Regards, Thomas