From owner-freebsd-fs@FreeBSD.ORG Tue Apr 24 14:30:21 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3C4E31065674 for ; Tue, 24 Apr 2012 14:30:21 +0000 (UTC) (envelope-from areilly@bigpond.net.au) Received: from nschwmtas05p.mx.bigpond.com (nschwmtas05p.mx.bigpond.com [61.9.189.149]) by mx1.freebsd.org (Postfix) with ESMTP id CE9958FC1D for ; Tue, 24 Apr 2012 14:30:20 +0000 (UTC) Received: from nschwcmgw06p ([61.9.190.166]) by nschwmtas05p.mx.bigpond.com with ESMTP id <20120424143014.DJWE2821.nschwmtas05p.mx.bigpond.com@nschwcmgw06p> for ; Tue, 24 Apr 2012 14:30:14 +0000 Received: from johnny.reilly.home ([124.188.161.100]) by nschwcmgw06p with BigPond Outbound id 1qWE1j0012AGJ5o01qWECU; Tue, 24 Apr 2012 14:30:14 +0000 X-Authority-Analysis: v=2.0 cv=IpCcgcDg c=1 sm=1 a=+rWFdGQzZE3xDYVtG1Y/Og==:17 a=z1TLwsU0kBEA:10 a=2A4K5xx_JT0A:10 a=kj9zAlcOel0A:10 a=cexIBkohAAAA:8 a=UmH_dSZQHSJ3RcGlep8A:9 a=4-aTtk1I5wHsH6S-Ba4A:7 a=CjuIK1q_8ugA:10 a=+rWFdGQzZE3xDYVtG1Y/Og==:117 Date: Wed, 25 Apr 2012 00:30:14 +1000 From: Andrew Reilly To: freebsd-fs@freebsd.org Message-ID: <20120424143014.GA2865@johnny.reilly.home> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Subject: Odd file system corruption in ZFS pool X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2012 14:30:21 -0000 Hi there, Had a power outage followed by an electrician visit, and the power supply to my server room yoyo'd a few times before I pulled the plugs. When I powered up again this afternoon I had ZFS pool corruption, according to zpool status. Several zpool scrub/reboot cycles later, I have the odd situation of a directory entry that shows up in glob expansion, but which claims not to be there when ls does a stat, or when I try to move or delete it. Any suggestions? Ls shows: $ ls ls: .Suppliers.2010: No such file or directory (Yes, this is in a Maildir managed jointly by qmail and dovecot.) There are also a bunch of message files in a different directory that find reports "unknown error 112" for, which doesn't look good. Zpool status -v tank says: pool: tank state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scan: scrub in progress since Wed Apr 25 00:12:48 2012 50.9G scanned out of 1.03T at 106M/s, 2h42m to go 0 repaired, 4.81% done config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 853 raidz1-0 ONLINE 0 0 3.33K gptid/b06b6337-e511-11e0-9d62-00270e0fb8e9 ONLINE 0 0 0 gptid/b6c7d5b0-e511-11e0-9d62-00270e0fb8e9 ONLINE 0 0 0 gptid/bbf6c485-e511-11e0-9d62-00270e0fb8e9 ONLINE 0 0 0 gptid/bf86a966-e511-11e0-9d62-00270e0fb8e9 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: tank/home:<0x0> Every time I run zpool scrub and reboot the CKSUM column goes to zero, and the errors go away and the status says that the previous scrub found no errors. Every time I look at the missing directory the CKSUM column gets larger... I do have a backup (I hope): on another zfs filesystem on a usb drive, constructed by sending and receiving snapshots in an incremental fashion. What I don't know is whether I actually need to use it or not. Everything seems to be working fine apart from this mysteriously untouchable Maildir directory and a few mail files, all of which I could recover from the backup if only I could remove the original. Should I just zfs destroy tank/home then create it again and send the last backup snapshots back? Would that remove the corruption? Oh, I'm running 9-stable, and my Zpool and zfs versions are : FreeBSD johnny.reilly.home 9.0-STABLE FreeBSD 9.0-STABLE #15: Sun Apr 22 11:37:17 EST 2012 root@johnny.reilly.home:/usr/obj/usr/src/sys/GENERIC amd64 ZFS filesystem version 5 ZFS storage pool version 28 Cheers, -- Andrew