From owner-freebsd-fs@FreeBSD.ORG Mon Feb 1 12:50:38 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DAF8106566B for ; Mon, 1 Feb 2010 12:50:38 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id D6C928FC0A for ; Mon, 1 Feb 2010 12:50:37 +0000 (UTC) Received: from cicely5.cicely.de ([10.1.1.7]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id o11CoU69027201 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 1 Feb 2010 13:50:30 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (cicely7.cicely.de [10.1.1.9]) by cicely5.cicely.de (8.14.2/8.14.2) with ESMTP id o11CoNvF032797 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 1 Feb 2010 13:50:23 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.14.2/8.14.2) with ESMTP id o11CoNWW031515; Mon, 1 Feb 2010 13:50:23 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.14.2/8.14.2/Submit) id o11CoLQP031514; Mon, 1 Feb 2010 13:50:21 +0100 (CET) (envelope-from ticso) Date: Mon, 1 Feb 2010 13:50:21 +0100 From: Bernd Walter To: Rich Message-ID: <20100201125020.GU21848@cicely7.cicely.de> References: <4B5B94B8.7070509@modulus.org> <5da0588e1001231638i349f8f17t297e970b08825441@mail.gmail.com> <5da0588e1001232017m6c67731fwaa1d71cd86800017@mail.gmail.com> <5da0588e1001232128w5a551674od0805c2ff0b884ad@mail.gmail.com> <5da0588e1001240544q61e3bebbka7ad1248343be26d@mail.gmail.com> <5da0588e1001312049p36b5facam478052cc6aeb4f1d@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5da0588e1001312049p36b5facam478052cc6aeb4f1d@mail.gmail.com> X-Operating-System: FreeBSD cicely7.cicely.de 7.0-STABLE i386 User-Agent: Mutt/1.5.11 X-Spam-Status: No, score=-4.4 required=5.0 tests=ALL_TRUSTED=-1.8, AWL=0.020, BAYES_00=-2.599 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on spamd.cicely.de Cc: freebsd-fs@freebsd.org Subject: Re: Errors on a file on a zpool: How to remove? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Feb 2010 12:50:38 -0000 On Sun, Jan 31, 2010 at 11:49:30PM -0500, Rich wrote: > Well, that's frustrating. > > I moved all the data off of that FS on the pool, onto a newly-created > FS on the pool. It took a few days. > > I just attempted to zfs destroy the old filesystem. > > The command hung for 30 minutes, at which point I decided that this > was likely not doing anything useful, especially since operations to > all but the latest-created filesystem on the pool now hung, as well as > zfs list. > > I got this fun thing out of /var/log/messages, too: > Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni > path=/dev/da5 offset=446136819712 size=8192 > Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni > path=/dev/da2 offset=320393101312 size=8192 > Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni > path=/dev/da4 offset=1231402180608 size=8192 > Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni > path=/dev/da5 offset=446136819712 size=8192 > Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni > path=/dev/da2 offset=320393101312 size=8192 > Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni > path=/dev/da4 offset=1231402180608 size=8192 > Jan 31 23:12:05 manticore root: ZFS: zpool I/O failure, zpool=rigatoni error=86 > Jan 31 23:12:05 manticore root: ZFS: vdev I/O failure, zpool=rigatoni > path= offset= size= error= > > The last one, of course, being fascinating. > > Okay, how annoying. I try sysctl debug.kdb.panic=1, and it hangs. Uh-oh. > > I reboot physically, and the system hangs on "Mounting local > filesystems..." for a long time. > > I power the machine off, unplug all 5 devices in that pool, and boot > the machine again. It boots rapidly and fine. > > I export the pool (now reporting that all 5 devices were missing, > unsurprisingly), power the machine off, plug the 5 devices back in, > and power on. > > Machine boots. zpool import rigatoni succeeds. zfs list now hangs. df > output shows no filesystems from that pool, and /var/log/messages got > another copy of what I just pasted above. I have had a strange corruption with an older FreeBSD version after a panic one day. On boot it excercised the harddisks for more than 20 hours before I finally gave up. The "solution" was to import the pool and manually mount the unbroken volumes. Disk excercise startet only when mounting the broken FS. It was possible to clone the most recent snapshot of the broken filesystem and use this, which was good because it was a few hours more recent than the last backup. I never deleted the broken FS, because the machine was already in transition to be replaced. At least you might be able to read the correct filesystems by hand mounting and have the pool available, which might allow trying a delete again. But I'm not sure if zfs import allows importing without mounting. -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.