From owner-freebsd-fs@FreeBSD.ORG Mon Feb 1 04:49:34 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CDC381065670 for ; Mon, 1 Feb 2010 04:49:34 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-fx0-f227.google.com (mail-fx0-f227.google.com [209.85.220.227]) by mx1.freebsd.org (Postfix) with ESMTP id 4C35C8FC15 for ; Mon, 1 Feb 2010 04:49:33 +0000 (UTC) Received: by fxm27 with SMTP id 27so114689fxm.3 for ; Sun, 31 Jan 2010 20:49:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=22A5P1934M+tofEks8jClyYYoP2yFAFjfhNWbQlGmd0=; b=JoiGrWBKKEWwCWwjT7xFVX7LKt/0RbAfgkBBPmdFu3fi/Meha1v8I1/yheNFglKwI/ +sJzcIKHafjGeoEaWWiZ2kCPNcDY9KH8a6mu8YJFc67QUt08ir/shpMo3ZN86onsK4ej Waduf3WcGVkWkuRvWNccXnTfwk4RCHC+9Mjus= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=OtL4Lp2lIYfBkBZZ4Su7vvRbmKUkYiPzQwyO4zUl8YZLO7rpWlw2bBV9zVYWD2I+Z6 Q+NoWTkCvJlT0mwAH4zx3LSMHWfhU0erTZ22Vm0IqhqSweyt+Lie3gvV2rdlzWdWNf6N NmEwM/itE53mcxUB8tT0IVhhN46Uab8//F4Zg= MIME-Version: 1.0 Received: by 10.239.189.7 with SMTP id r7mr624607hbh.116.1264999770438; Sun, 31 Jan 2010 20:49:30 -0800 (PST) In-Reply-To: <5da0588e1001240544q61e3bebbka7ad1248343be26d@mail.gmail.com> References: <5da0588e1001222223m773648am907267235bdcf882@mail.gmail.com> <4B5B94B8.7070509@modulus.org> <5da0588e1001231638i349f8f17t297e970b08825441@mail.gmail.com> <5da0588e1001232017m6c67731fwaa1d71cd86800017@mail.gmail.com> <5da0588e1001232128w5a551674od0805c2ff0b884ad@mail.gmail.com> <5da0588e1001240544q61e3bebbka7ad1248343be26d@mail.gmail.com> Date: Sun, 31 Jan 2010 23:49:30 -0500 Message-ID: <5da0588e1001312049p36b5facam478052cc6aeb4f1d@mail.gmail.com> From: Rich To: Wes Morgan Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-fs@freebsd.org Subject: Re: Errors on a file on a zpool: How to remove? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Feb 2010 04:49:35 -0000 Well, that's frustrating. I moved all the data off of that FS on the pool, onto a newly-created FS on the pool. It took a few days. I just attempted to zfs destroy the old filesystem. The command hung for 30 minutes, at which point I decided that this was likely not doing anything useful, especially since operations to all but the latest-created filesystem on the pool now hung, as well as zfs list. I got this fun thing out of /var/log/messages, too: Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni path=/dev/da5 offset=446136819712 size=8192 Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni path=/dev/da2 offset=320393101312 size=8192 Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni path=/dev/da4 offset=1231402180608 size=8192 Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni path=/dev/da5 offset=446136819712 size=8192 Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni path=/dev/da2 offset=320393101312 size=8192 Jan 31 23:12:05 manticore root: ZFS: checksum mismatch, zpool=rigatoni path=/dev/da4 offset=1231402180608 size=8192 Jan 31 23:12:05 manticore root: ZFS: zpool I/O failure, zpool=rigatoni error=86 Jan 31 23:12:05 manticore root: ZFS: vdev I/O failure, zpool=rigatoni path= offset= size= error= The last one, of course, being fascinating. Okay, how annoying. I try sysctl debug.kdb.panic=1, and it hangs. Uh-oh. I reboot physically, and the system hangs on "Mounting local filesystems..." for a long time. I power the machine off, unplug all 5 devices in that pool, and boot the machine again. It boots rapidly and fine. I export the pool (now reporting that all 5 devices were missing, unsurprisingly), power the machine off, plug the 5 devices back in, and power on. Machine boots. zpool import rigatoni succeeds. zfs list now hangs. df output shows no filesystems from that pool, and /var/log/messages got another copy of what I just pasted above. Thoughts? - Rich