From owner-freebsd-fs@FreeBSD.ORG Sat Jan 23 09:57:07 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5C6001065672 for ; Sat, 23 Jan 2010 09:57:07 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-iw0-f198.google.com (mail-iw0-f198.google.com [209.85.223.198]) by mx1.freebsd.org (Postfix) with ESMTP id 1F8F68FC17 for ; Sat, 23 Jan 2010 09:57:06 +0000 (UTC) Received: by iwn36 with SMTP id 36so1654480iwn.3 for ; Sat, 23 Jan 2010 01:57:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=Dbqdnk9dTYNAOUsRhYoCZa2YhXkKqiH92iNiBy3h+jM=; b=nBGw86FZEObhJp/CWQo8S+g1MGcj08MNbE45wy5SOodWK6isHossimpXTuk2yB14hG JOqsqVMVYS1MzosDmp/PQNm/t+U+lFDQgT0HUawAGVZlJ311oK5w/8cfav8BIJe2zsAB iw1t2i7xFtv8dV97kzD29zNwW3wHlIsH6XiHY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=fqDEcebsePZnOdVBHxJlUL/TKs/FtO3ymPnZ8UH2FZ6fhjEkE1xyHFsCPKr2McqwRG BfKwSKUdOQvZfLsRW3RifLSsTr2nqmNzLwWCOMBTynKM+894HsK/qPqKsWoNsmq7UaM9 F+2mlKJSiV5kfNQwiYJzHW9qcR1iNNMgr1cwg= MIME-Version: 1.0 Sender: artemb@gmail.com Received: by 10.231.143.148 with SMTP id v20mr550918ibu.14.1264240626013; Sat, 23 Jan 2010 01:57:06 -0800 (PST) In-Reply-To: <5da0588e1001230136j52442acfw306dccaa889af7bb@mail.gmail.com> References: <5da0588e1001222223m773648am907267235bdcf882@mail.gmail.com> <5da0588e1001230014k1b8a32f8v42046497265429ed@mail.gmail.com> <5da0588e1001230136j52442acfw306dccaa889af7bb@mail.gmail.com> Date: Sat, 23 Jan 2010 01:57:05 -0800 X-Google-Sender-Auth: e07966d0bd577faa Message-ID: From: Artem Belevich To: Rich Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs Subject: Re: Errors on a file on a zpool: How to remove? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2010 09:57:07 -0000 I guess it's a feature. Without redundancy all ZFS can do is report corruption. If corruption is in the file data, it cen be removed along with the file. If corruption is in directory or in metadata there's little ZFS can do because that particular portion of filesystem is inconsistent. The best that could be done is to keep those portions read-only and try not to crash right away. With 7TB of data with no redundancy of any kind you are asking for trouble even with all hardware functioning within spec. The math is rather scary: http://queue.acm.org/detail.cfm?id=3D1670144 --Artem On Sat, Jan 23, 2010 at 1:36 AM, Rich wrote: > I guess I'm asking for opinions on whether this behavior being > irreparable, to the point of not even being able to nuke the relevant > FS blocks from orbit because they're corrupt without using zdb, is > something that should be reported as a bug or enhancement request. > > I know the files are invalid at this point, and I'm okay with > sacrificing them - I just want the errors to go away. It knows the > checksums on the relevant data are invalid, I'd be okay with just > rewriting the blocks so that everything thinks they're unused blocks > and the files/directories are unlisted... > > ...but short of using zdb and rewriting them myself, I can't, and that > seems ridiculous. > > For a filesystem that insists it never needs fsck, telling someone to > resort to the filesystem's debug layer to remove damaged files is > horrifying. > > - Rich > > On Sat, Jan 23, 2010 at 4:31 AM, Artem Belevich wrote: >>> errors: Permanent errors have been detected in the following files: >>> =A0 =A0 =A0 =A0rigatoni/mirrors:<0x0> >> >> This looks similar to what I had. My wild guess would be that some >> metadata got corrupted. >> >> If you really want to fix it, you may need to get close and personal >> with zdb and =A0ZFS on-disk format spec here: >> http://hub.opensolaris.org/bin/download/Community+Group+zfs/docs/ondiskf= ormat0822.pdf >> >> Following video and PDF may help a bit: >> http://video.google.com/videoplay?docid=3D2325724487196148104# >> http://www.osdevcon.org/2008/files/osdevcon2008-max.pdf >> >> If all you want is to avoid nagging errors you can try going up the >> tree until you can do something with directory. Then move it somewhere >> where it would not get in the way, "chmod 000" it and perhaps even do >> "setflags schg" on it to prevent anyone from descending into directory >> with bad files. >> >> --Artem >> >> >> On Sat, Jan 23, 2010 at 12:14 AM, Rich wrote: >>> I already diagnosed the bad hardware - one of the two sticks of RAM >>> had gone bad, and fails memtest in the other machine. >>> >>> =A0pool: rigatoni >>> =A0state: ONLINE >>> status: One or more devices has experienced an error resulting in data >>> =A0 =A0 =A0 =A0corruption. =A0Applications may be affected. >>> action: Restore the file in question if possible. =A0Otherwise restore = the >>> =A0 =A0 =A0 =A0entire pool from backup. >>> =A0 see: http://www.sun.com/msg/ZFS-8000-8A >>> =A0scrub: scrub completed after 15h28m with 1 errors on Thu Jan 21 18:0= 9:25 2010 >>> config: >>> >>> =A0 =A0 =A0 =A0NAME =A0 =A0 =A0 =A0STATE =A0 =A0 READ WRITE CKSUM >>> =A0 =A0 =A0 =A0rigatoni =A0 =A0ONLINE =A0 =A0 =A0 0 =A0 =A0 0 =A0 =A0 1 >>> =A0 =A0 =A0 =A0 =A0da4 =A0 =A0 =A0 ONLINE =A0 =A0 =A0 0 =A0 =A0 0 =A0 = =A0 2 >>> =A0 =A0 =A0 =A0 =A0da5 =A0 =A0 =A0 ONLINE =A0 =A0 =A0 0 =A0 =A0 0 =A0 = =A0 2 >>> =A0 =A0 =A0 =A0 =A0da7 =A0 =A0 =A0 ONLINE =A0 =A0 =A0 0 =A0 =A0 0 =A0 = =A0 0 >>> =A0 =A0 =A0 =A0 =A0da6 =A0 =A0 =A0 ONLINE =A0 =A0 =A0 0 =A0 =A0 0 =A0 = =A0 0 >>> =A0 =A0 =A0 =A0 =A0da2 =A0 =A0 =A0 ONLINE =A0 =A0 =A0 0 =A0 =A0 0 =A0 = =A0 2 >>> >>> errors: Permanent errors have been detected in the following files: >>> >>> =A0 =A0 =A0 =A0rigatoni/mirrors:<0x0> >>> >>> Scrubbing repeatedly does nothing to remove the note about that error, >>> and I'd rather like to avoid trying to recreate a 7TB pool. >>> >>> - Rich >>> >>> On Sat, Jan 23, 2010 at 3:11 AM, Artem Belevich wrote= : >>>> The directory that those files are in may be corrupted. What does >>>> zpool status -v show? >>>> >>>> You may want to scrub the pool if you haven't done so yet. That would >>>> help to find all corrupted files. >>>> >>>> When plain files are corrupted, you should be able to remove them. You >>>> may also try to set atime=3Doff on the filesystem to avoid filesystem >>>> updates on reads. >>>> Some time back when I had zpool corruption I've found no way to remove >>>> corrupted directory that still had some files in it. In the end I had >>>> to rebuild the pool. >>>> >>>> BTW, given that your pool did get corrupted, perhaps it might be a >>>> good idea to start moving your data somewhere else rather than worry >>>> about how to remove corrupted files. If corruption is due to bad >>>> hardware, bad files would just keep popping up. >>>> >>>> --Artem >>>> >>>> >>>> >>>> On Fri, Jan 22, 2010 at 10:23 PM, Rich wrote: >>>>> Hey world, >>>>> I've got a series of files in a non-redundant zpool which all report >>>>> Input/Output Error on attempting to manipulate them in any way - stat= , >>>>> read, rm, anything. >>>>> >>>>> Whenever anything is attempted, the following style of thing is >>>>> printed to /var/log/messages: >>>>> Jan 23 01:22:34 manticore root: ZFS: checksum mismatch, zpool=3Drigat= oni >>>>> path=3D/dev/da4 offset=3D1231402180608 size=3D8192 >>>>> Jan 23 01:22:34 manticore root: ZFS: checksum mismatch, zpool=3Drigat= oni >>>>> path=3D/dev/da5 offset=3D446136819712 size=3D8192 >>>>> Jan 23 01:22:34 manticore root: ZFS: checksum mismatch, zpool=3Drigat= oni >>>>> path=3D/dev/da2 offset=3D320393101312 size=3D8192 >>>>> Jan 23 01:22:34 manticore root: ZFS: checksum mismatch, zpool=3Drigat= oni >>>>> path=3D/dev/da5 offset=3D446136819712 size=3D8192 >>>>> Jan 23 01:22:34 manticore root: ZFS: checksum mismatch, zpool=3Drigat= oni >>>>> path=3D/dev/da2 offset=3D320393101312 size=3D8192 >>>>> Jan 23 01:22:34 manticore root: ZFS: checksum mismatch, zpool=3Drigat= oni >>>>> path=3D/dev/da4 offset=3D1231402180608 size=3D8192 >>>>> Jan 23 01:22:35 manticore root: ZFS: zpool I/O failure, zpool=3Drigat= oni error=3D86 >>>>> >>>>> What can I do? I really would like to just purge all of these files >>>>> from orbit, since I can recreate them, but I can't seem to delete >>>>> them, and deleting the pool is a really inconvenient option, as I hav= e >>>>> other data on it. >>>>> >>>>> I'm running 8.0-RELEASE stock on amd64. >>>>> >>>>> Thanks! >>>>> >>>>> - Rich >>>>> _______________________________________________ >>>>> freebsd-fs@freebsd.org mailing list >>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>>>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>>>> >>>> >>> >>> >>> >>> -- >>> >>> Todo homem morre, mas nem todo homem vive. -- William Wallace >>> >> > > > > -- > > Yow! Am I in Milwaukee? >