From owner-freebsd-fs@FreeBSD.ORG Mon Feb 9 15:39:16 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EF56050E; Mon, 9 Feb 2015 15:39:16 +0000 (UTC) Received: from hades.sorbs.net (hades.sorbs.net [67.231.146.201]) by mx1.freebsd.org (Postfix) with ESMTP id D8B0716A; Mon, 9 Feb 2015 15:39:16 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from isux.com (firewall.isux.com [213.165.190.213]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0NJI00BBEH1LAS00@hades.sorbs.net>; Mon, 09 Feb 2015 07:44:10 -0800 (PST) Message-id: <54D8D4A1.9090106@sorbs.net> Date: Mon, 09 Feb 2015 16:39:13 +0100 From: Michelle Sullivan User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.24) Gecko/20100301 SeaMonkey/1.1.19 To: Stefan Esser Subject: Re: ZFS pool faulted (corrupt metadata) but the disk data appears ok... References: <54D3E9F6.20702@sorbs.net> <54D41608.50306@delphij.net> <54D41AAA.6070303@sorbs.net> <54D41C52.1020003@delphij.net> <54D424F0.9080301@sorbs.net> <54D47F94.9020404@freebsd.org> <54D4A552.7050502@sorbs.net> <54D4BB5A.30409@freebsd.org> <54D8B3D8.6000804@sorbs.net> <54D8CECE.60909@freebsd.org> In-reply-to: <54D8CECE.60909@freebsd.org> Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Feb 2015 15:39:17 -0000 Stefan Esser wrote: > Am 09.02.2015 um 14:19 schrieb Michelle Sullivan: > >> Stefan Esser wrote: >> >>> The point were zdb seg faults hints at the data structure that is >>> corrupt. You may get some output before the seg fault, if you add >>> a number of -v options (they add up to higher verbosity). >>> >>> Else, you may be able to look at the core and identify the function >>> that fails. You'll most probably need zdb and libzfs compiled with >>> "-g" to get any useful information from the core, though. >>> >>> For my failed pool, I noticed that internal assumptions were >>> violated, due to some free space occuring in more than one entry. >>> I had to special case the test in some function to ignore this >>> situation (I knew that I'd only ever wanted to mount that pool >>> R/O to rescue my data). But skipping the test did not suffice, >>> since another assert triggered (after skipping the NULL dereference, >>> the calculated sum of free space did not match the recorded sum, I >>> had to disable that assert, too). With these two patches I was able >>> to recover the pool starting at a TXG less than 100 transactions back, >>> which was sufficient for my purpose ... >>> >>> >> Question is will zdb 'fix' things or is it just a debug utility (for >> displaying)? >> > > The purpose of zdf is to access the pool without the need to import > it (which tends to crash the kernel) and to possibly identify a safe > TXG to go back to. Once you have found that zdb survives accesses to > critical data structures of your pool, you can then try to import the > pool to rescue your data. > > >> If it is just a debug and won't fix anything, I'm quite happy to roll >> back transactions, question is how (presumably after one finds the >> corrupt point - I'm quite happy to just do it by hand until I get >> success - it will save 2+months of work - I did get an output with a >> date/time that indicates where the rollback would go to...) >> >> In the mean time this appears to be working without crashing - it's been >> running days now... >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 4332 root 209 22 0 23770M 23277M uwait 1 549:07 11.04% >> zdb -AAA -L -uhdi -FX -e storage >> > > Options -u and -h do not take much time, -i depends on how much was > in the intent log (and recovery should be possible without, if your > kernel is not too old with regard to supported ZFS features). > > zdb -d takes a long time, and if it succeeds, you should be able to > recover your data. But zdb -m should also run to completion (and ISTR, > in my case that was where my kernel blew up trying to import the pool). > > Using the debugger to analyze the failed instruction let me work around > the inconsistency with two small patches (one skipped a consistency > check, the second fixed up the sum of free space which was miscalculated > due to the free block that lead to the panic being omitted). > > After I had these patches tested with zdb, I was able to import the pool > into a kernel that included these exact patches. You obviously do not > want to perform any other activities with the patched kernel, since it > lacks some internal checks - it is purely required for the one time > backup operation of the failed pool. > > > So, zdb and even the patches that make zdb dump your pool's internal > state will not directly lead to access to your data. But if you manage > to print all state with "zdb -dm", chances are very good, that you'll > be able to import the pool - possibly with temporary hacks to libzfs > that skip corrupt data elements (if not strictly required for read > accesses to your data). > > After that succeeded, you have a good chance to copy off your data > using a kernel that has the exact same patches in the ZFS driver ... > (if any are required, as in my case). > > Yeah -m crashes. The kernel will not crash it just refuses to import. zdb crashes with some options (or without others) ... question is how to get the import to succeed with a rollback of transactions to clean up the transaction DB - realistically there should be a maximum of 10 transactions that caused the issue as that's the length of the queue (kernel param) and data was lost up to that 10.... What I need to do is get it to recover to how it was before the metadb was corrupted - and if I am right it will be an incomplete write to the metadb that is blocking it (so the last transaction - or maybe last upto 10) are corrupted... Is it possible to discard 10 transactions and import - if so how? Is it possible to rebuild the metadb? The only accesses at the time to the array/fs was a network mount (NFS) where the connected client was torrenting (with less than 10k/s as I think there was only one active torrent at the time)... so if the torrented file is corrupt I can just discard it and start again, and if it's copy-on-write (as I thought zfs was) resetting it to the 'before write' version will just discard a piece of the file which will be replaced/re-downloaded as soon as it is restarted... I have not yet tried a 10.1 kernel - wrote the disk but it seems I can't use an apple superdrive as a boot rom on a PC... Michelle -- Michelle Sullivan http://www.mhix.org/