From owner-freebsd-fs@FreeBSD.ORG Fri Feb 6 11:21:08 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 743B8715 for ; Fri, 6 Feb 2015 11:21:08 +0000 (UTC) Received: from hades.sorbs.net (hades.sorbs.net [67.231.146.201]) by mx1.freebsd.org (Postfix) with ESMTP id 5EA0EF3B for ; Fri, 6 Feb 2015 11:21:07 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from isux.com (firewall.isux.com [213.165.190.213]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0NJC00K9AL3ARS00@hades.sorbs.net> for freebsd-fs@freebsd.org; Fri, 06 Feb 2015 03:26:00 -0800 (PST) Message-id: <54D4A3A0.2040408@sorbs.net> Date: Fri, 06 Feb 2015 12:21:04 +0100 From: Michelle Sullivan User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.24) Gecko/20100301 SeaMonkey/1.1.19 To: Xin Li Subject: Re: ZFS pool faulted (corrupt metadata) but the disk data appears ok... References: <54D3E9F6.20702@sorbs.net> <54D41608.50306@delphij.net> <54D41AAA.6070303@sorbs.net> <54D41C52.1020003@delphij.net> <54D424F0.9080301@sorbs.net> <54D457F0.8080502@delphij.net> In-reply-to: <54D457F0.8080502@delphij.net> Cc: "freebsd-fs@freebsd.org" , d@delphij.net X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 11:21:08 -0000 Xin Li wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA512 > > > > On 2/5/15 18:20, Michelle Sullivan wrote: > >> Xin Li wrote: On 02/05/15 17:36, Michelle Sullivan wrote: >> >> >>>>>> This suggests the pool was connected to a different system, >>>>>> is that the case? >>>>>> >>>>>> >>>>>> >>>>> No. >>>>> >>>>> >> Ok, that's good. Actually if you have two heads that writes to >> the same pool at the same time, it can easily enter an >> unrecoverable state. >> >> >> >>>>>> It's hard to tell right now, and we shall try all possible >>>>>> remedies but be prepared for the worst. >>>>>> >>>>>> >>>>> I am :( >>>>> >>>>> >> The next thing I would try is to: >> >> 1. move /boot/zfs/zpool.cache to somewhere else; >> >> >> >>> There isn't one. However 'cat'ing the inode I can see there was >>> one... >>> >>> <83>^LR^@^L^@^D^A.^@^@^@<80>^LR^@^A^D^B..^@^@<89>^LR^@^X^@^H^Ozpool.cache.tmp^@<89>^LR^@^A^H^Kzpool.cache^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ >>> >>> >>> > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > >> 2. zpool import -f -n -F -X storage and see if the system would >> give you a proposal. >> >> >> >>> This crashes (without -n) the machine out of memory.... there's >>> 32G of RAM. /boot/loader.conf contains: >>> >>> vfs.zfs.prefetch_disable=1 #vfs.zfs.arc_min="8G" >>> #vfs.zfs.arc_max="16G" #vm.kmem_size_max="8" #vm.kmem_size="6G" >>> vfs.zfs.txg.timeout="5" kern.maxvnodes=250000 >>> vfs.zfs.write_limit_override=1073741824 vboxdrv_load="YES" >>> > > Which release this is? write_limit_override have been removed quite a > while ago. > FreeBSD colossus 9.2-RELEASE-p15 FreeBSD 9.2-RELEASE-p15 #0: Mon Nov 3 20:31:29 UTC 2014 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 > I'd recommend using a fresh -CURRENT snapshot if possible (possibly > with -NODEBUG kernel). > I'm sorta afraid to try and upgrade it at this point. Michelle -- Michelle Sullivan http://www.mhix.org/