From owner-freebsd-fs@FreeBSD.ORG Fri Feb 6 16:17:47 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 12409D78 for ; Fri, 6 Feb 2015 16:17:47 +0000 (UTC) Received: from mail-vc0-x22a.google.com (mail-vc0-x22a.google.com [IPv6:2607:f8b0:400c:c03::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B967982D for ; Fri, 6 Feb 2015 16:17:46 +0000 (UTC) Received: by mail-vc0-f170.google.com with SMTP id kv7so5374462vcb.1 for ; Fri, 06 Feb 2015 08:17:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ucsc.edu; s=ucsc-google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=dUwkker78zaxcjn3l6rJIg6Rteu2rYz32Ap/01zlJlo=; b=HVOfTJ1ptDix8png3EsOZ96qKNCPzjfDx7qBxPsNdZ+rJT0DFZWN9qrxfmP1pzAJDq B2k0v8P6HiuEwNsI4atUZj3xtcLtte3sSmeeW4mJYVBNuG52QHKZIChEvYh56F5qo9le Peu1s8Qo4W7UCyJVTQ9GxUpJzog1hNeRYgUS8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=dUwkker78zaxcjn3l6rJIg6Rteu2rYz32Ap/01zlJlo=; b=EOK0R1dfhuvDsnzhpDOl/cwbnsFCHA4j2cVZn4iLVx0e8Odqtz+anN0EzXU7jMvbNl MXCcoZhMNqqiX96UmERCuvAi+Bf+jnl+MA7OP7zilS+xRks365suBde5GrAZuusXptPf gvro5Lyl/8r9PpvI4FdhQCmCuXoFTdqHdgu6p2CIOT8VadQ3BinDZcC85OE4shboVOqv APV6pklwmNO5QAT5YgK3aPhi1mNVQvEBqla3otFmnapsCFN/QnFY7/PGpwMCkhbEoFBy 9E1DEfMBTcIhjgpoBIedX5hH6x4Kd1ofOZ51mbl31tNT1wCpaKkpGWeBbK8HDZR/wDd4 fxgQ== X-Gm-Message-State: ALoCoQkQRMk38+Irb9flIivcX/FXR/4IVoXPAFyktXCePyoiUU8uZ15Nr3RwtB2Xq7xeYhkNUYR/ MIME-Version: 1.0 X-Received: by 10.52.168.105 with SMTP id zv9mr2199527vdb.33.1423239465370; Fri, 06 Feb 2015 08:17:45 -0800 (PST) Received: by 10.52.115.103 with HTTP; Fri, 6 Feb 2015 08:17:45 -0800 (PST) Received: by 10.52.115.103 with HTTP; Fri, 6 Feb 2015 08:17:45 -0800 (PST) In-Reply-To: <54D4A3A0.2040408@sorbs.net> References: <54D3E9F6.20702@sorbs.net> <54D41608.50306@delphij.net> <54D41AAA.6070303@sorbs.net> <54D41C52.1020003@delphij.net> <54D424F0.9080301@sorbs.net> <54D457F0.8080502@delphij.net> <54D4A3A0.2040408@sorbs.net> Date: Fri, 6 Feb 2015 08:17:45 -0800 Message-ID: Subject: Re: ZFS pool faulted (corrupt metadata) but the disk data appears ok... From: Michael Ware To: Michelle Sullivan Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-fs@freebsd.org" , d@delphij.net X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Feb 2015 16:17:47 -0000 You can mount using a live cd if you haven't tried yet. Maybe try the 10.1 iso and see if you have any luck. Mike On Feb 6, 2015 3:21 AM, "Michelle Sullivan" wrote: > Xin Li wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA512 > > > > > > > > On 2/5/15 18:20, Michelle Sullivan wrote: > > > >> Xin Li wrote: On 02/05/15 17:36, Michelle Sullivan wrote: > >> > >> > >>>>>> This suggests the pool was connected to a different system, > >>>>>> is that the case? > >>>>>> > >>>>>> > >>>>>> > >>>>> No. > >>>>> > >>>>> > >> Ok, that's good. Actually if you have two heads that writes to > >> the same pool at the same time, it can easily enter an > >> unrecoverable state. > >> > >> > >> > >>>>>> It's hard to tell right now, and we shall try all possible > >>>>>> remedies but be prepared for the worst. > >>>>>> > >>>>>> > >>>>> I am :( > >>>>> > >>>>> > >> The next thing I would try is to: > >> > >> 1. move /boot/zfs/zpool.cache to somewhere else; > >> > >> > >> > >>> There isn't one. However 'cat'ing the inode I can see there was > >>> one... > >>> > >>> > <83>^LR^@^L^@^D^A.^@^@^@<80>^LR^@^A^D^B..^@^@<89>^LR^@^X^@^H^Ozpool.c= ache.tmp^@<89>^LR^@^A^H^Kzpool.cache^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^= @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@= ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > >>> > >>> > >>> > > > ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^= @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@= ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^= @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@= ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^= @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@= ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^= @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@= ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^= @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ > > > >> 2. zpool import -f -n -F -X storage and see if the system would > >> give you a proposal. > >> > >> > >> > >>> This crashes (without -n) the machine out of memory.... there's > >>> 32G of RAM. /boot/loader.conf contains: > >>> > >>> vfs.zfs.prefetch_disable=3D1 #vfs.zfs.arc_min=3D"8G" > >>> #vfs.zfs.arc_max=3D"16G" #vm.kmem_size_max=3D"8" #vm.kmem_size=3D"6G" > >>> vfs.zfs.txg.timeout=3D"5" kern.maxvnodes=3D250000 > >>> vfs.zfs.write_limit_override=3D1073741824 vboxdrv_load=3D"YES" > >>> > > > > Which release this is? write_limit_override have been removed quite a > > while ago. > > > > FreeBSD colossus 9.2-RELEASE-p15 FreeBSD 9.2-RELEASE-p15 #0: Mon Nov 3 > 20:31:29 UTC 2014 > root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 > > > > I'd recommend using a fresh -CURRENT snapshot if possible (possibly > > with -NODEBUG kernel). > > > > I'm sorta afraid to try and upgrade it at this point. > > Michelle > > -- > Michelle Sullivan > http://www.mhix.org/ > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >