From owner-freebsd-fs@FreeBSD.ORG Thu Jun 6 22:04:40 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 94A0CB56 for ; Thu, 6 Jun 2013 22:04:40 +0000 (UTC) (envelope-from mxb@alumni.chalmers.se) Received: from mail-lb0-f170.google.com (mail-lb0-f170.google.com [209.85.217.170]) by mx1.freebsd.org (Postfix) with ESMTP id 1BF2317FA for ; Thu, 6 Jun 2013 22:04:39 +0000 (UTC) Received: by mail-lb0-f170.google.com with SMTP id t11so3621873lbd.29 for ; Thu, 06 Jun 2013 15:04:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=8tbn2dVNcK5B6YolsfHZfzlxWOCRi4AYyNhnfbFeeRs=; b=Q8vxrWS/kQ8H0QKF7aJy9z+Jy7s3lmHn5vvR503vb6LrQvde35ekTl66y5XHOejwUx WsIA25y8z/MFk6saCkncvt0ATMuCfsWOebm0oDNmvOKM+ESmv1a9tzaEljTMo+o84f5M reL2DAt90Ms0ol8wIFg7AQsuY6yB0HnkiLHMkbGuKdWb/XqOjM9WFa8Od97hxkoMX00K tNMa693XKlY4ECmZNbZLXEqdmheTU2j8RODQ0huqnP+8SyfkcNOZ712wGOXO3NCxTYtM aks+E2HAXokftKe+wONrkpFEVCjhimD+e/0tKowMDWW6UTQsB5UuUHOaYRfS8Nnlf4Fl YUxw== X-Received: by 10.152.18.234 with SMTP id z10mr18494303lad.29.1370556272501; Thu, 06 Jun 2013 15:04:32 -0700 (PDT) Received: from grey.home.unixconn.com (h-75-17.a183.priv.bahnhof.se. [46.59.75.17]) by mx.google.com with ESMTPSA id f8sm29440973lbr.10.2013.06.06.15.04.31 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 06 Jun 2013 15:04:31 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: zpool export/import on failover - The pool metadata is corrupted From: mxb In-Reply-To: <20130606193340.GA42205@icarus.home.lan> Date: Fri, 7 Jun 2013 00:04:30 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <34F6EC70-1812-44A2-AE6E-8DF8414404FE@alumni.chalmers.se> References: <20130606193340.GA42205@icarus.home.lan> To: Jeremy Chadwick X-Mailer: Apple Mail (2.1503) X-Gm-Message-State: ALoCoQkXYLDI3KNWZElELKN6Zo/tLXumtf7gW8xHdbPeokuN4ajAQcJY2HbMwSO+gaf5ngfUw3lz Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 22:04:40 -0000 root@nfs2:/root # uname -a FreeBSD nfs2 9.1-STABLE FreeBSD 9.1-STABLE #1: Tue May 21 21:42:21 CEST = 2013 root@nfs2:/usr/obj/usr/src/sys/GENERIC amd64 root@nfs2:/root # On 6 jun 2013, at 21:33, Jeremy Chadwick wrote: > On Thu, Jun 06, 2013 at 09:24:34PM +0200, mxb wrote: >>=20 >> Hello list, >>=20 >> I have two-head ZFS setup with external disk enclosure over SAS = expander. >> This is a failover setup with CARP and devd triggering spool = export/import. >> One of two nodes is preferred master. >>=20 >> Then master is rebooted, devd kicks in as of CARP becomes master and = the second node picks up ZFS-disks from external enclosure. >> Then master comes back, CARP becomes master, devd kicks in and pool = gets exported from the second node and imported on the first one. >>=20 >> However, I have experienced metadata corruption several times with = this setup. >> Note, that ZIL(mirrored) resides on external enclosure. Only L2ARC is = both local and external - da1,da2, da13s2, da14s2 >>=20 >> root@nfs2:/root # zpool import >> pool: jbod >> id: 17635654860276652744 >> state: FAULTED >> status: The pool metadata is corrupted. >> action: The pool cannot be imported due to damaged devices or data. >> see: http://illumos.org/msg/ZFS-8000-72 >> config: >>=20 >> jbod FAULTED corrupted data >> raidz3-0 ONLINE >> da3 ONLINE >> da4 ONLINE >> da5 ONLINE >> da6 ONLINE >> da7 ONLINE >> da8 ONLINE >> da9 ONLINE >> da10 ONLINE >> da11 ONLINE >> da12 ONLINE >> cache >> da1 >> da2 >> da13s2 >> da14s2 >> logs >> mirror-1 ONLINE >> da13s1 ONLINE >> da14s1 ONLINE >>=20 >> Any ideas what is going on? >=20 > uname -a please. >=20 > --=20 > | Jeremy Chadwick jdc@koitsu.org | > | UNIX Systems Administrator http://jdc.koitsu.org/ | > | Making life hard for others since 1977. PGP 4BD6C0CB | >=20