From owner-freebsd-fs@FreeBSD.ORG Fri Jun 7 08:03:51 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9C267180 for ; Fri, 7 Jun 2013 08:03:51 +0000 (UTC) (envelope-from ronald-freebsd8@klop.yi.org) Received: from smarthost1.greenhost.nl (smarthost1.greenhost.nl [195.190.28.81]) by mx1.freebsd.org (Postfix) with ESMTP id 36F3A1F01 for ; Fri, 7 Jun 2013 08:03:51 +0000 (UTC) Received: from smtp.greenhost.nl ([213.108.104.138]) by smarthost1.greenhost.nl with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.69) (envelope-from ) id 1UkrPO-0008MT-Ce for freebsd-fs@freebsd.org; Fri, 07 Jun 2013 09:48:27 +0200 Received: from [81.21.138.17] (helo=ronaldradial.versatec.local) by smtp.greenhost.nl with esmtpsa (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.72) (envelope-from ) id 1UkrPL-0001LA-K7 for freebsd-fs@freebsd.org; Fri, 07 Jun 2013 09:48:23 +0200 Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes To: freebsd-fs@freebsd.org Subject: Re: zpool export/import on failover - The pool metadata is corrupted References: Date: Fri, 07 Jun 2013 09:48:24 +0200 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: "Ronald Klop" Message-ID: In-Reply-To: User-Agent: Opera Mail/12.15 (Win32) X-Virus-Scanned: by clamav at smarthost1.samage.net X-Spam-Level: / X-Spam-Score: 0.8 X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.1 X-Scan-Signature: 897836312160ed0141c32cdc6ac56212 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Jun 2013 08:03:51 -0000 On Thu, 06 Jun 2013 21:24:34 +0200, mxb wrote: > > Hello list, > > I have two-head ZFS setup with external disk enclosure over SAS expander. > This is a failover setup with CARP and devd triggering spool > export/import. > One of two nodes is preferred master. > > Then master is rebooted, devd kicks in as of CARP becomes master and the > second node picks up ZFS-disks from external enclosure. > Then master comes back, CARP becomes master, devd kicks in and pool gets > exported from the second node and imported on the first one. > > However, I have experienced metadata corruption several times with this > setup. > Note, that ZIL(mirrored) resides on external enclosure. Only L2ARC is > both local and external - da1,da2, da13s2, da14s2 > > root@nfs2:/root # zpool import > pool: jbod > id: 17635654860276652744 > state: FAULTED > status: The pool metadata is corrupted. > action: The pool cannot be imported due to damaged devices or data. > see: http://illumos.org/msg/ZFS-8000-72 > config: > > jbod FAULTED corrupted data > raidz3-0 ONLINE > da3 ONLINE > da4 ONLINE > da5 ONLINE > da6 ONLINE > da7 ONLINE > da8 ONLINE > da9 ONLINE > da10 ONLINE > da11 ONLINE > da12 ONLINE > cache > da1 > da2 > da13s2 > da14s2 > logs > mirror-1 ONLINE > da13s1 ONLINE > da14s1 ONLINE > > Any ideas what is going on? > > //mxb I know the Oracle ZFS Appliance you can buy with clustering does a reboot of the node which should release the pool. The mechanism is called like this: http://en.wikipedia.org/wiki/STONITH Ronald.