From owner-freebsd-fs@FreeBSD.ORG Fri Oct 4 01:10:03 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 9B35785A for ; Fri, 4 Oct 2013 01:10:03 +0000 (UTC) (envelope-from ari@ish.com.au) Received: from fish.ish.com.au (eth5921.nsw.adsl.internode.on.net [59.167.240.32]) (using TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 58E712B5A for ; Fri, 4 Oct 2013 01:10:02 +0000 (UTC) Received: from ip-136.ish.com.au ([203.29.62.136]:63808) by fish.ish.com.au with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76) (envelope-from ) id 1VRtrD-0003j0-1T for freebsd-fs@freebsd.org; Fri, 04 Oct 2013 11:07:04 +1000 X-CTCH-RefID: str=0001.0A150203.524E14B7.00C6:SCFSTAT15613948, ss=1, re=-4.000, recu=0.000, reip=0.000, cl=1, cld=1, fgs=0 Message-ID: <524E14B6.9040808@ish.com.au> Date: Fri, 04 Oct 2013 11:07:02 +1000 From: Aristedes Maniatis User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:24.0) Gecko/20100101 Thunderbird/24.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: hast degraded X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Oct 2013 01:10:03 -0000 I am in a bit of a mess. I have a pair of servers running ZFS over HAST on four drives. Today the primary server reported that the hast array had changed role to secondary. And this started causing corruption to data since the zpool was still mounted and trying to write to this array. I was not able to make the hast primary again since when I did that it shows like this: # hastctl status Name Status Role Components ssd2 degraded primary /dev/da2 10.8.8.1 ssd3 degraded primary /dev/da3 10.8.8.1 ssd4 degraded primary /dev/da4 10.8.8.1 ssd5 degraded primary /dev/da5 10.8.8.1 On the slave server, I get exactly the same result. When I switch hast to primary role it is marked as degraded. Where to from here? How can I force the hast array to go back to primary role? I desparately need to recover from this point since the backups are missing some part of the data set. I had hoped that hast under a zfs raid would give me extra redundancy, however it appears it has instead given me a new way to corrupt all the data on both machines simultaneously. Cheers Ari -- --------------------------> Aristedes Maniatis ish http://www.ish.com.au Level 1, 30 Wilson Street Newtown 2042 Australia phone +61 2 9550 5001 fax +61 2 9550 4001 GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A