From owner-freebsd-fs@FreeBSD.ORG Sat May 11 11:16:46 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5F2541D5 for ; Sat, 11 May 2013 11:16:46 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-oa0-f49.google.com (mail-oa0-f49.google.com [209.85.219.49]) by mx1.freebsd.org (Postfix) with ESMTP id 2DCA42C0 for ; Sat, 11 May 2013 11:16:45 +0000 (UTC) Received: by mail-oa0-f49.google.com with SMTP id k14so4273442oag.8 for ; Sat, 11 May 2013 04:16:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=RNFjfjST/3NrUpYso+v3AuQoehwYbwH1js9viOwHTRA=; b=DJWTzILoiFIIiVEX/+xxxCK36X5gWkKGPKxcgHsWz7J1LkZCSRjmLMEiOhZIaILFQz nvXZ+RuSfdA6T212EefmNI6vX55oLDem7d9qDef4Q+M67EbmlQ3pA/LJmjHYmpNnLn8p Y957IuWVHLXlijnc2AZhCialUpxsCHWwJ4GyPN6X1MjgO1o9fBTaT4QsvkZV7R9JEAm1 XRM3DW0cFonANhHf92+v+oWDm+fDeqtFHxH1ra8xoH27dpAGR3VyImNq+JEIBuZViNG1 4PsyQiYP0LFOvnUXc2tDHd6bCZx3+qPOfY9pLlSMZl6i3/O7KGMyLhBqe50mR3lp2Po3 tNKg== MIME-Version: 1.0 X-Received: by 10.182.226.162 with SMTP id rt2mr9058153obc.9.1368271005292; Sat, 11 May 2013 04:16:45 -0700 (PDT) Received: by 10.76.96.49 with HTTP; Sat, 11 May 2013 04:16:45 -0700 (PDT) In-Reply-To: References: <518CFA05.6090706@gmail.com> Date: Sat, 11 May 2013 07:16:45 -0400 Message-ID: Subject: Re: Corrupted zpool import -f FAILS state FAULTED From: Outback Dingo To: Ronald Klop Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 May 2013 11:16:46 -0000 On Sat, May 11, 2013 at 4:44 AM, Ronald Klop wrote: > On Fri, 10 May 2013 16:07:36 +0200, Outback Dingo > wrote: > > On Fri, May 10, 2013 at 9:45 AM, Volodymyr Kostyrko > >wrote: >> >> 09.05.2013 15:31, Outback Dingo: >>> >>> ok zfsgurus, FreeBSD 9.1-STABLE box zpool import -f reports pool status >>> >>>> Faulted, one of more devices contains corrupted data, however its >>>> showing >>>> the guid as faulted in the poll, and not the actual disk device >>>> /dev/daX, >>>> the pool is a single vdev 24 disk raidz3. Essentially the hardward >>>> platform >>>> is a dual node system, with 8 enclosures connected to 24 SAS drives via >>>> 4 >>>> LSI cards. I am not currently using geom_multipath, but the box is zoned >>>> so >>>> that each node can see 50% of the drives, >>>> in case of Failure, carp kicks in and migrates "zpool import -af" the >>>> pools >>>> onto the other node. it seems as though somehow the pool is now seeing >>>> guid >>>> and not devices, not sure if they have switched devices ids due to a >>>> reboot. >>>> >>>> >>> Am not a zfs guru, but I'll try to help. >>> >>> Any console log snippets are welcome. What does "showing the guid as >>> faulted in the pool" looks like. >>> >>> What are the guids for all partitions? Do they interlap for different >>> nodes? >>> >>> ZFS recognizes devices by tasting they vdev labels and not by their >>> logical location and naming. It can safely report any vdev location - but >>> it requires the same set vdevs to bring pool online. >>> >> >> >> zdb shows valid data on the drives, no drives have been removed from the >> box >> whats confusing is why its using guids and not devices daX is what puzzles >> me >> camcontrol devlist and dmesg clearly show the devices are there. The SAS >> bus is shared >> so both nodes with 2 LSI controllers each see all drives. We were >> utilizing >> a failover script >> if nodeA dies, carp would kick the script to import the pool to nodeB, >> both >> nodes are in the >> same chassis and see all the enclosures and all the drives >> > > Are the machines configured the same? As in _exactly_ the same. Glabel > modules, hint files, sysctls, etc. > > yes, both nodes are identical, from sysctl.conf to loader.conf, ive also noticed that playing around with enclosure zoning on the system i can now see which strikes me as quite odd..... now im wondering if i have a controller flaking out. right now according to the zoning, gmultipath should see 24+ LUNS however it sees nothing. zpool import -f pool: backup id: 8548776274175948174 state: UNAVAIL status: The pool was last accessed by another system. action: The pool cannot be imported due to damaged devices or data. see: http://illumos.org/msg/ZFS-8000-EY config: backup UNAVAIL insufficient replicas raidz3-0 UNAVAIL insufficient replicas da32 ONLINE da30 ONLINE da29 ONLINE da3 ONLINE da4 ONLINE da5 ONLINE da6 ONLINE da7 ONLINE da8 ONLINE label/big4 ONLINE 18084052867377310822 UNAVAIL cannot open 2641768775090614171 UNAVAIL cannot open 8083525846528480855 UNAVAIL cannot open 8200855950201180014 UNAVAIL cannot open da37 ONLINE da11 ONLINE 4678398398699137944 UNAVAIL cannot open 18315550984013241979 UNAVAIL cannot open da22 ONLINE da23 ONLINE label/backup ONLINE da25 ONLINE da26 ONLINE da27 ONLINE > Ronald. > > ______________________________**_________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@**freebsd.org > " >