From owner-freebsd-fs@FreeBSD.ORG Thu Feb 5 22:32:19 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 448E14B7 for ; Thu, 5 Feb 2015 22:32:19 +0000 (UTC) Received: from hades.sorbs.net (hades.sorbs.net [67.231.146.201]) by mx1.freebsd.org (Postfix) with ESMTP id 312EC8CB for ; Thu, 5 Feb 2015 22:32:18 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from isux.com (firewall.isux.com [213.165.190.213]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0NJB00K1NLHYRS00@hades.sorbs.net> for freebsd-fs@freebsd.org; Thu, 05 Feb 2015 14:37:11 -0800 (PST) Message-id: <54D3EF70.6090703@sorbs.net> Date: Thu, 05 Feb 2015 23:32:16 +0100 From: Michelle Sullivan User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.24) Gecko/20100301 SeaMonkey/1.1.19 To: "freebsd-fs@freebsd.org" Subject: Re: ZFS pool faulted (corrupt metadata) but the disk data appears ok... References: <54D3E9F6.20702@sorbs.net> In-reply-to: <54D3E9F6.20702@sorbs.net> X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 22:32:19 -0000 Michelle Sullivan wrote: > Any clues on this? > > root@colossus:~ # zpool import > pool: storage > id: 10618504954404185222 > state: FAULTED > status: The pool metadata is corrupted. > action: The pool cannot be imported due to damaged devices or data. > The pool may be active on another system, but can be imported using > the '-f' flag. > see: http://illumos.org/msg/ZFS-8000-72 > config: > > storage FAULTED corrupted data > raidz2-0 ONLINE > mfid0 ONLINE > mfid15 ONLINE > mfid1 ONLINE > mfid2 ONLINE > mfid3 ONLINE > mfid4 ONLINE > mfid5 ONLINE > replacing-7 ONLINE > mfid13 ONLINE > mfid14 ONLINE > mfid6 ONLINE > mfid7 ONLINE > mfid8 ONLINE > mfid9 ONLINE > mfid10 ONLINE > mfid11 ONLINE > mfid12 ONLINE > root@colossus:~ # zpool import -Ff storage > cannot import 'storage': I/O error > Destroy and re-create the pool from > a backup source. > root@colossus:~ # zdb -l /dev/mfid0 > -------------------------------------------- > LABEL 0 > -------------------------------------------- > version: 5000 > name: 'storage' > state: 0 > txg: 1928241 > pool_guid: 10618504954404185222 > hostid: 4203774842 > hostname: 'colossus' > top_guid: 12489400212295803034 > guid: 3998695725653225547 > vdev_children: 1 > vdev_tree: > type: 'raidz' > id: 0 > guid: 12489400212295803034 > nparity: 2 > metaslab_array: 34 > metaslab_shift: 38 > ashift: 9 > asize: 45000449064960 > is_log: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 3998695725653225547 > path: '/dev/mfid0' > phys_path: '/dev/mfid0' > whole_disk: 1 > DTL: 168 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 10795471632546545577 > path: '/dev/mfid1' > phys_path: '/dev/mfid1' > whole_disk: 1 > DTL: 167 > create_txg: 4 > children[2]: > type: 'disk' > id: 2 > guid: 15820272272734706674 > path: '/dev/mfid2' > phys_path: '/dev/mfid2' > whole_disk: 1 > DTL: 166 > create_txg: 4 > children[3]: > type: 'disk' > id: 3 > guid: 3928579496187019848 > path: '/dev/mfid3' > phys_path: '/dev/mfid3' > whole_disk: 1 > DTL: 165 > create_txg: 4 > children[4]: > type: 'disk' > id: 4 > guid: 7125052278051590304 > path: '/dev/mfid4' > phys_path: '/dev/mfid4' > whole_disk: 1 > DTL: 164 > create_txg: 4 > children[5]: > type: 'disk' > id: 5 > guid: 14370198745088794709 > path: '/dev/mfid5' > phys_path: '/dev/mfid5' > whole_disk: 1 > DTL: 163 > create_txg: 4 > children[6]: > type: 'disk' > id: 6 > guid: 1843597351388951655 > path: '/dev/mfid6' > phys_path: '/dev/mfid6' > whole_disk: 1 > DTL: 162 > create_txg: 4 > children[7]: > type: 'replacing' > id: 7 > guid: 2914889727426054645 > whole_disk: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 10956220251832269421 > path: '/dev/mfid15' > phys_path: '/dev/mfid15' > whole_disk: 1 > DTL: 179 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 2463756237300743131 > path: '/dev/mfid13' > phys_path: '/dev/mfid13' > whole_disk: 1 > DTL: 181 > create_txg: 4 > resilvering: 1 > children[8]: > type: 'disk' > id: 8 > guid: 8864096842672670007 > path: '/dev/mfid7' > phys_path: '/dev/mfid7' > whole_disk: 1 > DTL: 160 > create_txg: 4 > children[9]: > type: 'disk' > id: 9 > guid: 4650681673751655245 > path: '/dev/mfid8' > phys_path: '/dev/mfid8' > whole_disk: 1 > DTL: 159 > create_txg: 4 > children[10]: > type: 'disk' > id: 10 > guid: 8432109430432996813 > path: '/dev/mfid9' > phys_path: '/dev/mfid9' > whole_disk: 1 > DTL: 158 > create_txg: 4 > children[11]: > type: 'disk' > id: 11 > guid: 414941847968750824 > path: '/dev/mfid10' > phys_path: '/dev/mfid10' > whole_disk: 1 > DTL: 157 > create_txg: 4 > children[12]: > type: 'disk' > id: 12 > guid: 7335375930620195352 > path: '/dev/mfid11' > phys_path: '/dev/mfid11' > whole_disk: 1 > DTL: 156 > create_txg: 4 > children[13]: > type: 'disk' > id: 13 > guid: 5100737174610362 > path: '/dev/mfid12' > phys_path: '/dev/mfid12' > whole_disk: 1 > DTL: 155 > create_txg: 4 > children[14]: > type: 'disk' > id: 14 > guid: 15695558693726858796 > path: '/dev/mfid14' > phys_path: '/dev/mfid14' > whole_disk: 1 > DTL: 174 > create_txg: 4 > features_for_read: > -------------------------------------------- > LABEL 1 > -------------------------------------------- > version: 5000 > name: 'storage' > state: 0 > txg: 1928241 > pool_guid: 10618504954404185222 > hostid: 4203774842 > hostname: 'colossus' > top_guid: 12489400212295803034 > guid: 3998695725653225547 > vdev_children: 1 > vdev_tree: > type: 'raidz' > id: 0 > guid: 12489400212295803034 > nparity: 2 > metaslab_array: 34 > metaslab_shift: 38 > ashift: 9 > asize: 45000449064960 > is_log: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 3998695725653225547 > path: '/dev/mfid0' > phys_path: '/dev/mfid0' > whole_disk: 1 > DTL: 168 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 10795471632546545577 > path: '/dev/mfid1' > phys_path: '/dev/mfid1' > whole_disk: 1 > DTL: 167 > create_txg: 4 > children[2]: > type: 'disk' > id: 2 > guid: 15820272272734706674 > path: '/dev/mfid2' > phys_path: '/dev/mfid2' > whole_disk: 1 > DTL: 166 > create_txg: 4 > children[3]: > type: 'disk' > id: 3 > guid: 3928579496187019848 > path: '/dev/mfid3' > phys_path: '/dev/mfid3' > whole_disk: 1 > DTL: 165 > create_txg: 4 > children[4]: > type: 'disk' > id: 4 > guid: 7125052278051590304 > path: '/dev/mfid4' > phys_path: '/dev/mfid4' > whole_disk: 1 > DTL: 164 > create_txg: 4 > children[5]: > type: 'disk' > id: 5 > guid: 14370198745088794709 > path: '/dev/mfid5' > phys_path: '/dev/mfid5' > whole_disk: 1 > DTL: 163 > create_txg: 4 > children[6]: > type: 'disk' > id: 6 > guid: 1843597351388951655 > path: '/dev/mfid6' > phys_path: '/dev/mfid6' > whole_disk: 1 > DTL: 162 > create_txg: 4 > children[7]: > type: 'replacing' > id: 7 > guid: 2914889727426054645 > whole_disk: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 10956220251832269421 > path: '/dev/mfid15' > phys_path: '/dev/mfid15' > whole_disk: 1 > DTL: 179 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 2463756237300743131 > path: '/dev/mfid13' > phys_path: '/dev/mfid13' > whole_disk: 1 > DTL: 181 > create_txg: 4 > resilvering: 1 > children[8]: > type: 'disk' > id: 8 > guid: 8864096842672670007 > path: '/dev/mfid7' > phys_path: '/dev/mfid7' > whole_disk: 1 > DTL: 160 > create_txg: 4 > children[9]: > type: 'disk' > id: 9 > guid: 4650681673751655245 > path: '/dev/mfid8' > phys_path: '/dev/mfid8' > whole_disk: 1 > DTL: 159 > create_txg: 4 > children[10]: > type: 'disk' > id: 10 > guid: 8432109430432996813 > path: '/dev/mfid9' > phys_path: '/dev/mfid9' > whole_disk: 1 > DTL: 158 > create_txg: 4 > children[11]: > type: 'disk' > id: 11 > guid: 414941847968750824 > path: '/dev/mfid10' > phys_path: '/dev/mfid10' > whole_disk: 1 > DTL: 157 > create_txg: 4 > children[12]: > type: 'disk' > id: 12 > guid: 7335375930620195352 > path: '/dev/mfid11' > phys_path: '/dev/mfid11' > whole_disk: 1 > DTL: 156 > create_txg: 4 > children[13]: > type: 'disk' > id: 13 > guid: 5100737174610362 > path: '/dev/mfid12' > phys_path: '/dev/mfid12' > whole_disk: 1 > DTL: 155 > create_txg: 4 > children[14]: > type: 'disk' > id: 14 > guid: 15695558693726858796 > path: '/dev/mfid14' > phys_path: '/dev/mfid14' > whole_disk: 1 > DTL: 174 > create_txg: 4 > features_for_read: > -------------------------------------------- > LABEL 2 > -------------------------------------------- > version: 5000 > name: 'storage' > state: 0 > txg: 1928241 > pool_guid: 10618504954404185222 > hostid: 4203774842 > hostname: 'colossus' > top_guid: 12489400212295803034 > guid: 3998695725653225547 > vdev_children: 1 > vdev_tree: > type: 'raidz' > id: 0 > guid: 12489400212295803034 > nparity: 2 > metaslab_array: 34 > metaslab_shift: 38 > ashift: 9 > asize: 45000449064960 > is_log: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 3998695725653225547 > path: '/dev/mfid0' > phys_path: '/dev/mfid0' > whole_disk: 1 > DTL: 168 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 10795471632546545577 > path: '/dev/mfid1' > phys_path: '/dev/mfid1' > whole_disk: 1 > DTL: 167 > create_txg: 4 > children[2]: > type: 'disk' > id: 2 > guid: 15820272272734706674 > path: '/dev/mfid2' > phys_path: '/dev/mfid2' > whole_disk: 1 > DTL: 166 > create_txg: 4 > children[3]: > type: 'disk' > id: 3 > guid: 3928579496187019848 > path: '/dev/mfid3' > phys_path: '/dev/mfid3' > whole_disk: 1 > DTL: 165 > create_txg: 4 > children[4]: > type: 'disk' > id: 4 > guid: 7125052278051590304 > path: '/dev/mfid4' > phys_path: '/dev/mfid4' > whole_disk: 1 > DTL: 164 > create_txg: 4 > children[5]: > type: 'disk' > id: 5 > guid: 14370198745088794709 > path: '/dev/mfid5' > phys_path: '/dev/mfid5' > whole_disk: 1 > DTL: 163 > create_txg: 4 > children[6]: > type: 'disk' > id: 6 > guid: 1843597351388951655 > path: '/dev/mfid6' > phys_path: '/dev/mfid6' > whole_disk: 1 > DTL: 162 > create_txg: 4 > children[7]: > type: 'replacing' > id: 7 > guid: 2914889727426054645 > whole_disk: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 10956220251832269421 > path: '/dev/mfid15' > phys_path: '/dev/mfid15' > whole_disk: 1 > DTL: 179 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 2463756237300743131 > path: '/dev/mfid13' > phys_path: '/dev/mfid13' > whole_disk: 1 > DTL: 181 > create_txg: 4 > resilvering: 1 > children[8]: > type: 'disk' > id: 8 > guid: 8864096842672670007 > path: '/dev/mfid7' > phys_path: '/dev/mfid7' > whole_disk: 1 > DTL: 160 > create_txg: 4 > children[9]: > type: 'disk' > id: 9 > guid: 4650681673751655245 > path: '/dev/mfid8' > phys_path: '/dev/mfid8' > whole_disk: 1 > DTL: 159 > create_txg: 4 > children[10]: > type: 'disk' > id: 10 > guid: 8432109430432996813 > path: '/dev/mfid9' > phys_path: '/dev/mfid9' > whole_disk: 1 > DTL: 158 > create_txg: 4 > children[11]: > type: 'disk' > id: 11 > guid: 414941847968750824 > path: '/dev/mfid10' > phys_path: '/dev/mfid10' > whole_disk: 1 > DTL: 157 > create_txg: 4 > children[12]: > type: 'disk' > id: 12 > guid: 7335375930620195352 > path: '/dev/mfid11' > phys_path: '/dev/mfid11' > whole_disk: 1 > DTL: 156 > create_txg: 4 > children[13]: > type: 'disk' > id: 13 > guid: 5100737174610362 > path: '/dev/mfid12' > phys_path: '/dev/mfid12' > whole_disk: 1 > DTL: 155 > create_txg: 4 > children[14]: > type: 'disk' > id: 14 > guid: 15695558693726858796 > path: '/dev/mfid14' > phys_path: '/dev/mfid14' > whole_disk: 1 > DTL: 174 > create_txg: 4 > features_for_read: > -------------------------------------------- > LABEL 3 > -------------------------------------------- > version: 5000 > name: 'storage' > state: 0 > txg: 1928241 > pool_guid: 10618504954404185222 > hostid: 4203774842 > hostname: 'colossus' > top_guid: 12489400212295803034 > guid: 3998695725653225547 > vdev_children: 1 > vdev_tree: > type: 'raidz' > id: 0 > guid: 12489400212295803034 > nparity: 2 > metaslab_array: 34 > metaslab_shift: 38 > ashift: 9 > asize: 45000449064960 > is_log: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 3998695725653225547 > path: '/dev/mfid0' > phys_path: '/dev/mfid0' > whole_disk: 1 > DTL: 168 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 10795471632546545577 > path: '/dev/mfid1' > phys_path: '/dev/mfid1' > whole_disk: 1 > DTL: 167 > create_txg: 4 > children[2]: > type: 'disk' > id: 2 > guid: 15820272272734706674 > path: '/dev/mfid2' > phys_path: '/dev/mfid2' > whole_disk: 1 > DTL: 166 > create_txg: 4 > children[3]: > type: 'disk' > id: 3 > guid: 3928579496187019848 > path: '/dev/mfid3' > phys_path: '/dev/mfid3' > whole_disk: 1 > DTL: 165 > create_txg: 4 > children[4]: > type: 'disk' > id: 4 > guid: 7125052278051590304 > path: '/dev/mfid4' > phys_path: '/dev/mfid4' > whole_disk: 1 > DTL: 164 > create_txg: 4 > children[5]: > type: 'disk' > id: 5 > guid: 14370198745088794709 > path: '/dev/mfid5' > phys_path: '/dev/mfid5' > whole_disk: 1 > DTL: 163 > create_txg: 4 > children[6]: > type: 'disk' > id: 6 > guid: 1843597351388951655 > path: '/dev/mfid6' > phys_path: '/dev/mfid6' > whole_disk: 1 > DTL: 162 > create_txg: 4 > children[7]: > type: 'replacing' > id: 7 > guid: 2914889727426054645 > whole_disk: 0 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 10956220251832269421 > path: '/dev/mfid15' > phys_path: '/dev/mfid15' > whole_disk: 1 > DTL: 179 > create_txg: 4 > children[1]: > type: 'disk' > id: 1 > guid: 2463756237300743131 > path: '/dev/mfid13' > phys_path: '/dev/mfid13' > whole_disk: 1 > DTL: 181 > create_txg: 4 > resilvering: 1 > children[8]: > type: 'disk' > id: 8 > guid: 8864096842672670007 > path: '/dev/mfid7' > phys_path: '/dev/mfid7' > whole_disk: 1 > DTL: 160 > create_txg: 4 > children[9]: > type: 'disk' > id: 9 > guid: 4650681673751655245 > path: '/dev/mfid8' > phys_path: '/dev/mfid8' > whole_disk: 1 > DTL: 159 > create_txg: 4 > children[10]: > type: 'disk' > id: 10 > guid: 8432109430432996813 > path: '/dev/mfid9' > phys_path: '/dev/mfid9' > whole_disk: 1 > DTL: 158 > create_txg: 4 > children[11]: > type: 'disk' > id: 11 > guid: 414941847968750824 > path: '/dev/mfid10' > phys_path: '/dev/mfid10' > whole_disk: 1 > DTL: 157 > create_txg: 4 > children[12]: > type: 'disk' > id: 12 > guid: 7335375930620195352 > path: '/dev/mfid11' > phys_path: '/dev/mfid11' > whole_disk: 1 > DTL: 156 > create_txg: 4 > children[13]: > type: 'disk' > id: 13 > guid: 5100737174610362 > path: '/dev/mfid12' > phys_path: '/dev/mfid12' > whole_disk: 1 > DTL: 155 > create_txg: 4 > children[14]: > type: 'disk' > id: 14 > guid: 15695558693726858796 > path: '/dev/mfid14' > phys_path: '/dev/mfid14' > whole_disk: 1 > DTL: 174 > create_txg: 4 > features_for_read: > > LSI9260-16i raid controller all drives in single disk RAID0, system > wasn't busy (=in use for writing, however was resilvering a replacement > drive after the HSP had taken over due to failed drive) and it got > rebooted... faulted on reboot. > > Recoverable? > > Regards, > > FYI before the first reboot: root@colossus:~ # zpool status -x pool: storage state: FAULTED status: The pool metadata is corrupted and the pool cannot be opened. action: Destroy and re-create the pool from a backup source. see: http://illumos.org/msg/ZFS-8000-72 scan: none requested config: NAME STATE READ WRITE CKSUM storage FAULTED 0 0 1 raidz2-0 ONLINE 0 0 7 mfid0 ONLINE 0 0 1 mfid1 ONLINE 0 0 0 mfid2 ONLINE 0 0 0 mfid3 ONLINE 0 0 0 mfid4 ONLINE 0 0 0 mfid5 ONLINE 0 0 0 mfid6 ONLINE 0 0 0 replacing-7 ONLINE 0 0 0 mfid14 ONLINE 0 0 0 mfid15 ONLINE 0 0 0 mfid7 ONLINE 0 0 0 mfid8 ONLINE 0 0 0 mfid9 ONLINE 0 0 0 mfid10 ONLINE 0 0 0 mfid11 ONLINE 0 0 0 mfid12 ONLINE 0 0 1 mfid13 ONLINE 0 0 0 root@colossus:~ # zpool clear -nF storage internal error: out of memory root@colossus:~ # "zpool import -Fn storage" reports no errors. Really don't care if a file or three is corrupted, just want the pool back as don't have a backup since Dec 3, 2014. -- Michelle Sullivan http://www.mhix.org/