From owner-freebsd-fs@FreeBSD.ORG Thu Feb 5 22:09:04 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 275C8ABA for ; Thu, 5 Feb 2015 22:09:04 +0000 (UTC) Received: from hades.sorbs.net (hades.sorbs.net [67.231.146.201]) by mx1.freebsd.org (Postfix) with ESMTP id 129F33B3 for ; Thu, 5 Feb 2015 22:09:03 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from isux.com (firewall.isux.com [213.165.190.213]) by hades.sorbs.net (Oracle Communications Messaging Server 7.0.5.29.0 64bit (built Jul 9 2013)) with ESMTPSA id <0NJB00K15KF0RS00@hades.sorbs.net> for freebsd-fs@freebsd.org; Thu, 05 Feb 2015 14:13:50 -0800 (PST) Message-id: <54D3E9F6.20702@sorbs.net> Date: Thu, 05 Feb 2015 23:08:54 +0100 From: Michelle Sullivan User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.24) Gecko/20100301 SeaMonkey/1.1.19 To: "freebsd-fs@freebsd.org" Subject: ZFS pool faulted (corrupt metadata) but the disk data appears ok... X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Feb 2015 22:09:04 -0000 Any clues on this? root@colossus:~ # zpool import pool: storage id: 10618504954404185222 state: FAULTED status: The pool metadata is corrupted. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: http://illumos.org/msg/ZFS-8000-72 config: storage FAULTED corrupted data raidz2-0 ONLINE mfid0 ONLINE mfid15 ONLINE mfid1 ONLINE mfid2 ONLINE mfid3 ONLINE mfid4 ONLINE mfid5 ONLINE replacing-7 ONLINE mfid13 ONLINE mfid14 ONLINE mfid6 ONLINE mfid7 ONLINE mfid8 ONLINE mfid9 ONLINE mfid10 ONLINE mfid11 ONLINE mfid12 ONLINE root@colossus:~ # zpool import -Ff storage cannot import 'storage': I/O error Destroy and re-create the pool from a backup source. root@colossus:~ # zdb -l /dev/mfid0 -------------------------------------------- LABEL 0 -------------------------------------------- version: 5000 name: 'storage' state: 0 txg: 1928241 pool_guid: 10618504954404185222 hostid: 4203774842 hostname: 'colossus' top_guid: 12489400212295803034 guid: 3998695725653225547 vdev_children: 1 vdev_tree: type: 'raidz' id: 0 guid: 12489400212295803034 nparity: 2 metaslab_array: 34 metaslab_shift: 38 ashift: 9 asize: 45000449064960 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 3998695725653225547 path: '/dev/mfid0' phys_path: '/dev/mfid0' whole_disk: 1 DTL: 168 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 10795471632546545577 path: '/dev/mfid1' phys_path: '/dev/mfid1' whole_disk: 1 DTL: 167 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 15820272272734706674 path: '/dev/mfid2' phys_path: '/dev/mfid2' whole_disk: 1 DTL: 166 create_txg: 4 children[3]: type: 'disk' id: 3 guid: 3928579496187019848 path: '/dev/mfid3' phys_path: '/dev/mfid3' whole_disk: 1 DTL: 165 create_txg: 4 children[4]: type: 'disk' id: 4 guid: 7125052278051590304 path: '/dev/mfid4' phys_path: '/dev/mfid4' whole_disk: 1 DTL: 164 create_txg: 4 children[5]: type: 'disk' id: 5 guid: 14370198745088794709 path: '/dev/mfid5' phys_path: '/dev/mfid5' whole_disk: 1 DTL: 163 create_txg: 4 children[6]: type: 'disk' id: 6 guid: 1843597351388951655 path: '/dev/mfid6' phys_path: '/dev/mfid6' whole_disk: 1 DTL: 162 create_txg: 4 children[7]: type: 'replacing' id: 7 guid: 2914889727426054645 whole_disk: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 10956220251832269421 path: '/dev/mfid15' phys_path: '/dev/mfid15' whole_disk: 1 DTL: 179 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 2463756237300743131 path: '/dev/mfid13' phys_path: '/dev/mfid13' whole_disk: 1 DTL: 181 create_txg: 4 resilvering: 1 children[8]: type: 'disk' id: 8 guid: 8864096842672670007 path: '/dev/mfid7' phys_path: '/dev/mfid7' whole_disk: 1 DTL: 160 create_txg: 4 children[9]: type: 'disk' id: 9 guid: 4650681673751655245 path: '/dev/mfid8' phys_path: '/dev/mfid8' whole_disk: 1 DTL: 159 create_txg: 4 children[10]: type: 'disk' id: 10 guid: 8432109430432996813 path: '/dev/mfid9' phys_path: '/dev/mfid9' whole_disk: 1 DTL: 158 create_txg: 4 children[11]: type: 'disk' id: 11 guid: 414941847968750824 path: '/dev/mfid10' phys_path: '/dev/mfid10' whole_disk: 1 DTL: 157 create_txg: 4 children[12]: type: 'disk' id: 12 guid: 7335375930620195352 path: '/dev/mfid11' phys_path: '/dev/mfid11' whole_disk: 1 DTL: 156 create_txg: 4 children[13]: type: 'disk' id: 13 guid: 5100737174610362 path: '/dev/mfid12' phys_path: '/dev/mfid12' whole_disk: 1 DTL: 155 create_txg: 4 children[14]: type: 'disk' id: 14 guid: 15695558693726858796 path: '/dev/mfid14' phys_path: '/dev/mfid14' whole_disk: 1 DTL: 174 create_txg: 4 features_for_read: -------------------------------------------- LABEL 1 -------------------------------------------- version: 5000 name: 'storage' state: 0 txg: 1928241 pool_guid: 10618504954404185222 hostid: 4203774842 hostname: 'colossus' top_guid: 12489400212295803034 guid: 3998695725653225547 vdev_children: 1 vdev_tree: type: 'raidz' id: 0 guid: 12489400212295803034 nparity: 2 metaslab_array: 34 metaslab_shift: 38 ashift: 9 asize: 45000449064960 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 3998695725653225547 path: '/dev/mfid0' phys_path: '/dev/mfid0' whole_disk: 1 DTL: 168 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 10795471632546545577 path: '/dev/mfid1' phys_path: '/dev/mfid1' whole_disk: 1 DTL: 167 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 15820272272734706674 path: '/dev/mfid2' phys_path: '/dev/mfid2' whole_disk: 1 DTL: 166 create_txg: 4 children[3]: type: 'disk' id: 3 guid: 3928579496187019848 path: '/dev/mfid3' phys_path: '/dev/mfid3' whole_disk: 1 DTL: 165 create_txg: 4 children[4]: type: 'disk' id: 4 guid: 7125052278051590304 path: '/dev/mfid4' phys_path: '/dev/mfid4' whole_disk: 1 DTL: 164 create_txg: 4 children[5]: type: 'disk' id: 5 guid: 14370198745088794709 path: '/dev/mfid5' phys_path: '/dev/mfid5' whole_disk: 1 DTL: 163 create_txg: 4 children[6]: type: 'disk' id: 6 guid: 1843597351388951655 path: '/dev/mfid6' phys_path: '/dev/mfid6' whole_disk: 1 DTL: 162 create_txg: 4 children[7]: type: 'replacing' id: 7 guid: 2914889727426054645 whole_disk: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 10956220251832269421 path: '/dev/mfid15' phys_path: '/dev/mfid15' whole_disk: 1 DTL: 179 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 2463756237300743131 path: '/dev/mfid13' phys_path: '/dev/mfid13' whole_disk: 1 DTL: 181 create_txg: 4 resilvering: 1 children[8]: type: 'disk' id: 8 guid: 8864096842672670007 path: '/dev/mfid7' phys_path: '/dev/mfid7' whole_disk: 1 DTL: 160 create_txg: 4 children[9]: type: 'disk' id: 9 guid: 4650681673751655245 path: '/dev/mfid8' phys_path: '/dev/mfid8' whole_disk: 1 DTL: 159 create_txg: 4 children[10]: type: 'disk' id: 10 guid: 8432109430432996813 path: '/dev/mfid9' phys_path: '/dev/mfid9' whole_disk: 1 DTL: 158 create_txg: 4 children[11]: type: 'disk' id: 11 guid: 414941847968750824 path: '/dev/mfid10' phys_path: '/dev/mfid10' whole_disk: 1 DTL: 157 create_txg: 4 children[12]: type: 'disk' id: 12 guid: 7335375930620195352 path: '/dev/mfid11' phys_path: '/dev/mfid11' whole_disk: 1 DTL: 156 create_txg: 4 children[13]: type: 'disk' id: 13 guid: 5100737174610362 path: '/dev/mfid12' phys_path: '/dev/mfid12' whole_disk: 1 DTL: 155 create_txg: 4 children[14]: type: 'disk' id: 14 guid: 15695558693726858796 path: '/dev/mfid14' phys_path: '/dev/mfid14' whole_disk: 1 DTL: 174 create_txg: 4 features_for_read: -------------------------------------------- LABEL 2 -------------------------------------------- version: 5000 name: 'storage' state: 0 txg: 1928241 pool_guid: 10618504954404185222 hostid: 4203774842 hostname: 'colossus' top_guid: 12489400212295803034 guid: 3998695725653225547 vdev_children: 1 vdev_tree: type: 'raidz' id: 0 guid: 12489400212295803034 nparity: 2 metaslab_array: 34 metaslab_shift: 38 ashift: 9 asize: 45000449064960 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 3998695725653225547 path: '/dev/mfid0' phys_path: '/dev/mfid0' whole_disk: 1 DTL: 168 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 10795471632546545577 path: '/dev/mfid1' phys_path: '/dev/mfid1' whole_disk: 1 DTL: 167 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 15820272272734706674 path: '/dev/mfid2' phys_path: '/dev/mfid2' whole_disk: 1 DTL: 166 create_txg: 4 children[3]: type: 'disk' id: 3 guid: 3928579496187019848 path: '/dev/mfid3' phys_path: '/dev/mfid3' whole_disk: 1 DTL: 165 create_txg: 4 children[4]: type: 'disk' id: 4 guid: 7125052278051590304 path: '/dev/mfid4' phys_path: '/dev/mfid4' whole_disk: 1 DTL: 164 create_txg: 4 children[5]: type: 'disk' id: 5 guid: 14370198745088794709 path: '/dev/mfid5' phys_path: '/dev/mfid5' whole_disk: 1 DTL: 163 create_txg: 4 children[6]: type: 'disk' id: 6 guid: 1843597351388951655 path: '/dev/mfid6' phys_path: '/dev/mfid6' whole_disk: 1 DTL: 162 create_txg: 4 children[7]: type: 'replacing' id: 7 guid: 2914889727426054645 whole_disk: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 10956220251832269421 path: '/dev/mfid15' phys_path: '/dev/mfid15' whole_disk: 1 DTL: 179 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 2463756237300743131 path: '/dev/mfid13' phys_path: '/dev/mfid13' whole_disk: 1 DTL: 181 create_txg: 4 resilvering: 1 children[8]: type: 'disk' id: 8 guid: 8864096842672670007 path: '/dev/mfid7' phys_path: '/dev/mfid7' whole_disk: 1 DTL: 160 create_txg: 4 children[9]: type: 'disk' id: 9 guid: 4650681673751655245 path: '/dev/mfid8' phys_path: '/dev/mfid8' whole_disk: 1 DTL: 159 create_txg: 4 children[10]: type: 'disk' id: 10 guid: 8432109430432996813 path: '/dev/mfid9' phys_path: '/dev/mfid9' whole_disk: 1 DTL: 158 create_txg: 4 children[11]: type: 'disk' id: 11 guid: 414941847968750824 path: '/dev/mfid10' phys_path: '/dev/mfid10' whole_disk: 1 DTL: 157 create_txg: 4 children[12]: type: 'disk' id: 12 guid: 7335375930620195352 path: '/dev/mfid11' phys_path: '/dev/mfid11' whole_disk: 1 DTL: 156 create_txg: 4 children[13]: type: 'disk' id: 13 guid: 5100737174610362 path: '/dev/mfid12' phys_path: '/dev/mfid12' whole_disk: 1 DTL: 155 create_txg: 4 children[14]: type: 'disk' id: 14 guid: 15695558693726858796 path: '/dev/mfid14' phys_path: '/dev/mfid14' whole_disk: 1 DTL: 174 create_txg: 4 features_for_read: -------------------------------------------- LABEL 3 -------------------------------------------- version: 5000 name: 'storage' state: 0 txg: 1928241 pool_guid: 10618504954404185222 hostid: 4203774842 hostname: 'colossus' top_guid: 12489400212295803034 guid: 3998695725653225547 vdev_children: 1 vdev_tree: type: 'raidz' id: 0 guid: 12489400212295803034 nparity: 2 metaslab_array: 34 metaslab_shift: 38 ashift: 9 asize: 45000449064960 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 3998695725653225547 path: '/dev/mfid0' phys_path: '/dev/mfid0' whole_disk: 1 DTL: 168 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 10795471632546545577 path: '/dev/mfid1' phys_path: '/dev/mfid1' whole_disk: 1 DTL: 167 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 15820272272734706674 path: '/dev/mfid2' phys_path: '/dev/mfid2' whole_disk: 1 DTL: 166 create_txg: 4 children[3]: type: 'disk' id: 3 guid: 3928579496187019848 path: '/dev/mfid3' phys_path: '/dev/mfid3' whole_disk: 1 DTL: 165 create_txg: 4 children[4]: type: 'disk' id: 4 guid: 7125052278051590304 path: '/dev/mfid4' phys_path: '/dev/mfid4' whole_disk: 1 DTL: 164 create_txg: 4 children[5]: type: 'disk' id: 5 guid: 14370198745088794709 path: '/dev/mfid5' phys_path: '/dev/mfid5' whole_disk: 1 DTL: 163 create_txg: 4 children[6]: type: 'disk' id: 6 guid: 1843597351388951655 path: '/dev/mfid6' phys_path: '/dev/mfid6' whole_disk: 1 DTL: 162 create_txg: 4 children[7]: type: 'replacing' id: 7 guid: 2914889727426054645 whole_disk: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 10956220251832269421 path: '/dev/mfid15' phys_path: '/dev/mfid15' whole_disk: 1 DTL: 179 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 2463756237300743131 path: '/dev/mfid13' phys_path: '/dev/mfid13' whole_disk: 1 DTL: 181 create_txg: 4 resilvering: 1 children[8]: type: 'disk' id: 8 guid: 8864096842672670007 path: '/dev/mfid7' phys_path: '/dev/mfid7' whole_disk: 1 DTL: 160 create_txg: 4 children[9]: type: 'disk' id: 9 guid: 4650681673751655245 path: '/dev/mfid8' phys_path: '/dev/mfid8' whole_disk: 1 DTL: 159 create_txg: 4 children[10]: type: 'disk' id: 10 guid: 8432109430432996813 path: '/dev/mfid9' phys_path: '/dev/mfid9' whole_disk: 1 DTL: 158 create_txg: 4 children[11]: type: 'disk' id: 11 guid: 414941847968750824 path: '/dev/mfid10' phys_path: '/dev/mfid10' whole_disk: 1 DTL: 157 create_txg: 4 children[12]: type: 'disk' id: 12 guid: 7335375930620195352 path: '/dev/mfid11' phys_path: '/dev/mfid11' whole_disk: 1 DTL: 156 create_txg: 4 children[13]: type: 'disk' id: 13 guid: 5100737174610362 path: '/dev/mfid12' phys_path: '/dev/mfid12' whole_disk: 1 DTL: 155 create_txg: 4 children[14]: type: 'disk' id: 14 guid: 15695558693726858796 path: '/dev/mfid14' phys_path: '/dev/mfid14' whole_disk: 1 DTL: 174 create_txg: 4 features_for_read: LSI9260-16i raid controller all drives in single disk RAID0, system wasn't busy (=in use for writing, however was resilvering a replacement drive after the HSP had taken over due to failed drive) and it got rebooted... faulted on reboot. Recoverable? Regards, -- Michelle Sullivan http://www.mhix.org/