From owner-freebsd-fs@FreeBSD.ORG Sun Oct 4 21:22:44 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E19F81065707 for ; Sun, 4 Oct 2009 21:22:43 +0000 (UTC) (envelope-from aaron@goflexitllc.com) Received: from mail.goflexitllc.com (mail.goflexitllc.com [70.38.81.12]) by mx1.freebsd.org (Postfix) with ESMTP id 6E35A8FC16 for ; Sun, 4 Oct 2009 21:22:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=goflexitllc.com; h=message-id:date:from:mime-version:to:cc:subject:references :in-reply-to:content-type; s=zeta; bh=oNEbOcTSndoEiixECZvuGfKqDb k=; b=WJhJEEbOQR2Mi2OrX9qaM7E6uqN4kceaVvScaWU7gM1MttrUBJWOvKclWZ nToe/vWCcFl8yv7l7WWOzC7PzGi7inTsdCXw5E6lOKFJD12UAUvsGACIG/cojmjD bueV0V DomainKey-Signature: a=rsa-sha1; c=nofws; d=goflexitllc.com; h= message-id:date:from:mime-version:to:cc:subject:references :in-reply-to:content-type; q=dns; s=zeta; b=DdQFd1zrEbDIEzkrAY3I 1i4nIEEjf5Nkw3ysnl8gRKnmfFXS9XTOewSB2WggPHX14uuRO6aAX3X887THRD/g JuOJHSlGRr+W15i2G3MXW30B8Yxby0yMrZDiIQdo6plh Received: (qmail 65227 invoked by uid 89); 4 Oct 2009 21:22:40 -0000 Received: (simscan 1.4.1 ppid 65219 pid 65224 t 0.6505s) (scanners: regex: 1.4.1 attach: 1.4.1 clamav: 0.95.2/m:51/d:9840); 04 Oct 0109 21:22:39 -0000 Received: from temp4.wavelinx.net (HELO ?172.16.1.128?) (aaron@goflexitllc.com@69.27.151.4) by mail.goflexitllc.com with ESMTPA; 4 Oct 2009 21:22:39 -0000 Message-ID: <4AC91216.9070200@goflexitllc.com> Date: Sun, 04 Oct 2009 16:22:30 -0500 From: Aaron Hurt User-Agent: Thunderbird 2.0.0.22 (X11/20090719) MIME-Version: 1.0 To: Artem Belevich References: <683849754.20091001110503@pyro.de> <20091004174746.GF1660@garage.freebsd.pl> <4AC8FFE9.90606@goflexitllc.com> In-Reply-To: Content-Type: multipart/mixed; boundary="------------050008030608050201080905" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, Pawel Jakub Dawidek Subject: Re: Help needed! ZFS I/O error recovery? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2009 21:22:44 -0000 This is a multi-part message in MIME format. --------------050008030608050201080905 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Artem Belevich wrote: > Similar issue has been discussed on various zfs-related lists. > > Following post contains number of useful links on the issue itself and > the ways to recover. > http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg26704.html > > --Artem > Thank you for the links, I read through them and read victor's post...however, I am not even able to get the command he mentioned to start on my particular machine: schroeder# zdb -bbccsv 5666150306755460943 zdb: can't open 5666150306755460943: Input/output error I do have some more hope than before after reading those threads, but I am unsure of how to proceed past this point. Not sure if it will help but here is the zdb -ll for all the devices in this pool. schroeder# zdb -ll /dev/ad4s1 -------------------------------------------- LABEL 0 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=14725838678957352699 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 1 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=14725838678957352699 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 2 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=14725838678957352699 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 3 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=14725838678957352699 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 schroeder# zdb -ll /dev/ad6s1 -------------------------------------------- LABEL 0 -------------------------------------------- version=13 name='tank0' state=0 txg=478204 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=14258118189710079071 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 1 -------------------------------------------- version=13 name='tank0' state=0 txg=478204 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=14258118189710079071 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 2 -------------------------------------------- version=13 name='tank0' state=0 txg=478204 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=14258118189710079071 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 3 -------------------------------------------- version=13 name='tank0' state=0 txg=478204 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=14258118189710079071 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 schroeder# zdb -ll /dev/ad8s1 -------------------------------------------- LABEL 0 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=3285817555281532863 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 1 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=3285817555281532863 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 2 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=3285817555281532863 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 3 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=3285817555281532863 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 schroeder# zdb -ll /dev/ad10s1 -------------------------------------------- LABEL 0 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=4760880554491391388 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 1 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=4760880554491391388 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 2 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=4760880554491391388 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 -------------------------------------------- LABEL 3 -------------------------------------------- version=13 name='tank0' state=0 txg=478240 pool_guid=5666150306755460943 hostid=2440235366 hostname='unset' top_guid=2037719367177844310 guid=4760880554491391388 vdev_tree type='raidz' id=0 guid=2037719367177844310 nparity=1 metaslab_array=23 metaslab_shift=32 ashift=9 asize=1600332496896 is_log=0 children[0] type='disk' id=0 guid=14725838678957352699 path='/dev/ad4s1' whole_disk=0 DTL=323 children[1] type='disk' id=1 guid=14258118189710079071 path='/dev/ad6s1' whole_disk=0 DTL=327 removed=1 children[2] type='disk' id=2 guid=3285817555281532863 path='/dev/ad8s1' whole_disk=0 DTL=326 children[3] type='disk' id=3 guid=4760880554491391388 path='/dev/ad10s1' whole_disk=0 DTL=325 schroeder# Thank You, Aaron --------------050008030608050201080905--