From owner-freebsd-fs@FreeBSD.ORG Sun Jan 24 05:16:02 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 68F2E106566B for ; Sun, 24 Jan 2010 05:16:02 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-qy0-f201.google.com (mail-qy0-f201.google.com [209.85.221.201]) by mx1.freebsd.org (Postfix) with ESMTP id 1435C8FC16 for ; Sun, 24 Jan 2010 05:16:01 +0000 (UTC) Received: by qyk39 with SMTP id 39so1372413qyk.27 for ; Sat, 23 Jan 2010 21:16:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:date:from:to:cc :subject:in-reply-to:message-id:references:user-agent :x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; bh=wiixVMc3sBnB2SwTNDLUDFPD/zmjop0gwcwkgOF2hiQ=; b=p+JL+oeX/8yr03WsHng35ByiXwTLIq2nH/LEYFJdPUYiswBbhCOPWOyiRLrlKALhMf BRGoa4T4FqgF8Koc+Y4iyFlzND+VCwfB0F6SMtArvW9BVjmLPa1V+F0Jrorm4aN8eQDZ Qt6DEI3fJ6By/sMjpC+4ftgfek9GysE6yI9Hk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; b=PCZuImhqRUEfqhUoVBLh/Xsu3J0qoMURvpQprBVHd+/Cj6QGxMgG5lyAG6LIZmpadG /X05db/m2nzGovIfs7Eyo8iyIRcum3SwZXfE0jkZQt66mt9szRpLaGdY2H4hKwZn4+fU zPrkawR1kjgm4a1advT1Ofo7y/uF9YyR9B8no= Received: by 10.224.39.149 with SMTP id g21mr3243034qae.52.1264310161402; Sat, 23 Jan 2010 21:16:01 -0800 (PST) Received: from centel.dataix.local (ppp-21.4.dialinfree.com [209.172.21.4]) by mx.google.com with ESMTPS id 21sm3372677qyk.12.2010.01.23.21.15.51 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sat, 23 Jan 2010 21:15:55 -0800 (PST) Sender: "J. Hellenthal" Date: Sun, 24 Jan 2010 00:15:16 -0500 From: jhell To: Rich In-Reply-To: <5da0588e1001232017m6c67731fwaa1d71cd86800017@mail.gmail.com> Message-ID: References: <5da0588e1001222223m773648am907267235bdcf882@mail.gmail.com> <5da0588e1001231415t403f29ceq6e8dcd16edb4a28@mail.gmail.com> <5da0588e1001231541l246769eao410c5ea6ccca0de4@mail.gmail.com> <5da0588e1001231615t37c22575uedaae938be40f530@mail.gmail.com> <4B5B94B8.7070509@modulus.org> <5da0588e1001231638i349f8f17t297e970b08825441@mail.gmail.com> <5da0588e1001232017m6c67731fwaa1d71cd86800017@mail.gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: freebsd-fs@freebsd.org Subject: Re: Errors on a file on a zpool: How to remove? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jan 2010 05:16:02 -0000 On Sat, 23 Jan 2010 23:17, rincebrain@ wrote: >> Can you try the following please and then report back. >> >> Setting failmode to continue might let you continue a removal. >> zpool set failmode=continue what_is_it_rigatoni? >> >> This should be inherited by effected drives. >> zfs set checksum=off what_is_it_rigatoni? >> >> Remove the said effected files at this point. > > [root@manticore ~]# zpool set failmode=continue rigatoni > [root@manticore ~]# zfs set checksum=off rigatoni > [root@manticore ~]# mv > /mirrors/geocities/from-asheesh-home/swebb3/www.geocities.com/JMcgo1979 > /mirrors/geocities/from-asheesh-home/swebb3/www.geocities.com/JMcgo19791 > mv: rename /mirrors/geocities/from-asheesh-home/swebb3/www.geocities.com/JMcgo1979 > to /mirrors/geocities/from-asheesh-home/swebb3/www.geocities.com/JMcgo19791: > Input/output error > [root@manticore ~]# rm -f > /mirrors/geocities/from-asheesh-home/swebb3/www.geocities.com/JMcgo1979 > rm: /mirrors/geocities/from-asheesh-home/swebb3/www.geocities.com/JMcgo1979: > Input/output error > > :/ > > - Rich > I had my fingers crossed for you to... :( >From what I see and what was already mentioned earlier in this thread is meta data corruption but the checksum errors do not span across the whole pool of vdevs. These are, correct me if I am wrong USB mass storage devices ? SSD ? In the arrangement of the devices on the system are da2,4,5 on the same hub and da6,7 on another ? If this is the case you may have consolidated your errors down to being a USB problem and narrowed down to where they are connected to. What happened to da1,3 ? Were these once connected to the system ? and if so did you start noticing this problem occur roughly about the same period they were removed ? Will zpool(8) allow a removal of a vdevs in operation and copy the data to the leftover vdevs if there is free space ? As in: (maybe possibly with -f) zpool remove da2 zpool remove da4 zpool remove da5 Maybe if it is a possibility, replace each with another daN device if one is available and continue operations from this point to see if you can most likely rm the files in question as they are unlikely to be recoverable. Fingers crossed, -- jhell