From owner-freebsd-fs@FreeBSD.ORG Sun Jan 24 05:51:58 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F3331065672 for ; Sun, 24 Jan 2010 05:51:58 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-qy0-f201.google.com (mail-qy0-f201.google.com [209.85.221.201]) by mx1.freebsd.org (Postfix) with ESMTP id EC21A8FC0A for ; Sun, 24 Jan 2010 05:51:57 +0000 (UTC) Received: by qyk39 with SMTP id 39so1379227qyk.27 for ; Sat, 23 Jan 2010 21:51:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:date:from:to:cc :subject:in-reply-to:message-id:references:user-agent :x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; bh=cSCoQyGyhPlNcbByJePmL188GZTugxmrzrJuU9zMcoQ=; b=GKQFCtehSDrwKXJxC//g8gRZMgg0qAMwGtd4VtkqAf/U6c9LCkJPixmuMsl43dTDc3 tEMR4tAWtNP1PduTmnZdy6C+FZ/fbGBuMm+tMMxf/d5lj3uZRdrUNSPgd+HAEJ5gU7ex CsKWXj70UlG71vG3AIF0rTJb7PwOnEvbPaOh0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:date:from:to:cc:subject:in-reply-to:message-id:references :user-agent:x-openpgp-key-id:x-openpgp-key-fingerprint:mime-version :content-type; b=yEOTn5WbgsDxalBrwK3oPchmWr/RwTPfVECQRc7KlphgHGFHW6p6zKCCj9FlAnuQnX SZKdyOK59NU2tZsYQuGduBRW7H8jfZbLKK3zz5Ey6KAaX9LIqJIIVETAuI5eF9xNS8zx HCQoStF+xn6+vYypTpy5/HvGam35KIHB27dMs= Received: by 10.224.81.204 with SMTP id y12mr3208857qak.358.1264312317178; Sat, 23 Jan 2010 21:51:57 -0800 (PST) Received: from centel.dataix.local (ppp-21.4.dialinfree.com [209.172.21.4]) by mx.google.com with ESMTPS id 20sm1540083qyk.13.2010.01.23.21.51.35 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sat, 23 Jan 2010 21:51:56 -0800 (PST) Sender: "J. Hellenthal" Date: Sun, 24 Jan 2010 00:51:13 -0500 From: jhell To: Rich In-Reply-To: <5da0588e1001232128w5a551674od0805c2ff0b884ad@mail.gmail.com> Message-ID: References: <5da0588e1001222223m773648am907267235bdcf882@mail.gmail.com> <5da0588e1001231541l246769eao410c5ea6ccca0de4@mail.gmail.com> <5da0588e1001231615t37c22575uedaae938be40f530@mail.gmail.com> <4B5B94B8.7070509@modulus.org> <5da0588e1001231638i349f8f17t297e970b08825441@mail.gmail.com> <5da0588e1001232017m6c67731fwaa1d71cd86800017@mail.gmail.com> <5da0588e1001232128w5a551674od0805c2ff0b884ad@mail.gmail.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-OpenPGP-Key-Id: 0x89D8547E X-OpenPGP-Key-Fingerprint: 85EF E26B 07BB 3777 76BE B12A 9057 8789 89D8 547E MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-fs@freebsd.org Subject: Re: Errors on a file on a zpool: How to remove? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jan 2010 05:51:58 -0000 On Sun, 24 Jan 2010 00:28, rincebrain@ wrote: > On Sun, Jan 24, 2010 at 12:15 AM, jhell wrote: >> From what I see and what was already mentioned earlier in this thread is >> meta data corruption but the checksum errors do not span across the whole >> pool of vdevs. These are, correct me if I am wrong USB mass storage devices >> ? SSD ? > > 1.5T Seagate 7200RPM drives. > >> In the arrangement of the devices on the system are da2,4,5 on the same hub >> and da6,7 on another ? If this is the case you may have consolidated your >> errors down to being a USB problem and narrowed down to where they are >> connected to. > > ...no. > > All five are on the same SATA controller. These behaviors persist > independent of which SATA controller they are plugged into, and I've > tried all seven in the machine. > >> What happened to da1,3 ? Were these once connected to the system ? and if so >> did you start noticing this problem occur roughly about the same period they >> were removed ? > > da1,3 are being used in another disk pool, and were never a part of this pool. > > This is not an issue of a faulty SATA controller or SATA drives. > > This is an issue of "there was a single faulty stick of RAM in the machine". > Yeah I read this earlier, My apologies it slipped while I was writing "mind went into multi-write single read mode". > I have sixteen disks in this machine. These three are having issues > only on these particular files, and only on these files, not on random > portions of the disk. The disks never report read errors - the ZFS > layer is what reports them. SMART is not reporting any difficulties in > reading any sectors of these disks. > > > I could be mistaken, but I do not believe there to be a faulty > controller in play at this time. I've rotated the drives among the > spares of the 24 ports on the SATA controller in question, as well as > the on-motherboard controller, and this behavior has persisted. > > - Rich > As I was thinking earlier... you mentioned you scrubbed multiple times with no difference. When I was mentioning the attempt to remove/replace I was thinking this will cause a "re-silvering" of the drives possibly fixing meta-data for the effected disks if good meta-data still exists somewhere. Might be worth a shot but I would start with the replace of the devices that are showing the errors until you can clear the errors successfully without them showing up again and/or until you have replaced all disks. Best of luck. -- jhell