From owner-freebsd-fs@FreeBSD.ORG Sat Oct 25 03:47:44 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BBA643B7; Sat, 25 Oct 2014 03:47:44 +0000 (UTC) Received: from mail-vc0-x22f.google.com (mail-vc0-x22f.google.com [IPv6:2607:f8b0:400c:c03::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 65A5D920; Sat, 25 Oct 2014 03:47:44 +0000 (UTC) Received: by mail-vc0-f175.google.com with SMTP id id10so921732vcb.34 for ; Fri, 24 Oct 2014 20:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=0BnmbHrX/Kv7dQq2OxzsMW3F+oLauUCmo0wzo8PXHmc=; b=SIuMNvb31uqKWYArdO0v9AwPrx44etjqwYgmfZc6SNZRRAUHtU754uXK8cGyOOrShm YjDv9gJAYjTkhRE5/1jFVritYQ0Pn8eH06H6TeLwnzfW/De/J2W9iLzj96ZmzR3rJAlY GYG68eeex6LQlixTmO87OFpqvoX56PzmPwkgPiZvNTFT1AGN1OmLwr+2Nv+9SwCNy8p9 Mzj8ZcCSP97+KgjcizR5eyrVFgJhC2RSdOhUUvdU9Oui8LDQ/5BCDYNuqSvjcWdU/blD 0RP6PIcdsE7mXqkCKr5lkLY+7wZXQ4cgdnKZ5buzHjShPSLJvOaQPWj8XTWnWRaxQQE3 y6Xg== MIME-Version: 1.0 X-Received: by 10.52.121.73 with SMTP id li9mr3236351vdb.34.1414208863229; Fri, 24 Oct 2014 20:47:43 -0700 (PDT) Received: by 10.220.238.14 with HTTP; Fri, 24 Oct 2014 20:47:43 -0700 (PDT) In-Reply-To: <544B12B8.8060302@freebsd.org> References: <544B12B8.8060302@freebsd.org> Date: Fri, 24 Oct 2014 23:47:43 -0400 Message-ID: Subject: Re: ZFS errors on the array but not the disk. From: Zaphod Beeblebrox To: Steven Hartland Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 03:47:44 -0000 Thanks for the heads up. I'm following releng/10.1 and 271683 seems to be part of that, but a good catch/guess. On Fri, Oct 24, 2014 at 11:02 PM, Steven Hartland wrote: > There was an issue which would cause resilver restarts fixed by *265253* < > https://svnweb.freebsd.org/base?view=revision&revision=265253> which was > MFC'ed to stable/10 by *271683* base?view=revision&revision=271683>so you'll want to make sure your > latter than that. > > > On 24/10/2014 19:42, Zaphod Beeblebrox wrote: > >> I manually replaced a disk... and the array was scrubbed recently. >> Interestingly, I seem to be in the "endless loop" of resilvering problem. >> Not much I can find on it. but resilvering will complete and I can then >> run another scrub. It will complete, too. Then rebooting causes another >> resilvering. >> >> Another odd data point: it seems as if the things that show up as "errors" >> change from resilvering to resilvering. >> >> One bug, it would seem, is that once ZFS has detected an error... another >> scrub can reset it, but no attempt is made to read-through the error if >> you >> access the object directly. >> >> On Fri, Oct 24, 2014 at 11:33 AM, Alan Somers >> wrote: >> >> On Thu, Oct 23, 2014 at 11:37 PM, Zaphod Beeblebrox >>> wrote: >>> >>>> What does it mean when checksum errors appear on the array (and the >>>> vdev) >>>> but not on any of the disks? See the paste below. One would think that >>>> there isn't some ephemeral data stored somewhere that is not one of the >>>> disks, yet "cksum" errors show only on the vdev and the array lines. >>>> >>> Help? >>> >>>> [2:17:316]root@virtual:/vr2/torrent/in> zpool status >>>> pool: vr2 >>>> state: ONLINE >>>> status: One or more devices is currently being resilvered. The pool >>>> will >>>> continue to function, possibly in a degraded state. >>>> action: Wait for the resilver to complete. >>>> scan: resilver in progress since Thu Oct 23 23:11:29 2014 >>>> 1.53T scanned out of 22.6T at 62.4M/s, 98h23m to go >>>> 119G resilvered, 6.79% done >>>> config: >>>> >>>> NAME STATE READ WRITE CKSUM >>>> vr2 ONLINE 0 0 36 >>>> raidz1-0 ONLINE 0 0 72 >>>> label/vr2-d0 ONLINE 0 0 0 >>>> label/vr2-d1 ONLINE 0 0 0 >>>> gpt/vr2-d2c ONLINE 0 0 0 block size: 512B >>>> configured, 4096B native (resilvering) >>>> gpt/vr2-d3b ONLINE 0 0 0 block size: 512B >>>> configured, 4096B native >>>> gpt/vr2-d4a ONLINE 0 0 0 block size: 512B >>>> configured, 4096B native >>>> ada14 ONLINE 0 0 0 >>>> label/vr2-d6 ONLINE 0 0 0 >>>> label/vr2-d7c ONLINE 0 0 0 >>>> label/vr2-d8 ONLINE 0 0 0 >>>> raidz1-1 ONLINE 0 0 0 >>>> gpt/vr2-e0 ONLINE 0 0 0 block size: 512B >>>> configured, 4096B native >>>> gpt/vr2-e1 ONLINE 0 0 0 block size: 512B >>>> configured, 4096B native >>>> gpt/vr2-e2 ONLINE 0 0 0 block size: 512B >>>> configured, 4096B native >>>> gpt/vr2-e3 ONLINE 0 0 0 >>>> gpt/vr2-e4 ONLINE 0 0 0 block size: 512B >>>> configured, 4096B native >>>> gpt/vr2-e5 ONLINE 0 0 0 block size: 512B >>>> configured, 4096B native >>>> gpt/vr2-e6 ONLINE 0 0 0 block size: 512B >>>> configured, 4096B native >>>> gpt/vr2-e7 ONLINE 0 0 0 block size: 512B >>>> configured, 4096B native >>>> >>>> errors: 43 data errors, use '-v' for a list >>>> >>> The checksum errors will appear on the raidz vdev instead of a leaf if >>> vdev_raidz.c can't determine which leaf vdev was responsible. This >>> could happen if two or more leaf vdevs return bad data for the same >>> block, which would also lead to unrecoverable data errors. I see that >>> you have some unrecoverable data errors, so maybe that's what happened >>> to you. >>> >>> Subtle design bugs in ZFS can also lead to vdev_raidz.c being unable >>> to determine which child was responsible for a checksum error. >>> However, I've only seen that happen when a raidz vdev has a mirror >>> child. That can only happen if the child is a spare or replacing >>> vdev. Did you activate any spares, or did you manually replace a >>> vdev? >>> >>> -Alan >>> >>> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >> >> >> > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >