From owner-freebsd-fs@FreeBSD.ORG Fri Oct 24 18:42:35 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2B689AB7; Fri, 24 Oct 2014 18:42:35 +0000 (UTC) Received: from mail-vc0-x22f.google.com (mail-vc0-x22f.google.com [IPv6:2607:f8b0:400c:c03::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B9236C6B; Fri, 24 Oct 2014 18:42:34 +0000 (UTC) Received: by mail-vc0-f175.google.com with SMTP id id10so577411vcb.6 for ; Fri, 24 Oct 2014 11:42:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=yacKNvcBnyT3CZOWf0Z7ax56esPKFstuCx7hDfawYfI=; b=bjkpgzh4wtQg5hKzfMigHqNXmQWQSwdfsPV/pAXVkUf8O4iUCfDm4/EgTjbc68MRZW EJ+VyqNNMB5qXTUKrnK5FZX0LZQ+3pmV8phmqWUT5Ug0eyyYZhSZJxqnnRpFT4woz+1k le4NoRlCt3Qf6sdRNIOZy/xAcEEcNgPNv1uU+z499h0Ps6RawnT9Y9odu/XgotAygWpE XF0+Ws6L+PXw3j4jdGA5ZkXl2QW4YzB3PX93RFRcK75YCCvAxAjpMFRBO1imZ1pW244M Xi/Vrw3iKykBGgVceTMOhfnXX8rn2NvoaiyfiYWUAxYWq/gyuLBzz4PPqYWNYBlnrwqX mucg== MIME-Version: 1.0 X-Received: by 10.221.46.4 with SMTP id um4mr4068777vcb.23.1414176153132; Fri, 24 Oct 2014 11:42:33 -0700 (PDT) Received: by 10.220.238.14 with HTTP; Fri, 24 Oct 2014 11:42:32 -0700 (PDT) In-Reply-To: References: Date: Fri, 24 Oct 2014 14:42:32 -0400 Message-ID: Subject: Re: ZFS errors on the array but not the disk. From: Zaphod Beeblebrox To: Alan Somers Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: freebsd-fs , FreeBSD Hackers X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2014 18:42:35 -0000 I manually replaced a disk... and the array was scrubbed recently. Interestingly, I seem to be in the "endless loop" of resilvering problem. Not much I can find on it. but resilvering will complete and I can then run another scrub. It will complete, too. Then rebooting causes another resilvering. Another odd data point: it seems as if the things that show up as "errors" change from resilvering to resilvering. One bug, it would seem, is that once ZFS has detected an error... another scrub can reset it, but no attempt is made to read-through the error if you access the object directly. On Fri, Oct 24, 2014 at 11:33 AM, Alan Somers wrote: > On Thu, Oct 23, 2014 at 11:37 PM, Zaphod Beeblebrox > wrote: > > What does it mean when checksum errors appear on the array (and the vdev) > > but not on any of the disks? See the paste below. One would think that > > there isn't some ephemeral data stored somewhere that is not one of the > > disks, yet "cksum" errors show only on the vdev and the array lines. > Help? > > > > [2:17:316]root@virtual:/vr2/torrent/in> zpool status > > pool: vr2 > > state: ONLINE > > status: One or more devices is currently being resilvered. The pool will > > continue to function, possibly in a degraded state. > > action: Wait for the resilver to complete. > > scan: resilver in progress since Thu Oct 23 23:11:29 2014 > > 1.53T scanned out of 22.6T at 62.4M/s, 98h23m to go > > 119G resilvered, 6.79% done > > config: > > > > NAME STATE READ WRITE CKSUM > > vr2 ONLINE 0 0 36 > > raidz1-0 ONLINE 0 0 72 > > label/vr2-d0 ONLINE 0 0 0 > > label/vr2-d1 ONLINE 0 0 0 > > gpt/vr2-d2c ONLINE 0 0 0 block size: 512B > > configured, 4096B native (resilvering) > > gpt/vr2-d3b ONLINE 0 0 0 block size: 512B > > configured, 4096B native > > gpt/vr2-d4a ONLINE 0 0 0 block size: 512B > > configured, 4096B native > > ada14 ONLINE 0 0 0 > > label/vr2-d6 ONLINE 0 0 0 > > label/vr2-d7c ONLINE 0 0 0 > > label/vr2-d8 ONLINE 0 0 0 > > raidz1-1 ONLINE 0 0 0 > > gpt/vr2-e0 ONLINE 0 0 0 block size: 512B > > configured, 4096B native > > gpt/vr2-e1 ONLINE 0 0 0 block size: 512B > > configured, 4096B native > > gpt/vr2-e2 ONLINE 0 0 0 block size: 512B > > configured, 4096B native > > gpt/vr2-e3 ONLINE 0 0 0 > > gpt/vr2-e4 ONLINE 0 0 0 block size: 512B > > configured, 4096B native > > gpt/vr2-e5 ONLINE 0 0 0 block size: 512B > > configured, 4096B native > > gpt/vr2-e6 ONLINE 0 0 0 block size: 512B > > configured, 4096B native > > gpt/vr2-e7 ONLINE 0 0 0 block size: 512B > > configured, 4096B native > > > > errors: 43 data errors, use '-v' for a list > > The checksum errors will appear on the raidz vdev instead of a leaf if > vdev_raidz.c can't determine which leaf vdev was responsible. This > could happen if two or more leaf vdevs return bad data for the same > block, which would also lead to unrecoverable data errors. I see that > you have some unrecoverable data errors, so maybe that's what happened > to you. > > Subtle design bugs in ZFS can also lead to vdev_raidz.c being unable > to determine which child was responsible for a checksum error. > However, I've only seen that happen when a raidz vdev has a mirror > child. That can only happen if the child is a spare or replacing > vdev. Did you activate any spares, or did you manually replace a > vdev? > > -Alan >