From nobody Thu Dec 14 22:35:07 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SrnHD3lT7z53ssQ for ; Thu, 14 Dec 2023 22:35:20 +0000 (UTC) (envelope-from delphij@gmail.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SrnHD1QjFz3Fxj for ; Thu, 14 Dec 2023 22:35:20 +0000 (UTC) (envelope-from delphij@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-a22deb95d21so9381666b.3 for ; Thu, 14 Dec 2023 14:35:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1702593319; x=1703198119; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Jczh/1jsTAL5f7l/qRRSp44gHyLsW+bWHkFdY3cJsP8=; b=ADLniGOrECOeoO3L1lQpQ3lXYunLLh+QvSTOiuOxFpk8mshfWJHte4V13ekXPw3dJe SqhWH/742NkAcITtbHDStWAK9p2Ygh5TrvpURRTYfg9RchssT2rUC7lqtv3RVgFetZ7t H23sPTcl3DtD6rBUKwHeIUsxco+vsL9GXzqqQZ8M0PfZCJN58Swu7K5dFAm/XI75R7Bm JGoozNsG5Azsc03ZMbLJjEEL0xqQ+4DDziKwH5jT5YwfjwqdHiuk0OBZWqHzwYjLtxgU uRALNhlHPoj23uKWgUOiynOz8cWODVMUVSFgpX9Q0kM0Zgy8nUNLGcY2pPr+5T/gDk7d 1dQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702593319; x=1703198119; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Jczh/1jsTAL5f7l/qRRSp44gHyLsW+bWHkFdY3cJsP8=; b=JC9k7p/yNvKNl1iOCihar5RMihPwOSBMhbZTrx1tlYS+ImU4Q1DKIDrrjl1bdivVFU DfZjkQrvRri65fMIaPj2VS4lT2Gr6m30hjGUgi91sgbUMatkqwqcxG9hy1c3hw45e5v1 uTu4FOi2gu6hj6CcU8LFLKgdD87Y9/ZQhxfVOpTUJtLmYWdtueK+6ojC+dYfyG4mrWja +y1t7fYP3myyUxKQycfdRLFCpuFCFAEHxVeNlzEx3Nm9AhHpjKNpRuVf8rkBd4RugG+7 cukjM3kwSE+Hpcfa4yROTZhM0PZ39WTfxFvoIml/Y/6n5XS8tqaZ+zxP0jUbmboNlBbq cxcw== X-Gm-Message-State: AOJu0YxLQWdM2iPOtDzOGO39NnK8u4B4ZNDn625KEDHFbRICBHEft7u+ PtDrQC3thcSxU0PELQu75QMmryXFtH+udXysaNk= X-Google-Smtp-Source: AGHT+IEVPRx/hnm902FlBXllBljXg93AvisIgwLjHn6stNNzaqFBY1/i0LMzrQsV8Q2d5X9shhU7/yi9JDckd1vrnJI= X-Received: by 2002:a17:906:212:b0:a01:9364:ec62 with SMTP id 18-20020a170906021200b00a019364ec62mr6018898ejd.8.1702593318635; Thu, 14 Dec 2023 14:35:18 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: <787CB64A-1687-49C3-9063-2CE3B6F957EF@le-fay.org> <1656A349-9E7D-4798-89D1-ECE39126FBC8@le-fay.org> In-Reply-To: <1656A349-9E7D-4798-89D1-ECE39126FBC8@le-fay.org> From: Xin LI Date: Thu, 14 Dec 2023 14:35:07 -0800 Message-ID: Subject: Re: unusual ZFS issue To: Lexi Winter Cc: Pete Wright , "freebsd-fs@freebsd.org" Content-Type: multipart/alternative; boundary="000000000000f44e6b060c7fe5bc" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4SrnHD1QjFz3Fxj --000000000000f44e6b060c7fe5bc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Dec 14, 2023 at 2:29=E2=80=AFPM Lexi Winter wrote= : > On 14 Dec 2023, at 22:25, Xin LI wrote: > > Try "zpool status -x" and see if it would show something useful? > > the output seems to be the same as =E2=80=98zpool status -v=E2=80=99: > > # zpool status -xv > pool: data > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A > scan: scrub in progress since Thu Dec 14 18:58:21 2023 > 11.5T / 18.8T scanned at 962M/s, 8.71T / 18.8T issued at 726M/s > 0B repaired, 46.41% done, 04:02:02 to go > config: > > NAME STATE READ WRITE CKSUM > data ONLINE 0 0 0 > raidz2-0 ONLINE 0 0 0 > da4p1 ONLINE 0 0 0 > da6p1 ONLINE 0 0 0 > da5p1 ONLINE 0 0 0 > da7p1 ONLINE 0 0 0 > da1p1 ONLINE 0 0 0 > da0p1 ONLINE 0 0 0 > da3p1 ONLINE 0 0 0 > da2p1 ONLINE 0 0 0 > logs > mirror-2 ONLINE 0 0 0 > ada0p4 ONLINE 0 0 0 > ada1p4 ONLINE 0 0 0 > cache > ada1p5 ONLINE 0 0 0 > ada0p5 ONLINE 0 0 0 > > errors: Permanent errors have been detected in the following files: > This is strange, I'd expect some non-zero values above; did you 'zpool clear' before this? Note that this is permanent damage (otherwise ZFS will automatically "heal" the pool by overwriting the bad data with good copies, and your applications shouldn't see these). You can delete these files or datasets and restore them from backup. If you are not using ECC memory, they may occasionally cause damage like this one if you are really unlucky, by the way. i think this is expected since -x just filters the output to show pools > with errors? > Yeah.... -x shows only pools that are not healthy. If you have only one pool and that's the pool you are seeing issues, the output should be identical. Cheers, --000000000000f44e6b060c7fe5bc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Dec 14, 2023 at 2:29=E2=80= =AFPM Lexi Winter <lexi@le-fay.org> wrote:
On = 14 Dec 2023, at 22:25, Xin LI <delphij@gmail.com> wrote:
> Try "zpool status -x" and see if it would show something use= ful?

the output seems to be the same as =E2=80=98zpool status -v=E2=80=99:

# zpool status -xv
=C2=A0 pool: data
=C2=A0state: ONLINE
status: One or more devices has experienced an error resulting in data
=C2=A0 =C2=A0 =C2=A0 =C2=A0 corruption.=C2=A0 Applications may be affected.=
action: Restore the file in question if possible.=C2=A0 Otherwise restore t= he
=C2=A0 =C2=A0 =C2=A0 =C2=A0 entire pool from backup.
=C2=A0 =C2=A0see: https://openzfs.github.io/op= enzfs-docs/msg/ZFS-8000-8A
=C2=A0 scan: scrub in progress since Thu Dec 14 18:58:21 2023
=C2=A0 =C2=A0 =C2=A0 =C2=A0 11.5T / 18.8T scanned at 962M/s, 8.71T / 18.8T = issued at 726M/s
=C2=A0 =C2=A0 =C2=A0 =C2=A0 0B repaired, 46.41% done, 04:02:02 to go
config:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 STATE=C2=A0 =C2= =A0 =C2=A0READ WRITE CKSUM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 data=C2=A0 =C2=A0 =C2=A0 =C2=A0 ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 raidz2-0=C2=A0 ONLINE=C2=A0 =C2=A0 =C2= =A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da4p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da6p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da5p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da7p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da1p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da0p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da3p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da2p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 logs
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 mirror-2=C2=A0 ONLINE=C2=A0 =C2=A0 =C2= =A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ada0p4=C2=A0 ONLINE=C2=A0 =C2=A0 = =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ada1p4=C2=A0 ONLINE=C2=A0 =C2=A0 = =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 cache
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ada1p5=C2=A0 =C2=A0 ONLINE=C2=A0 =C2=A0 = =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ada0p5=C2=A0 =C2=A0 ONLINE=C2=A0 =C2=A0 = =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00

errors: Permanent errors have been detected in the following files:

This is strange, I'd expect some non-zero valu= es above; did you 'zpool clear' before this?

Note tha= t this is permanent damage (otherwise ZFS will automatically "heal&quo= t; the pool by overwriting the bad data with good copies, and your applicat= ions shouldn't see these).=C2=A0 You can delete these files or datasets= and restore them from backup.=C2=A0 If you are not using ECC memory, they = may occasionally cause damage like this one if you are really unlucky, by t= he way.

i think this is expected since -x just filters the output to show pools wit= h errors?

Yeah.... -x shows only pools that are not healthy.= =C2=A0 If you have only one pool and that's the pool you are seeing iss= ues, the output should be identical.

Cheers,
--000000000000f44e6b060c7fe5bc--