From nobody Thu Dec 14 21:17:06 2023 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SrlYB1LT2z53mn0 for ; Thu, 14 Dec 2023 21:17:18 +0000 (UTC) (envelope-from lexi@le-fay.org) Received: from thyme.eden.le-Fay.ORG (THYME.EDEN.LE-FAY.ORG [IPv6:2001:8b0:aab5:107::10]) by mx1.freebsd.org (Postfix) with ESMTP id 4SrlY933TTz4np9 for ; Thu, 14 Dec 2023 21:17:17 +0000 (UTC) (envelope-from lexi@le-fay.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=le-fay.org header.s=thyme header.b=hzbHQfjK; spf=pass (mx1.freebsd.org: domain of lexi@le-fay.org designates 2001:8b0:aab5:107::10 as permitted sender) smtp.mailfrom=lexi@le-fay.org; dmarc=none Received: from iris.eden.le-Fay.ORG (IRIS.EDEN.LE-FAY.ORG [IPv6:2001:8b0:aab5:106::18]) by thyme.eden.le-Fay.ORG (Postfix) with ESMTP id EBA6E22B37 for ; Thu, 14 Dec 2023 21:17:16 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=le-fay.org; s=thyme; t=1702588636; bh=pOaT8w0ednxBkgmx6kcB/IVR52BqanMuakNQuPgZBXY=; h=From:Subject:Date:To; b=hzbHQfjKPJjY+lmbaSIZR7j5Dffay0ZmATj5Ctdi17Ef4fONfBqSK/Bv0Mph2pmih tPOx4G4W4nhLT2dMM+9cevPFQmBivUWRneG7R74gTp9KgNTwnCkUsfd51d3eoo8djc 1GFKq0RANXCal3XMfLSSRlkjgGV6QZopogg7yGxM= Received: from smtpclient.apple (unknown [IPv6:2001:8b0:aab5:104:bdc0:5de2:bc0e:fba9]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by iris.eden.le-Fay.ORG (Postfix) with ESMTPSA id E03ACD14F for ; Thu, 14 Dec 2023 21:17:16 +0000 (GMT) From: Lexi Winter Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.300.61.1.2\)) Subject: unusual ZFS issue Message-Id: <787CB64A-1687-49C3-9063-2CE3B6F957EF@le-fay.org> Date: Thu, 14 Dec 2023 21:17:06 +0000 To: "freebsd-fs@freebsd.org" X-Mailer: Apple Mail (2.3774.300.61.1.2) X-Spamd-Result: default: False [-2.90 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2001:8b0:aab5:107::10:c]; R_DKIM_ALLOW(-0.20)[le-fay.org:s=thyme]; MIME_GOOD(-0.10)[text/plain]; RCVD_NO_TLS_LAST(0.10)[]; MLMMJ_DEST(0.00)[freebsd-fs@FreeBSD.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:20712, ipnet:2001:8b0::/32, country:GB]; TO_DN_EQ_ADDR_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DKIM_TRACE(0.00)[le-fay.org:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[le-fay.org]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[le-fay.org:dkim] X-Rspamd-Queue-Id: 4SrlY933TTz4np9 X-Spamd-Bar: -- hi list, i=E2=80=99ve just hit this ZFS error: # zfs list -rt snapshot data/vm/media/disk1 cannot iterate filesystems: I/O error NAME USED AVAIL = REFER MOUNTPOINT data/vm/media/disk1@autosnap_2023-12-13_12:00:00_hourly 0B - = 6.42G - data/vm/media/disk1@autosnap_2023-12-14_10:16:00_hourly 0B - = 6.46G - data/vm/media/disk1@autosnap_2023-12-14_11:17:00_hourly 0B - = 6.46G - data/vm/media/disk1@autosnap_2023-12-14_12:04:00_monthly 0B - = 6.46G - data/vm/media/disk1@autosnap_2023-12-14_12:15:00_hourly 0B - = 6.46G - data/vm/media/disk1@autosnap_2023-12-14_13:14:00_hourly 0B - = 6.46G - data/vm/media/disk1@autosnap_2023-12-14_14:38:00_hourly 0B - = 6.46G - data/vm/media/disk1@autosnap_2023-12-14_15:11:00_hourly 0B - = 6.46G - data/vm/media/disk1@autosnap_2023-12-14_17:12:00_hourly 316K - = 6.47G - data/vm/media/disk1@autosnap_2023-12-14_17:29:00_daily 2.70M - = 6.47G - the pool itself also reports an error: # zpool status -v pool: data state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub in progress since Thu Dec 14 18:58:21 2023 11.5T / 18.8T scanned at 1.46G/s, 6.25T / 18.8T issued at 809M/s 0B repaired, 33.29% done, 04:30:20 to go config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 da4p1 ONLINE 0 0 0 da6p1 ONLINE 0 0 0 da5p1 ONLINE 0 0 0 da7p1 ONLINE 0 0 0 da1p1 ONLINE 0 0 0 da0p1 ONLINE 0 0 0 da3p1 ONLINE 0 0 0 da2p1 ONLINE 0 0 0 logs mirror-2 ONLINE 0 0 0 ada0p4 ONLINE 0 0 0 ada1p4 ONLINE 0 0 0 cache ada1p5 ONLINE 0 0 0 ada0p5 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: (it doesn=E2=80=99t list any files, the output ends there.) my assumption is that this indicates some sort of metadata corruption = issue, but i can=E2=80=99t find anything that might have caused it. = none of the disks report any errors, and while all the disks are on the = same SAS controller, i would have expected controller errors to be = flagged as CKSUM errors. my best guess is that this might be caused by a CPU or memory issue, but = the system has ECC memory and hasn=E2=80=99t reported any issues. - has anyone else encountered anything like this? - i=E2=80=99m a bit worried that if i reboot, the system won=E2=80=99t = be able to re-import the pool due to the =E2=80=9Ccannot iterate = filesystems=E2=80=9D errors; is that a concern? the system is running: FreeBSD hemlock.eden.le-fay.org 14.0-RELEASE-p2 FreeBSD 14.0-RELEASE-p2 = #4 releng/14.0-n265396-06497fbd52e2: Fri Dec 8 06:14:12 GMT 2023 = root@hemlock.eden.le-fay.org:/data/src/obj/data/src/releng/14.0/amd64.amd6= 4/sys/HEMLOCK amd64 thanks.=