Date: Thu, 14 Dec 2023 14:25:13 -0800 From: Xin LI <delphij@gmail.com> To: Pete Wright <pete@nomadlogic.org> Cc: Lexi Winter <lexi@le-fay.org>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: unusual ZFS issue Message-ID: <CAGMYy3sp-6T0GYEBVG1WSkn4hzvQd15XuwD8bvx1qg1Qys%2BpCg@mail.gmail.com> In-Reply-To: <ec82ce34-6463-4200-960a-c820605d0cfd@nomadlogic.org> References: <787CB64A-1687-49C3-9063-2CE3B6F957EF@le-fay.org> <twivq3fadiotyfpblbqzfulx2wvel5povfic7626hktylqr4gg@uxipr5j7igs5> <C210159C-4747-45A1-9FB1-1708D6A21DE8@le-fay.org> <ec82ce34-6463-4200-960a-c820605d0cfd@nomadlogic.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000008bebbd060c7fc204
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Try "zpool status -x" and see if it would show something useful?
Cheers,
On Thu, Dec 14, 2023 at 2:10=E2=80=AFPM Pete Wright <pete@nomadlogic.org> w=
rote:
>
>
> On 12/14/23 2:05 PM, Lexi Winter wrote:
> > On 14 Dec 2023, at 22:02, Pete Wright <pete@nomadlogic.org> wrote:
> >> On Thu, Dec 14, 2023 at 09:17:06PM +0000, Lexi Winter wrote:
> >>> hi list,
> >>>
> >>> i=E2=80=99ve just hit this ZFS error:
> >>>
> >>> # zfs list -rt snapshot data/vm/media/disk1
> >>> cannot iterate filesystems: I/O error
> >>
> >> hrm, i wonder if you see any errors in dmesg or /var/log/messages abou=
t
> a
> >> device failing?
> >
> > nothing that looks relevant in the last few days (the problem appeared
> last night, Dec 13th):
> >
> > Dec 11 15:44:21 hemlock kernel: ix1: link state changed to DOWN
> > Dec 11 15:44:21 hemlock kernel: ix1.107: link state changed to DOWN
> > Dec 11 15:44:35 hemlock kernel: ix1: link state changed to UP
> > Dec 11 15:44:35 hemlock kernel: ix1.107: link state changed to UP
> > Dec 11 15:44:47 hemlock kernel: nfsrv_cache_session: no session
> IPaddr=3D2001:8b0:aab5:ffff::2, check NFS clients for unique /etc/hostid'=
s
> > Dec 11 15:44:47 hemlock syslogd: last message repeated 1 times
> > Dec 11 17:00:48 hemlock kernel: tcp_vnet_init: WARNING: unable to
> initialise TCP stats
> > Dec 11 17:00:48 hemlock kernel: lo0: link state changed to UP
> > Dec 12 06:17:23 hemlock ntpd[25836]: leapsecond file
> ('/var/db/ntpd.leap-seconds.list'): will expire in less than 16 days
> > Dec 13 06:17:23 hemlock ntpd[25836]: leapsecond file
> ('/var/db/ntpd.leap-seconds.list'): will expire in less than 15 days
> > Dec 14 06:17:23 hemlock ntpd[25836]: leapsecond file
> ('/var/db/ntpd.leap-seconds.list'): will expire in less than 14 days
> > Dec 14 16:30:12 hemlock smbd[98264]: [2023/12/14 16:30:12.404883, 0]
> ../../source3/smbd/server.c:1741(main)
> > Dec 14 16:30:12 hemlock smbd[98264]: smbd version 4.16.11 started.
> > Dec 14 16:30:12 hemlock smbd[98264]: Copyright Andrew Tridgell and th=
e
> Samba Team 1992-2022
> >
> > i=E2=80=99ve also checked the disks with smartctl and i didn=E2=80=99t =
see any errors
> there. (a couple of devices have corrected read errors, but that=E2=80=
=99s
> expected given their age - and if it *was* a disk error i=E2=80=99d expec=
t it to
> show up as a checksum error).
> >
>
> dang, was hoping something obvious would pop up there or with smartctl.
> hopefully others here have some ideas about trying to find the root
> cause before a restart.
>
> -pete
>
> --
> Pete Wright
> pete@nomadlogic.org
>
>
--0000000000008bebbd060c7fc204
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:monospac=
e,monospace">Try "zpool status -x" and see if it would show somet=
hing useful?</div><div class=3D"gmail_default" style=3D"font-family:monospa=
ce,monospace"><br></div><div class=3D"gmail_default" style=3D"font-family:m=
onospace,monospace">Cheers,<br></div></div><br><div class=3D"gmail_quote"><=
div dir=3D"ltr" class=3D"gmail_attr">On Thu, Dec 14, 2023 at 2:10=E2=80=AFP=
M Pete Wright <<a href=3D"mailto:pete@nomadlogic.org">pete@nomadlogic.or=
g</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin=
:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"=
><br>
<br>
On 12/14/23 2:05 PM, Lexi Winter wrote:<br>
> On 14 Dec 2023, at 22:02, Pete Wright <<a href=3D"mailto:pete@nomad=
logic.org" target=3D"_blank">pete@nomadlogic.org</a>> wrote:<br>
>> On Thu, Dec 14, 2023 at 09:17:06PM +0000, Lexi Winter wrote:<br>
>>> hi list,<br>
>>><br>
>>> i=E2=80=99ve just hit this ZFS error:<br>
>>><br>
>>> # zfs list -rt snapshot data/vm/media/disk1<br>
>>> cannot iterate filesystems: I/O error<br>
>><br>
>> hrm, i wonder if you see any errors in dmesg or /var/log/messages =
about a<br>
>> device failing?<br>
> <br>
> nothing that looks relevant in the last few days (the problem appeared=
last night, Dec 13th):<br>
> <br>
> Dec 11 15:44:21 hemlock kernel: ix1: link state changed to DOWN<br>
> Dec 11 15:44:21 hemlock kernel: ix1.107: link state changed to DOWN<br=
>
> Dec 11 15:44:35 hemlock kernel: ix1: link state changed to UP<br>
> Dec 11 15:44:35 hemlock kernel: ix1.107: link state changed to UP<br>
> Dec 11 15:44:47 hemlock kernel: nfsrv_cache_session: no session IPaddr=
=3D2001:8b0:aab5:ffff::2, check NFS clients for unique /etc/hostid's<br=
>
> Dec 11 15:44:47 hemlock syslogd: last message repeated 1 times<br>
> Dec 11 17:00:48 hemlock kernel: tcp_vnet_init: WARNING: unable to init=
ialise TCP stats<br>
> Dec 11 17:00:48 hemlock kernel: lo0: link state changed to UP<br>
> Dec 12 06:17:23 hemlock ntpd[25836]: leapsecond file ('/var/db/ntp=
d.leap-seconds.list'): will expire in less than 16 days<br>
> Dec 13 06:17:23 hemlock ntpd[25836]: leapsecond file ('/var/db/ntp=
d.leap-seconds.list'): will expire in less than 15 days<br>
> Dec 14 06:17:23 hemlock ntpd[25836]: leapsecond file ('/var/db/ntp=
d.leap-seconds.list'): will expire in less than 14 days<br>
> Dec 14 16:30:12 hemlock smbd[98264]: [2023/12/14 16:30:12.404883,=C2=
=A0 0] ../../source3/smbd/server.c:1741(main)<br>
> Dec 14 16:30:12 hemlock smbd[98264]:=C2=A0 =C2=A0smbd version 4.16.11 =
started.<br>
> Dec 14 16:30:12 hemlock smbd[98264]:=C2=A0 =C2=A0Copyright Andrew Trid=
gell and the Samba Team 1992-2022<br>
> <br>
> i=E2=80=99ve also checked the disks with smartctl and i didn=E2=80=99t=
see any errors there.=C2=A0 (a couple of devices have corrected read error=
s, but that=E2=80=99s expected given their age - and if it *was* a disk err=
or i=E2=80=99d expect it to show up as a checksum error).<br>
> <br>
<br>
dang, was hoping something obvious would pop up there or with smartctl. <br=
>
hopefully others here have some ideas about trying to find the root <br>
cause before a restart.<br>
<br>
-pete<br>
<br>
-- <br>
Pete Wright<br>
<a href=3D"mailto:pete@nomadlogic.org" target=3D"_blank">pete@nomadlogic.or=
g</a><br>
<br>
</blockquote></div>
--0000000000008bebbd060c7fc204--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGMYy3sp-6T0GYEBVG1WSkn4hzvQd15XuwD8bvx1qg1Qys%2BpCg>