From nobody Thu Nov 28 13:52:35 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xzd6m1SdXz5fkcD for ; Thu, 28 Nov 2024 13:52:48 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xzd6l6gQHz4rBK for ; Thu, 28 Nov 2024 13:52:47 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-aa1e6ecd353so49638366b.1 for ; Thu, 28 Nov 2024 05:52:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1732801966; x=1733406766; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kkm2deqqq4BGJzJRtiy/u4PLzZArKAB5Gkvo/D29CaE=; b=EkVWnxxitXE/NODhjzJJtp3pOotVP8iVM2xZDNwWJxW9/2vsMr0tNGU6ruJ9Vrg1+b ELSsVa9/Lkpd+IMY+AIEOky3mPLe8XcmbbqUkPKD9BpQUEfY7jmAwN8keNBf0HnxZzAA pLcLEyIGLr7sFchQwSC+xyuB8KwTGSDyHrPW30yMT3ScxAXEUJaKkvlvoqAAJynBLX9L r6nL5f03BvF6fIZVzd9fkMQuzeHz/ThyGoiBMFwi1zoCzORMil3+2kt+3hTBA4kijipM Up/lXVgdDbjDFbtzbJ9a5uvZU1VLiStvlexfaxdm83YpK81pD5bcDUZF026E+XkxuZ4q +hPA== X-Gm-Message-State: AOJu0Yw2WbjwjygF2Tm2PWOkN5EKay+Me41W8YfEmYN7miE1/+xCuGJ9 Z7yXlRgmtxxmxmj/HNyCzUyOFyy40UwEWVAhb1EV1VdSsIv2T76xxQhvWL7QbwxEZ6mvisXaDRI TvZ1T+JQYC7q7j7eCVUk5kueywFBE/w== X-Gm-Gg: ASbGncuLxEOEXH1VXQQm1AY2nkdrGxD/GZ9mT0vnjtSce5UlInH/13/BB2O95xVh7Ji 7iaeYo9ddI+xsHq+XOqSMY9osy2zV9kWqa6aYP0LzcCXNTDugMWTry3AqFJ0PibNu X-Google-Smtp-Source: AGHT+IFVazhxOXfm/uBW5b3RwUVbDIBJ7iDkFrbg1zD90k30Gzt39JK2r31aX9vmIHf2gwyAkm1lK/5k4FjaVQGco88= X-Received: by 2002:a05:6402:3553:b0:5cf:f361:1c11 with SMTP id 4fb4d7f45d1cf-5d080c07623mr7653162a12.20.1732801966172; Thu, 28 Nov 2024 05:52:46 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <5798b0db-bc73-476a-908a-dd1f071bfe43@blastwave.org> In-Reply-To: <5798b0db-bc73-476a-908a-dd1f071bfe43@blastwave.org> From: Alan Somers Date: Thu, 28 Nov 2024 07:52:35 -0600 Message-ID: Subject: Re: zpools no longer exist after boot To: Dennis Clarke Cc: Current FreeBSD Content-Type: multipart/alternative; boundary="000000000000a8f3c40627f96554" X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Queue-Id: 4Xzd6l6gQHz4rBK X-Spamd-Bar: ---- --000000000000a8f3c40627f96554 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Nov 28, 2024, 7:06=E2=80=AFAM Dennis Clarke = wrote: > > This is a baffling problem wherein two zpools no longer exist after > boot. This is : > > titan# uname -apKU > FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1 > main-n273749-4b65481ac68a-dirty: Wed Nov 20 15:08:52 GMT 2024 > root@titan:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 amd64 > 1500027 1500027 > titan# > > titan# zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP > HEALTH ALTROOT > t0 444G 91.2G 353G - - 27% 20% 1.00x > ONLINE - > titan# > > The *only* zpool that seems to exist in any reliable way is the little > NVME based unit for booting. The other two zpools vanished and yet the > devices exist just fine : > > titan# > titan# camcontrol devlist > at scbus0 target 0 lun 0 (pass0,ada0) > at scbus1 target 0 lun 0 (pass1,ada1) > at scbus2 target 0 lun 0 (ses0,pass2) > at scbus6 target 0 lun 0 (ses1,pass3) > at scbus7 target 0 lun 1 > (pass4,nda0) > at scbus8 target 0 lun 0 (da0,pass5) > titan# > titan# nvmecontrol devlist > nvme0: SAMSUNG MZVKW512HMJP-000L7 > nvme0ns1 (488386MB) > titan# > titan# zpool status t0 > pool: t0 > state: ONLINE > status: Some supported and requested features are not enabled on the pool= . > The pool can still be used, but some features are unavailable. > action: Enable all features using 'zpool upgrade'. Once this is done, > the pool may no longer be accessible by software that does not > support > the features. See zpool-features(7) for details. > scan: scrub repaired 0B in 00:00:44 with 0 errors on Wed Feb 7 > 09:56:40 2024 > config: > > NAME STATE READ WRITE CKSUM > t0 ONLINE 0 0 0 > nda0p3 ONLINE 0 0 0 > > errors: No known data errors > titan# > > > Initially I thought the problem was related to cachefile being empty for > these zpools. However if I set the cachefile to something reasonable > then the cachefile property vanishes at a reboot. The file, of course, > exists just fine : > > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile - default > titan# > titan# zpool set cachefile=3D"/var/log/zpool_cache" proteus > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile /var/log/zpool_cache local > titan# ls -ladb /var/log/zpool_cache > -rw-r--r-- 1 root wheel 1440 Nov 28 11:45 /var/log/zpool_cache > titan# > > So there we have 1440 bytes of data in that file. > > titan# zpool set cachefile=3D"/var/log/zpool_cache" t0 > titan# zpool get cachefile t0 > NAME PROPERTY VALUE SOURCE > t0 cachefile /var/log/zpool_cache local > titan# > titan# ls -ladb /var/log/zpool_cache > -rw-r--r-- 1 root wheel 2880 Nov 28 11:46 /var/log/zpool_cache > titan# > > Now we have 2 * 1440 bytes =3D 2880 bytes of some zpool cache data. > > titan# zpool set cachefile=3D"/var/log/zpool_cache" leaf > titan# zpool get cachefile leaf > NAME PROPERTY VALUE SOURCE > leaf cachefile /var/log/zpool_cache local > titan# > titan# zpool get cachefile t0 > NAME PROPERTY VALUE SOURCE > t0 cachefile /var/log/zpool_cache local > titan# > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile /var/log/zpool_cache local > titan# > titan# reboot > > From here on ... the only zpool that exists after boot is the local > little NVME samsung unit. > > So here I can import those pools and then see that the cachefile > property has been wiped out : > > titan# > titan# zpool import proteus > titan# zpool import leaf > titan# > titan# zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP > HEALTH ALTROOT > leaf 18.2T 984K 18.2T - - 0% 0% 1.00x > ONLINE - > proteus 1.98T 361G 1.63T - - 1% 17% 1.00x > ONLINE - > t0 444G 91.2G 353G - - 27% 20% 1.00x > ONLINE - > titan# > titan# zpool get cachefile leaf > NAME PROPERTY VALUE SOURCE > leaf cachefile - default > titan# > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile - default > titan# > titan# zpool get cachefile t0 > NAME PROPERTY VALUE SOURCE > t0 cachefile - default > titan# > titan# ls -l /var/log/zpool_cache > -rw-r--r-- 1 root wheel 4960 Nov 28 11:52 /var/log/zpool_cache > titan# > > The cachefile exists and seems to have grown in size. > > However a reboot will once again provide nothing but the t0 pool. > > Baffled. > > Any thoughts would be welcome. > > -- > -- > Dennis Clarke > RISC-V/SPARC/PPC/ARM/CISC > UNIX and Linux spoken > Do you have zfs_enable=3D"YES" set in /etc/rc.conf? If not then nothing wil= l get imported. Regarding the cachefile property, it's expected that "zpool import" will change it, unless you do "zpool import -O cachefile=3Dwhatever". > --000000000000a8f3c40627f96554 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Thu, Nov 28, 2024, 7:06=E2=80=AFAM Dennis Clarke <dclarke@blastwave.org> wrote:
=

This is a baffling problem wherein two zpools no longer exist after
boot. This is :

titan# uname -apKU
FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1
main-n273749-4b65481ac68a-dirty: Wed Nov 20 15:08:52 GMT 2024
root@titan:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 amd64 1500027 1500027
titan#

titan# zpool list
NAME=C2=A0 =C2=A0SIZE=C2=A0 ALLOC=C2=A0 =C2=A0FREE=C2=A0 CKPOINT=C2=A0 EXPA= NDSZ=C2=A0 =C2=A0FRAG=C2=A0 =C2=A0 CAP=C2=A0 DEDUP
HEALTH=C2=A0 ALTROOT
t0=C2=A0 =C2=A0 =C2=A0444G=C2=A0 91.2G=C2=A0 =C2=A0353G=C2=A0 =C2=A0 =C2=A0= =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 27%=C2=A0 =C2=A0 = 20%=C2=A0 1.00x
ONLINE=C2=A0 -
titan#

The *only* zpool that seems to exist in any reliable way is the little
NVME based unit for booting. The other two zpools vanished and yet the
devices exist just fine :

titan#
titan# camcontrol devlist
<ST20000NM007D-3DJ103 SN03>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at scbus0 targ= et 0 lun 0 (pass0,ada0)
<ST20000NM007D-3DJ103 SN03>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at scbus1 targ= et 0 lun 0 (pass1,ada1)
<AHCI SGPIO Enclosure 2.00 0001>=C2=A0 =C2=A0at scbus2 target 0 lun 0= (ses0,pass2)
<AHCI SGPIO Enclosure 2.00 0001>=C2=A0 =C2=A0at scbus6 target 0 lun 0= (ses1,pass3)
<SAMSUNG MZVKW512HMJP-000L7 6L6QCXA7>=C2=A0 at scbus7 target 0 lun 1 = (pass4,nda0)
<FREEBSD CTLDISK 0001>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at scbus8 target 0 lun 0 (da0,pass5)
titan#
titan# nvmecontrol devlist
=C2=A0 nvme0: SAMSUNG MZVKW512HMJP-000L7
=C2=A0 =C2=A0 =C2=A0nvme0ns1 (488386MB)
titan#
titan# zpool status t0
=C2=A0 =C2=A0pool: t0
=C2=A0 state: ONLINE
status: Some supported and requested features are not enabled on the pool.<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The pool can still be used, but some feat= ures are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is don= e,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0the pool may no longer be accessible by s= oftware that does not
support
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0the features. See zpool-features(7) for d= etails.
=C2=A0 =C2=A0scan: scrub repaired 0B in 00:00:44 with 0 errors on Wed Feb= =C2=A0 7
09:56:40 2024
config:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 STATE=C2= =A0 =C2=A0 =C2=A0READ WRITE CKSUM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0t0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ONLI= NE=C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0nda0p3=C2=A0 =C2=A0 ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00

errors: No known data errors
titan#


Initially I thought the problem was related to cachefile being empty for these zpools. However if I set the cachefile to something reasonable
then the cachefile property vanishes at a reboot.=C2=A0 The file, of course= ,
exists just fine :

titan# zpool get cachefile proteus
NAME=C2=A0 =C2=A0 =C2=A0PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 SOUR= CE
proteus=C2=A0 cachefile=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default titan#
titan# zpool set cachefile=3D"/var/log/zpool_cache" proteus
titan# zpool get cachefile proteus
NAME=C2=A0 =C2=A0 =C2=A0PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SOURCE
proteus=C2=A0 cachefile=C2=A0 /var/log/zpool_cache=C2=A0 local
titan# ls -ladb /var/log/zpool_cache
-rw-r--r--=C2=A0 1 root wheel 1440 Nov 28 11:45 /var/log/zpool_cache
titan#

So there we have 1440 bytes of data in that file.

titan# zpool set cachefile=3D"/var/log/zpool_cache" t0
titan# zpool get cachefile t0
NAME=C2=A0 PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0SOURCE
t0=C2=A0 =C2=A0 cachefile=C2=A0 /var/log/zpool_cache=C2=A0 local
titan#
titan# ls -ladb /var/log/zpool_cache
-rw-r--r--=C2=A0 1 root wheel 2880 Nov 28 11:46 /var/log/zpool_cache
titan#

Now we have 2 * 1440 bytes =3D 2880 bytes of some zpool cache data.

titan# zpool set cachefile=3D"/var/log/zpool_cache" leaf
titan# zpool get cachefile leaf
NAME=C2=A0 PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0SOURCE
leaf=C2=A0 cachefile=C2=A0 /var/log/zpool_cache=C2=A0 local
titan#
titan# zpool get cachefile t0
NAME=C2=A0 PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0SOURCE
t0=C2=A0 =C2=A0 cachefile=C2=A0 /var/log/zpool_cache=C2=A0 local
titan#
titan# zpool get cachefile proteus
NAME=C2=A0 =C2=A0 =C2=A0PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SOURCE
proteus=C2=A0 cachefile=C2=A0 /var/log/zpool_cache=C2=A0 local
titan#
titan# reboot

=C2=A0From here on ... the only zpool that exists after boot is the local little NVME samsung unit.

So here I can import those pools and then see that the cachefile
property has been wiped out :

titan#
titan# zpool import proteus
titan# zpool import leaf
titan#
titan# zpool list
NAME=C2=A0 =C2=A0 =C2=A0 SIZE=C2=A0 ALLOC=C2=A0 =C2=A0FREE=C2=A0 CKPOINT=C2= =A0 EXPANDSZ=C2=A0 =C2=A0FRAG=C2=A0 =C2=A0 CAP=C2=A0 DEDUP
HEALTH=C2=A0 ALTROOT
leaf=C2=A0 =C2=A0 =C2=A018.2T=C2=A0 =C2=A0984K=C2=A0 18.2T=C2=A0 =C2=A0 =C2= =A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A00%=C2=A0= =C2=A0 =C2=A00%=C2=A0 1.00x
ONLINE=C2=A0 -
proteus=C2=A0 1.98T=C2=A0 =C2=A0361G=C2=A0 1.63T=C2=A0 =C2=A0 =C2=A0 =C2=A0= -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A01%=C2=A0 =C2=A0 17= %=C2=A0 1.00x
ONLINE=C2=A0 -
t0=C2=A0 =C2=A0 =C2=A0 =C2=A0 444G=C2=A0 91.2G=C2=A0 =C2=A0353G=C2=A0 =C2= =A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 27%=C2= =A0 =C2=A0 20%=C2=A0 1.00x
ONLINE=C2=A0 -
titan#
titan# zpool get cachefile leaf
NAME=C2=A0 PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 SOURCE
leaf=C2=A0 cachefile=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default
titan#
titan# zpool get cachefile proteus
NAME=C2=A0 =C2=A0 =C2=A0PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 SOUR= CE
proteus=C2=A0 cachefile=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default titan#
titan# zpool get cachefile t0
NAME=C2=A0 PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 SOURCE
t0=C2=A0 =C2=A0 cachefile=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=
titan#
titan# ls -l /var/log/zpool_cache
-rw-r--r--=C2=A0 1 root wheel 4960 Nov 28 11:52 /var/log/zpool_cache
titan#

The cachefile exists and seems to have grown in size.

However a reboot will once again provide nothing but the t0 pool.

Baffled.

Any thoughts would be welcome.

--
--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken

Do you have zfs_enable=3D"YES" set in /etc/r= c.conf? If not then nothing will get imported.=C2=A0

Regarding the cachefile property, it's exp= ected that "zpool import" will change it, unless you do "zpo= ol import -O cachefile=3Dwhatever".
--000000000000a8f3c40627f96554--