Date: Thu, 28 Nov 2024 07:52:35 -0600 From: Alan Somers <asomers@freebsd.org> To: Dennis Clarke <dclarke@blastwave.org> Cc: Current FreeBSD <freebsd-current@freebsd.org> Subject: Re: zpools no longer exist after boot Message-ID: <CAOtMX2hKCYrx92SBLQOtekKiBWMgBy_n93ZGQ_NVLq=6puRhOg@mail.gmail.com> In-Reply-To: <5798b0db-bc73-476a-908a-dd1f071bfe43@blastwave.org> References: <5798b0db-bc73-476a-908a-dd1f071bfe43@blastwave.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000a8f3c40627f96554 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Nov 28, 2024, 7:06=E2=80=AFAM Dennis Clarke <dclarke@blastwave.org>= wrote: > > This is a baffling problem wherein two zpools no longer exist after > boot. This is : > > titan# uname -apKU > FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1 > main-n273749-4b65481ac68a-dirty: Wed Nov 20 15:08:52 GMT 2024 > root@titan:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 amd64 > 1500027 1500027 > titan# > > titan# zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP > HEALTH ALTROOT > t0 444G 91.2G 353G - - 27% 20% 1.00x > ONLINE - > titan# > > The *only* zpool that seems to exist in any reliable way is the little > NVME based unit for booting. The other two zpools vanished and yet the > devices exist just fine : > > titan# > titan# camcontrol devlist > <ST20000NM007D-3DJ103 SN03> at scbus0 target 0 lun 0 (pass0,ada0) > <ST20000NM007D-3DJ103 SN03> at scbus1 target 0 lun 0 (pass1,ada1) > <AHCI SGPIO Enclosure 2.00 0001> at scbus2 target 0 lun 0 (ses0,pass2) > <AHCI SGPIO Enclosure 2.00 0001> at scbus6 target 0 lun 0 (ses1,pass3) > <SAMSUNG MZVKW512HMJP-000L7 6L6QCXA7> at scbus7 target 0 lun 1 > (pass4,nda0) > <FREEBSD CTLDISK 0001> at scbus8 target 0 lun 0 (da0,pass5) > titan# > titan# nvmecontrol devlist > nvme0: SAMSUNG MZVKW512HMJP-000L7 > nvme0ns1 (488386MB) > titan# > titan# zpool status t0 > pool: t0 > state: ONLINE > status: Some supported and requested features are not enabled on the pool= . > The pool can still be used, but some features are unavailable. > action: Enable all features using 'zpool upgrade'. Once this is done, > the pool may no longer be accessible by software that does not > support > the features. See zpool-features(7) for details. > scan: scrub repaired 0B in 00:00:44 with 0 errors on Wed Feb 7 > 09:56:40 2024 > config: > > NAME STATE READ WRITE CKSUM > t0 ONLINE 0 0 0 > nda0p3 ONLINE 0 0 0 > > errors: No known data errors > titan# > > > Initially I thought the problem was related to cachefile being empty for > these zpools. However if I set the cachefile to something reasonable > then the cachefile property vanishes at a reboot. The file, of course, > exists just fine : > > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile - default > titan# > titan# zpool set cachefile=3D"/var/log/zpool_cache" proteus > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile /var/log/zpool_cache local > titan# ls -ladb /var/log/zpool_cache > -rw-r--r-- 1 root wheel 1440 Nov 28 11:45 /var/log/zpool_cache > titan# > > So there we have 1440 bytes of data in that file. > > titan# zpool set cachefile=3D"/var/log/zpool_cache" t0 > titan# zpool get cachefile t0 > NAME PROPERTY VALUE SOURCE > t0 cachefile /var/log/zpool_cache local > titan# > titan# ls -ladb /var/log/zpool_cache > -rw-r--r-- 1 root wheel 2880 Nov 28 11:46 /var/log/zpool_cache > titan# > > Now we have 2 * 1440 bytes =3D 2880 bytes of some zpool cache data. > > titan# zpool set cachefile=3D"/var/log/zpool_cache" leaf > titan# zpool get cachefile leaf > NAME PROPERTY VALUE SOURCE > leaf cachefile /var/log/zpool_cache local > titan# > titan# zpool get cachefile t0 > NAME PROPERTY VALUE SOURCE > t0 cachefile /var/log/zpool_cache local > titan# > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile /var/log/zpool_cache local > titan# > titan# reboot > > From here on ... the only zpool that exists after boot is the local > little NVME samsung unit. > > So here I can import those pools and then see that the cachefile > property has been wiped out : > > titan# > titan# zpool import proteus > titan# zpool import leaf > titan# > titan# zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP > HEALTH ALTROOT > leaf 18.2T 984K 18.2T - - 0% 0% 1.00x > ONLINE - > proteus 1.98T 361G 1.63T - - 1% 17% 1.00x > ONLINE - > t0 444G 91.2G 353G - - 27% 20% 1.00x > ONLINE - > titan# > titan# zpool get cachefile leaf > NAME PROPERTY VALUE SOURCE > leaf cachefile - default > titan# > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile - default > titan# > titan# zpool get cachefile t0 > NAME PROPERTY VALUE SOURCE > t0 cachefile - default > titan# > titan# ls -l /var/log/zpool_cache > -rw-r--r-- 1 root wheel 4960 Nov 28 11:52 /var/log/zpool_cache > titan# > > The cachefile exists and seems to have grown in size. > > However a reboot will once again provide nothing but the t0 pool. > > Baffled. > > Any thoughts would be welcome. > > -- > -- > Dennis Clarke > RISC-V/SPARC/PPC/ARM/CISC > UNIX and Linux spoken > Do you have zfs_enable=3D"YES" set in /etc/rc.conf? If not then nothing wil= l get imported. Regarding the cachefile property, it's expected that "zpool import" will change it, unless you do "zpool import -O cachefile=3Dwhatever". > --000000000000a8f3c40627f96554 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"auto"><div><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D= "gmail_attr">On Thu, Nov 28, 2024, 7:06=E2=80=AFAM Dennis Clarke <<a hre= f=3D"mailto:dclarke@blastwave.org">dclarke@blastwave.org</a>> wrote:<br>= </div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b= order-left:1px solid rgb(204,204,204);padding-left:1ex"><br> This is a baffling problem wherein two zpools no longer exist after<br> boot. This is :<br> <br> titan# uname -apKU<br> FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1 <br> main-n273749-4b65481ac68a-dirty: Wed Nov 20 15:08:52 GMT 2024 <br> root@titan:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 amd64 <br= > 1500027 1500027<br> titan#<br> <br> titan# zpool list<br> NAME=C2=A0 =C2=A0SIZE=C2=A0 ALLOC=C2=A0 =C2=A0FREE=C2=A0 CKPOINT=C2=A0 EXPA= NDSZ=C2=A0 =C2=A0FRAG=C2=A0 =C2=A0 CAP=C2=A0 DEDUP <br> HEALTH=C2=A0 ALTROOT<br> t0=C2=A0 =C2=A0 =C2=A0444G=C2=A0 91.2G=C2=A0 =C2=A0353G=C2=A0 =C2=A0 =C2=A0= =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 27%=C2=A0 =C2=A0 = 20%=C2=A0 1.00x <br> ONLINE=C2=A0 -<br> titan#<br> <br> The *only* zpool that seems to exist in any reliable way is the little<br> NVME based unit for booting. The other two zpools vanished and yet the<br> devices exist just fine :<br> <br> titan#<br> titan# camcontrol devlist<br> <ST20000NM007D-3DJ103 SN03>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at scbus0 targ= et 0 lun 0 (pass0,ada0)<br> <ST20000NM007D-3DJ103 SN03>=C2=A0 =C2=A0 =C2=A0 =C2=A0 at scbus1 targ= et 0 lun 0 (pass1,ada1)<br> <AHCI SGPIO Enclosure 2.00 0001>=C2=A0 =C2=A0at scbus2 target 0 lun 0= (ses0,pass2)<br> <AHCI SGPIO Enclosure 2.00 0001>=C2=A0 =C2=A0at scbus6 target 0 lun 0= (ses1,pass3)<br> <SAMSUNG MZVKW512HMJP-000L7 6L6QCXA7>=C2=A0 at scbus7 target 0 lun 1 = (pass4,nda0)<br> <FREEBSD CTLDISK 0001>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0at scbus8 target 0 lun 0 (da0,pass5)<br> titan#<br> titan# nvmecontrol devlist<br> =C2=A0 nvme0: SAMSUNG MZVKW512HMJP-000L7<br> =C2=A0 =C2=A0 =C2=A0nvme0ns1 (488386MB)<br> titan#<br> titan# zpool status t0<br> =C2=A0 =C2=A0pool: t0<br> =C2=A0 state: ONLINE<br> status: Some supported and requested features are not enabled on the pool.<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0The pool can still be used, but some feat= ures are unavailable.<br> action: Enable all features using 'zpool upgrade'. Once this is don= e,<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0the pool may no longer be accessible by s= oftware that does not <br> support<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0the features. See zpool-features(7) for d= etails.<br> =C2=A0 =C2=A0scan: scrub repaired 0B in 00:00:44 with 0 errors on Wed Feb= =C2=A0 7 <br> 09:56:40 2024<br> config:<br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 STATE=C2= =A0 =C2=A0 =C2=A0READ WRITE CKSUM<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0t0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ONLI= NE=C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br= > =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0nda0p3=C2=A0 =C2=A0 ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> <br> errors: No known data errors<br> titan#<br> <br> <br> Initially I thought the problem was related to cachefile being empty for<br= > these zpools. However if I set the cachefile to something reasonable<br> then the cachefile property vanishes at a reboot.=C2=A0 The file, of course= , <br> exists just fine :<br> <br> titan# zpool get cachefile proteus<br> NAME=C2=A0 =C2=A0 =C2=A0PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 SOUR= CE<br> proteus=C2=A0 cachefile=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default<b= r> titan#<br> titan# zpool set cachefile=3D"/var/log/zpool_cache" proteus<br> titan# zpool get cachefile proteus<br> NAME=C2=A0 =C2=A0 =C2=A0PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SOURCE<br> proteus=C2=A0 cachefile=C2=A0 /var/log/zpool_cache=C2=A0 local<br> titan# ls -ladb /var/log/zpool_cache<br> -rw-r--r--=C2=A0 1 root wheel 1440 Nov 28 11:45 /var/log/zpool_cache<br> titan#<br> <br> So there we have 1440 bytes of data in that file.<br> <br> titan# zpool set cachefile=3D"/var/log/zpool_cache" t0<br> titan# zpool get cachefile t0<br> NAME=C2=A0 PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0SOURCE<br> t0=C2=A0 =C2=A0 cachefile=C2=A0 /var/log/zpool_cache=C2=A0 local<br> titan#<br> titan# ls -ladb /var/log/zpool_cache<br> -rw-r--r--=C2=A0 1 root wheel 2880 Nov 28 11:46 /var/log/zpool_cache<br> titan#<br> <br> Now we have 2 * 1440 bytes =3D 2880 bytes of some zpool cache data.<br> <br> titan# zpool set cachefile=3D"/var/log/zpool_cache" leaf<br> titan# zpool get cachefile leaf<br> NAME=C2=A0 PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0SOURCE<br> leaf=C2=A0 cachefile=C2=A0 /var/log/zpool_cache=C2=A0 local<br> titan#<br> titan# zpool get cachefile t0<br> NAME=C2=A0 PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0SOURCE<br> t0=C2=A0 =C2=A0 cachefile=C2=A0 /var/log/zpool_cache=C2=A0 local<br> titan#<br> titan# zpool get cachefile proteus<br> NAME=C2=A0 =C2=A0 =C2=A0PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SOURCE<br> proteus=C2=A0 cachefile=C2=A0 /var/log/zpool_cache=C2=A0 local<br> titan#<br> titan# reboot<br> <br> =C2=A0From here on ... the only zpool that exists after boot is the local<b= r> little NVME samsung unit.<br> <br> So here I can import those pools and then see that the cachefile <br> property has been wiped out :<br> <br> titan#<br> titan# zpool import proteus<br> titan# zpool import leaf<br> titan#<br> titan# zpool list<br> NAME=C2=A0 =C2=A0 =C2=A0 SIZE=C2=A0 ALLOC=C2=A0 =C2=A0FREE=C2=A0 CKPOINT=C2= =A0 EXPANDSZ=C2=A0 =C2=A0FRAG=C2=A0 =C2=A0 CAP=C2=A0 DEDUP <br> HEALTH=C2=A0 ALTROOT<br> leaf=C2=A0 =C2=A0 =C2=A018.2T=C2=A0 =C2=A0984K=C2=A0 18.2T=C2=A0 =C2=A0 =C2= =A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A00%=C2=A0= =C2=A0 =C2=A00%=C2=A0 1.00x <br> ONLINE=C2=A0 -<br> proteus=C2=A0 1.98T=C2=A0 =C2=A0361G=C2=A0 1.63T=C2=A0 =C2=A0 =C2=A0 =C2=A0= -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A01%=C2=A0 =C2=A0 17= %=C2=A0 1.00x <br> ONLINE=C2=A0 -<br> t0=C2=A0 =C2=A0 =C2=A0 =C2=A0 444G=C2=A0 91.2G=C2=A0 =C2=A0353G=C2=A0 =C2= =A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 27%=C2= =A0 =C2=A0 20%=C2=A0 1.00x <br> ONLINE=C2=A0 -<br> titan#<br> titan# zpool get cachefile leaf<br> NAME=C2=A0 PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 SOURCE<br> leaf=C2=A0 cachefile=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default<br> titan#<br> titan# zpool get cachefile proteus<br> NAME=C2=A0 =C2=A0 =C2=A0PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 SOUR= CE<br> proteus=C2=A0 cachefile=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default<b= r> titan#<br> titan# zpool get cachefile t0<br> NAME=C2=A0 PROPERTY=C2=A0 =C2=A0VALUE=C2=A0 =C2=A0 =C2=A0 SOURCE<br> t0=C2=A0 =C2=A0 cachefile=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default= <br> titan#<br> titan# ls -l /var/log/zpool_cache<br> -rw-r--r--=C2=A0 1 root wheel 4960 Nov 28 11:52 /var/log/zpool_cache<br> titan#<br> <br> The cachefile exists and seems to have grown in size.<br> <br> However a reboot will once again provide nothing but the t0 pool.<br> <br> Baffled.<br> <br> Any thoughts would be welcome.<br> <br> -- <br> --<br> Dennis Clarke<br> RISC-V/SPARC/PPC/ARM/CISC<br> UNIX and Linux spoken<br></blockquote></div></div><div dir=3D"auto"><br></d= iv><div dir=3D"auto">Do you have zfs_enable=3D"YES" set in /etc/r= c.conf? If not then nothing will get imported.=C2=A0</div><div dir=3D"auto"= ><br></div><div dir=3D"auto">Regarding the cachefile property, it's exp= ected that "zpool import" will change it, unless you do "zpo= ol import -O cachefile=3Dwhatever".</div><div dir=3D"auto"><div class= =3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px = 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> </blockquote></div></div></div> --000000000000a8f3c40627f96554--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2hKCYrx92SBLQOtekKiBWMgBy_n93ZGQ_NVLq=6puRhOg>