Date: Thu, 28 Nov 2024 14:49:59 +0100 (CET) From: Ronald Klop <ronald-lists@klop.ws> To: Dennis Clarke <dclarke@blastwave.org> Cc: Current FreeBSD <freebsd-current@freebsd.org> Subject: Re: zpools no longer exist after boot Message-ID: <1784014555.6851.1732801799724@localhost> In-Reply-To: <5798b0db-bc73-476a-908a-dd1f071bfe43@blastwave.org>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_6850_62102694.1732801799721 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Are the other disks available at the moment the boot process does zpool import? Regards, Ronald Van: Dennis Clarke <dclarke@blastwave.org> Datum: 28 november 2024 14:06 Aan: Current FreeBSD <freebsd-current@freebsd.org> Onderwerp: zpools no longer exist after boot > > > > This is a baffling problem wherein two zpools no longer exist after > boot. This is : > > titan# uname -apKU > FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1 main-n273749-4b65481ac68a-dirty: Wed Nov 20 15:08:52 GMT 2024 root@titan:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 amd64 1500027 1500027 > titan# > > titan# zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > t0 444G 91.2G 353G - - 27% 20% 1.00x ONLINE - > titan# > > The *only* zpool that seems to exist in any reliable way is the little > NVME based unit for booting. The other two zpools vanished and yet the > devices exist just fine : > > titan# > titan# camcontrol devlist > at scbus0 target 0 lun 0 (pass0,ada0) > at scbus1 target 0 lun 0 (pass1,ada1) > 0001=""> at scbus2 target 0 lun 0 (ses0,pass2) > 0001=""> at scbus6 target 0 lun 0 (ses1,pass3) > at scbus7 target 0 lun 1 (pass4,nda0) > at scbus8 target 0 lun 0 (da0,pass5) > titan# > titan# nvmecontrol devlist > nvme0: SAMSUNG MZVKW512HMJP-000L7 > nvme0ns1 (488386MB) > titan# > titan# zpool status t0 > pool: t0 > state: ONLINE > status: Some supported and requested features are not enabled on the pool. > The pool can still be used, but some features are unavailable. > action: Enable all features using 'zpool upgrade'. Once this is done, > the pool may no longer be accessible by software that does not support > the features. See zpool-features(7) for details. > scan: scrub repaired 0B in 00:00:44 with 0 errors on Wed Feb 7 09:56:40 2024 > config: > > NAME STATE READ WRITE CKSUM > t0 ONLINE 0 0 0 > nda0p3 ONLINE 0 0 0 > > errors: No known data errors > titan# > > > Initially I thought the problem was related to cachefile being empty for > these zpools. However if I set the cachefile to something reasonable > then the cachefile property vanishes at a reboot. The file, of course, exists just fine : > > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile - default > titan# > titan# zpool set cachefile="/var/log/zpool_cache" proteus > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile /var/log/zpool_cache local > titan# ls -ladb /var/log/zpool_cache > -rw-r--r-- 1 root wheel 1440 Nov 28 11:45 /var/log/zpool_cache > titan# > > So there we have 1440 bytes of data in that file. > > titan# zpool set cachefile="/var/log/zpool_cache" t0 > titan# zpool get cachefile t0 > NAME PROPERTY VALUE SOURCE > t0 cachefile /var/log/zpool_cache local > titan# > titan# ls -ladb /var/log/zpool_cache > -rw-r--r-- 1 root wheel 2880 Nov 28 11:46 /var/log/zpool_cache > titan# > > Now we have 2 * 1440 bytes = 2880 bytes of some zpool cache data. > > titan# zpool set cachefile="/var/log/zpool_cache" leaf > titan# zpool get cachefile leaf > NAME PROPERTY VALUE SOURCE > leaf cachefile /var/log/zpool_cache local > titan# > titan# zpool get cachefile t0 > NAME PROPERTY VALUE SOURCE > t0 cachefile /var/log/zpool_cache local > titan# > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile /var/log/zpool_cache local > titan# > titan# reboot > > From here on ... the only zpool that exists after boot is the local > little NVME samsung unit. > > So here I can import those pools and then see that the cachefile property has been wiped out : > > titan# > titan# zpool import proteus > titan# zpool import leaf > titan# > titan# zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > leaf 18.2T 984K 18.2T - - 0% 0% 1.00x ONLINE - > proteus 1.98T 361G 1.63T - - 1% 17% 1.00x ONLINE - > t0 444G 91.2G 353G - - 27% 20% 1.00x ONLINE - > titan# > titan# zpool get cachefile leaf > NAME PROPERTY VALUE SOURCE > leaf cachefile - default > titan# > titan# zpool get cachefile proteus > NAME PROPERTY VALUE SOURCE > proteus cachefile - default > titan# > titan# zpool get cachefile t0 > NAME PROPERTY VALUE SOURCE > t0 cachefile - default > titan# > titan# ls -l /var/log/zpool_cache > -rw-r--r-- 1 root wheel 4960 Nov 28 11:52 /var/log/zpool_cache > titan# > > The cachefile exists and seems to have grown in size. > > However a reboot will once again provide nothing but the t0 pool. > > Baffled. > > Any thoughts would be welcome. > > -- > -- > Dennis Clarke > RISC-V/SPARC/PPC/ARM/CISC > UNIX and Linux spoken > > > > > ------=_Part_6850_62102694.1732801799721 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit <html><head></head><body>Are the other disks available at the moment the boot process does zpool import?<div><br></div><div>Regards,</div><div>Ronald</div><div><br><p><small><strong>Van:</strong> Dennis Clarke <dclarke@blastwave.org><br><strong>Datum:</strong> 28 november 2024 14:06<br><strong>Aan:</strong> Current FreeBSD <freebsd-current@freebsd.org><br><strong>Onderwerp:</strong> zpools no longer exist after boot<br></small></p><blockquote style="margin-left: 5px; border-left: 3px solid #ccc; margin-right: 0px; padding-left: 5px;"><div class="MessageRFC822Viewer do_not_remove" id="P"><!-- P --> <!-- processMimeMessage --><div class="TextPlainViewer do_not_remove" id="P.P"><!-- P.P --><br> This is a baffling problem wherein two zpools no longer exist after<br> boot. This is :<br> <br> titan# uname -apKU<br> FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1 main-n273749-4b65481ac68a-dirty: Wed Nov 20 15:08:52 GMT 2024 root@titan:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 amd64 1500027 1500027<br> titan#<br> <br> titan# zpool list<br> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT<br> t0 444G 91.2G 353G - - 27% 20% 1.00x ONLINE -<br> titan#<br> <br> The *only* zpool that seems to exist in any reliable way is the little<br> NVME based unit for booting. The other two zpools vanished and yet the<br> devices exist just fine :<br> <br> titan#<br> titan# camcontrol devlist<br> <st20000nm007d-3dj103 sn03=""> at scbus0 target 0 lun 0 (pass0,ada0)<br> <st20000nm007d-3dj103 sn03=""> at scbus1 target 0 lun 0 (pass1,ada1)<br> <ahci sgpio="" enclosure="" 2.00="" 0001=""> at scbus2 target 0 lun 0 (ses0,pass2)<br> <ahci sgpio="" enclosure="" 2.00="" 0001=""> at scbus6 target 0 lun 0 (ses1,pass3)<br> <samsung mzvkw512hmjp-000l7="" 6l6qcxa7=""> at scbus7 target 0 lun 1 (pass4,nda0)<br> <freebsd ctldisk="" 0001=""> at scbus8 target 0 lun 0 (da0,pass5)<br> titan#<br> titan# nvmecontrol devlist<br> nvme0: SAMSUNG MZVKW512HMJP-000L7<br> nvme0ns1 (488386MB)<br> titan#<br> titan# zpool status t0<br> pool: t0<br> state: ONLINE<br> status: Some supported and requested features are not enabled on the pool.<br> The pool can still be used, but some features are unavailable.<br> action: Enable all features using 'zpool upgrade'. Once this is done,<br> the pool may no longer be accessible by software that does not support<br> the features. See zpool-features(7) for details.<br> scan: scrub repaired 0B in 00:00:44 with 0 errors on Wed Feb 7 09:56:40 2024<br> config:<br> <br> NAME STATE READ WRITE CKSUM<br> t0 ONLINE 0 0 0<br> nda0p3 ONLINE 0 0 0<br> <br> errors: No known data errors<br> titan#<br> <br> <br> Initially I thought the problem was related to cachefile being empty for<br> these zpools. However if I set the cachefile to something reasonable<br> then the cachefile property vanishes at a reboot. The file, of course, exists just fine :<br> <br> titan# zpool get cachefile proteus<br> NAME PROPERTY VALUE SOURCE<br> proteus cachefile - default<br> titan#<br> titan# zpool set cachefile="/var/log/zpool_cache" proteus<br> titan# zpool get cachefile proteus<br> NAME PROPERTY VALUE SOURCE<br> proteus cachefile /var/log/zpool_cache local<br> titan# ls -ladb /var/log/zpool_cache<br> -rw-r--r-- 1 root wheel 1440 Nov 28 11:45 /var/log/zpool_cache<br> titan#<br> <br> So there we have 1440 bytes of data in that file.<br> <br> titan# zpool set cachefile="/var/log/zpool_cache" t0<br> titan# zpool get cachefile t0<br> NAME PROPERTY VALUE SOURCE<br> t0 cachefile /var/log/zpool_cache local<br> titan#<br> titan# ls -ladb /var/log/zpool_cache<br> -rw-r--r-- 1 root wheel 2880 Nov 28 11:46 /var/log/zpool_cache<br> titan#<br> <br> Now we have 2 * 1440 bytes = 2880 bytes of some zpool cache data.<br> <br> titan# zpool set cachefile="/var/log/zpool_cache" leaf<br> titan# zpool get cachefile leaf<br> NAME PROPERTY VALUE SOURCE<br> leaf cachefile /var/log/zpool_cache local<br> titan#<br> titan# zpool get cachefile t0<br> NAME PROPERTY VALUE SOURCE<br> t0 cachefile /var/log/zpool_cache local<br> titan#<br> titan# zpool get cachefile proteus<br> NAME PROPERTY VALUE SOURCE<br> proteus cachefile /var/log/zpool_cache local<br> titan#<br> titan# reboot<br> <br> From here on ... the only zpool that exists after boot is the local<br> little NVME samsung unit.<br> <br> So here I can import those pools and then see that the cachefile property has been wiped out :<br> <br> titan#<br> titan# zpool import proteus<br> titan# zpool import leaf<br> titan#<br> titan# zpool list<br> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT<br> leaf 18.2T 984K 18.2T - - 0% 0% 1.00x ONLINE -<br> proteus 1.98T 361G 1.63T - - 1% 17% 1.00x ONLINE -<br> t0 444G 91.2G 353G - - 27% 20% 1.00x ONLINE -<br> titan#<br> titan# zpool get cachefile leaf<br> NAME PROPERTY VALUE SOURCE<br> leaf cachefile - default<br> titan#<br> titan# zpool get cachefile proteus<br> NAME PROPERTY VALUE SOURCE<br> proteus cachefile - default<br> titan#<br> titan# zpool get cachefile t0<br> NAME PROPERTY VALUE SOURCE<br> t0 cachefile - default<br> titan#<br> titan# ls -l /var/log/zpool_cache<br> -rw-r--r-- 1 root wheel 4960 Nov 28 11:52 /var/log/zpool_cache<br> titan#<br> <br> The cachefile exists and seems to have grown in size.<br> <br> However a reboot will once again provide nothing but the t0 pool.<br> <br> Baffled.<br> <br> Any thoughts would be welcome.<br> <br> -- <br> --<br> Dennis Clarke<br> RISC-V/SPARC/PPC/ARM/CISC<br> UNIX and Linux spoken<br> <br> </freebsd></samsung></ahci></ahci></st20000nm007d-3dj103></st20000nm007d-3dj103></div><!-- TextPlainViewer --> <hr> </div><!-- MessageRFC822Viewer --> </blockquote><br><br><br></div></body></html> ------=_Part_6850_62102694.1732801799721--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1784014555.6851.1732801799724>