From nobody Thu Nov 28 13:58:29 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XzdFN0RCtz5fkbb for ; Thu, 28 Nov 2024 13:58:32 +0000 (UTC) (envelope-from SRS0=O5G7=SX=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4XzdFM2SvFz4sT2 for ; Thu, 28 Nov 2024 13:58:31 +0000 (UTC) (envelope-from SRS0=O5G7=SX=klop.ws=ronald-lists@realworks.nl) Authentication-Results: mx1.freebsd.org; none Date: Thu, 28 Nov 2024 14:58:29 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1732802309; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=gMIe/uasxtbsqZL0vvAZcxrB5N8fUd3nvlt4+l7Jkjc=; b=kDyVGQnk/ecaGnu4sDPWFvUM0OLBwdgpdRsQcgK6Io0SIlL3+OEur33kcu6SiHme2YrjQB FBaumEMz+B4fhk3Eh6femtpoGqIDpwyqjWfdXb6Ztrg5SZzqPgqj1VLjAmrvfu0DXGet39 i/pGipQekQymXco6O0gd/wT/PHrdA79gStn9dqaGPMeG0lxJ+zox+4myrnxvpCx4gZ08cA ZvUy1IBV9llXdNCjdIAz+fBHMwFWgQTIP8Tu5T+3jLtQla8sxGNlL/oh3lVXyB58lkXghD UgakRUsSv6/+X9F6Zrmh2O4yx4SY8TUEdF5noPNXf+asYm25uS/R7zK3jrP/iw== From: Ronald Klop To: Ronald Klop Cc: Current FreeBSD , Dennis Clarke Message-ID: <1764191396.6959.1732802309600@localhost> In-Reply-To: <1784014555.6851.1732801799724@localhost> Subject: Re: zpools no longer exist after boot List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_6958_1940271980.1732802309596" X-Mailer: Realworks (730.92) Importance: Normal X-Priority: 3 (Normal) X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL] X-Rspamd-Queue-Id: 4XzdFM2SvFz4sT2 X-Spamd-Bar: ---- ------=_Part_6958_1940271980.1732802309596 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Btw: The /etc/rc.d/zpool script looks into these cachefiles: for cachefile in /etc/zfs/zpool.cache /boot/zfs/zpool.cache; do I didn=E2=80=99t check where the cachefile pool property is used.=20 Hope this helps resolving the issue. Or maybe helps you to provide more inf= ormation about your setup.=20 Regards, Ronald.=20 Van: Ronald Klop Datum: 28 november 2024 14:50 Aan: Dennis Clarke CC: Current FreeBSD Onderwerp: Re: zpools no longer exist after boot >=20 >=20 >=20 > Are the other disks available at the moment the boot process does zpool i= mport? >=20 > Regards, > Ronald >=20 > Van: Dennis Clarke=20 > Datum: 28 november 2024 14:06 > Aan: Current FreeBSD=20 > Onderwerp: zpools no longer exist after boot >=20 >>=20 >> This is a baffling problem wherein two zpools no longer exist after >> boot. This is : >>=20 >> titan# uname -apKU >> FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1 main-n273749-4b65481a= c68a-dirty: Wed Nov 20 15:08:52 GMT 2024 root@titan:/usr/obj/usr/src/amd64.= amd64/sys/GENERIC-NODEBUG amd64 amd64 1500027 1500027 >> titan# >>=20 >> titan# zpool list >> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH= ALTROOT >> t0 444G 91.2G 353G - - 27% 20% 1.00x ONLINE= - >> titan# >>=20 >> The *only* zpool that seems to exist in any reliable way is the little >> NVME based unit for booting. The other two zpools vanished and yet the >> devices exist just fine : >>=20 >> titan# >> titan# camcontrol devlist >> at scbus0 target 0 lun 0 (pass0,ada0) >> at scbus1 target 0 lun 0 (pass1,ada1) >> at scbus2 target 0 lun 0 (ses0,pass2) >> at scbus6 target 0 lun 0 (ses1,pass3) >> at scbus7 target 0 lun 1 (pass4,nda0) >> at scbus8 target 0 lun 0 (da0,pass5) >> titan# >> titan# nvmecontrol devlist >> nvme0: SAMSUNG MZVKW512HMJP-000L7 >> nvme0ns1 (488386MB) >> titan# >> titan# zpool status t0 >> pool: t0 >> state: ONLINE >> status: Some supported and requested features are not enabled on the poo= l. >> The pool can still be used, but some features are unavailable. >> action: Enable all features using 'zpool upgrade'. Once this is done, >> the pool may no longer be accessible by software that does not = support >> the features. See zpool-features(7) for details. >> scan: scrub repaired 0B in 00:00:44 with 0 errors on Wed Feb 7 09:56= :40 2024 >> config: >>=20 >> NAME STATE READ WRITE CKSUM >> t0 ONLINE 0 0 0 >> nda0p3 ONLINE 0 0 0 >>=20 >> errors: No known data errors >> titan# >>=20 >>=20 >> Initially I thought the problem was related to cachefile being empty for >> these zpools. However if I set the cachefile to something reasonable >> then the cachefile property vanishes at a reboot. The file, of course, = exists just fine : >>=20 >> titan# zpool get cachefile proteus >> NAME PROPERTY VALUE SOURCE >> proteus cachefile - default >> titan# >> titan# zpool set cachefile=3D"/var/log/zpool_cache" proteus >> titan# zpool get cachefile proteus >> NAME PROPERTY VALUE SOURCE >> proteus cachefile /var/log/zpool_cache local >> titan# ls -ladb /var/log/zpool_cache >> -rw-r--r-- 1 root wheel 1440 Nov 28 11:45 /var/log/zpool_cache >> titan# >>=20 >> So there we have 1440 bytes of data in that file. >>=20 >> titan# zpool set cachefile=3D"/var/log/zpool_cache" t0 >> titan# zpool get cachefile t0 >> NAME PROPERTY VALUE SOURCE >> t0 cachefile /var/log/zpool_cache local >> titan# >> titan# ls -ladb /var/log/zpool_cache >> -rw-r--r-- 1 root wheel 2880 Nov 28 11:46 /var/log/zpool_cache >> titan# >>=20 >> Now we have 2 * 1440 bytes =3D 2880 bytes of some zpool cache data. >>=20 >> titan# zpool set cachefile=3D"/var/log/zpool_cache" leaf >> titan# zpool get cachefile leaf >> NAME PROPERTY VALUE SOURCE >> leaf cachefile /var/log/zpool_cache local >> titan# >> titan# zpool get cachefile t0 >> NAME PROPERTY VALUE SOURCE >> t0 cachefile /var/log/zpool_cache local >> titan# >> titan# zpool get cachefile proteus >> NAME PROPERTY VALUE SOURCE >> proteus cachefile /var/log/zpool_cache local >> titan# >> titan# reboot >>=20 >> From here on ... the only zpool that exists after boot is the local >> little NVME samsung unit. >>=20 >> So here I can import those pools and then see that the cachefile propert= y has been wiped out : >>=20 >> titan# >> titan# zpool import proteus >> titan# zpool import leaf >> titan# >> titan# zpool list >> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEA= LTH ALTROOT >> leaf 18.2T 984K 18.2T - - 0% 0% 1.00x ONL= INE - >> proteus 1.98T 361G 1.63T - - 1% 17% 1.00x ONL= INE - >> t0 444G 91.2G 353G - - 27% 20% 1.00x ONL= INE - >> titan# >> titan# zpool get cachefile leaf >> NAME PROPERTY VALUE SOURCE >> leaf cachefile - default >> titan# >> titan# zpool get cachefile proteus >> NAME PROPERTY VALUE SOURCE >> proteus cachefile - default >> titan# >> titan# zpool get cachefile t0 >> NAME PROPERTY VALUE SOURCE >> t0 cachefile - default >> titan# >> titan# ls -l /var/log/zpool_cache >> -rw-r--r-- 1 root wheel 4960 Nov 28 11:52 /var/log/zpool_cache >> titan# >>=20 >> The cachefile exists and seems to have grown in size. >>=20 >> However a reboot will once again provide nothing but the t0 pool. >>=20 >> Baffled. >>=20 >> Any thoughts would be welcome. >>=20 >> --=20 >> -- >> Dennis Clarke >> RISC-V/SPARC/PPC/ARM/CISC >> UNIX and Linux spoken >>=20 >>=20 >>=20 >>=20 >>=20 >=20 >=20 >=20 >=20 >=20 >=20 ------=_Part_6958_1940271980.1732802309596 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable Btw:

The /etc/rc.d/zpool script= looks into these cachefiles:

for cachefile in /et= c/zfs/zpool.cache /boot/zfs/zpool.cache; do

I didn= =E2=80=99t check where the cachefile pool property is used. 

Hope this helps resolving the issue. Or maybe helps you to = provide more information about your setup. 

R= egards,
Ronald. 

Van: Ronald Klop <ronald-lists@klop.ws>
Datum: 28 n= ovember 2024 14:50
Aan: Dennis Clarke <dclarke@blast= wave.org>
CC: Current FreeBSD <freebsd-current@fr= eebsd.org>
Onderwerp: Re: zpools no longer exist aft= er boot

Are the other disks available at the moment the boot process does zpool imp= ort?

Reg= ards,
Ronald

Van: Dennis Clarke
Datum: 28 november 2024 14:06
Aan: Current FreeBSD
Onderwerp: zpools no longer exist after boot
<= /dclarke@blastwave.org>


This is a baffling problem wherein two zpools no longer exist after
boot. This is :

titan# uname -apKU
FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1 main-n273749-4b65481ac68= a-dirty: Wed Nov 20 15:08:52 GMT 2024 root@titan:/usr/obj/usr/src/amd64.amd= 64/sys/GENERIC-NODEBUG amd64 amd64 1500027 1500027
titan#

titan# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPA= NDSZ   FRAG    CAP  DEDUP HEALTH  ALTROO= T
t0     444G  91.2G   353G   &n= bsp;    -        &nb= sp;-    27%    20%  1.00x ONLINE  -=
titan#

The *only* zpool that seems to exist in any reliable way is the little
NVME based unit for booting. The other two zpools vanished and yet the
devices exist just fine :

titan#
titan# camcontrol devlist
       at scbus0 target 0 lun 0 (pass0,= ada0)
       at scbus1 target 0 lun 0 (pass1,= ada1)
  at scbus2 target 0 lun 0 (ses0,pass2)
  at scbus6 target 0 lun 0 (ses1,pass3)
 at scbus7 target 0 lun 1 (pass4,nda0)
            at= scbus8 target 0 lun 0 (da0,pass5)
titan#
titan# nvmecontrol devlist
  nvme0: SAMSUNG MZVKW512HMJP-000L7
     nvme0ns1 (488386MB)
titan#
titan# zpool status t0
   pool: t0
  state: ONLINE
status: Some supported and requested features are not enabled on the pool.<= br>          The pool can still be= used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
         the pool may no longe= r be accessible by software that does not support
         the features. See zpo= ol-features(7) for details.
   scan: scrub repaired 0B in 00:00:44 with 0 errors on Wed = Feb  7 09:56:40 2024
config:

         NAME   &nbs= p;    STATE     READ WRITE CKSUM          t0    =       ONLINE      &n= bsp;0     0     0
           nda0p3 &n= bsp;  ONLINE       0   &n= bsp; 0     0

errors: No known data errors
titan#


Initially I thought the problem was related to cachefile being empty for these zpools. However if I set the cachefile to something reasonable
then the cachefile property vanishes at a reboot.  The file, of course= , exists just fine :

titan# zpool get cachefile proteus
NAME     PROPERTY   VALUE    &= nbsp; SOURCE
proteus  cachefile  -        &= nbsp; default
titan#
titan# zpool set cachefile=3D"/var/log/zpool_cache" proteus
titan# zpool get cachefile proteus
NAME     PROPERTY   VALUE    &= nbsp;           &nbs= p;SOURCE
proteus  cachefile  /var/log/zpool_cache  local
titan# ls -ladb /var/log/zpool_cache
-rw-r--r--  1 root wheel 1440 Nov 28 11:45 /var/log/zpool_cache
titan#

So there we have 1440 bytes of data in that file.

titan# zpool set cachefile=3D"/var/log/zpool_cache" t0
titan# zpool get cachefile t0
NAME  PROPERTY   VALUE       &= nbsp;         SOURCE
t0    cachefile  /var/log/zpool_cache  local
titan#
titan# ls -ladb /var/log/zpool_cache
-rw-r--r--  1 root wheel 2880 Nov 28 11:46 /var/log/zpool_cache
titan#

Now we have 2 * 1440 bytes =3D 2880 bytes of some zpool cache data.

titan# zpool set cachefile=3D"/var/log/zpool_cache" leaf
titan# zpool get cachefile leaf
NAME  PROPERTY   VALUE       &= nbsp;         SOURCE
leaf  cachefile  /var/log/zpool_cache  local
titan#
titan# zpool get cachefile t0
NAME  PROPERTY   VALUE       &= nbsp;         SOURCE
t0    cachefile  /var/log/zpool_cache  local
titan#
titan# zpool get cachefile proteus
NAME     PROPERTY   VALUE    &= nbsp;           &nbs= p;SOURCE
proteus  cachefile  /var/log/zpool_cache  local
titan#
titan# reboot

 From here on ... the only zpool that exists after boot is the local little NVME samsung unit.

So here I can import those pools and then see that the cachefile property h= as been wiped out :

titan#
titan# zpool import proteus
titan# zpool import leaf
titan#
titan# zpool list
NAME      SIZE  ALLOC   FREE  = CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP H= EALTH  ALTROOT
leaf     18.2T   984K  18.2T   = ;     -        =  -     0%     0%  1.00x O= NLINE  -
proteus  1.98T   361G  1.63T     &n= bsp;  -         -  &= nbsp;  1%    17%  1.00x ONLINE  -
t0        444G  91.2G   3= 53G        -     &nb= sp;   -    27%    20%  1.= 00x ONLINE  -
titan#
titan# zpool get cachefile leaf
NAME  PROPERTY   VALUE      SOURCE<= br> leaf  cachefile  -        &nbs= p; default
titan#
titan# zpool get cachefile proteus
NAME     PROPERTY   VALUE    &= nbsp; SOURCE
proteus  cachefile  -        &= nbsp; default
titan#
titan# zpool get cachefile t0
NAME  PROPERTY   VALUE      SOURCE<= br> t0    cachefile  -       =    default
titan#
titan# ls -l /var/log/zpool_cache
-rw-r--r--  1 root wheel 4960 Nov 28 11:52 /var/log/zpool_cache
titan#

The cachefile exists and seems to have grown in size.

However a reboot will once again provide nothing but the t0 pool.

Baffled.

Any thoughts would be welcome.

-- 
--
Dennis Clarke
RISC-V/SPARC/PPC/ARM/CISC
UNIX and Linux spoken







------=_Part_6958_1940271980.1732802309596--