From nobody Thu Nov 28 13:05:39 2024 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Xzc4S6gWwz5fhLp for ; Thu, 28 Nov 2024 13:05:44 +0000 (UTC) (envelope-from dclarke@blastwave.org) Received: from mail.oetec.com (mail.oetec.com [108.160.241.186]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mail.oetec.com", Issuer "E6" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Xzc4S1Cxmz4kJF for ; Thu, 28 Nov 2024 13:05:44 +0000 (UTC) (envelope-from dclarke@blastwave.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=blastwave.org header.s=default header.b=hAlCXcfE; spf=pass (mx1.freebsd.org: domain of dclarke@blastwave.org designates 108.160.241.186 as permitted sender) smtp.mailfrom=dclarke@blastwave.org; dmarc=pass (policy=quarantine) header.from=blastwave.org Received: from [172.16.35.3] (pool-99-253-118-250.cpe.net.cable.rogers.com [99.253.118.250]) (authenticated bits=0) by mail.oetec.com (8.17.1/8.17.1) with ESMTPSA id 4ASD5d8N026619 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT) for ; Thu, 28 Nov 2024 08:05:41 -0500 (EST) (envelope-from dclarke@blastwave.org) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=blastwave.org; s=default; t=1732799141; bh=rwYNqpusMvqLR/mcUHo5FVUgXHL0k65I13RTn1m3tFg=; h=Date:To:From:Subject; b=hAlCXcfEzANsT+3L3v0CW9IjGLQBZUL6b0XudoDOjuj8Btx0s/ESBwDVxiot1f8Kf yFOXN51ViGhAxp3B+DXebFfHabF5PjWisQinOuVNGkldEWRRgSrAB2UNi5AwpXwKZU Jg0Fa5uPP1YL/9PN2Ypt1ytdKTWxUEn4lI5u4XRYSCUqSYZ5Brq9jwp/7blbqig++p I1dfqri5nm2cr6XLaZjg5wjdCplWEQ9Ah0yzUs76h/s49IGSWW6YBliMov869FpJEq YiBtzSnC/XhPnVeh/sghVPdDgdK5nnjUZrLxsQoI4ILx0yxhI2ImUnHxAPDQmXN8Qp wUKg2W2yjqOrw== Message-ID: <5798b0db-bc73-476a-908a-dd1f071bfe43@blastwave.org> Date: Thu, 28 Nov 2024 08:05:39 -0500 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Current FreeBSD Content-Language: en-CA From: Dennis Clarke Subject: zpools no longer exist after boot Organization: GENUNIX Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-oetec-MailScanner-Information: Please contact the ISP for more information X-oetec-MailScanner-ID: 4ASD5d8N026619 X-oetec-MailScanner: Found to be clean X-oetec-MailScanner-From: dclarke@blastwave.org X-Spam-Status: No X-Spamd-Result: default: False [-4.69 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[blastwave.org,quarantine]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCVD_IN_DNSWL_MED(-0.20)[108.160.241.186:from]; R_DKIM_ALLOW(-0.20)[blastwave.org:s=default]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; XM_UA_NO_VERSION(0.01)[]; TO_DN_ALL(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[blastwave.org:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:812, ipnet:108.160.240.0/20, country:CA]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4Xzc4S1Cxmz4kJF X-Spamd-Bar: ---- This is a baffling problem wherein two zpools no longer exist after boot. This is : titan# uname -apKU FreeBSD titan 15.0-CURRENT FreeBSD 15.0-CURRENT #1 main-n273749-4b65481ac68a-dirty: Wed Nov 20 15:08:52 GMT 2024 root@titan:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG amd64 amd64 1500027 1500027 titan# titan# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT t0 444G 91.2G 353G - - 27% 20% 1.00x ONLINE - titan# The *only* zpool that seems to exist in any reliable way is the little NVME based unit for booting. The other two zpools vanished and yet the devices exist just fine : titan# titan# camcontrol devlist at scbus0 target 0 lun 0 (pass0,ada0) at scbus1 target 0 lun 0 (pass1,ada1) at scbus2 target 0 lun 0 (ses0,pass2) at scbus6 target 0 lun 0 (ses1,pass3) at scbus7 target 0 lun 1 (pass4,nda0) at scbus8 target 0 lun 0 (da0,pass5) titan# titan# nvmecontrol devlist nvme0: SAMSUNG MZVKW512HMJP-000L7 nvme0ns1 (488386MB) titan# titan# zpool status t0 pool: t0 state: ONLINE status: Some supported and requested features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: scrub repaired 0B in 00:00:44 with 0 errors on Wed Feb 7 09:56:40 2024 config: NAME STATE READ WRITE CKSUM t0 ONLINE 0 0 0 nda0p3 ONLINE 0 0 0 errors: No known data errors titan# Initially I thought the problem was related to cachefile being empty for these zpools. However if I set the cachefile to something reasonable then the cachefile property vanishes at a reboot. The file, of course, exists just fine : titan# zpool get cachefile proteus NAME PROPERTY VALUE SOURCE proteus cachefile - default titan# titan# zpool set cachefile="/var/log/zpool_cache" proteus titan# zpool get cachefile proteus NAME PROPERTY VALUE SOURCE proteus cachefile /var/log/zpool_cache local titan# ls -ladb /var/log/zpool_cache -rw-r--r-- 1 root wheel 1440 Nov 28 11:45 /var/log/zpool_cache titan# So there we have 1440 bytes of data in that file. titan# zpool set cachefile="/var/log/zpool_cache" t0 titan# zpool get cachefile t0 NAME PROPERTY VALUE SOURCE t0 cachefile /var/log/zpool_cache local titan# titan# ls -ladb /var/log/zpool_cache -rw-r--r-- 1 root wheel 2880 Nov 28 11:46 /var/log/zpool_cache titan# Now we have 2 * 1440 bytes = 2880 bytes of some zpool cache data. titan# zpool set cachefile="/var/log/zpool_cache" leaf titan# zpool get cachefile leaf NAME PROPERTY VALUE SOURCE leaf cachefile /var/log/zpool_cache local titan# titan# zpool get cachefile t0 NAME PROPERTY VALUE SOURCE t0 cachefile /var/log/zpool_cache local titan# titan# zpool get cachefile proteus NAME PROPERTY VALUE SOURCE proteus cachefile /var/log/zpool_cache local titan# titan# reboot From here on ... the only zpool that exists after boot is the local little NVME samsung unit. So here I can import those pools and then see that the cachefile property has been wiped out : titan# titan# zpool import proteus titan# zpool import leaf titan# titan# zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT leaf 18.2T 984K 18.2T - - 0% 0% 1.00x ONLINE - proteus 1.98T 361G 1.63T - - 1% 17% 1.00x ONLINE - t0 444G 91.2G 353G - - 27% 20% 1.00x ONLINE - titan# titan# zpool get cachefile leaf NAME PROPERTY VALUE SOURCE leaf cachefile - default titan# titan# zpool get cachefile proteus NAME PROPERTY VALUE SOURCE proteus cachefile - default titan# titan# zpool get cachefile t0 NAME PROPERTY VALUE SOURCE t0 cachefile - default titan# titan# ls -l /var/log/zpool_cache -rw-r--r-- 1 root wheel 4960 Nov 28 11:52 /var/log/zpool_cache titan# The cachefile exists and seems to have grown in size. However a reboot will once again provide nothing but the t0 pool. Baffled. Any thoughts would be welcome. -- -- Dennis Clarke RISC-V/SPARC/PPC/ARM/CISC UNIX and Linux spoken