From nobody Sat Sep 16 15:43:49 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RnwMs1M74z4tW0N for ; Sat, 16 Sep 2023 15:44:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-21.consmr.mail.gq1.yahoo.com (sonic310-21.consmr.mail.gq1.yahoo.com [98.137.69.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RnwMq67gsz4cgB for ; Sat, 16 Sep 2023 15:44:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=HcgJC1aK; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1694879045; bh=aoKPLPM5y3dPHQYJCXshWiGtAMorY2twpx3KSBwS6hc=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=HcgJC1aK7Y239nATgfVVKbSolYVO/IFlL5tY2Mf5eXyfntugKAXTPvCe/lv4TTcWqRxoX3Ft+tgpcy6SBbsijpsFHj4ht2Pk2fmxMCKrrW1aOiUuLfD1TMkMlAk8oqIhMOOdC5SGdThqpP05uarWSPWbdD4IEvMB32IJz5LwVAuEI5nIlcuixjuD9B5slcKG+Dz3gBMD5RcJOavzj+re3xSfLCIwXc/bOi0RH5v7/BYWT8+j6BgNODE8ZfDfxHTBU/W78lnN/TzF9B52+0iKufd4FVWzbtQ8SEJaPVvpChatGhh9+dj/9zTWZVPk3iQ3Ms0cskD39D65ozXaTHlt2A== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1694879045; bh=XDCl9GSAjv8r+3NLLhL+1EU39Fo9p9Nx3OMPfeH2NWH=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Mbbgyya3So48QuU80CBWnM98vt/IjIRzdUWIlBE+nkeba9EylCWLtetlFy9XXkxc7bLCR+iqrrSaLqY1pf1F61KYXLrFM9T1EvLeUfQXRSC8NAYsTecnNxJ/6CJyAM+mAIsRTxZuMlJI0i6CW+YvBuzOctHMW1P/ffHW34Slb1oht6cXWZtSCLGiK06CI0JNcyAcBQfyUQ0i8F1MvNh4MmwCrPmUKLxFIM7G0TWmqaf0wJUp49i4ZrRSbQeRJVytuke3X/oE+GLkAS9J5vkZA9bsLlA6BgQotctaWYzbxqK5d4dbtEYng+AxeYbw0Foc7j1eUUhzyk2Onl7GP6uOzA== X-YMail-OSG: C.b5LOwVM1kvbcz.Ogad41gnxldPkxwzlKFBR8BDGdcgcszELzzNs_2cKpwxb7_ .HRJy6IWLLpVrxSuFsSnL_.xJdYsRRDwfYBqjvERXUfjY.iFrSyNOvEew2UalPEdvDaJ75eNvq_J WfBZM1BMt0OScorZRMGjGqDfq5G2zPVdgCejc6ofVHlkaoGZkRTu.O8ZRaWGqD_FQjrFb_yuqAu9 ZnF6RshvSfCcZASX_3nwj8jXXBiD2PxF2oh7Dadt6Ztmtvqt1KJZzi9Ue30dUmEZtVg2vWhtx9dJ bcT0ThBa1sHCpHZoooBgf8VN.gCpBiMcaKIw9YpRzZAMIndsCYm3Ujea_6k6vTdMETqavyqWfQUj TVQ4HEsvSmJAJrngDGfsoAkgr3CN5L9D0faDRzGv06IB_kLBoXTYYpE2YeSozLqsvqANDmBxseJu rPabB977UPRzCg40OaQMaxfu4uRJPTbs0OU5tPAEkI16xkgDKF5a5Jv649Kw0e6SQrj4J3VilMal giCzVVfixg4ywW0nKtyEHZq_ip0h8m6c.yGhkUAdbnRlZ9iJiL_T3aSnBO8CoGjvL8bhFvzd8iWm X6jWdKrU7a.lFdxgvSdAVr.02BU2fDxSmZks6B74rXOKEVUfh0RVwnjHQkYgXkmrZnLPZOhmaAN_ O_XriilIfyQHlbgik42C6sVS.Yf4YegrWQwuTeYsvjctB5qKRsCrNn2sIqEnVROAkJnbpSaB9LS3 .yEiM.iMYLXkIKeoE.C8av6jL85zS8PhG_wOxK2L1PDUMEm75T.144jPqyDSNhk7wdWih4AoXR0l zK1Q2gclgYsrDa6X1Lfr6hrZ1mRhUNwfO1IO8ooOgE2RfwIUIwxjTrtST1yk1WHEJpQTiGJ16QBi 9NULNu.D9i9xt2oJuD9hFNDBahcfdzs2ugl.c3W8Fntpf4IkQzpI2gCdAU9W0OmwJSb6BcPNDjIz HZ2MuIonik2bGbMLVwCo.e6P1nr0G2Ue_Djeqk.6TEh99dRV3fbbGUIp7_qjQjmAiY6YOuwryeye l4TQuiPn2kQqJbFqdvpPEZjby.In0KgJROZZWnWUeJ81yYKK9w1RWEoW0xVqBTk5y4vRXOEg5VvO dvMaW8YS8Bs74RchrdgyMrwgFateBHP5KmnBf1G47uoJNpsPLjqG.9dYl3htKpm1K99bHbPXIals Y._lPEkUJfk7H13Ksxs.lQIgQJgVoHCQ3cliG0bPsb_TT_1jDmLZeAza7WiMazoRuq4pO2tg0_Pe UQ3Tj1B_JKrwBtbSyZ2KueekLOq4_zeMHwzDEs7ewaSsPLW1oihALPsRf_zAos5Ls818eDAI_PoS xOKSl0emvvCpu0yrL6dbtJ_M8QUvg0KWTHM0uUSGLLvFPQ2xRZKwMkae5aTpx.xkSwRBbc16KjwR 3XZFDgIlq9uHLCoU6qElfOMCunl_7AVM6qpG_T.xj2I5xU.c1MG8PdFgtx5g.WqGr1jTAjFoBfQi oQr_PWF3PzgrgWK1jAz4GC5XI8Gfv_rUmrJ2VhEx6yTdScxR8DO26J7NIE6OVL4aJ.7KXhUPVWHe sC3R7DjaMq3pXFl9MO0SwyNzZz90louwoiqoAOoPxTaPbDsA4WQFWPV2Y1LKYuqY5Ih1.uO66NxN kY5TQtFYLhgZ9Z5y9kNi9AHbMoEA80OJZv8ZZXoZPrqhgdeqfwx9mnBUvdWY_v_rPKvXisaesR1m cNQcj6yfQ.r4ezZGxnsMxfNlnT8t2elsTCL9xrGvpHQGHg3b2Pe.Fo9tDCtXK7BjagNsbZdyjpR8 Wj2ayoPpkrjjgC94kNrSeVipgv3VZ93B0Z2cleKqw_lXlGXA1V49mLOC152wvygrtuYELv.42iN_ .uDSFQY3FsNp2YJoP2zjRb213G3eq_qdBpCCpmwPQv1diRgW.po3DyynaRckGgoaEzOZGsMSLCrh S.p13ot0kSiTSk2iJeVFH7y5.82AsXnWhK4daIvuz7hw4DsH1mHPJPsNO__B2x9uAgZPMekmNUsz tohQxOk0dymErok3nnIQHr4WJ78C86Ypo3O2sS3VehKD3k_GX8Hcaiep7LMhkJlO_pPT5mLG9RW5 lCh0Cv1LblelBBam3_1l3ZvYlqqbCB3CczCNwsU3fDPEf6AYugSU9d5LpPQJNTR1bbutd7r5MR1c 6v1rl0QWyVbwKPxmLp8oMvZMzB_ZPeEgLe_zrG1oyX28tLp09MiY0Mat8_Ct8C.jl8b99uy5TOO2 _dpbK9C6djE.qy3CzwtnHEtyH4qZA6N5gekNqy5CdiHru7D7XAGNAcpj32y7UkOKyo90Xrq3P5Kz 6 X-Sonic-MF: X-Sonic-ID: 62f9fbd5-9b11-44a5-ae17-32416568eece Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Sat, 16 Sep 2023 15:44:05 +0000 Received: by hermes--production-gq1-77657878bb-9z6zl (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID d26a89bd86f7a9014c2ff8b232463fd3; Sat, 16 Sep 2023 15:44:00 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature Message-Id: <7EEF3435-064D-4C3C-98E4-2B27A788DB43@yahoo.com> Date: Sat, 16 Sep 2023 08:43:49 -0700 To: void , Current FreeBSD X-Mailer: Apple Mail (2.3731.700.6) References: <7EEF3435-064D-4C3C-98E4-2B27A788DB43.ref@yahoo.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.47 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.973]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.147:from]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.147:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_ALL(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCPT_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[f-m.fm,freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RnwMq67gsz4cgB void wrote on Date: Sat, 16 Sep 2023 12:12:02 UTC : > On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote: >=20 > >Yes. The boot loader comes from the host. It must know how to read = ZFS.=20 >=20 > It knows how to read zfs. I expect Warner was indicating: you have a (efi?) loader that knows how to deal with the features listed in: sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd being active but not with some new feature(s) listed in: sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 being active. The following are the "read-only-compatibile no" features that are new in openzfs-2.2 compared to openzfs-2.1-freebsd : blake3 ednor head_errlog vdev_zaps_v2 So any of those being active leads to lack of even read-only activity being compatible. (Although, the loader's subset of the potential overall activity might allow ignoring some specific "read-only-compatibile no" status examples.) For reference: # diff -u99 = /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-fr= eebsd = /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 --- = /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-fr= eebsd 2021-06-24 20:08:57.206621000 -0700 +++ = /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 = 2023-06-10 15:59:25.354999000 -0700 @@ -1,34 +1,40 @@ -# Features supported by OpenZFS 2.1 on FreeBSD +# Features supported by OpenZFS 2.2 on Linux and FreeBSD allocation_classes async_destroy +blake3 +block_cloning bookmark_v2 bookmark_written bookmarks device_rebuild device_removal draid +edonr embedded_data empty_bpobj enabled_txg encryption extensible_dataset filesystem_limits +head_errlog hole_birth large_blocks large_dnode livelist log_spacemap lz4_compress multi_vdev_crash_dump obsolete_counts project_quota redacted_datasets redaction_bookmarks resilver_defer sha512 skein spacemap_histogram spacemap_v2 userobj_accounting +vdev_zaps_v2 +zilsaxattr zpool_checkpoint zstd_compress (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does not exist yet. Thus were I had the diff look.) > On the host in question, there are many guests, > some with zfs-boot, some not, just file-based. But with what openzfs features active vs. not active in each case? > What the host is not, is zfs-on-root. It boots from ssd (ada0). > The vdevs are on a sas disk array. >=20 > >So either your bootable partitions must not have = com.klarasystems:vdev_zaps_v2 > >in your BEs or you must have a new user boot. I think you can just = install > >the one from 14, but haven't tried it. >=20 > Can you briefly explain how I'd install the one from 14 please? I do not use bhyve so I do not even know if the context is using the efi loader from a msdosfs vs. not. For efi loaders, copying from one msdosfs with a sufficient vintage to the one with the wrong vintage (replacing) is sufficient. For reference (from an aarch64 context): # find /boot/efi/EFI/ -print /boot/efi/EFI/ /boot/efi/EFI/FREEBSD /boot/efi/EFI/FREEBSD/loader.efi /boot/efi/EFI/BOOT /boot/efi/EFI/BOOT/bootaa64.efi There may well be only: EFI/BOOT/bootaa64.efi for all I know. =46rom an amd64 context: # find /boot/efi/EFI/ -print /boot/efi/EFI/ /boot/efi/EFI/FREEBSD /boot/efi/EFI/FREEBSD/loader.efi /boot/efi/EFI/BOOT /boot/efi/EFI/BOOT/bootx64.efi There may well be only: EFI/BOOT/bootx64.efi for all I know. (I set things up to have the EFI capitalization so that referencing efi/ vs. EFI/ in my context is unique for the mount point. vs. the msdosfs directory.) =3D=3D=3D Mark Millard marklmi at yahoo.com