From nobody Sat Sep 16 16:35:01 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RnxVp5jxqz4sTVk for ; Sat, 16 Sep 2023 16:35:14 +0000 (UTC) (envelope-from freebsd@grem.de) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RnxVp3nhCz4qYb for ; Sat, 16 Sep 2023 16:35:14 +0000 (UTC) (envelope-from freebsd@grem.de) Authentication-Results: mx1.freebsd.org; none Received: by mail.evolve.de (OpenSMTPD) with ESMTP id d0c572b3; Sat, 16 Sep 2023 16:35:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=grem.de; h=date:from:to:cc :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=20180501; bh=hZk1t5hh ur30LcmUzCYapMB4uaw=; b=puH1lzBN3aVs5D7UG0i5UmwIl4psAR26f3vY+Wre A7WqCJWEP0+tCuF09INQkU0iLxIPGZHs9eV/2PuPtjHFOmQpp4U57KYPhiIMFV+O b/k1FFJGJ+Y6H8f54AviphWs77uRLW/zxRUNL4qBXlJ3biuJSTdaTO0csV5jPvD1 x48OcGgJ4/xAQ3vy6xmrtt6USfs/IKMu4YqJoakmiQJfToyhT7FPbar+64fVwdel 5KyaXxV1jVVPi6BPUxK1sVt09D2I/M7S9Crg+5TAULY5N6811MIZRuwZhTi6wjXX pADBjPdBVz08gpzW+g3g0HEQOxFAbdruPir61d1jEiaGFw== DomainKey-Signature: a=rsa-sha1; c=nofws; d=grem.de; h=date:from:to:cc :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=20180501; b=j6 TEGrli7BjUL9bWCyYyXvrAz4/usR7Y7QfbY07TUCOCdIG66ol5LTyKWfAwNPhDqI xrpGN/qS6CyzENR+kQZaZdtvAQf2E18sSh5cbCHLpcKsfQd9xEXs1Mf7+WcvvywH TCVYD82YdtVhIKn8VRP1+cz6WcjlgQdWLAn1D/MgQ7zCZ0WhUJhCTK1mwQ0RU5rj ZezAKOunaddG5XKGpDy4J+NoA1/v6La1exOd4tx4JJTYZKg3FR01ifZdJLVmDnj2 rsGpYD/smMBJKBVo5qF9g6KCpbT/PcLAP7zH207Y6sm7JhA1lFhHUg7mxOOUB5V8 gk9MSydGrOjj4WGZ8rEg== Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id 2e59e5e8 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Sat, 16 Sep 2023 16:35:12 +0000 (UTC) Date: Sat, 16 Sep 2023 18:35:01 +0200 From: Michael Gmelin To: Warner Losh Cc: Toomas Soome , Mark Millard , void , Current FreeBSD Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature Message-ID: <20230916183501.1e4a131e@bsd64.grem.de> In-Reply-To: References: <7EEF3435-064D-4C3C-98E4-2B27A788DB43.ref@yahoo.com> <7EEF3435-064D-4C3C-98E4-2B27A788DB43@yahoo.com> <4D51E8E6-8AF0-4773-A9BA-D53C08B744EA@me.com> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE] X-Rspamd-Queue-Id: 4RnxVp3nhCz4qYb As this is the continuation of a thread I started in June, let me top post again the solution I used back then: """ For completeness sake, this is how I boot 14.0 on 13.2 using sysutils/vm-bhyve: ISO=3DFreeBSD-14.0-CURRENT-amd64-20230608-653738e895ba-263444-bootonly.iso export ISO cd /mountpoint/for/pool/vm vm iso https://download.freebsd.org/snapshots/ISO-IMAGES/14.0/$ISO mkdir .loaders tar --strip-components 1 -C .loaders -xf .iso/$ISO boot/userboot.so mv .loaders/userboot.so .loaders/userboot14.so vm create test14 sysrc -f test14/test14.conf memory=3D1G sysrc -f test14/test14.conf \ bhyveload_loader=3D"$(realpath .loaders/userboot14.so)" OS installation is done the usual way (using tmux instead of cu in this example): pkg install -y tmux sysrc -f .config/system.conf console=3Dtmux vm install test14 $ISO tmux attach -t test14 """ You can find thew whole thread here: https://lists.freebsd.org/archives/freebsd-current/2023-June/003835.html Best Michael On Sat, 16 Sep 2023 17:22:20 +0100 Warner Losh wrote: > On Sat, Sep 16, 2023 at 5:11=E2=80=AFPM Toomas Soome wrot= e: >=20 > > > > =20 > > > On 16. Sep 2023, at 18:43, Mark Millard wrote: > > > > > > void wrote on > > > Date: Sat, 16 Sep 2023 12:12:02 UTC : > > > =20 > > >> On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote: > > >> =20 > > >>> Yes. The boot loader comes from the host. It must know how to > > >>> read =20 > > ZFS. =20 > > >> > > >> It knows how to read zfs. =20 > > > > > > I expect Warner was indicating: you have a (efi?) loader that > > > knows how to deal with the features listed in: > > > > > > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd > > > > > > being active but not with some new feature(s) listed in: > > > > > > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 > > > > > > being active. > > > > > > The following are the "read-only-compatibile no" features > > > that are new in openzfs-2.2 compared to openzfs-2.1-freebsd : > > > > > > blake3 > > > ednor > > > head_errlog > > > vdev_zaps_v2 > > > > > > So any of those being active leads to lack of even read-only > > > activity being compatible. (Although, the loader's subset > > > of the potential overall activity might allow ignoring some > > > specific "read-only-compatibile no" status examples.) > > > > > > For reference: > > > > > > # diff -u99 =20 > > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1= -freebsd > > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 > > =20 > > > --- =20 > > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1= -freebsd > > 2021-06-24 20:08:57.206621000 -0700 =20 > > > +++ =20 > > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 > > 2023-06-10 15:59:25.354999000 -0700 =20 > > > @@ -1,34 +1,40 @@ > > > -# Features supported by OpenZFS 2.1 on FreeBSD > > > +# Features supported by OpenZFS 2.2 on Linux and FreeBSD > > > allocation_classes > > > async_destroy > > > +blake3 > > > +block_cloning > > > bookmark_v2 > > > bookmark_written > > > bookmarks > > > device_rebuild > > > device_removal > > > draid > > > +edonr > > > embedded_data > > > empty_bpobj > > > enabled_txg > > > encryption > > > extensible_dataset > > > filesystem_limits > > > +head_errlog > > > hole_birth > > > large_blocks > > > large_dnode > > > livelist > > > log_spacemap > > > lz4_compress > > > multi_vdev_crash_dump > > > obsolete_counts > > > project_quota > > > redacted_datasets > > > redaction_bookmarks > > > resilver_defer > > > sha512 > > > skein > > > spacemap_histogram > > > spacemap_v2 > > > userobj_accounting > > > +vdev_zaps_v2 > > > +zilsaxattr > > > zpool_checkpoint > > > zstd_compress > > > > > > (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does > > > not exist yet. Thus were I had the diff look.) > > > =20 > > >> On the host in question, there are many guests, > > >> some with zfs-boot, some not, just file-based. =20 > > > > > > But with what openzfs features active vs. not active > > > in each case? > > > =20 > > >> What the host is not, is zfs-on-root. It boots from ssd (ada0). > > >> The vdevs are on a sas disk array. > > >> =20 > > >>> So either your bootable partitions must not have =20 > > com.klarasystems:vdev_zaps_v2 =20 > > >>> in your BEs or you must have a new user boot. I think you can > > >>> just =20 > > install =20 > > >>> the one from 14, but haven't tried it. =20 > > >> > > >> Can you briefly explain how I'd install the one from 14 please? =20 > > > > > > > > > I do not use bhyve so I do not even know if the > > > context is using the efi loader from a msdosfs > > > vs. not. For efi loaders, copying from one msdosfs > > > with a sufficient vintage to the one with the wrong > > > vintage (replacing) is sufficient. =20 > > > > bhyve in freebsd is traditionally using /boot/userboot.so, I > > believe. =20 >=20 >=20 > Yes. We use the *HOSTS* (running FreeBSD 13) /boot/userboot.so to > boot the FreeBSD 14 > image. Since we're not using the boot loader from the target image to > load it for bhyve, > the loader we're using has to understand the ZFS dataset that it's > booting off of. FreeBSD > 13's userboot.so doesn't support all the bells and whistles that the > ZFS folks have added > to 14. >=20 > So, either you have to turn off those features (which I got no clue > how to do in the > normal installer), or you have to update userboot.so to the FreeBSD 14 > version (which > I think had a good chance of actually running on FreeBSD 13 since it > has no 'system' > references, which are confined to bhyveload). >=20 > Warner >=20 >=20 > > > > > > # find /boot/efi/EFI/ -print > > > /boot/efi/EFI/ > > > /boot/efi/EFI/FREEBSD > > > /boot/efi/EFI/FREEBSD/loader.efi > > > /boot/efi/EFI/BOOT > > > /boot/efi/EFI/BOOT/bootaa64.efi > > > > > > There may well be only: > > > > > > EFI/BOOT/bootaa64.efi > > > > > > for all I know. > > > > > > From an amd64 context: > > > > > > # find /boot/efi/EFI/ -print > > > /boot/efi/EFI/ > > > /boot/efi/EFI/FREEBSD > > > /boot/efi/EFI/FREEBSD/loader.efi > > > /boot/efi/EFI/BOOT > > > /boot/efi/EFI/BOOT/bootx64.efi > > > > > > There may well be only: > > > > > > EFI/BOOT/bootx64.efi > > > > > > for all I know. > > > > > > (I set things up to have the EFI capitalization > > > so that referencing efi/ vs. EFI/ in my context > > > is unique for the mount point. vs. the msdosfs > > > directory.) > > > > > > =3D=3D=3D > > > Mark Millard > > > marklmi at yahoo.com > > > > > > =20 > > > > > > =20 --=20 Michael Gmelin