From nobody Fri Oct 10 13:02:20 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cjn340jdZz6C0mW; Fri, 10 Oct 2025 13:02:40 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4cjn315xX7z4B6V; Fri, 10 Oct 2025 13:02:37 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=YZkwz3Ad; dkim=none ("invalid DKIM record") header.d=cse.huji.ac.il header.s=57791128 header.b=yb5MKauU; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=References:To:Cc:In-Reply-To:Date:Subject:Mime-Version:Content-Type:Message-Id:From; bh=wNbeC/rR88sXC7ZpwbWWLe8PmhyhBreTgamM7JWJNUM=; b=YZkwz3AdyFwiYI1QBYgRzGR4dM3fA2ugGf0JLcnFimKtC2dGG7jPLL9UjrQz4WEgwoyzRenpEYna1Q0cSlYfzS9BcKwvy6GNWLyCdb9i66I3PgYs8OtsorPxCHIk+Ic6fWT2yzQFz49sOREQiGkgAlsg7PlNAiTEGnBUe3dmVUzb91W8kpcMroFqmwvKZi6U706mwuXA1pJZcKt2ShbnAlHFstjrgcW97WKjO9FHoR0Xp8T81ZLbIYJFiv2MvPI23fRma9TTz4SUj5NZsYohQWGLB/qHSVlzhJd5u4px6tNRHnjRK+7g9wISsGD1nh0PprLFALiCBKPFwzN6hC8Ghg==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cse.huji.ac.il; s=57791128; h=References:To:Cc:In-Reply-To:Date:Subject:Mime-Version:Content-Type:Message-Id:From; bh=wNbeC/rR88sXC7ZpwbWWLe8PmhyhBreTgamM7JWJNUM=; b=yb5MKauUrzQMKjOLpKbl6YjlOe21weQ2P1Yc/zfwNaGw8RuffVBTbq+ju9MNCpxX64nFQWp4x34nzJocH86oNKBaoO9IcpQdD6JL/sC4GcSDo4wv1ERKFLY8Ap9xKT6z0VA7gFxQdQzBD4vyTMGGP7TZ/CSXDYRYJEkhLxN1EcwWYFSe3UuM/zObf/vb8Hz/IiiOk6PRS65q1f0WP4deRjr6wLJNypHuEayjCM1wDfjbHyGh4jmOmBl8OYKnKfiNlOr2NaKibh0cUDxWDMp8Ehqk30k8uFpWY6cIWNaRjr65jHIWQs+dfT8KbYelX93eT6W0tEX8X+h4H7mS4IbuvA==; Received: from imac.bk.cs.huji.ac.il ([132.65.179.42] helo=smtpclient.apple) by kabab.cs.huji.ac.il with esmtp id 1v7ClW-000M9q-8k; Fri, 10 Oct 2025 16:02:34 +0300 From: Daniel Braniss Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_B24AFD8D-C67C-4BAD-B1C7-AB53FB33034A" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3864.100.1.1.5\)) Subject: Re: ALPHA-3 diskless + bhyve issues Date: Fri, 10 Oct 2025 16:02:20 +0300 In-Reply-To: Cc: "Bjoern A. Zeeb" , Freebsd-stable List , virtualization@freebsd.org To: Rick Macklem References: <342F2780-9C83-4CEE-BE3E-AE72C9AE12BA@cs.huji.ac.il> <639210F6-87B6-4B34-AB9B-8EAB46682B80@cs.huji.ac.il> X-Mailer: Apple Mail (2.3864.100.1.1.5) X-Spamd-Bar: - X-Spamd-Result: default: False [-1.60 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; MV_CASE(0.50)[]; ONCE_RECEIVED(0.20)[]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DKIM_TRACE(0.00)[cs.huji.ac.il:+,cse.huji.ac.il:~]; R_SPF_NA(0.00)[no SPF record]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_PERMFAIL(0.00)[cse.huji.ac.il:s=57791128]; RCPT_COUNT_THREE(0.00)[4]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_MIXED(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org,virtualization@freebsd.org]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FREEFALL_USER(0.00)[danny]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4cjn315xX7z4B6V --Apple-Mail=_B24AFD8D-C67C-4BAD-B1C7-AB53FB33034A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 10 Oct 2025, at 15:44, Rick Macklem wrote: >=20 > On Thu, Oct 9, 2025 at 11:15=E2=80=AFPM Daniel Braniss = > wrote: >>=20 >>=20 >>=20 >> On 10 Oct 2025, at 00:33, Bjoern A. Zeeb = wrote: >>=20 >> On Tue, 23 Sep 2025, Daniel Braniss wrote: >>=20 >> So after many years, I tried to install instead of upgrading, >> and after much fiddling (if anyone is interested I can write how I = did i) I got it to >> install, now here is my problem: >> when the the local disk was empty, loader.efi is called (this via = dhcp) and it boots diskless, >> but now that the disk is populated, it boots from it. >>=20 >> Q: how can I have it boot diskless again? >>=20 >>=20 >> I haven't seen an answer. Have you tried the freebsd-virtualization = list? >>=20 >> I assume you could interrupt the boot process if fast enough (but = it's the blink of an eye); >> so you'll most likely be more successful putting a startup.nsh and = possibly an EFI shell in the EFI System Partition (ESP) and script it? = Could be the bhyve firmware brings the efi shell along already? >>=20 >> /bz >>=20 >> -- >> Bjoern A. Zeeb = r15:7 >>=20 >>=20 >> hi, >> no answer yet, but after some experimenting: >> boots diskless: >> if no local disk - dah >> if local disk and no EFI partition >> if local disk has the above but no valid root partition >> and finally if root partition and efi but no bootx64.efi ( i just = move it to bootx64.efix) > Can you try the case where there is a non-bootable ZFS partition? >=20 > When I have a non-bootable ZFS partition with a zpool in it, bhyve = decides that > is the root and fails to boot the UFS root fs. I need to: >=20 > currdev=3Ddisk0s1 > boot >=20 > to get it to boot. >=20 > I don't know if this is what you might be seeing? rick Not really, in the past I could change from diskless to local root by = changing the filename option in the dhcp, so that if filename is loader.efi it would boot diskless, and if = boot1.efi it would boot off the local disk. Now loader.efi will first try and boot from the local disk. Btw, I think = this is documented. Cheers, Danny >=20 >>=20 >> thanks, >> danny >> ps: i guess i forgot to mention that the host is running in bhyve = (14.3) >> pass: it=E2=80=99s now alpha-5 --Apple-Mail=_B24AFD8D-C67C-4BAD-B1C7-AB53FB33034A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 10 Oct 2025, at 15:44, Rick Macklem = <rick.macklem@gmail.com> wrote:

On Thu, Oct 9, 2025 at 11:15=E2=80=AFPM = Daniel Braniss <danny@cs.huji.ac.il> wrote:



On 10 Oct 2025, at 00:33, Bjoern A. = Zeeb <bzeeb-lists@lists.zabbadoz.net> wrote:

On Tue, 23 Sep = 2025, Daniel Braniss wrote:

So after many years, I tried to = install instead of upgrading,
and after much fiddling (if anyone is = interested I can write how I did i) I got it to
install, now here is = my problem:
when the the local disk was empty, loader.efi is called = (this via dhcp) and it boots diskless,
but now that the disk is = populated, it boots from it.

Q: how can I have it boot diskless = again?


I haven't seen an answer.  Have you tried the = freebsd-virtualization list?

I assume you could interrupt the = boot process if fast enough (but it's the blink of an eye);
so you'll = most likely be more successful putting a startup.nsh and possibly an EFI = shell in the EFI System Partition (ESP) and script it? Could be the = bhyve firmware brings the efi shell along = already?

/bz

--
Bjoern A. Zeeb =             &n= bsp;           &nbs= p;            =             &n= bsp;  r15:7


hi,
no answer yet, but after some = experimenting:
boots diskless:
if no local disk - dah
if local = disk and no EFI partition
if local disk has the above but no valid = root partition
and finally if root partition and efi but no = bootx64.efi ( i just move it to bootx64.efix)
Can you try the case where there is a = non-bootable ZFS partition?

When I = have a non-bootable ZFS partition with a zpool in it, bhyve decides = that
is the root and fails to = boot the UFS root fs. I need to:

currdev=3Ddisk0s1
boot

to get = it to boot.

I don't know if this is = what you might be seeing? rick


Not really, in = the past I could change from diskless to local root by changing the = filename option in the dhcp,
so that if filename is loader.efi = it would boot diskless, and if boot1.efi it would boot off the local = disk.
Now loader.efi will first try and boot from the local = disk. Btw, I think this is documented.
Cheers,
= Danny



thanks,
danny
ps: i guess i forgot to mention that the = host is running in bhyve (14.3)
pass: it=E2=80=99s now = alpha-5

= --Apple-Mail=_B24AFD8D-C67C-4BAD-B1C7-AB53FB33034A--