From nobody Thu May 28 03:57:39 2026 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gQt4X11Q4z6dxwQ for ; Thu, 28 May 2026 03:58:04 +0000 (UTC) (envelope-from markovic@internode.on.net) Received: from omr-01.pc5.atmailcloud.com (omr-01.pc5.atmailcloud.com [103.150.252.78]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4gQt4P3T9sz3HM1 for ; Thu, 28 May 2026 03:57:56 +0000 (UTC) (envelope-from markovic@internode.on.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=internode.on.net header.s=202309 header.b="U8/4gnd "; dmarc=pass (policy=reject) header.from=internode.on.net; spf=pass (mx1.freebsd.org: domain of markovic@internode.on.net designates 103.150.252.78 as permitted sender) smtp.mailfrom=markovic@internode.on.net DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=internode.on.net; s=202309; h=To:Date:Message-Id:Subject:Mime-Version: Content-Type:From; bh=y13Tygw2ssI5fFSkmd62p/a9+ISlQCdAUBd3HMulrak=; b=U8/4gnd fgdbmMimSF2iTLcT/PaSPUhQHsaujnVLkydv2hsHtGBG/B3YPWZ40R5hhgATj24ma/z7ghCBO3RMA tSKZslKlEbbVYWYzypjByebakLn3Hq3rXFT9E2kmrf44nt5Flh0E9V9KNvARPfIh4xEb1bQokALDu yOUYQblrgo2QXSpWJH7nUT50GdrztOAYgYc7pirpnjv3lJ9mCN+DDAIXMgkdvYobpj0oDVUDF1SeL eLLrOZdnz0YKuFKoJw5wQ3N9iToxZs1ot7QSZAXRFRV/5KV+XQfGg51JeJr9o209yNuTMpiTjZ+aR 5D1os3YZFVUfPGnZQIjdUTXKN0Q==; Received: from cmr-kakadu01.internal.pc5.atmailcloud.com (cmr-kakadu01.internal.pc5.atmailcloud.com [192.168.1.3]) by omr-01.pc5.atmailcloud.com (Exim/cmr-kakadu01.i-0bb6ff6c5581ec300) with ESMTPS (envelope-from ) id 1wSRsY-00000001Sjz-0Hm5 for ; Thu, 28 May 2026 03:57:54 +0000 Received: from [145.79.197.93] (helo=smtpclient.apple) by cmr-kakadu01.i-0bb6ff6c5581ec300 with esmtpa (envelope-from ) id 1wSRsV-00000007t6A-2bYh for freebsd-virtualization@freebsd.org; Thu, 28 May 2026 03:57:53 +0000 From: marko Content-Type: multipart/alternative; boundary="Apple-Mail=_B8CBD535-A8EB-42E0-87FE-5F9D19BECC6A" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.300.87.4.3\)) Subject: bhyve: boot custom kernel without workaround Message-Id: <04FCC0F3-3E82-41BB-B6B1-791459D7505F@internode.on.net> Date: Wed, 27 May 2026 20:57:39 -0700 To: freebsd-virtualization@freebsd.org X-Mailer: Apple Mail (2.3826.300.87.4.3) X-Atmail-Id: markovic@internode.on.net X-atmailcloud-spam-action: X-AVAS-Report: FROM_HAS_DN(0.00) FROM_EQ_ENVFROM(0.00) UNKNOWN_SENDER(0.50) TO_DN_NONE(0.00) TO_MATCH_ENVRCPT_ALL(0.00) CLOUDMARK_NOT_SPAM(-5.00) MIME_GOOD(-0.10) MV_CASE(0.50) MID_RHS_MATCH_FROM(0.00) HAS_WP_URI(0.00) PD_HOST_FLUX_CHECK(0.30) RCVD_VIA_SMTP_AUTH(0.00) ARC_NA(0.00) ASN(0.00) RCVD_COUNT_ONE(0.00) MIME_TRACE(0.00) RCVD_NO_TLS_LAST(0.10) FREEMAIL_FROM(0.00) RCPT_COUNT_ONE(0.00) CYREN_UNKNOWN(0.00) FREEMAIL_ENVFROM(0.00) X-atmailcloud-route: unknown X-Spamd-Result: default: False [-3.46 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.962]; DMARC_POLICY_ALLOW(-0.50)[internode.on.net,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[internode.on.net:s=202309]; R_SPF_ALLOW(-0.20)[+ip4:103.150.252.0/24]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DKIM_TRACE(0.00)[internode.on.net:+]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ASN(0.00)[asn:16509, ipnet:103.150.252.0/24, country:US]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[internode.on.net]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; HAS_WP_URI(0.00)[]; FREEMAIL_ENVFROM(0.00)[internode.on.net] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4gQt4P3T9sz3HM1 --Apple-Mail=_B8CBD535-A8EB-42E0-87FE-5F9D19BECC6A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Seeking guidance on what I might be doing wrong, and how to avoid the = manual workaround (reload-conf) specified below: I've built a custom kernel based off 15.0-STABLE then installed it to a = raw disk (kernfs.raw), which I intend to boot from in a bhyve container = per: = https://freebsdfoundation.org/wp-content/uploads/2021/11/Kernel_Developmen= t_Recipes.pdf Command: bhyve -c 1 -m 512M -H -A -P -G :1234 \ -s 0:0,hostbridge \ -s 1:0,lpc \ -s 2:0,virtio-blk,FreeBSD-15.0-STABLE-amd64.raw \ -s 3:0,virtio-blk,kernfs.raw \ -l com1,stdio \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ myvm However, the custom kernel image on the additional partition is not = loaded by default in the bhyve container even though I've applied the = rc.conf, sysctl.conf and fstab modifications as specified in the pdf. I can boot into the custom kernel stored on the additional virtio-blk = device with the following workaround: Launch bhyve vm and modify /boot/loader.conf, point `kernel` at the = additional partition with the custom kernel: kernel=3D"disk1p1:/boot/kernel" Now restart vm and: - drop into boot loader - OK reload-conf - OK boot uname -a now shows the custom kernel=E2=80=99s label. Troubleshooting: I've dumped the relevant boot loader variables before and after = `reload-conf` Before: OK show ... kernel=3Dkernel kernel_options=3D kernels_autodetect=3DYES loaddev=3Ddisk0p4: ... After: OK reload-conf: OK show ... kernel=3Ddisk1p1:/boot/kernel <=3D=3D=3D=3D updated kernel_options=3D kernels_autodetect=3DYES loaddev=3Ddisk0p4: ... My impression from Mark Johnston's article is that the modified = sysctl.conf and fstab files should've caused the vm to boot off of the = custom kernel partition on the 2nd virtio-blk device (by ensuring it=E2=80= =99s mounted at /boot/kernel/kernel). I=E2=80=99ve looked through the forums which helped identify the = workaround, but hoping to avoid it every time I spin up the container. Thanks, Mark --Apple-Mail=_B8CBD535-A8EB-42E0-87FE-5F9D19BECC6A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

Seeking guidance on what I might be doing wrong, and how = to avoid the manual workaround (reload-conf) specified below:


I've built a custom = kernel based off 15.0-STABLE then installed it to a raw disk = (kernfs.raw), which I intend to boot from in a bhyve container = per:


https://freebsdfoundation.org/wp-content/uploads/20= 21/11/Kernel_Development_Recipes.pdf


Command:


bhyve -c 1 -m 512M -H = -A -P -G :1234 \

-s 0:0,hostbridge \

-s 1:0,lpc = \

-s = 2:0,virtio-blk,FreeBSD-15.0-STABLE-amd64.raw \

-s = 3:0,virtio-blk,kernfs.raw \

-l com1,stdio \

-l = bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \

myvm


However, the custom kernel image on the = additional partition is not loaded by default in the bhyve container = even though I've applied the rc.conf, sysctl.conf and fstab = modifications as specified in the pdf.


I can boot into the = custom kernel stored on the additional virtio-blk device with the = following workaround:


Launch bhyve vm and = modify /boot/loader.conf, point `kernel` at the additional partition = with the custom kernel:


kernel=3D"disk1p1:/boot/kernel"


Now restart vm = and:


- drop into boot = loader

- OK reload-conf

- OK = boot


uname -a now shows the custom kernel=E2=80=99= s label.


Troubleshooting:

I've dumped the relevant boot loader = variables before and after `reload-conf`


Before:


OK show

...

kernel=3Dkernel

kernel_options=3D

kernels_autodetect=3DYES

loaddev=3Ddisk0p4:

...



After:


OK = reload-conf:

OK show

...

kernel=3Ddisk1p1:/boot/kernel    =         <=3D=3D=3D=3D  updated

kernel_options=3D

kernels_autodetect=3DYES

loaddev=3Ddisk0p4:

...



My impression from Mark Johnston's article = is that the modified sysctl.conf and fstab files should've caused the vm = to boot off of the custom kernel partition on the 2nd virtio-blk device = (by ensuring it=E2=80=99s mounted at /boot/kernel/kernel).


I=E2=80=99ve looked through the forums which helped = identify the workaround, but hoping to avoid it every time I spin up the = container.


Thanks,

Mark

= --Apple-Mail=_B8CBD535-A8EB-42E0-87FE-5F9D19BECC6A--