Date: Thu, 4 Jun 2026 13:23:17 +0200 From: Mario Marietto <marietto2008@gmail.com> To: FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: =?UTF-8?Q?=2A=2A=5BHelp=5D_bhyve=3A_bootrom=5Falloc=3A_vm=5Fmmap=5Fmapseg=3A_Inval?= =?UTF-8?Q?id_argument_=E2=80=94_NVIDIA_passthrough_with_Corvin=27s_branch_on_F?= =?UTF-8?Q?reeBSD_15=2E0=2A=2A?= Message-ID: <CA%2B1FSiiGdW4JCi=XA68Wtr1pDbMNAwTpwq9P8TixdDGRHSf%2B6g@mail.gmail.com>
index | next in thread | raw e-mail
[-- Attachment #1 --] <https://www.reddit.com/r/freebsd/?f=flair_name%3A%22help%20needed%22>; Hi everyone, I'm trying to get NVIDIA GPU passthrough working with bhyve on FreeBSD 15.0-RELEASE-p5, using Corvin Köhne's nvidia-wip branch: https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/15.0/nvidia-wip The VM fails to start with the following error before the guest even boots: bhyve: bootrom_alloc: vm_mmap_mapseg: Invalid argument --- **Hardware** - Motherboard: Gigabyte Z390 AORUS PRO - CPU: Intel Core i9-9900K - GPU: ZOTAC RTX 2080 Ti (0x10de:0x1e04, subvendor 0x19da, subdevice 0x2503) --- **Host** - FreeBSD 15.0-RELEASE-p5 (kernel GENERIC, amd64) - vmm.ko compiled from Corvin's `phab/corvink/15.0/nvidia-wip` branch - bhyve compiled from the same branch - libvmmapi.so.6 produced by the Corvin branch build --- **bhyve command** --- bhyve-lin -S -c sockets=8,cores=1,threads=1 -m 32G -w -H -A \ -s 0,hostbridge \ -s 1,ahci-hd,/mnt/zroot-133/bhyve/img/Linux/Ubuntu-2404-KDE6-Wayland.img,bootindex=1 \ -s 7,passthru,5/0/0 \ -s 8:0,passthru,2/0/0 \ -s 8:1,passthru,2/0/1 \ -s 8:2,passthru,2/0/2 \ -s 8:3,passthru,2/0/3 \ -s 11,hda,play=/dev/dsp,rec=/dev/dsp \ -s 13,virtio-net,tap4 \ -s 14,virtio-9p,sharename=/ \ -s 29,fbuf,tcp=0.0.0.0:5900,w=1650,h=900,wait \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:4 < /dev/null 2>&1 & --- **Root cause analysis (via ktrace)** --- Tracing bhyve with `ktrace` reveals the following sequence: 1. `openat("/dev/vmmctl", O_RDWR)` → success 2. `ioctl(VM_MMAP_GETNEXT)` → success 3. `ioctl(VM_MMAP_MEMSEG)` → **errno 22 (EINVAL)** The failure is inside `bootrom_alloc()` → `vm_mmap_memseg()` when trying to map the `VM_BOOTROM` memory segment. Digging into `lib/libvmmapi/vmmapi.c`, the function `vm_alloc_memseg()` first calls `vm_get_memseg()` to check if the segment already exists. If it returns success with `len == 0` (segment not yet allocated), it should proceed to call `VM_ALLOC_MEMSEG` ioctl to create the segment. However, `VM_ALLOC_MEMSEG` is **never called** in the ktrace output. The result is that `seg->object == NULL` in the kernel when `vm_mmap_memseg()` is later called for `VM_BOOTROM`, which returns `EINVAL`. --- **Workaround found** --- Replacing `libvmmapi.so.6` (from the Corvin build) with the system stock `libvmmapi.so.7` fixes the bootrom issue — the VM boots successfully. However, this introduces a new problem: `bhyvectl --vm=name --create` returns `Operation not permitted` (EPERM) because the stock libvmmapi uses different ioctls that vmm.ko Corvin doesn't handle correctly. What is the correct fix for `vm_alloc_memseg()` in the Corvin libvmmapi to ensure `VM_ALLOC_MEMSEG` is properly called for `VM_BOOTROM`? Is there a known patch or workaround that keeps compatibility with vmm.ko Corvin while fixing the bootrom allocation? Thanks! --- *Relevant source: `lib/libvmmapi/vmmapi.c`, function `vm_alloc_memseg()` and `vm_create_devmem()`* *vmm kernel side: `sys/dev/vmm/vmm_mem.c`, function `vm_mmap_memseg()`* -- Mario. [-- Attachment #2 --] <div dir="ltr"><div><span name="credit-bar"><div id="gmail-pdp-credit-bar" class="gmail-flex gmail-justify-between gmail-text-12 gmail-px-md gmail-xs:px-0 gmail-relative gmail-pb-2xs gmail-pt-md"></div></span> <span name="ssr-post-content-header"><div></div></span> <div class="gmail-flex"> <div class="gmail-flex gmail-flex-col gmail-grow gmail-max-w-full"><span name="title"></span><span class="gmail-pdp-flair-slot" name="post-flair"><div class="gmail-mb-2xs gmail-px-md gmail-xs:px-0"> <span><span><a href="https://www.reddit.com/r/freebsd/?f=flair_name%3A%22help%20needed%22" class="gmail-no-decoration"> <span class="gmail-bg-tone-4 gmail-inline-block gmail-truncate gmail-max-w-full gmail-text-12 gmail-font-normal gmail-align-text-bottom gmail-box-border gmail-px-[6px] gmail-rounded-5 gmail-leading-4 gmail-text-global-black" style="background-color:rgb(255,214,53)"> <div aria-label="Flair: help needed" class="gmail-flair-content gmail-[&_.flair-image]:align-bottom gmail-max-w-full gmail-overflow-hidden gmail-whitespace-nowrap gmail-text-ellipsis" dir="auto"></div> </span> </a></span></span> </div></span> <span name="mod-content-state-indicators"></span> </div> <div> <span name="thumbnail"></span></div> </div> <span name="post-media-container" class="gmail-xs:pt-md"></span> <span name="text-body"><span><span name="text-body"><div class="gmail-text-neutral-content"> <div class="gmail-mb-sm gmail-mb-xs gmail-px-md gmail-xs:px-0 gmail-overflow-hidden"> <div id="gmail-t3_1twi976-post-rtjson-content" class="gmail-md gmail-text-14-scalable gmail-pb-2xs gmail-[--emote-size:20px]" dir="auto"> <p dir="auto"> Hi everyone, </p><p dir="auto"> I'm trying to get NVIDIA GPU passthrough working with bhyve on FreeBSD 15.0-RELEASE-p5, using Corvin Köhne's nvidia-wip branch: </p><p dir="auto"> <a class="gmail-relative gmail-pointer-events-auto gmail-a gmail-text-body-2 gmail-underline gmail-cursor-pointer" href="https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/15.0/nvidia-wip" rel="noopener nofollow ugc" target="_blank">https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/15.0/nvidia-wip</a>; </p><p dir="auto"> The VM fails to start with the following error before the guest even boots: </p><p dir="auto"> <code>bhyve: bootrom_alloc: vm_mmap_mapseg: Invalid argument</code> </p><p dir="auto"> --- </p><blockquote> <p dir="auto"> **Hardware** </p> </blockquote><p dir="auto"> - Motherboard: Gigabyte Z390 AORUS PRO </p><p dir="auto"> - CPU: Intel Core i9-9900K </p><p dir="auto"> - GPU: ZOTAC RTX 2080 Ti (0x10de:0x1e04, subvendor 0x19da, subdevice 0x2503) </p><p dir="auto"> --- </p><blockquote> <p dir="auto"> **Host** </p> </blockquote><p dir="auto"> - FreeBSD 15.0-RELEASE-p5 (kernel GENERIC, amd64) </p><p dir="auto"> - vmm.ko compiled from Corvin's `phab/corvink/15.0/nvidia-wip` branch </p><p dir="auto"> - bhyve compiled from the same branch </p><p dir="auto"> - libvmmapi.so.6 produced by the Corvin branch build </p><p dir="auto"> --- </p><blockquote> <p dir="auto"> **bhyve command** </p> </blockquote><p dir="auto"> --- </p><p dir="auto"> <code>bhyve-lin -S -c sockets=8,cores=1,threads=1 -m 32G -w -H -A \</code> </p><p dir="auto"> <code>-s 0,hostbridge \</code> </p><p dir="auto"> <code>-s 1,ahci-hd,/mnt/zroot-133/bhyve/img/Linux/Ubuntu-2404-KDE6-Wayland.img,bootindex=1 \</code> </p><p dir="auto"> <code>-s 7,passthru,5/0/0 \</code> </p><p dir="auto"> <code>-s 8:0,passthru,2/0/0 \</code> </p><p dir="auto"> <code>-s 8:1,passthru,2/0/1 \</code> </p><p dir="auto"> <code>-s 8:2,passthru,2/0/2 \</code> </p><p dir="auto"> <code>-s 8:3,passthru,2/0/3 \</code> </p><p dir="auto"> <code>-s 11,hda,play=/dev/dsp,rec=/dev/dsp \</code> </p><p dir="auto"> <code>-s 13,virtio-net,tap4 \</code> </p><p dir="auto"> <code>-s 14,virtio-9p,sharename=/ \</code> </p><p dir="auto"> <code>-s 29,fbuf,tcp=<a href="http://0.0.0.0:5900">0.0.0.0:5900</a>,w=1650,h=900,wait \</code> </p><p dir="auto"> <code>-s 30,xhci,tablet \</code> </p><p dir="auto"> <code>-s 31,lpc \</code> </p><p dir="auto"> <code>-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \</code> </p><p dir="auto"> <code>vm0:4 < /dev/null 2>&1 &</code></p><p dir="auto"><code><br></code></p><p dir="auto"> --- </p><blockquote> <p dir="auto"> **Root cause analysis (via ktrace)** </p> </blockquote><p dir="auto"> --- </p><p dir="auto"> Tracing bhyve with `ktrace` reveals the following sequence:</p><p dir="auto"><br></p><ol><li> <p dir="auto"> `openat("/dev/vmmctl", O_RDWR)` → success </p> </li><li> <p dir="auto"> `ioctl(VM_MMAP_GETNEXT)` → success </p> </li><li> <p dir="auto"> `ioctl(VM_MMAP_MEMSEG)` → **errno 22 (EINVAL)**</p></li></ol><div><br></div><p dir="auto"> The failure is inside `bootrom_alloc()` → `vm_mmap_memseg()` when trying to map the `VM_BOOTROM` memory segment. </p><p dir="auto"> Digging into `lib/libvmmapi/vmmapi.c`, the function `vm_alloc_memseg()` first calls `vm_get_memseg()` to check if the segment already exists. If it returns success with `len == 0` (segment not yet allocated), it should proceed to call</p><p dir="auto"><br></p><p dir="auto"> `VM_ALLOC_MEMSEG` ioctl to create the segment. However, `VM_ALLOC_MEMSEG` is **never called** in the ktrace output.</p><p dir="auto"><br></p><p dir="auto"> The result is that `seg->object == NULL` in the kernel when `vm_mmap_memseg()` is later called for `VM_BOOTROM`, which returns `EINVAL`. </p><p dir="auto"> --- </p><blockquote> <p dir="auto"> **Workaround found** </p> </blockquote><p dir="auto"> ---</p><p dir="auto"><br></p><p dir="auto"> Replacing `libvmmapi.so.6` (from the Corvin build) with the system stock `libvmmapi.so.7` fixes the bootrom issue — the VM boots successfully. However, this introduces a new problem: `bhyvectl --vm=name --create` returns `Operation not permitted` (EPERM) because the stock libvmmapi uses different ioctls that vmm.ko Corvin doesn't handle correctly. </p><p dir="auto"> What is the correct fix for `vm_alloc_memseg()` in the Corvin libvmmapi to ensure `VM_ALLOC_MEMSEG` is properly called for `VM_BOOTROM`? Is there a known patch or workaround that keeps compatibility with vmm.ko Corvin while fixing the bootrom allocation? </p><p dir="auto"> Thanks! </p><p dir="auto"> ---</p><p dir="auto"><br></p><p dir="auto"> *Relevant source: `lib/libvmmapi/vmmapi.c`, function `vm_alloc_memseg()` and `vm_create_devmem()`* </p><p dir="auto"> *vmm kernel side: `sys/dev/vmm/vmm_mem.c`, function `vm_mmap_memseg()`* </p> </div> </div> </div></span></span></span><br clear="all"></div><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Mario.<br></div></div>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSiiGdW4JCi=XA68Wtr1pDbMNAwTpwq9P8TixdDGRHSf%2B6g>