From nobody Wed Jun 10 16:26:52 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gbB513TFzz6gkw9 for ; Wed, 10 Jun 2026 16:27:17 +0000 (UTC) (envelope-from Stephan.Althaus@Duedinghausen.eu) Received: from mo4-p05-ob.smtp.rzone.de (mo4-p05-ob.smtp.rzone.de [81.169.146.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "*.smtp.rzone.de", Issuer "Telekom Security ServerID OV Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4gbB4z2scMz3lrt for ; Wed, 10 Jun 2026 16:27:15 +0000 (UTC) (envelope-from Stephan.Althaus@Duedinghausen.eu) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hoewweken.de header.s=strato-dkim-0002 header.b=NQL161kv; dkim=pass header.d=hoewweken.de header.s=strato-dkim-0003 header.b=FUItAQkS; dkim=pass header.d=duedinghausen.eu header.s=default header.b=Jpte7Aez; dmarc=pass (policy=reject) header.from=duedinghausen.eu; spf=none (mx1.freebsd.org: domain of Stephan.Althaus@Duedinghausen.eu has no SPF policy when checking 81.169.146.183) smtp.mailfrom=Stephan.Althaus@Duedinghausen.eu; arc=pass ("strato.com:s=strato-dkim-0002:i=1") ARC-Seal: i=1; a=rsa-sha256; t=1781108832; cv=none; d=strato.com; s=strato-dkim-0002; b=golDra36lcUxvz7a20qFGHSQtbcEbj80RC+apq6EkKKRAbI7tLxJk1x69SHuSJZYSD SG+ZS+AXJoJ8vW1NByJqq0rqOPk0or1UWxQxxfC5uFYkEXrpZsUkXizfPxgbhI9j0FzL eJLDLL18qFZO3GRWNNwZF7fODL4c8qunJm7xpCu8+Q72x+77l9zmC8A31Jn7K0MNdxJw PVVCwLdJgvUND92YMSl5iuUyYEjFLNk7ZqzR+dsAWvBS/BHs+TEq3PngJehfhQK6bNPw qo1PeUBCc37gdS6WHBUMDtAHzi+HiW4qtiNeToAhc2OYauPdLC85K0lBn5kvP2RJYUwy MxMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; t=1781108832; s=strato-dkim-0002; d=strato.com; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=eLA/j/UlTM0DxDfy4lOoC9NkliaNmMXeqLZ+21a+SBo=; b=kpgbv+i9qpVEM2UgEixDZHxQeDjJ/rl37pfWpQcjtlBA7LpxVVjw/KsUo3h217GMr5 a9mU7lBTenFP7Y6s/ljdalHS4I0vplTjGhrExEDfXG9vodgs/gvUGiPxbBS8Rvvd7KHF x5vOL+EIxzcZAQiM8kjIAW36rcf8zRsXp1w+6qNJsvd4itnMmtu5Sg6AVhx+MMdVB4iB jol3xUA0GGTepG3tZL5oC5C+ct80jN+jzOanoxcItIGTZbCMro2UcxhS1ZSzYJzOieJv YwdaPJaQCYeb0OnM28dtBLeG0M4jkZGi9nSVQVLbrUtKigmlkrUnW7Z9IdAFZkpOPEfo eSbg== ARC-Authentication-Results: i=1; strato.com; arc=none; dkim=pass header.d="duedinghausen.eu" header.s="default" header.a="rsa-sha256" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1781108832; s=strato-dkim-0002; d=hoewweken.de; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=eLA/j/UlTM0DxDfy4lOoC9NkliaNmMXeqLZ+21a+SBo=; b=NQL161kv/8kpRwaFs6uOTBlbz+iN+6O7bkYSb+wLp4xCUlkengIm124Y0aQgEwMh7A utI+0RUEfjX7udAQ4UiaAD9WMnZoGC+nAduOQqq+J/sezMd8ncM+I7EKZwkbSBlDMw32 6FbpS8yR/DINtssI4Zca6AAB5K6kBap3lucNQH+LNAra+8MnigEFWsys8UduY1bg/Oc4 8BT1R3zhr9vC3Cy4REW6B9NcLIKpDAtgBJWDv7Ly1IvzjNMvtBVwHAdAQ3Z3wrI2IZ8V 6RSf0bCQQ3Pjt33ThPTA0tFo6jQDUg597jJqO5F2E/XiLIJWEVV8RYQ1Tqsy5e/atAqX K86g== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; t=1781108832; s=strato-dkim-0003; d=hoewweken.de; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=eLA/j/UlTM0DxDfy4lOoC9NkliaNmMXeqLZ+21a+SBo=; b=FUItAQkSobsdb3/A1IdFyEfx/SguRcNu7hmQXBEvuGxkyNvonPn7+gee25nz+cYe6E jp983JPXkmTNbu9snXAg== X-RZG-AUTH: ":O2kGeEG7b/pS1EW2TmikjLDsYYueHLp2aWg0q38nsxvThUrH35SBqNi8UP9PRt5KqGBhEbw=" X-RZG-CLASS-ID: mo05 Received: from www.duedinghausen.eu by smtp.strato.de (RZmta 55.0.1 DYNA|AUTH) with ESMTPSA id qeddbf25AGRCWXb (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits)) (Client did not present a certificate) for ; Wed, 10 Jun 2026 18:27:12 +0200 (CEST) Received: from [192.168.2.63] (p4fea3da3.dip0.t-ipconnect.de [79.234.61.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519MLKEM768 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) (Authenticated sender: steven) by www.duedinghausen.eu (Postfix) with ESMTPSA id 4E7588613 for ; Wed, 10 Jun 2026 18:27:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=duedinghausen.eu; s=default; t=1781108832; bh=2QVWXVaIRR9zcMjKCnyr3HaX9kBT/qWNwQ4JzcB8Kpk=; h=Date:Subject:To:References:From:In-Reply-To; b=Jpte7AezRehj257hUBbShyoFzcgqVbeWTHGdg6HcbzRzdj/JdTCfHOtAPIgypbTNu 8011YR+Jc0Ue/yuyPoa457E5zqDc4XfgnfAJ78uhehobVFU76ZEiLZ/S8EREzdtVhw CdvIVkSwKprqfPuNBFMEwYdzhMYDhdeJLWqRITQcokdGwOYDd/XFg9h/vdDF31R0uJ W4WZmeYTH446kI2PEWbcIPc/P9oNqapqcRrF9Oip6mxcCDvox2XAOTwprkJrKJvOT3 AB+MP0H9kcjJMQHlpLduUl47k03U+1k6COHeUJHtSIJAYnstDRTaLD6vPne0KW+r/h 9lrX98Qfavjhn3Z+qv5PPzewUKRdmBkFwn5XgAcfMWfYeT2q0IQhjr56eD/dD+loFF 8QeCQJZLWUkfY30E2ONrjM9FzNS8yNM2rJ82SBxOjpzyvAGQ30GYLqPlxb76ZngU9e cASmDTsBAgHv3SV2Y5PgZ0iO2117hjNaDxDJV6qWn4gOhv4Qls/aWWm3x7lWo+h++G DE6Eqqg/Mnje5gjU3Y3IXgllk+AYxX7G+SkCqqf1/7RMMsjAZtdLrqd7YorcLbv0FQ cfeBfDh7QPNWiXAQRQ9XI5jivk0wKTsp7GXnVMTTc9cqtpUhwVCVOnXMwIrWPS0LLC +G/7Lg2rJI2GOkq6vldjozsM= Message-ID: <6e6addbc-14e7-4025-98b9-0d3007f6d17d@Duedinghausen.eu> Date: Wed, 10 Jun 2026 18:26:52 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: =?UTF-8?Q?Re=3A_**=5BHelp=5D_bhyve=3A_bootrom=5Falloc=3A_vm=5Fmmap?= =?UTF-8?Q?=5Fmapseg=3A_Invalid_argument_=E2=80=94_NVIDIA_passthrough_with_C?= =?UTF-8?Q?orvin=27s_branch_on_FreeBSD_15=2E0**?= To: virtualization@freebsd.org References: <1bbf28a7-7025-4d83-937d-72d8583048ca@shrew.net> <368dc4ea87fea1c1082c5e54e4140f3cb43075a9.camel@FreeBSD.org> From: Stephan Althaus Content-Language: en-US In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-4.89 / 15.00]; ARC_ALLOW(-1.00)[strato.com:s=strato-dkim-0002:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; DMARC_POLICY_ALLOW(-0.50)[duedinghausen.eu,reject]; R_DKIM_ALLOW(-0.20)[hoewweken.de:s=strato-dkim-0002,hoewweken.de:s=strato-dkim-0003,duedinghausen.eu:s=default]; RCVD_IN_DNSWL_LOW(-0.10)[81.169.146.183:from]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[hoewweken.de:+,duedinghausen.eu:+]; RCVD_TLS_ALL(0.00)[]; ASN(0.00)[asn:6724, ipnet:81.169.144.0/22, country:DE]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_NA(0.00)[no SPF record]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_TWO(0.00)[2]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; MIME_TRACE(0.00)[0:+] X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4gbB4z2scMz3lrt On 6/10/26 16:57, Eva Winterschön wrote: > Hi Mario, > > I have perhaps too many nvidia GPUs, from Maxwell generation to Kepler, Pascal, Turing, Ampere, and GB10 series, with consumer RTX in the mix but mostly datacenter grade SKUs from those series. Some are in systems, some are in boxes, some will never be used, some have passive cooling, some have active, etc. Otherwise to say, generations of GPUs are important for testing compatibility and operational consistency. I can potentially run the same tests if there's more detailed information. > > Back when I had time for anything other than wasting my attention span on cleaning up the security messes from May's Github CVE, I was running a lot of test-bench and workload profiling on 14.x with bhyve passthrough for nvidia as well as CUDA support (without virtualization involved). While reading these threads I wonder about the following: > > 1. What does your test harness look like? > 2. What is your kernel config? > 3. What version and config of everything is in use on the OS (hardware specs, uefi with/without csm enabled, x2acpi opt-out, pcie lane settings, aspm settings, bhyve global settings, qemu settings, vm specific settings, pciconf output, uefi loader settings/flags, loader.conf settings, rc.conf settings, pkg versions, driver versions, firmware version for the MB and BMC, sysctl.conf settings, firmware version on the GPU, model of GPU, nvidia driver version, etc) > 4. Are you tracking and keeping syslog entries from the start to end of the test runs? > 5. Are you using additional process debugging for bhyve, kernel debug symbol support, using drace + flamegraphs etc. > 6. Which method of patching is used, what exactly was observed every time a test iteration occurred? how idempotent and repeatable are your tests? > > There's a lot of work to get from "doesn't work" to "it works", including the necessary definition of "works" (are we talking, the VM can see the GPU, or the VM can use the GPU, the VM can see and use the GPU and also execute CUDA workloads on the GPU). > > Thanks, > > Eva > > > > On Wed, Jun 10, 2026, at 02:18, Mario Marietto wrote: >> --> Does it occur on stock 15? . >> >> Error happened with stock 15 upgraded until 15p9. >> >> ---> Nvidia GPU passthrough should work on stock 15, so Please test it >> without patching any files. >> >> I asked Claude to create the patches because it didn't work with stock >> 15. With the patches created by Claude that I have applied it worked >> again. >> >> Mario. >> >> On Wed, Jun 10, 2026 at 10:59 AM Corvin Köhne wrote: >>> On Tue, 2026-06-09 at 08:40 +0200, Mario Marietto wrote: >>>> --- >>>>> https://github.com/freebsd/freebsd-src/commit/effcd0ded0ef1c92bef29c1e9bbf51 >>>> 7c97bb45b8 >>>> >>>> oh ok. That's not the bug I was talking about. You gave no reply about that >>>> (bhyve: bootrom_alloc: vm_mmap_mapseg: Invalid argument on FreeBSD 15*- >>>> RELEASE). >>>> >>> Does it occur on stock 15? Nvidia GPU passthrough should work on stock 15, so >>> please test it without patching any files. >>> >>> >>> -- >>> Best regards, >>> Corvin >> >> -- >> Mario. HI! As a starting point, you could get relevant  informations with: # pciconf -lbcevV # grep vmm /boot/loader.conf # grep ppt /boot/loader.conf # acpidump -t | grep DMAR # kldstat # grep -iE '(vmm|ppt|APIC|DMAR|IOMMU|mapp)' /var/log/messages Regards, Stephan