From nobody Mon Jan 6 08:10:32 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YRRgt0xBVz5k8VJ for ; Mon, 06 Jan 2025 08:10:34 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YRRgt0MW4z47Hr; Mon, 6 Jan 2025 08:10:34 +0000 (UTC) (envelope-from corvink@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1736151034; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=I1mh08et75SWA9UQ/flD504P2/0uoIxi4Vj08Q7ls08=; b=KjSYA5zxAKT1V0s9chFCWYurM6EmY22YLhdq3qEAlGAZ1Q+bVEkacQNgfFmqDdqkK6JB7f LpNaK8Df/euKuvh57cZ317K8t3rDbX955DG15+RROxsDIiXc8Fdn51rneFk9sGxPDXD4b1 e/iR2TUYSpYm3BggUHoVglkedtzkitY5qdmxTYnIW+xXQjA6kSkyP8a677WQYfYI33Gmv4 lDOOaEN5D+8nZgxiXfkW+AV7c5Y8wOGJEXyVFkFg0r/lxwI7+TVIrxg7ZiprhtRjrqrXLb xQN3EBEesaAz9vfnd3o+LWKt8ilRfJclkA3GcK08zMfJlUj0YBz6cOwubgeKNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1736151034; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=I1mh08et75SWA9UQ/flD504P2/0uoIxi4Vj08Q7ls08=; b=dohO9aXF8G6gSBIIHehIL2qvpXd/pNfI4FCWAd8+IWHMtIq9brIVc9iG50q0GLDJB11Km5 UQHzbN+DDa4haA99STjgSbdn8EjoFCFH5hDDVG1A7+42oF40M5Qg+BQr9NTS/1kA3D5RHI tFjz+PJjn0i3YWWTYczCDI4bB38yd1d0ldjbwWfufDtcDvDkgZufZXqJSiZTivKhZOQtP6 PSWL7sPEM6hgIUHq281Oa/gGAAGVXEYSENrD+cm8FnqlYII54tA1bYSP55M4aQFUMB3zVI oKbCEz2qu0WRtVJMaFtOlxwHt+5JRdkUW6S/Dw8NEs00vlWLvjLJJrcmdOg3ew== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1736151034; a=rsa-sha256; cv=none; b=mGTPNoH+EGOBIUxtc9lDkd0ECuWyXd+DTHSW1D26OhbfbvxE0aB2cWBLKEdXkHpvoshy2F jFrZ4EKQidu0acVOHlZPT3WMGqHAGVtLDcNLzSczO3zUm1VOPhpzwOw5Vbzpd3hFs40Pvh WTv+6JHxr/gF5/T37oOZBhvZtQF+PhpX3vG5i4BDT4V3jH9kzfJsZRWWCZEJj12JmQ2UqR mTIqdqghcH4zixAPszJuFcYMx0rJ53LJOGHqt9h21Y0kb4COgNswgEZVJPta2AI0ci7Brw BZ7xZ/7qFwgU6wu4lrOnCVqnPGslTKCkm6fZslSFYVyb/8+zT3ZU5Jj3eiFEvA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [IPv6:2001:9e8:dc01:5900:fc72:1b65:c1e1:a77c] (unknown [IPv6:2001:9e8:dc01:5900:fc72:1b65:c1e1:a77c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: corvink) by smtp.freebsd.org (Postfix) with ESMTPSA id 4YRRgs3sV2zHvL; Mon, 06 Jan 2025 08:10:33 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Message-ID: Subject: Re: bhyve/passthru for Intel dGPU (ARC A380)? From: Corvin =?ISO-8859-1?Q?K=F6hne?= To: Peter Wood , Mario Marietto Cc: freebsd-virtualization@freebsd.org Date: Mon, 06 Jan 2025 09:10:32 +0100 In-Reply-To: References: Autocrypt: addr=corvink@FreeBSD.org; prefer-encrypt=mutual; keydata=mQINBGNjZaIBEADDTrDNf+0pwiuRPBdClcnZW83dH1UhuOi0u+A1J2SatEBbNaFVtXXAa vewCTuyV/ZbNidjlhq3R/pWyiKjFKvs5dj7PMCw+3z2D5OWpMdHg7TrB+fbdFPOEsu0zQVKNaO+pS KCfN0Re0m7bL3wuvl7PXvBufRwA3Guo1P4j3TXWaEkuso7VupTvE25zVGg9ONHrGOjA9RUy+Yg4Se 3NLgtUdjBgA21SBQTDvRQV4fDmVenlwvWeE0Xm8FcDcpQb6sJTihaDku78mi3Ux1HCk7rTcepVEB0 xIB6qmFxv0sLlDmVv6Z6qg1y/Q5m23Pgz60o3TulMPV4F+3Itm8ifU+wgVSzBZbD29GYkd7LKqMkF bhvfSBk+5db3vbYY5OD//+LTM5AV7e2AhXuXMvG1UNBqXqSJTTSy6KZz+qmPQO0zos0dq46p8o82l KiBEGD2Hu0p+u0OyV+MmRYo1NIBFVbOPXp2MvUVl5II0UIJ3+N9gLBmfGA+HEpVO8PnvdoT/5NQ7m 8JK1rQHzjiDub/iDPAYMqKH4C0eZ/7zO0fuY5FeRNtuNtpH1Bw/+7/5RJH7bcKkfGHHEp15FJUrGH gWNydoDLB9QBprwQc8FEldDXBjzOMXIgh6FGKLNu6DswvIPGy6M3u7DXwDakCXz+c9Ym0oFihLzZx WntrsxdswD/CwARAQABtCdDb3J2aW4gS8O2aG5lIDxjb3J2aW4ua29laG5lQGdtYWlsLmNvbT6JAl QEEwEIAD4WIQSC9FKVreba38fZT0bYVNpWMV4CagUCZArUIwIbAwUJCWYBgAULCQgHAgYVCgkICwI EFgIDAQIeAQIXgAAKCRDYVNpWMV4CalcID/44k2i/mqSSi4W6FAobSF1nFLtP/pfcRNJriWKx2UF7 cfFMKyg7Nilg7FhLb5FDB1umUW2nFfchFPTUp4FfKzgRvPzIMg4RIRcVtTpYbl3z7zs9ZXD8qS//i ibbiUG3quncm6tO2x1jLZD3ORC+8MuLGXhYQIa4O5vVF2SBHdb/U6P+wsrF+U+OpRdEdQ/4Xu9S02 kltzBGgArjcexdhUqEqW01KCCSH0+qgfN1NE+9L934ZOB+cai2b9apPbCOGuV6KcUKMj4z0RWInhl XIyMqtmhdix/P/GjrQ1REVNdp74JeweSSedM15wwc6YLMXPrtOnnExyZ1gyNFYaub+Mdo3ZQ+8386 3B0C9IlpzEW0K8bYlZrl4WBNdcMOyByETAxgQmBgP6ZFErTtaeUOH1nX2FyR6o2GLSahRgngQmnRE zScTpPbBEkMwJMDAO+rbjjGxjeKSUwD1WOfbI6QZj+MS/uBk2p08kgN7fQaCEwj6jqML/IIE+FZ9I In6TNG0hChD384VHO+YioLBno1Atgi4Q7JUWSRIHQXZW+StQajFrWPPyKQwCe1MwqpKoMpX/q8IZB lzwJgZS8ShLeFZjtzOt1jgM99TD91Neonf9OzjTSbfo7sJviSWoICMhB/MvDZjj+naMVF86uGFxnI EsjVKyBxlJd4TRHnLYPTIHFKBLQjQ29ydmluIEvDtmhuZSA8Y29ydmlua0BGcmVlQlNELm9yZz6JA lQEEwEKAD4WIQSC9FKVreba38fZT0bYVNpWMV4CagUCY2NlogIbAwUJCWYBgAULCQgHAwUVCgkICw UWAwIBAAIeAQIXgAAKCRDYVNpWMV4CavfDEACCFnXpR7H9eOgP+GJMNPtK6i9/xnqdyXi8uCZIN0h YwjN4Xzo9SMLOf4UUlQEveOB+bGqbRfHd/fGKnrlXiPd0SGpKWJC21gqL/DsIH0J8I3Whth+O8tfP WeFy0oCsvBaaGFLIrDfoIgHF9i/gqEe48xhN42weB02Z3mdR1L0d7ME/BLwS0mCXe9Zh3uHw63S6x YB3Wsjptxe/ph6TpQDUKWtRJkjC6BqXPBdThpbbfIRWmjZbp2fKEJPvtRXS14+gbUqWeJ4xCvprA3 +ae7vtrp91X775yngyW3XTw5cmDiJIjykH8+zhEIoNQXNBpFrehkQDYrcM+WoE6NGSJo+3VJvSRWh UGWDVrxdTYNkbIjmTNlkI12NINC007DiuV7OF9XHWgrbbylvuZvbODmbJRdhTFy9upAUygX1/xUAQ EIMqMiJmyTdv8i0IbZ611WElQx9XHgGeZgM6+39/laN8FwspM6gE/4NzZHIZN0LEBOerZqoF+Il6e ccQpoEWx7nb/RilJp3dUUyvkBnJWg+AJByosg857kvvmDnZ3UB+bejWpcfFvnbkiKPUBPDO7tWPb5 r2yFDpDe/Vg5x+sRbkkXGUFD6Rx0p/ZiRIneVVg8emTzhDR0IL/BZkq/uVJkdaphZli1F/31cNgf6 ZJYnjxlk86uiYSySZQR0dKLDqq7QlQ29ydmluIEvDtmhuZSA8Yy5rb2VobmVAYmVja2hvZmYuY29t PokCVAQTAQgAPhYhBIL0UpWt5trfx9lPRthU2lYxXgJqBQJkCtScAhsDBQkJZgGABQsJCAcCBhUKC QgLAgQWAgMBAh4BAheAAAoJENhU2lYxXgJqI+QQAML5PTR7KpUFV3SLG60LQJGEOHUfDmJYczxBFb IAq1U4hIbivopu1AdLty7oDDrIjCVoa2/Cy34dd99O7lhLvUmZFB/zDSUtbUg2zhDkU0YSZ11Fdrl Wzky2tFaQRgxpDvWlUP0baa3Pd4dPDRiIUI6AOSR2SL6XANk6sJh56gLVM6G8yyafGsxSyDYg6Z78 EEMFejHwB+KP2DdsahupzM+F97HeC1+bOHYxtqN+2hEkPLtQWizyumPqNg5FvZhwe7yO8V95hF3Rh uDO+9aJT+WLLvcZEb/L1bI04IvZ5FWgCLI7Levd/DuOtZI8gWapHhqGZRbXB2fuJkCoKCl6V67h/7 aWhU3LjFTsC5siJyrxPjapKcIk8a7PqZDswNCKR+24LJ5D59mPgEOnsiCCVpik1WE/kgD+rOu9dQx jpjKwuKowf4EJP1KYNkYtoy1HthzyTOqPwqXC3IUl0GVPO3xw6MhUM3irCVFruC+ecVrv85Rd37vr duT6JvgW92xjWegMsamtkDZH8Ik/cmYhH0K/qEc6OZVNea/4PTEZe0uxODJ6pbMd80AJyGqDPPVeA gWJtEIG4k6IS8XyD5v1QJtlpDVpPwP/bbFnVc3h3Oatfn8Etm1KAqYvNwyO+om2PkF1p732uapDZd LwksVmgc9s79+9pSpeP5DbIeMzhrIK Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-avojERmarfoZl3SV3vsk" User-Agent: Evolution 3.54.2 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 --=-avojERmarfoZl3SV3vsk Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Peter, unfortunately, this iGPU stuff is a bit messy. For some reason, Intel is un= able to design them in a platform independent way. That's why bhyve has to probe= for it and apply some quirks to it in it's GVT-d code. It's not expected that t= hose quirks are required for dedicated GPUs. So, please try to skip the GVT-d ch= eck you've already found. If you want to pass the option rom to the guest, you can use the rom option= of passthru devices: -s 1/2/3,passthru,1/2/3,rom=3D/path/to/rom -- Kind regards, Corvin On Sun, 2025-01-05 at 10:38 +0000, Peter Wood wrote: > Hello Mario, >=20 > Thanks for the response. Interesting, is the expectation that an dGPU ARC > would work with the gvt-d code in place? Which would reenforce I'm fighti= ng > the motherboard/BIOS. >=20 > The BIOS is infact in CSM, as you've suggested, but for reason - though t= his > isn't a -virtualization problem, though I'll explain it for any future re= ader: >=20 > My experience of the ROMED8-2T is when in pure UEFI after the FreeBSD loa= der > starts the framebuffer console there are no further updates via the AST25= 00 if > the Intel ARC is present, I suspect the framebuffer is being started on t= he > Intel ARC - but I don't have a monitor capable of checking that near the > server. >=20 > The BIOS (3.5 and 3.8 checked) only offers configuration of the preferred > graphical output if the BIOS is CSM, and even then the framebuffer only s= eems > to stay on the AST2500 if the video option rom is set to legacy mode. >=20 > For pure EFI, I haven't dug into loader yet, but what I have observed is = that > the ROMED8-2T does not output the EFI variable ConOut, which having a qui= ck > read would probably guide the loader where to push the framebuffer. I don= 't > know if it's possible for me to write that var to EFI, but may be worth > looking into. >=20 > As an aside, with CSM (video legacy option rom) I did actually patch bhyv= e to > remove the gvt-d check, and the linux guest did boot with the passthru - = I > assume with the card in legacy mode... the linux kernel was happy with it > until it tried to read the option rom, which I haven't exported yet - but= I'd > prefer to fix this the correct way if possible. >=20 > P. >=20 >=20 > On Sat, 4 Jan 2025 at 20:34, Mario Marietto wrot= e: > > Hi Peter, > > =C2=A0 > > Please make sure your GPU is booted in UEFI and not CSM mode. This can = be > > changed in the host BIOS. > > Cheers.=20 > >=20 > >=20 > > On Sat, Jan 4, 2025 at 9:24=E2=80=AFPM Peter Wood = wrote: > > > Happy new year all. > > >=20 > > > I've been using bhyve happily for a year or two now, multiple machine= s > > > with VT-d running happily passing SAS cards and USB cards into VMs. > > >=20 > > > I've reached the point where I want to pass a GPU in for accelerated > > > encoding/decoding/etc (scrypted, tdarr, jellyfin). I picked up an Int= el > > > ARC A380, as it's encoders/decoders are pretty decent for my use case= - > > > I'd also seen that there had been success with people using the iGPU'= s in > > > Intel CPUs successfully. > > >=20 > > > Unfortunately after attaching the GPUs PCI device to ppt, and attempt= ing > > > to start a VM with it attached, I'm greated by an error that seems to > > > focus on iGPUs? > > >=20 > > > /usr/sbin/bhyve -A -H -w -u -S -c 2 -m 8G -l com1,/dev/nmdm202B -l > > > bootrom,/usr/local/share/uefi- > > > firmware/BHYVE_UEFI.fd,/usr/local/var/cache/bmd/k8s-worker-3.vars -s > > > 0,hostbridge -s 1,lpc -s 2,nvme,/dev/zvol/vm/k8s-worker-3 -s 3,ahci- > > > cd,/mnt/vm/isos/ubuntu-24.04-live-server-amd64.iso -s 5,virtio-net,ta= p9 -s > > > 4,passthru,4/0/0 -s 6,fbuf,tcp=3D0.0.0.0:6007,w=3D1280,h=3D720,vga=3D= io -s > > > 7,xhci,tablet test > > > bhyve: Warning: Unable to reuse host address of Graphics Stolen Memor= y. > > > GPU passthrough might not work properly. > > > bhyve: gvt_d_setup_opregion: Invalid OpRegion signature > > > bhyve: gvt_d_init: Unable to setup OpRegion > > > Device emulation initialization error: No such file or directory > > >=20 > > > The machine is an AMD EPYC 7343 on a ASRock ROMED8-2T, it has a dedic= ated > > > onboard GPU (attached to the BMC), which the BIOS is configured to > > > encourage operating systems to use as the primary display - and sure > > > enough the loader and BSD console are presented through the BMCs ASPE= ED > > > AST2500. > > >=20 > > > Digging through the freebsd source tree, it appears that that pci_gvt= -d.c > > > is responsible for this, if it's Intel and a Display then attempt the= set > > > up of graphics memory. > > > https://github.com/freebsd/freebsd-src/blob/b662ca1d6cd82044c6cb79075= e1830b97594bef3/usr.sbin/bhyve/amd64/pci_gvt-d.c#L44 > > >=20 > > > Has anyone experimented with this? Can I just patch this out, rebuild > > > bhyve and expect a chance of success? > > >=20 > > > Cheers, > > >=20 > > > P. > > > --=20 > > > Peter Wood > > > peter@alastria.net > > >=20 > >=20 > >=20 > > --=20 > > Mario. >=20 >=20 > --=20 > Peter Wood > peter@alastria.net >=20 --=20 Kind regards, Corvin --=-avojERmarfoZl3SV3vsk Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmd7j/gACgkQ2FTaVjFe AmrFJg//aR0fFMQwh61Ogs+/d//AkS8XXsYYp6jkEJH3kxec7QvorB1rY/1oYIXn RwCbMwmk1SwHtkrWhGeUdI/G0sQy6rXV4s4ycEni6NhwTflp2ZnblGDvFAwh/yhT hS7eU7LXA5R83Hcy7/3EDFK575XaoT+to6c8+lckcbTbmvIuQfgQ5oHxh8nRZb0f 9mpi5ij7zwtcI3Skhu4NfheyXDNaHFPusD/S4O7dCf2NutURtIZDJ/ZUKm9Lt35H tLBFSlCFnkE+mwLG4FbpBJDNkKlKINaUQJXFaS6YGIFeZZ7P/PMIl7z9y4AuNEox PLCDFEF95B0sLL183tfUtVhUfv1F5eqE8A+pE1YUGMhf2BbAMYJnQ8iSBV1hH5Cu vG8NP7IaEkvUm3sltPf6rYtFmNioOaFAvU3bwPItTOjEtUy8+Oied8d51sujOg8F ZX3Zl4LZ8RUZ/Cpfq2aFlnBEAvHbhXmRO6CVreF8m0rlCwTTeRXNPOr076WCUeoL 9NQQ9bvU37EwhWTrI7S5qtk4J1OU1D17YHC+XUPSivBzffkihoB94hEgRM4rFTEk DpUGIW2gTP0dOn6GKdyNcULJvW5fU+jgiYZgqCtkULgm9dUv1hEGs5MMGfcFO2YQ AvSO7qpoFTH948dXViFI/whKX/X7UnsbHEdqC2p+XvwUMep5/e4= =IEUd -----END PGP SIGNATURE----- --=-avojERmarfoZl3SV3vsk--