Date: Mon, 06 Jan 2025 09:10:32 +0100 From: Corvin =?ISO-8859-1?Q?K=F6hne?= <corvink@FreeBSD.org> To: Peter Wood <peter@alastria.net>, Mario Marietto <marietto2008@gmail.com> Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve/passthru for Intel dGPU (ARC A380)? Message-ID: <d2e6638855f263e8f613ba06c37e091c710c27be.camel@FreeBSD.org> In-Reply-To: <CAD-E2icA%2BHkWZJL2JVjU28ECpc8BNG1KXkpQj4Zw5nB8yPfahQ@mail.gmail.com> References: <CAD-E2if_q6JreqPWiFBgPc=KHeP12Pq_E2R4m3ZxvGC3g87ZHA@mail.gmail.com> <CA%2B1FSihc5EiBUSFjxoUViAYzZ3qbo%2BqpssrBkuEvQK1=O9W6uw@mail.gmail.com> <CAD-E2icA%2BHkWZJL2JVjU28ECpc8BNG1KXkpQj4Zw5nB8yPfahQ@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] Hi Peter, unfortunately, this iGPU stuff is a bit messy. For some reason, Intel is unable to design them in a platform independent way. That's why bhyve has to probe for it and apply some quirks to it in it's GVT-d code. It's not expected that those quirks are required for dedicated GPUs. So, please try to skip the GVT-d check you've already found. If you want to pass the option rom to the guest, you can use the rom option of passthru devices: -s 1/2/3,passthru,1/2/3,rom=/path/to/rom -- Kind regards, Corvin On Sun, 2025-01-05 at 10:38 +0000, Peter Wood wrote: > Hello Mario, > > Thanks for the response. Interesting, is the expectation that an dGPU ARC > would work with the gvt-d code in place? Which would reenforce I'm fighting > the motherboard/BIOS. > > The BIOS is infact in CSM, as you've suggested, but for reason - though this > isn't a -virtualization problem, though I'll explain it for any future reader: > > My experience of the ROMED8-2T is when in pure UEFI after the FreeBSD loader > starts the framebuffer console there are no further updates via the AST2500 if > the Intel ARC is present, I suspect the framebuffer is being started on the > Intel ARC - but I don't have a monitor capable of checking that near the > server. > > The BIOS (3.5 and 3.8 checked) only offers configuration of the preferred > graphical output if the BIOS is CSM, and even then the framebuffer only seems > to stay on the AST2500 if the video option rom is set to legacy mode. > > For pure EFI, I haven't dug into loader yet, but what I have observed is that > the ROMED8-2T does not output the EFI variable ConOut, which having a quick > read would probably guide the loader where to push the framebuffer. I don't > know if it's possible for me to write that var to EFI, but may be worth > looking into. > > As an aside, with CSM (video legacy option rom) I did actually patch bhyve to > remove the gvt-d check, and the linux guest did boot with the passthru - I > assume with the card in legacy mode... the linux kernel was happy with it > until it tried to read the option rom, which I haven't exported yet - but I'd > prefer to fix this the correct way if possible. > > P. > > > On Sat, 4 Jan 2025 at 20:34, Mario Marietto <marietto2008@gmail.com> wrote: > > Hi Peter, > > > > Please make sure your GPU is booted in UEFI and not CSM mode. This can be > > changed in the host BIOS. > > Cheers. > > > > > > On Sat, Jan 4, 2025 at 9:24 PM Peter Wood <peter@alastria.net> wrote: > > > Happy new year all. > > > > > > I've been using bhyve happily for a year or two now, multiple machines > > > with VT-d running happily passing SAS cards and USB cards into VMs. > > > > > > I've reached the point where I want to pass a GPU in for accelerated > > > encoding/decoding/etc (scrypted, tdarr, jellyfin). I picked up an Intel > > > ARC A380, as it's encoders/decoders are pretty decent for my use case - > > > I'd also seen that there had been success with people using the iGPU's in > > > Intel CPUs successfully. > > > > > > Unfortunately after attaching the GPUs PCI device to ppt, and attempting > > > to start a VM with it attached, I'm greated by an error that seems to > > > focus on iGPUs? > > > > > > /usr/sbin/bhyve -A -H -w -u -S -c 2 -m 8G -l com1,/dev/nmdm202B -l > > > bootrom,/usr/local/share/uefi- > > > firmware/BHYVE_UEFI.fd,/usr/local/var/cache/bmd/k8s-worker-3.vars -s > > > 0,hostbridge -s 1,lpc -s 2,nvme,/dev/zvol/vm/k8s-worker-3 -s 3,ahci- > > > cd,/mnt/vm/isos/ubuntu-24.04-live-server-amd64.iso -s 5,virtio-net,tap9 -s > > > 4,passthru,4/0/0 -s 6,fbuf,tcp=0.0.0.0:6007,w=1280,h=720,vga=io -s > > > 7,xhci,tablet test > > > bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory. > > > GPU passthrough might not work properly. > > > bhyve: gvt_d_setup_opregion: Invalid OpRegion signature > > > bhyve: gvt_d_init: Unable to setup OpRegion > > > Device emulation initialization error: No such file or directory > > > > > > The machine is an AMD EPYC 7343 on a ASRock ROMED8-2T, it has a dedicated > > > onboard GPU (attached to the BMC), which the BIOS is configured to > > > encourage operating systems to use as the primary display - and sure > > > enough the loader and BSD console are presented through the BMCs ASPEED > > > AST2500. > > > > > > Digging through the freebsd source tree, it appears that that pci_gvt-d.c > > > is responsible for this, if it's Intel and a Display then attempt the set > > > up of graphics memory. > > > https://github.com/freebsd/freebsd-src/blob/b662ca1d6cd82044c6cb79075e1830b97594bef3/usr.sbin/bhyve/amd64/pci_gvt-d.c#L44 > > > > > > Has anyone experimented with this? Can I just patch this out, rebuild > > > bhyve and expect a chance of success? > > > > > > Cheers, > > > > > > P. > > > -- > > > Peter Wood > > > peter@alastria.net > > > > > > > > > -- > > Mario. > > > -- > Peter Wood > peter@alastria.net > -- Kind regards, Corvin [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmd7j/gACgkQ2FTaVjFe AmrFJg//aR0fFMQwh61Ogs+/d//AkS8XXsYYp6jkEJH3kxec7QvorB1rY/1oYIXn RwCbMwmk1SwHtkrWhGeUdI/G0sQy6rXV4s4ycEni6NhwTflp2ZnblGDvFAwh/yhT hS7eU7LXA5R83Hcy7/3EDFK575XaoT+to6c8+lckcbTbmvIuQfgQ5oHxh8nRZb0f 9mpi5ij7zwtcI3Skhu4NfheyXDNaHFPusD/S4O7dCf2NutURtIZDJ/ZUKm9Lt35H tLBFSlCFnkE+mwLG4FbpBJDNkKlKINaUQJXFaS6YGIFeZZ7P/PMIl7z9y4AuNEox PLCDFEF95B0sLL183tfUtVhUfv1F5eqE8A+pE1YUGMhf2BbAMYJnQ8iSBV1hH5Cu vG8NP7IaEkvUm3sltPf6rYtFmNioOaFAvU3bwPItTOjEtUy8+Oied8d51sujOg8F ZX3Zl4LZ8RUZ/Cpfq2aFlnBEAvHbhXmRO6CVreF8m0rlCwTTeRXNPOr076WCUeoL 9NQQ9bvU37EwhWTrI7S5qtk4J1OU1D17YHC+XUPSivBzffkihoB94hEgRM4rFTEk DpUGIW2gTP0dOn6GKdyNcULJvW5fU+jgiYZgqCtkULgm9dUv1hEGs5MMGfcFO2YQ AvSO7qpoFTH948dXViFI/whKX/X7UnsbHEdqC2p+XvwUMep5/e4= =IEUd -----END PGP SIGNATURE-----help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?d2e6638855f263e8f613ba06c37e091c710c27be.camel>