Date: Sun, 5 Jan 2025 10:38:49 +0000 From: Peter Wood <peter@alastria.net> To: Mario Marietto <marietto2008@gmail.com> Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve/passthru for Intel dGPU (ARC A380)? Message-ID: <CAD-E2icA%2BHkWZJL2JVjU28ECpc8BNG1KXkpQj4Zw5nB8yPfahQ@mail.gmail.com> In-Reply-To: <CA%2B1FSihc5EiBUSFjxoUViAYzZ3qbo%2BqpssrBkuEvQK1=O9W6uw@mail.gmail.com> References: <CAD-E2if_q6JreqPWiFBgPc=KHeP12Pq_E2R4m3ZxvGC3g87ZHA@mail.gmail.com> <CA%2B1FSihc5EiBUSFjxoUViAYzZ3qbo%2BqpssrBkuEvQK1=O9W6uw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000ae3784062af31e72 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Mario, Thanks for the response. Interesting, is the expectation that an dGPU ARC would work with the gvt-d code in place? Which would reenforce I'm fighting the motherboard/BIOS. The BIOS is infact in CSM, as you've suggested, but for reason - though this isn't a -virtualization problem, though I'll explain it for any future reader: My experience of the ROMED8-2T is when in pure UEFI after the FreeBSD loader starts the framebuffer console there are no further updates via the AST2500 if the Intel ARC is present, I suspect the framebuffer is being started on the Intel ARC - but I don't have a monitor capable of checking that near the server. The BIOS (3.5 and 3.8 checked) only offers configuration of the preferred graphical output if the BIOS is CSM, and even then the framebuffer only seems to stay on the AST2500 if the video option rom is set to legacy mode. For pure EFI, I haven't dug into loader yet, but what I have observed is that the ROMED8-2T does not output the EFI variable ConOut, which having a quick read would probably guide the loader where to push the framebuffer. I don't know if it's possible for me to write that var to EFI, but may be worth looking into. As an aside, with CSM (video legacy option rom) I did actually patch bhyve to remove the gvt-d check, and the linux guest did boot with the passthru - I assume with the card in legacy mode... the linux kernel was happy with it until it tried to read the option rom, which I haven't exported yet - but I'd prefer to fix this the correct way if possible. P. On Sat, 4 Jan 2025 at 20:34, Mario Marietto <marietto2008@gmail.com> wrote: > Hi Peter, > > > > Please make sure your GPU is booted in UEFI and not CSM mode. This can be > changed in the host BIOS. > > Cheers. > > > > On Sat, Jan 4, 2025 at 9:24=E2=80=AFPM Peter Wood <peter@alastria.net> wr= ote: > >> Happy new year all. >> >> I've been using bhyve happily for a year or two now, multiple machines >> with VT-d running happily passing SAS cards and USB cards into VMs. >> >> I've reached the point where I want to pass a GPU in for accelerated >> encoding/decoding/etc (scrypted, tdarr, jellyfin). I picked up an Intel = ARC >> A380, as it's encoders/decoders are pretty decent for my use case - I'd >> also seen that there had been success with people using the iGPU's in In= tel >> CPUs successfully. >> >> Unfortunately after attaching the GPUs PCI device to ppt, and attempting >> to start a VM with it attached, I'm greated by an error that seems to fo= cus >> on iGPUs? >> >> /usr/sbin/bhyve -A -H -w -u -S -c 2 -m 8G -l com1,/dev/nmdm202B -l >> bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/usr/local/var/cach= e/bmd/k8s-worker-3.vars >> -s 0,hostbridge -s 1,lpc -s 2,nvme,/dev/zvol/vm/k8s-worker-3 -s >> 3,ahci-cd,/mnt/vm/isos/ubuntu-24.04-live-server-amd64.iso -s >> 5,virtio-net,tap9 -s 4,passthru,4/0/0 -s 6,fbuf,tcp=3D0.0.0.0:6007,w=3D1= 280,h=3D720,vga=3Dio >> -s 7,xhci,tablet test >> bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory. >> GPU passthrough might not work properly. >> bhyve: gvt_d_setup_opregion: Invalid OpRegion signature >> bhyve: gvt_d_init: Unable to setup OpRegion >> Device emulation initialization error: No such file or directory >> >> The machine is an AMD EPYC 7343 on a ASRock ROMED8-2T, it has a dedicate= d >> onboard GPU (attached to the BMC), which the BIOS is configured to >> encourage operating systems to use as the primary display - and sure eno= ugh >> the loader and BSD console are presented through the BMCs ASPEED AST2500= . >> >> Digging through the freebsd source tree, it appears that that pci_gvt-d.= c >> is responsible for this, if it's Intel and a Display then attempt the se= t >> up of graphics memory. >> >> https://github.com/freebsd/freebsd-src/blob/b662ca1d6cd82044c6cb79075e18= 30b97594bef3/usr.sbin/bhyve/amd64/pci_gvt-d.c#L44 >> >> Has anyone experimented with this? Can I just patch this out, rebuild >> bhyve and expect a chance of success? >> >> Cheers, >> >> P. >> -- >> *Peter Wood* >> peter@alastria.net >> >> > > -- > Mario. > --=20 *Peter Wood* peter@alastria.net --000000000000ae3784062af31e72 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hello Mario,</div><div><br></div><div>Thanks for the = response. Interesting, is the expectation that an dGPU ARC would work with = the gvt-d code in place? Which would reenforce I'm fighting the motherb= oard/BIOS.<br></div><div><br></div><div>The BIOS is infact in CSM, as you&#= 39;ve suggested, but for reason - though this isn't a -virtualization p= roblem, though I'll explain it for any future reader:<br></div><div><br= ></div><div>My experience of the ROMED8-2T is when in pure UEFI after the F= reeBSD loader starts the framebuffer console there are no further updates v= ia the AST2500 if the Intel ARC is present, I suspect the framebuffer is be= ing started on the Intel ARC - but I don't have a monitor capable of ch= ecking that near the server.<br></div><br><div>The BIOS (3.5 and 3.8 checke= d) only offers configuration of the preferred graphical output if the BIOS = is CSM, and even then the framebuffer only seems to stay on the AST2500 if = the video option rom is set to legacy mode.</div><div><br></div><div>For pu= re EFI, I haven't dug into loader yet, but what I have observed is that= the ROMED8-2T does not output the EFI variable ConOut, which having a quic= k read would probably guide the loader where to push the framebuffer. I don= 't know if it's possible for me to write that var to EFI, but may b= e worth looking into.</div><div><br></div><div>As an aside, with CSM (video= legacy option rom) I did actually patch bhyve to remove the gvt-d check, a= nd the linux guest did boot with the passthru - I assume with the card in l= egacy mode... the linux kernel was happy with it until it tried to read the= option rom, which I haven't exported yet - but I'd prefer to fix t= his the correct way if possible.</div><div><br></div><div>P.<br></div><div>= <br></div></div><br><div class=3D"gmail_quote gmail_quote_container"><div d= ir=3D"ltr" class=3D"gmail_attr">On Sat, 4 Jan 2025 at 20:34, Mario Marietto= <<a href=3D"mailto:marietto2008@gmail.com">marietto2008@gmail.com</a>&g= t; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p= x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div d= ir=3D"ltr"><p class=3D"MsoNormal">Hi Peter,</p> <p class=3D"MsoNormal"><u></u>=C2=A0<u></u></p> <p class=3D"MsoNormal">Please make sure your GPU is booted in UEFI and not = CSM mode. This can be changed in the <span>host</span> BIOS.</p> <p class=3D"MsoNormal">Cheers. <br></p><p class=3D"MsoNormal"><br></p></div= ><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Sa= t, Jan 4, 2025 at 9:24=E2=80=AFPM Peter Wood <<a href=3D"mailto:peter@al= astria.net" target=3D"_blank">peter@alastria.net</a>> wrote:<br></div><b= lockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-le= ft:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div>Happy= new year all.<br></div><div><br></div><div>I've been using bhyve happi= ly for a year or two now, multiple machines with VT-d running happily passi= ng SAS cards and USB cards into VMs.</div><div><br></div><div>I've reac= hed the point where I want to pass a GPU in for accelerated encoding/decodi= ng/etc (scrypted, tdarr, jellyfin). I picked up an Intel ARC A380, as it= 9;s encoders/decoders are pretty decent for my use case - I'd also seen= that there had been success with people using the iGPU's in Intel CPUs= successfully.</div><div><br></div><div>Unfortunately after attaching the G= PUs PCI device to ppt, and attempting to start a VM with it attached, I'= ;m greated by an error that seems to focus on iGPUs?<br></div><div><br></di= v><div>/usr/sbin/bhyve -A -H -w -u -S -c 2 -m 8G -l com1,/dev/nmdm202B -l b= ootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/usr/local/var/cache/bm= d/k8s-worker-3.vars -s 0,hostbridge -s 1,lpc -s 2,nvme,/dev/zvol/vm/k8s-wor= ker-3 -s 3,ahci-cd,/mnt/vm/isos/ubuntu-24.04-live-server-amd64.iso -s 5,vir= tio-net,tap9 -s 4,passthru,4/0/0 -s 6,fbuf,tcp=3D<a href=3D"http://0.0.0.0:= 6007" target=3D"_blank">0.0.0.0:6007</a>,w=3D1280,h=3D720,vga=3Dio -s 7,xhc= i,tablet test<br>bhyve: Warning: Unable to reuse host address of Graphics S= tolen Memory. GPU passthrough might not work properly.<br>bhyve: gvt_d_setu= p_opregion: Invalid OpRegion signature<br>bhyve: gvt_d_init: Unable to setu= p OpRegion<br>Device emulation initialization error: No such file or direct= ory<br></div><div><br></div><div>The machine is an AMD EPYC 7343 on a ASRoc= k ROMED8-2T, it has a dedicated onboard GPU (attached to the BMC), which th= e BIOS is configured to encourage operating systems to use as the primary d= isplay - and sure enough the loader and BSD console are presented through t= he BMCs ASPEED AST2500.</div><div><br></div><div>Digging through the freebsd source= tree, it appears that that pci_gvt-d.c is responsible for this, if it'= s Intel and a Display then attempt the set up of graphics memory.</div><div= ><a href=3D"https://github.com/freebsd/freebsd-src/blob/b662ca1d6cd82044c6c= b79075e1830b97594bef3/usr.sbin/bhyve/amd64/pci_gvt-d.c#L44" target=3D"_blan= k">https://github.com/freebsd/freebsd-src/blob/b662ca1d6cd82044c6cb79075e18= 30b97594bef3/usr.sbin/bhyve/amd64/pci_gvt-d.c#L44</a></div><div><br></div><= div>Has anyone experimented with this? Can I just patch this out, rebuild b= hyve and expect a chance of success?</div><div><br></div><div>Cheers,</div>= <div><br></div><div>P.</div><span class=3D"gmail_signature_prefix">-- </spa= n><br><div dir=3D"ltr" class=3D"gmail_signature"><div dir=3D"ltr"><div><b>P= eter Wood</b></div><div><div style=3D"font-size:12.8px"><a href=3D"mailto:p= eter@alastria.net" target=3D"_blank">peter@alastria.net</a></div></div><div= ><br></div></div></div></div> </blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si= gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Ma= rio.<br></div> </blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si= gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><d= iv dir=3D"ltr"><div><b>Peter Wood</b></div><div><div style=3D"font-size:12.= 8px"><a href=3D"mailto:peter@alastria.net" target=3D"_blank">peter@alastria= .net</a></div></div><div><br></div></div></div> --000000000000ae3784062af31e72--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAD-E2icA%2BHkWZJL2JVjU28ECpc8BNG1KXkpQj4Zw5nB8yPfahQ>