Date: Mon, 6 Jan 2025 16:05:08 +0100 From: Mario Marietto <marietto2008@gmail.com> To: =?UTF-8?Q?Corvin_K=C3=B6hne?= <corvink@freebsd.org> Cc: Peter Wood <peter@alastria.net>, freebsd-virtualization@freebsd.org Subject: Re: bhyve/passthru for Intel dGPU (ARC A380)? Message-ID: <CA%2B1FSihc=gTcttUZVY4-3jVF905CY09pU65iYqAqNJ73yA4xbg@mail.gmail.com> In-Reply-To: <d2f9347053741ceab43d925ad22448f325559974.camel@FreeBSD.org> References: <CAD-E2if_q6JreqPWiFBgPc=KHeP12Pq_E2R4m3ZxvGC3g87ZHA@mail.gmail.com> <CA%2B1FSihc5EiBUSFjxoUViAYzZ3qbo%2BqpssrBkuEvQK1=O9W6uw@mail.gmail.com> <CAD-E2icA%2BHkWZJL2JVjU28ECpc8BNG1KXkpQj4Zw5nB8yPfahQ@mail.gmail.com> <d2e6638855f263e8f613ba06c37e091c710c27be.camel@FreeBSD.org> <CAD-E2ieq0P03tTkuAdvg-D35ua9PQT6XuT9GHOPLc7Aqa6xGvw@mail.gmail.com> <d2f9347053741ceab43d925ad22448f325559974.camel@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000759f67062b0af68e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I want here to reiterate the importance of creating a version of Bhyve that is able to accelerate qemu. Please can some developer "hear" me ? you know that this idea is good and fundamental for bhyve and for FreeBSD,for every user who loves this system. On Mon, Jan 6, 2025 at 3:41=E2=80=AFPM Corvin K=C3=B6hne <corvink@freebsd.o= rg> wrote: > On Mon, 2025-01-06 at 13:56 +0000, Peter Wood wrote: > > Thanks for the feedback Corvin, and thank you for the hard work you've > been > > putting into GPU passthru. > > > > I'm reaching the end of my limited knowledge here, and I have no > expectation > > of any further assistance - but as a status update, BIOS in CSM (with > legacy > > video op rom, so I see the console). > > > > > If you want to pass the option rom to the guest, you can use the rom > option > > > of > > > passthru devices: > > > > > > -s 1/2/3,passthru,1/2/3,rom=3D/path/to/rom > > > > > > > > > I extracted the option ROM using linux, I was able to use the > > /sys/devices/pci*/rom route to extract it, it seems valid at a glance > (768k > > dump) - but no idea how to really tell. > > > > Using the patched bhyve executable to bypass gvt-d: -s > > 4/0/0,passthru,4/0/0,rom=3D/mnt/vm/intel-arc-a380.bin -s > 5/0/0,passthru,5/0/0 > > (5/0/0 is a separate audio device exposing the audio channels of the HD= MI > > ports). > > > > Sadly initialization of the GPU in the linux (Ubuntu 24.04 / linux > 6.8.0) VM > > still fails: > > [ 2.508656] i915 0000:00:04.0: enabling device (0000 -> 0002) > > [ 2.520226] i915 0000:00:04.0: [drm] Local memory IO size: > > 0x000000017c800000 > > [ 2.520232] i915 0000:00:04.0: [drm] Local memory available: > > 0x000000017c800000 > > [ 2.540148] i915 0000:00:04.0: vgaarb: VGA decodes changed: > > olddecodes=3Dio+mem,decodes=3Dnone:owns=3Dnone > > [ 2.550829] i915 0000:00:04.0: [drm] Finished loading DMC firmware > > i915/dg2_dmc_ver2_08.bin (v2.8) > > [ 2.564885] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture alloc siz= e > > changed from 32768 to 36864 > > [ 2.565855] i915 0000:00:04.0: [drm] GT0: GuC firmware > i915/dg2_guc_70.bin > > version 70.20.0 > > [ 2.565859] i915 0000:00:04.0: [drm] GT0: HuC firmware > i915/dg2_huc_gsc.bin > > version 7.10.3 > > [ 2.565979] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture alloc siz= e > > changed from 32768 to 36864 > > [ 2.567001] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status = =3D > > 0x40000056, time =3D 0ms, freq =3D 2300MHz, ret =3D 0 > > [ 2.567006] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status: > Reset =3D > > 0, BootROM =3D 0x2B, UKernel =3D 0x00, MIA =3D 0x00, Auth =3D 0x01 > > [ 2.567009] i915 0000:00:04.0: [drm] GT0: GUC: firmware production > part > > check failure > > [ 2.567077] i915 0000:00:04.0: [drm] *ERROR* GT0: GuC initialization > failed > > -ENOEXEC > > [ 2.567610] i915 0000:00:04.0: [drm] *ERROR* GT0: Enabling uc failed > (-5) > > [ 2.567949] i915 0000:00:04.0: [drm] *ERROR* GT0: Failed to > initialize GPU, > > declaring it wedged! > > [ 2.570106] i915 0000:00:04.0: [drm:add_taint_for_CI [i915]] CI > tainted:0x9 > > by intel_gt_set_wedged_on_init+0x34/0x50 [i915] > > [ 2.587048] [drm] Initialized i915 1.6.0 20230929 for 0000:00:04.0 o= n > minor > > 1 > > > > If possible, it might be a good idea to check if it's running on a Linux > host > with QEMU properly. If yes, we may be able to check if QEMU has some > special > quirks for those devices (don't see one yet). > > > Interestingly intel_gpu_top will interact with the card to a degree, it > shows > > a render utilization (of 0%), but none of the other card capabilities. > There > > are some very similar errors in Google which may suggest it's may not b= e > a > > bhyve/passthru issue, though it could be. I need to spin up a new VM > with more > > bleeding edge linux (or maybe even Win11) to see if it can talk to the > card. > > > > https://github.com/intel-analytics/ipex-llm/issues/12122 > > > > I'll post if I get any further, but I suspect this is the end for now. > > > > Hmm, the issues you've posted is related to resizable BARs. I'm not > familiar > with it but afaik, bhyve isn't able to emulate resizable BARs yet. > > Btw. resizable BARs are somehow supported by QEMU, so it might be worth > giving > it a try: > > > https://gitlab.com/qemu-project/qemu/-/commit/b5048a4cbfa0362abc720b5198f= e9a35441bf5fe > > > Peter. > > -- > > Peter Wood > > peter@alastria.net > > > > -- > Kind regards, > Corvin > --=20 Mario. --000000000000759f67062b0af68e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>I want here to reiterate the importance of creating a= version of Bhyve that is able to accelerate qemu. <br></div><div>Please ca= n some developer "hear" me ? you know that this idea is good and = fundamental for bhyve and for FreeBSD,for every user who loves this system.= <br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gm= ail_attr">On Mon, Jan 6, 2025 at 3:41=E2=80=AFPM Corvin K=C3=B6hne <<a h= ref=3D"mailto:corvink@freebsd.org" target=3D"_blank">corvink@freebsd.org</a= >> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px= 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On = Mon, 2025-01-06 at 13:56 +0000, Peter Wood wrote:<br> > Thanks for the feedback Corvin, and thank you for the hard work you= 9;ve been<br> > putting into GPU passthru.<br> > <br> > I'm reaching the end of my limited knowledge here, and I have no e= xpectation<br> > of any further assistance - but as a status update, BIOS in CSM (with = legacy<br> > video op rom, so I see the console).<br> > <br> > > If you want to pass the option rom to the guest, you can use the = rom option<br> > > of<br> > > passthru devices:<br> > > <br> > > -s 1/2/3,passthru,1/2/3,rom=3D/path/to/rom<br> > > <br> > <br> > <br> > I extracted the option ROM using linux, I was able to use the<br> > /sys/devices/pci*/rom route to extract it, it seems valid at a glance = (768k<br> > dump) - but no idea how to really tell.<br> > <br> > Using the patched bhyve executable to bypass gvt-d: -s<br> > 4/0/0,passthru,4/0/0,rom=3D/mnt/vm/intel-arc-a380.bin -s 5/0/0,passthr= u,5/0/0<br> > (5/0/0 is a separate audio device exposing the audio channels of the H= DMI<br> > ports).<br> > <br> > Sadly initialization of the GPU in the linux (Ubuntu 24.04 / linux 6.8= .0) VM<br> > still fails:<br> > [ =C2=A0 =C2=A02.508656] i915 0000:00:04.0: enabling device (0000 ->= ; 0002)<br> > [ =C2=A0 =C2=A02.520226] i915 0000:00:04.0: [drm] Local memory IO size= :<br> > 0x000000017c800000<br> > [ =C2=A0 =C2=A02.520232] i915 0000:00:04.0: [drm] Local memory availab= le:<br> > 0x000000017c800000<br> > [ =C2=A0 =C2=A02.540148] i915 0000:00:04.0: vgaarb: VGA decodes change= d:<br> > olddecodes=3Dio+mem,decodes=3Dnone:owns=3Dnone<br> > [ =C2=A0 =C2=A02.550829] i915 0000:00:04.0: [drm] Finished loading DMC= firmware<br> > i915/dg2_dmc_ver2_08.bin (v2.8)<br> > [ =C2=A0 =C2=A02.564885] i915 0000:00:04.0: [drm] GT0: GUC: ADS captur= e alloc size<br> > changed from 32768 to 36864<br> > [ =C2=A0 =C2=A02.565855] i915 0000:00:04.0: [drm] GT0: GuC firmware i9= 15/dg2_guc_70.bin<br> > version 70.20.0<br> > [ =C2=A0 =C2=A02.565859] i915 0000:00:04.0: [drm] GT0: HuC firmware i9= 15/dg2_huc_gsc.bin<br> > version 7.10.3<br> > [ =C2=A0 =C2=A02.565979] i915 0000:00:04.0: [drm] GT0: GUC: ADS captur= e alloc size<br> > changed from 32768 to 36864<br> > [ =C2=A0 =C2=A02.567001] i915 0000:00:04.0: [drm] GT0: GUC: load faile= d: status =3D<br> > 0x40000056, time =3D 0ms, freq =3D 2300MHz, ret =3D 0<br> > [ =C2=A0 =C2=A02.567006] i915 0000:00:04.0: [drm] GT0: GUC: load faile= d: status: Reset =3D<br> > 0, BootROM =3D 0x2B, UKernel =3D 0x00, MIA =3D 0x00, Auth =3D 0x01<br> > [ =C2=A0 =C2=A02.567009] i915 0000:00:04.0: [drm] GT0: GUC: firmware p= roduction part<br> > check failure<br> > [ =C2=A0 =C2=A02.567077] i915 0000:00:04.0: [drm] *ERROR* GT0: GuC ini= tialization failed<br> > -ENOEXEC<br> > [ =C2=A0 =C2=A02.567610] i915 0000:00:04.0: [drm] *ERROR* GT0: Enablin= g uc failed (-5)<br> > [ =C2=A0 =C2=A02.567949] i915 0000:00:04.0: [drm] *ERROR* GT0: Failed = to initialize GPU,<br> > declaring it wedged!<br> > [ =C2=A0 =C2=A02.570106] i915 0000:00:04.0: [drm:add_taint_for_CI [i91= 5]] CI tainted:0x9<br> > by intel_gt_set_wedged_on_init+0x34/0x50 [i915]<br> > [ =C2=A0 =C2=A02.587048] [drm] Initialized i915 1.6.0 20230929 for 000= 0:00:04.0 on minor<br> > 1<br> > <br> <br> If possible, it might be a good idea to check if it's running on a Linu= x host<br> with QEMU properly. If yes, we may be able to check if QEMU has some specia= l<br> quirks for those devices (don't see one yet).<br> <br> > Interestingly intel_gpu_top will interact with the card to a degree, i= t shows<br> > a render utilization (of 0%), but none of the other card capabilities.= There<br> > are some very similar errors in Google which may suggest it's may = not be a<br> > bhyve/passthru issue, though it could be. I need to spin up a new VM w= ith more<br> > bleeding edge linux (or maybe even Win11) to see if it can talk to the= card.<br> > <br> > <a href=3D"https://github.com/intel-analytics/ipex-llm/issues/12122" r= el=3D"noreferrer" target=3D"_blank">https://github.com/intel-analytics/ipex= -llm/issues/12122</a><br> > <br> > I'll post if I get any further, but I suspect this is the end for = now.<br> > <br> <br> Hmm, the issues you've posted is related to resizable BARs. I'm not= familiar<br> with it but afaik, bhyve isn't able to emulate resizable BARs yet.<br> <br> Btw. resizable BARs are somehow supported by QEMU, so it might be worth giv= ing<br> it a try:<br> <br> <a href=3D"https://gitlab.com/qemu-project/qemu/-/commit/b5048a4cbfa0362abc= 720b5198fe9a35441bf5fe" rel=3D"noreferrer" target=3D"_blank">https://gitlab= .com/qemu-project/qemu/-/commit/b5048a4cbfa0362abc720b5198fe9a35441bf5fe</a= ><br> <br> > Peter.<br> > -- <br> > Peter Wood<br> > <a href=3D"mailto:peter@alastria.net" target=3D"_blank">peter@alastria= .net</a><br> > <br> <br> -- <br> Kind regards,<br> Corvin<br> </blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si= gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature">Ma= rio.<br></div> --000000000000759f67062b0af68e--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSihc=gTcttUZVY4-3jVF905CY09pU65iYqAqNJ73yA4xbg>