Date: Mon, 6 Jan 2025 16:05:08 +0100 From: Mario Marietto <marietto2008@gmail.com> To: =?UTF-8?Q?Corvin_K=C3=B6hne?= <corvink@freebsd.org> Cc: Peter Wood <peter@alastria.net>, freebsd-virtualization@freebsd.org Subject: Re: bhyve/passthru for Intel dGPU (ARC A380)? Message-ID: <CA%2B1FSihc=gTcttUZVY4-3jVF905CY09pU65iYqAqNJ73yA4xbg@mail.gmail.com> In-Reply-To: <d2f9347053741ceab43d925ad22448f325559974.camel@FreeBSD.org> References: <CAD-E2if_q6JreqPWiFBgPc=KHeP12Pq_E2R4m3ZxvGC3g87ZHA@mail.gmail.com> <CA%2B1FSihc5EiBUSFjxoUViAYzZ3qbo%2BqpssrBkuEvQK1=O9W6uw@mail.gmail.com> <CAD-E2icA%2BHkWZJL2JVjU28ECpc8BNG1KXkpQj4Zw5nB8yPfahQ@mail.gmail.com> <d2e6638855f263e8f613ba06c37e091c710c27be.camel@FreeBSD.org> <CAD-E2ieq0P03tTkuAdvg-D35ua9PQT6XuT9GHOPLc7Aqa6xGvw@mail.gmail.com> <d2f9347053741ceab43d925ad22448f325559974.camel@FreeBSD.org>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] I want here to reiterate the importance of creating a version of Bhyve that is able to accelerate qemu. Please can some developer "hear" me ? you know that this idea is good and fundamental for bhyve and for FreeBSD,for every user who loves this system. On Mon, Jan 6, 2025 at 3:41 PM Corvin Köhne <corvink@freebsd.org> wrote: > On Mon, 2025-01-06 at 13:56 +0000, Peter Wood wrote: > > Thanks for the feedback Corvin, and thank you for the hard work you've > been > > putting into GPU passthru. > > > > I'm reaching the end of my limited knowledge here, and I have no > expectation > > of any further assistance - but as a status update, BIOS in CSM (with > legacy > > video op rom, so I see the console). > > > > > If you want to pass the option rom to the guest, you can use the rom > option > > > of > > > passthru devices: > > > > > > -s 1/2/3,passthru,1/2/3,rom=/path/to/rom > > > > > > > > > I extracted the option ROM using linux, I was able to use the > > /sys/devices/pci*/rom route to extract it, it seems valid at a glance > (768k > > dump) - but no idea how to really tell. > > > > Using the patched bhyve executable to bypass gvt-d: -s > > 4/0/0,passthru,4/0/0,rom=/mnt/vm/intel-arc-a380.bin -s > 5/0/0,passthru,5/0/0 > > (5/0/0 is a separate audio device exposing the audio channels of the HDMI > > ports). > > > > Sadly initialization of the GPU in the linux (Ubuntu 24.04 / linux > 6.8.0) VM > > still fails: > > [ 2.508656] i915 0000:00:04.0: enabling device (0000 -> 0002) > > [ 2.520226] i915 0000:00:04.0: [drm] Local memory IO size: > > 0x000000017c800000 > > [ 2.520232] i915 0000:00:04.0: [drm] Local memory available: > > 0x000000017c800000 > > [ 2.540148] i915 0000:00:04.0: vgaarb: VGA decodes changed: > > olddecodes=io+mem,decodes=none:owns=none > > [ 2.550829] i915 0000:00:04.0: [drm] Finished loading DMC firmware > > i915/dg2_dmc_ver2_08.bin (v2.8) > > [ 2.564885] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture alloc size > > changed from 32768 to 36864 > > [ 2.565855] i915 0000:00:04.0: [drm] GT0: GuC firmware > i915/dg2_guc_70.bin > > version 70.20.0 > > [ 2.565859] i915 0000:00:04.0: [drm] GT0: HuC firmware > i915/dg2_huc_gsc.bin > > version 7.10.3 > > [ 2.565979] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture alloc size > > changed from 32768 to 36864 > > [ 2.567001] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status = > > 0x40000056, time = 0ms, freq = 2300MHz, ret = 0 > > [ 2.567006] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status: > Reset = > > 0, BootROM = 0x2B, UKernel = 0x00, MIA = 0x00, Auth = 0x01 > > [ 2.567009] i915 0000:00:04.0: [drm] GT0: GUC: firmware production > part > > check failure > > [ 2.567077] i915 0000:00:04.0: [drm] *ERROR* GT0: GuC initialization > failed > > -ENOEXEC > > [ 2.567610] i915 0000:00:04.0: [drm] *ERROR* GT0: Enabling uc failed > (-5) > > [ 2.567949] i915 0000:00:04.0: [drm] *ERROR* GT0: Failed to > initialize GPU, > > declaring it wedged! > > [ 2.570106] i915 0000:00:04.0: [drm:add_taint_for_CI [i915]] CI > tainted:0x9 > > by intel_gt_set_wedged_on_init+0x34/0x50 [i915] > > [ 2.587048] [drm] Initialized i915 1.6.0 20230929 for 0000:00:04.0 on > minor > > 1 > > > > If possible, it might be a good idea to check if it's running on a Linux > host > with QEMU properly. If yes, we may be able to check if QEMU has some > special > quirks for those devices (don't see one yet). > > > Interestingly intel_gpu_top will interact with the card to a degree, it > shows > > a render utilization (of 0%), but none of the other card capabilities. > There > > are some very similar errors in Google which may suggest it's may not be > a > > bhyve/passthru issue, though it could be. I need to spin up a new VM > with more > > bleeding edge linux (or maybe even Win11) to see if it can talk to the > card. > > > > https://github.com/intel-analytics/ipex-llm/issues/12122 > > > > I'll post if I get any further, but I suspect this is the end for now. > > > > Hmm, the issues you've posted is related to resizable BARs. I'm not > familiar > with it but afaik, bhyve isn't able to emulate resizable BARs yet. > > Btw. resizable BARs are somehow supported by QEMU, so it might be worth > giving > it a try: > > > https://gitlab.com/qemu-project/qemu/-/commit/b5048a4cbfa0362abc720b5198fe9a35441bf5fe > > > Peter. > > -- > > Peter Wood > > peter@alastria.net > > > > -- > Kind regards, > Corvin > -- Mario. [-- Attachment #2 --] <div dir="ltr"><div>I want here to reiterate the importance of creating a version of Bhyve that is able to accelerate qemu. <br></div><div>Please can some developer "hear" me ? you know that this idea is good and fundamental for bhyve and for FreeBSD,for every user who loves this system.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 6, 2025 at 3:41 PM Corvin Köhne <<a href="mailto:corvink@freebsd.org" target="_blank">corvink@freebsd.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, 2025-01-06 at 13:56 +0000, Peter Wood wrote:<br> > Thanks for the feedback Corvin, and thank you for the hard work you've been<br> > putting into GPU passthru.<br> > <br> > I'm reaching the end of my limited knowledge here, and I have no expectation<br> > of any further assistance - but as a status update, BIOS in CSM (with legacy<br> > video op rom, so I see the console).<br> > <br> > > If you want to pass the option rom to the guest, you can use the rom option<br> > > of<br> > > passthru devices:<br> > > <br> > > -s 1/2/3,passthru,1/2/3,rom=/path/to/rom<br> > > <br> > <br> > <br> > I extracted the option ROM using linux, I was able to use the<br> > /sys/devices/pci*/rom route to extract it, it seems valid at a glance (768k<br> > dump) - but no idea how to really tell.<br> > <br> > Using the patched bhyve executable to bypass gvt-d: -s<br> > 4/0/0,passthru,4/0/0,rom=/mnt/vm/intel-arc-a380.bin -s 5/0/0,passthru,5/0/0<br> > (5/0/0 is a separate audio device exposing the audio channels of the HDMI<br> > ports).<br> > <br> > Sadly initialization of the GPU in the linux (Ubuntu 24.04 / linux 6.8.0) VM<br> > still fails:<br> > [ 2.508656] i915 0000:00:04.0: enabling device (0000 -> 0002)<br> > [ 2.520226] i915 0000:00:04.0: [drm] Local memory IO size:<br> > 0x000000017c800000<br> > [ 2.520232] i915 0000:00:04.0: [drm] Local memory available:<br> > 0x000000017c800000<br> > [ 2.540148] i915 0000:00:04.0: vgaarb: VGA decodes changed:<br> > olddecodes=io+mem,decodes=none:owns=none<br> > [ 2.550829] i915 0000:00:04.0: [drm] Finished loading DMC firmware<br> > i915/dg2_dmc_ver2_08.bin (v2.8)<br> > [ 2.564885] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture alloc size<br> > changed from 32768 to 36864<br> > [ 2.565855] i915 0000:00:04.0: [drm] GT0: GuC firmware i915/dg2_guc_70.bin<br> > version 70.20.0<br> > [ 2.565859] i915 0000:00:04.0: [drm] GT0: HuC firmware i915/dg2_huc_gsc.bin<br> > version 7.10.3<br> > [ 2.565979] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture alloc size<br> > changed from 32768 to 36864<br> > [ 2.567001] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status =<br> > 0x40000056, time = 0ms, freq = 2300MHz, ret = 0<br> > [ 2.567006] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status: Reset =<br> > 0, BootROM = 0x2B, UKernel = 0x00, MIA = 0x00, Auth = 0x01<br> > [ 2.567009] i915 0000:00:04.0: [drm] GT0: GUC: firmware production part<br> > check failure<br> > [ 2.567077] i915 0000:00:04.0: [drm] *ERROR* GT0: GuC initialization failed<br> > -ENOEXEC<br> > [ 2.567610] i915 0000:00:04.0: [drm] *ERROR* GT0: Enabling uc failed (-5)<br> > [ 2.567949] i915 0000:00:04.0: [drm] *ERROR* GT0: Failed to initialize GPU,<br> > declaring it wedged!<br> > [ 2.570106] i915 0000:00:04.0: [drm:add_taint_for_CI [i915]] CI tainted:0x9<br> > by intel_gt_set_wedged_on_init+0x34/0x50 [i915]<br> > [ 2.587048] [drm] Initialized i915 1.6.0 20230929 for 0000:00:04.0 on minor<br> > 1<br> > <br> <br> If possible, it might be a good idea to check if it's running on a Linux host<br> with QEMU properly. If yes, we may be able to check if QEMU has some special<br> quirks for those devices (don't see one yet).<br> <br> > Interestingly intel_gpu_top will interact with the card to a degree, it shows<br> > a render utilization (of 0%), but none of the other card capabilities. There<br> > are some very similar errors in Google which may suggest it's may not be a<br> > bhyve/passthru issue, though it could be. I need to spin up a new VM with more<br> > bleeding edge linux (or maybe even Win11) to see if it can talk to the card.<br> > <br> > <a href="https://github.com/intel-analytics/ipex-llm/issues/12122" rel="noreferrer" target="_blank">https://github.com/intel-analytics/ipex-llm/issues/12122</a><br>; > <br> > I'll post if I get any further, but I suspect this is the end for now.<br> > <br> <br> Hmm, the issues you've posted is related to resizable BARs. I'm not familiar<br> with it but afaik, bhyve isn't able to emulate resizable BARs yet.<br> <br> Btw. resizable BARs are somehow supported by QEMU, so it might be worth giving<br> it a try:<br> <br> <a href="https://gitlab.com/qemu-project/qemu/-/commit/b5048a4cbfa0362abc720b5198fe9a35441bf5fe" rel="noreferrer" target="_blank">https://gitlab.com/qemu-project/qemu/-/commit/b5048a4cbfa0362abc720b5198fe9a35441bf5fe</a><br>; <br> > Peter.<br> > -- <br> > Peter Wood<br> > <a href="mailto:peter@alastria.net" target="_blank">peter@alastria.net</a><br> > <br> <br> -- <br> Kind regards,<br> Corvin<br> </blockquote></div><div><br clear="all"></div><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature">Mario.<br></div>help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSihc=gTcttUZVY4-3jVF905CY09pU65iYqAqNJ73yA4xbg>