From nobody Mon Jan 6 15:05:08 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YRctz2Gs0z5kgcm for ; Mon, 06 Jan 2025 15:05:47 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YRctz00mMz4qRb; Mon, 6 Jan 2025 15:05:47 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-2ef89dbd8eeso15474838a91.0; Mon, 06 Jan 2025 07:05:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736175945; x=1736780745; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=agIpc47/nvAXs/rNmE58ZWV5Y3c6Ydu2+JuC/N5oF0k=; b=dzvJs+BrmtYpKdRDCGfNE4Ei3u89OUKr5Hq0JMciHq2iuZJPjusxnl7ypRNj0xz2JL h44pudySyIyvfunT4yY1WBaajf2LCMl/UAWOpOgUsY6LAT74s62sPfy7kYitbbIQ4ibm 8a/X1y2NSTLd/VXv4amFqjEzucIeR+vWxNxtnE/KmWKS8MDlzQ4jh4tJFcGz4boPAl7I HlqlJkhE3zk+pTZtg6jsozYzgJtqYCy4rGNJY7/9uhigUrY//Mu5iHSopGStkDz6BydU SpD33nMv4kb8vNqHtbT0qemMpUp2IEcLWRneelFV7naQaCZJuFXe8/Xbe4EFeMHpMYQG 0AQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736175945; x=1736780745; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=agIpc47/nvAXs/rNmE58ZWV5Y3c6Ydu2+JuC/N5oF0k=; b=R1E+PcdRfAP8Pce7cjpI15LQ5X2V8kavP1d4be+CLhoh5MRtiPPu0rk5cFpn7c3t6L NuSBuHEjjzrjPT2O54/zkWgccVrZKbblvpILKFo5MKU1jBpZg2izIRDYdI4eao/uR+B1 bLT+DJp836+thMCLFV9JdRvGNuWF/dzNh9dbpEwKtrNOQhsRpLqr5I/+BBzMuou5MDt8 /dPyHm3WOUc2VRF9UWiMtJeQ3xLQkLZywVODnzwmylIxAC7F58aCFvypBRS9L0rJljhR edSF6yknkIZoCmblk4tvM1uN+FdWs/F9IzTOs7V97zTz91qhy70CG/6uasrf/Yf2B7Mo S3FQ== X-Forwarded-Encrypted: i=1; AJvYcCVcuIVSjOKdMhh8yOlIjRiB+35zF2cNjqqDXRR8PQ5E8rbXE77lvT0A6GpjZTuU43sowuJVv+f+EsSXQptK0DTcMJaCaId5@freebsd.org X-Gm-Message-State: AOJu0YzCqdIr3el8b7bz45NJSkYnLdofw6xlAi4LYL1/MJvpslDuKrtd CD4Yuo+3fSXvElnjJv23MkRSpZsr7XD06aZIiRENIZqNuteW9s21Ag4fwuHa34lT8bknpL0reE6 iW5d6AWJ30J81WbEqk4/TqJUrITBuJzso X-Gm-Gg: ASbGncuk4dqQt5CqYF/JNHKk+OSjf/MOhxPLSHKDUxWJGWKZ2PCK71Q15q7Z/hBs4Ph go6BkcaIvFWp/BtZ//hZtEIY1gqF6L1Cph77HkNk= X-Google-Smtp-Source: AGHT+IENrIRdSA0dO+KMjnso1K7tbVWyPppkWiMeqGJHon89GYsLFEW5S+99LR+C/+kr1E+bWS380+dVd0mOjSBa274= X-Received: by 2002:a17:90b:2807:b0:2ee:599e:f411 with SMTP id 98e67ed59e1d1-2f452eeb5a3mr76662150a91.34.1736175944832; Mon, 06 Jan 2025 07:05:44 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 6 Jan 2025 16:05:08 +0100 Message-ID: Subject: Re: bhyve/passthru for Intel dGPU (ARC A380)? To: =?UTF-8?Q?Corvin_K=C3=B6hne?= Cc: Peter Wood , freebsd-virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000759f67062b0af68e" X-Rspamd-Queue-Id: 4YRctz00mMz4qRb X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] --000000000000759f67062b0af68e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I want here to reiterate the importance of creating a version of Bhyve that is able to accelerate qemu. Please can some developer "hear" me ? you know that this idea is good and fundamental for bhyve and for FreeBSD,for every user who loves this system. On Mon, Jan 6, 2025 at 3:41=E2=80=AFPM Corvin K=C3=B6hne wrote: > On Mon, 2025-01-06 at 13:56 +0000, Peter Wood wrote: > > Thanks for the feedback Corvin, and thank you for the hard work you've > been > > putting into GPU passthru. > > > > I'm reaching the end of my limited knowledge here, and I have no > expectation > > of any further assistance - but as a status update, BIOS in CSM (with > legacy > > video op rom, so I see the console). > > > > > If you want to pass the option rom to the guest, you can use the rom > option > > > of > > > passthru devices: > > > > > > -s 1/2/3,passthru,1/2/3,rom=3D/path/to/rom > > > > > > > > > I extracted the option ROM using linux, I was able to use the > > /sys/devices/pci*/rom route to extract it, it seems valid at a glance > (768k > > dump) - but no idea how to really tell. > > > > Using the patched bhyve executable to bypass gvt-d: -s > > 4/0/0,passthru,4/0/0,rom=3D/mnt/vm/intel-arc-a380.bin -s > 5/0/0,passthru,5/0/0 > > (5/0/0 is a separate audio device exposing the audio channels of the HD= MI > > ports). > > > > Sadly initialization of the GPU in the linux (Ubuntu 24.04 / linux > 6.8.0) VM > > still fails: > > [ 2.508656] i915 0000:00:04.0: enabling device (0000 -> 0002) > > [ 2.520226] i915 0000:00:04.0: [drm] Local memory IO size: > > 0x000000017c800000 > > [ 2.520232] i915 0000:00:04.0: [drm] Local memory available: > > 0x000000017c800000 > > [ 2.540148] i915 0000:00:04.0: vgaarb: VGA decodes changed: > > olddecodes=3Dio+mem,decodes=3Dnone:owns=3Dnone > > [ 2.550829] i915 0000:00:04.0: [drm] Finished loading DMC firmware > > i915/dg2_dmc_ver2_08.bin (v2.8) > > [ 2.564885] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture alloc siz= e > > changed from 32768 to 36864 > > [ 2.565855] i915 0000:00:04.0: [drm] GT0: GuC firmware > i915/dg2_guc_70.bin > > version 70.20.0 > > [ 2.565859] i915 0000:00:04.0: [drm] GT0: HuC firmware > i915/dg2_huc_gsc.bin > > version 7.10.3 > > [ 2.565979] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture alloc siz= e > > changed from 32768 to 36864 > > [ 2.567001] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status = =3D > > 0x40000056, time =3D 0ms, freq =3D 2300MHz, ret =3D 0 > > [ 2.567006] i915 0000:00:04.0: [drm] GT0: GUC: load failed: status: > Reset =3D > > 0, BootROM =3D 0x2B, UKernel =3D 0x00, MIA =3D 0x00, Auth =3D 0x01 > > [ 2.567009] i915 0000:00:04.0: [drm] GT0: GUC: firmware production > part > > check failure > > [ 2.567077] i915 0000:00:04.0: [drm] *ERROR* GT0: GuC initialization > failed > > -ENOEXEC > > [ 2.567610] i915 0000:00:04.0: [drm] *ERROR* GT0: Enabling uc failed > (-5) > > [ 2.567949] i915 0000:00:04.0: [drm] *ERROR* GT0: Failed to > initialize GPU, > > declaring it wedged! > > [ 2.570106] i915 0000:00:04.0: [drm:add_taint_for_CI [i915]] CI > tainted:0x9 > > by intel_gt_set_wedged_on_init+0x34/0x50 [i915] > > [ 2.587048] [drm] Initialized i915 1.6.0 20230929 for 0000:00:04.0 o= n > minor > > 1 > > > > If possible, it might be a good idea to check if it's running on a Linux > host > with QEMU properly. If yes, we may be able to check if QEMU has some > special > quirks for those devices (don't see one yet). > > > Interestingly intel_gpu_top will interact with the card to a degree, it > shows > > a render utilization (of 0%), but none of the other card capabilities. > There > > are some very similar errors in Google which may suggest it's may not b= e > a > > bhyve/passthru issue, though it could be. I need to spin up a new VM > with more > > bleeding edge linux (or maybe even Win11) to see if it can talk to the > card. > > > > https://github.com/intel-analytics/ipex-llm/issues/12122 > > > > I'll post if I get any further, but I suspect this is the end for now. > > > > Hmm, the issues you've posted is related to resizable BARs. I'm not > familiar > with it but afaik, bhyve isn't able to emulate resizable BARs yet. > > Btw. resizable BARs are somehow supported by QEMU, so it might be worth > giving > it a try: > > > https://gitlab.com/qemu-project/qemu/-/commit/b5048a4cbfa0362abc720b5198f= e9a35441bf5fe > > > Peter. > > -- > > Peter Wood > > peter@alastria.net > > > > -- > Kind regards, > Corvin > --=20 Mario. --000000000000759f67062b0af68e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I want here to reiterate the importance of creating a= version of Bhyve that is able to accelerate qemu.
Please ca= n some developer "hear" me ? you know that this idea is good and = fundamental for bhyve and for FreeBSD,for every user who loves this system.=

On Mon, Jan 6, 2025 at 3:41=E2=80=AFPM Corvin K=C3=B6hne <corvink@freebsd.org> wrote:
On = Mon, 2025-01-06 at 13:56 +0000, Peter Wood wrote:
> Thanks for the feedback Corvin, and thank you for the hard work you= 9;ve been
> putting into GPU passthru.
>
> I'm reaching the end of my limited knowledge here, and I have no e= xpectation
> of any further assistance - but as a status update, BIOS in CSM (with = legacy
> video op rom, so I see the console).
>
> > If you want to pass the option rom to the guest, you can use the = rom option
> > of
> > passthru devices:
> >
> > -s 1/2/3,passthru,1/2/3,rom=3D/path/to/rom
> >
>
>
> I extracted the option ROM using linux, I was able to use the
> /sys/devices/pci*/rom route to extract it, it seems valid at a glance = (768k
> dump) - but no idea how to really tell.
>
> Using the patched bhyve executable to bypass gvt-d: -s
> 4/0/0,passthru,4/0/0,rom=3D/mnt/vm/intel-arc-a380.bin -s 5/0/0,passthr= u,5/0/0
> (5/0/0 is a separate audio device exposing the audio channels of the H= DMI
> ports).
>
> Sadly initialization of the GPU in the linux (Ubuntu 24.04 / linux 6.8= .0) VM
> still fails:
> [ =C2=A0 =C2=A02.508656] i915 0000:00:04.0: enabling device (0000 ->= ; 0002)
> [ =C2=A0 =C2=A02.520226] i915 0000:00:04.0: [drm] Local memory IO size= :
> 0x000000017c800000
> [ =C2=A0 =C2=A02.520232] i915 0000:00:04.0: [drm] Local memory availab= le:
> 0x000000017c800000
> [ =C2=A0 =C2=A02.540148] i915 0000:00:04.0: vgaarb: VGA decodes change= d:
> olddecodes=3Dio+mem,decodes=3Dnone:owns=3Dnone
> [ =C2=A0 =C2=A02.550829] i915 0000:00:04.0: [drm] Finished loading DMC= firmware
> i915/dg2_dmc_ver2_08.bin (v2.8)
> [ =C2=A0 =C2=A02.564885] i915 0000:00:04.0: [drm] GT0: GUC: ADS captur= e alloc size
> changed from 32768 to 36864
> [ =C2=A0 =C2=A02.565855] i915 0000:00:04.0: [drm] GT0: GuC firmware i9= 15/dg2_guc_70.bin
> version 70.20.0
> [ =C2=A0 =C2=A02.565859] i915 0000:00:04.0: [drm] GT0: HuC firmware i9= 15/dg2_huc_gsc.bin
> version 7.10.3
> [ =C2=A0 =C2=A02.565979] i915 0000:00:04.0: [drm] GT0: GUC: ADS captur= e alloc size
> changed from 32768 to 36864
> [ =C2=A0 =C2=A02.567001] i915 0000:00:04.0: [drm] GT0: GUC: load faile= d: status =3D
> 0x40000056, time =3D 0ms, freq =3D 2300MHz, ret =3D 0
> [ =C2=A0 =C2=A02.567006] i915 0000:00:04.0: [drm] GT0: GUC: load faile= d: status: Reset =3D
> 0, BootROM =3D 0x2B, UKernel =3D 0x00, MIA =3D 0x00, Auth =3D 0x01
> [ =C2=A0 =C2=A02.567009] i915 0000:00:04.0: [drm] GT0: GUC: firmware p= roduction part
> check failure
> [ =C2=A0 =C2=A02.567077] i915 0000:00:04.0: [drm] *ERROR* GT0: GuC ini= tialization failed
> -ENOEXEC
> [ =C2=A0 =C2=A02.567610] i915 0000:00:04.0: [drm] *ERROR* GT0: Enablin= g uc failed (-5)
> [ =C2=A0 =C2=A02.567949] i915 0000:00:04.0: [drm] *ERROR* GT0: Failed = to initialize GPU,
> declaring it wedged!
> [ =C2=A0 =C2=A02.570106] i915 0000:00:04.0: [drm:add_taint_for_CI [i91= 5]] CI tainted:0x9
> by intel_gt_set_wedged_on_init+0x34/0x50 [i915]
> [ =C2=A0 =C2=A02.587048] [drm] Initialized i915 1.6.0 20230929 for 000= 0:00:04.0 on minor
> 1
>

If possible, it might be a good idea to check if it's running on a Linu= x host
with QEMU properly. If yes, we may be able to check if QEMU has some specia= l
quirks for those devices (don't see one yet).

> Interestingly intel_gpu_top will interact with the card to a degree, i= t shows
> a render utilization (of 0%), but none of the other card capabilities.= There
> are some very similar errors in Google which may suggest it's may = not be a
> bhyve/passthru issue, though it could be. I need to spin up a new VM w= ith more
> bleeding edge linux (or maybe even Win11) to see if it can talk to the= card.
>
> https://github.com/intel-analytics/ipex= -llm/issues/12122
>
> I'll post if I get any further, but I suspect this is the end for = now.
>

Hmm, the issues you've posted is related to resizable BARs. I'm not= familiar
with it but afaik, bhyve isn't able to emulate resizable BARs yet.

Btw. resizable BARs are somehow supported by QEMU, so it might be worth giv= ing
it a try:

https://gitlab= .com/qemu-project/qemu/-/commit/b5048a4cbfa0362abc720b5198fe9a35441bf5fe

> Peter.
> --
> Peter Wood
> peter@alastria= .net
>

--
Kind regards,
Corvin


--
Ma= rio.
--000000000000759f67062b0af68e--