From nobody Mon Jan 6 14:40:59 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YRcLP0RV6z5kdcY for ; Mon, 06 Jan 2025 14:41:01 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YRcLN3Mxkz4nkv; Mon, 6 Jan 2025 14:41:00 +0000 (UTC) (envelope-from corvink@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1736174460; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=FxlVfClbRmJUaBdq9k8cuoHraHryEtzToIHEDv1sJF8=; b=aYSu6z9eq1FOZMiZTG8MWviRQwCMyMoc91tjN4FBfxz3a90FYJo4E4EX6AFxyC8gN2wPXr Gajfr+sqw4+Ei4AimPXtqNowh/OySGyhj91RQ2OCHRK4pKXrHcot5d0I4OiWwLqUjk52UG W/PjqgamDdd72kSXjR22lfKjuWxVpnEFRprUb9C1plt8Jt/9+PZMJj/5t0wP+Q46DN8AId bgt8Aucye5GAzLsg2HOZ0JpHBVt0hmsxdY54HEpfO9RiMfeTNKzPiZAkGZmPQQxvPp9b4L dYOZ/504tbbrkoWXHK8M2NQvXE9lCfMDaWQcX4MGQiOr4s1BYHRmBVRymSTLGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1736174460; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=FxlVfClbRmJUaBdq9k8cuoHraHryEtzToIHEDv1sJF8=; b=CcdebA2JNY/1XeNW3gVWBXoWcszFbAKAEOrpP786ulrEeMseLcbJrIxiwLQjlRdtutdXNa cDr/luObGKrIDGMvSPTq9fmldOBfwaipe7h47T6950w4NI6C2TG5MxUhkjOgaQe4mohZAx Zf3g6+PTJABAIvXvkaoxgRUlRL4TaJDvlYPIzMV7+F0uVtsOhX2gu/ae2onkm1kdZAwzlt V8GkVTVh93GvORZExsyLiPWgw3Ogbmkr7hAKZWEGWFp02mHri/aO82WkSKk9CAlIH5SU32 mL8DhQD6NbPazr5y2b7jY+DKs5hw7ft4ADbzRUijoS0YFmMZr3qssNvDcdYENg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1736174460; a=rsa-sha256; cv=none; b=ynj8M2tTMTjsLFVGliK1+uHVSxUVZRHRkQeFRvdQkGlOCoCFwmk7uxyQqywHg3gPXxfpa4 FELPvK48tTMOBVKO6eebVeVRrF82UHNeRyDuKAaEBiP9H4Z4EZPBx9+Rhx4RI6+b+wzGAh 6MIHcxNOFAWwKnj2O7XBFJIK85W7bk5EQcv/M2OymQD3l3Lydk7iKSG0aWQjbIveCcc6Tg yf7U+J5tItl5Vp/kmtUk9xRaaGzCQ9+PucOC7bS3fgV51iNmxiUa0y3zeuwwZtatlPnUpB dYyux5tPobP+mKWXlk0EBlm9QVeVCeDU5TmWG9TjzXjmCpWabHMo9elYExDZcA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from [IPv6:2001:9e8:dc01:5900:fc72:1b65:c1e1:a77c] (unknown [IPv6:2001:9e8:dc01:5900:fc72:1b65:c1e1:a77c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: corvink) by smtp.freebsd.org (Postfix) with ESMTPSA id 4YRcLN0qxnzRPM; Mon, 06 Jan 2025 14:41:00 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Message-ID: Subject: Re: bhyve/passthru for Intel dGPU (ARC A380)? From: Corvin =?ISO-8859-1?Q?K=F6hne?= To: Peter Wood Cc: freebsd-virtualization@freebsd.org Date: Mon, 06 Jan 2025 15:40:59 +0100 In-Reply-To: References: Autocrypt: addr=corvink@FreeBSD.org; prefer-encrypt=mutual; keydata=mQINBGNjZaIBEADDTrDNf+0pwiuRPBdClcnZW83dH1UhuOi0u+A1J2SatEBbNaFVtXXAa vewCTuyV/ZbNidjlhq3R/pWyiKjFKvs5dj7PMCw+3z2D5OWpMdHg7TrB+fbdFPOEsu0zQVKNaO+pS KCfN0Re0m7bL3wuvl7PXvBufRwA3Guo1P4j3TXWaEkuso7VupTvE25zVGg9ONHrGOjA9RUy+Yg4Se 3NLgtUdjBgA21SBQTDvRQV4fDmVenlwvWeE0Xm8FcDcpQb6sJTihaDku78mi3Ux1HCk7rTcepVEB0 xIB6qmFxv0sLlDmVv6Z6qg1y/Q5m23Pgz60o3TulMPV4F+3Itm8ifU+wgVSzBZbD29GYkd7LKqMkF bhvfSBk+5db3vbYY5OD//+LTM5AV7e2AhXuXMvG1UNBqXqSJTTSy6KZz+qmPQO0zos0dq46p8o82l KiBEGD2Hu0p+u0OyV+MmRYo1NIBFVbOPXp2MvUVl5II0UIJ3+N9gLBmfGA+HEpVO8PnvdoT/5NQ7m 8JK1rQHzjiDub/iDPAYMqKH4C0eZ/7zO0fuY5FeRNtuNtpH1Bw/+7/5RJH7bcKkfGHHEp15FJUrGH gWNydoDLB9QBprwQc8FEldDXBjzOMXIgh6FGKLNu6DswvIPGy6M3u7DXwDakCXz+c9Ym0oFihLzZx WntrsxdswD/CwARAQABtCdDb3J2aW4gS8O2aG5lIDxjb3J2aW4ua29laG5lQGdtYWlsLmNvbT6JAl QEEwEIAD4WIQSC9FKVreba38fZT0bYVNpWMV4CagUCZArUIwIbAwUJCWYBgAULCQgHAgYVCgkICwI EFgIDAQIeAQIXgAAKCRDYVNpWMV4CalcID/44k2i/mqSSi4W6FAobSF1nFLtP/pfcRNJriWKx2UF7 cfFMKyg7Nilg7FhLb5FDB1umUW2nFfchFPTUp4FfKzgRvPzIMg4RIRcVtTpYbl3z7zs9ZXD8qS//i ibbiUG3quncm6tO2x1jLZD3ORC+8MuLGXhYQIa4O5vVF2SBHdb/U6P+wsrF+U+OpRdEdQ/4Xu9S02 kltzBGgArjcexdhUqEqW01KCCSH0+qgfN1NE+9L934ZOB+cai2b9apPbCOGuV6KcUKMj4z0RWInhl XIyMqtmhdix/P/GjrQ1REVNdp74JeweSSedM15wwc6YLMXPrtOnnExyZ1gyNFYaub+Mdo3ZQ+8386 3B0C9IlpzEW0K8bYlZrl4WBNdcMOyByETAxgQmBgP6ZFErTtaeUOH1nX2FyR6o2GLSahRgngQmnRE zScTpPbBEkMwJMDAO+rbjjGxjeKSUwD1WOfbI6QZj+MS/uBk2p08kgN7fQaCEwj6jqML/IIE+FZ9I In6TNG0hChD384VHO+YioLBno1Atgi4Q7JUWSRIHQXZW+StQajFrWPPyKQwCe1MwqpKoMpX/q8IZB lzwJgZS8ShLeFZjtzOt1jgM99TD91Neonf9OzjTSbfo7sJviSWoICMhB/MvDZjj+naMVF86uGFxnI EsjVKyBxlJd4TRHnLYPTIHFKBLQjQ29ydmluIEvDtmhuZSA8Y29ydmlua0BGcmVlQlNELm9yZz6JA lQEEwEKAD4WIQSC9FKVreba38fZT0bYVNpWMV4CagUCY2NlogIbAwUJCWYBgAULCQgHAwUVCgkICw UWAwIBAAIeAQIXgAAKCRDYVNpWMV4CavfDEACCFnXpR7H9eOgP+GJMNPtK6i9/xnqdyXi8uCZIN0h YwjN4Xzo9SMLOf4UUlQEveOB+bGqbRfHd/fGKnrlXiPd0SGpKWJC21gqL/DsIH0J8I3Whth+O8tfP WeFy0oCsvBaaGFLIrDfoIgHF9i/gqEe48xhN42weB02Z3mdR1L0d7ME/BLwS0mCXe9Zh3uHw63S6x YB3Wsjptxe/ph6TpQDUKWtRJkjC6BqXPBdThpbbfIRWmjZbp2fKEJPvtRXS14+gbUqWeJ4xCvprA3 +ae7vtrp91X775yngyW3XTw5cmDiJIjykH8+zhEIoNQXNBpFrehkQDYrcM+WoE6NGSJo+3VJvSRWh UGWDVrxdTYNkbIjmTNlkI12NINC007DiuV7OF9XHWgrbbylvuZvbODmbJRdhTFy9upAUygX1/xUAQ EIMqMiJmyTdv8i0IbZ611WElQx9XHgGeZgM6+39/laN8FwspM6gE/4NzZHIZN0LEBOerZqoF+Il6e ccQpoEWx7nb/RilJp3dUUyvkBnJWg+AJByosg857kvvmDnZ3UB+bejWpcfFvnbkiKPUBPDO7tWPb5 r2yFDpDe/Vg5x+sRbkkXGUFD6Rx0p/ZiRIneVVg8emTzhDR0IL/BZkq/uVJkdaphZli1F/31cNgf6 ZJYnjxlk86uiYSySZQR0dKLDqq7QlQ29ydmluIEvDtmhuZSA8Yy5rb2VobmVAYmVja2hvZmYuY29t PokCVAQTAQgAPhYhBIL0UpWt5trfx9lPRthU2lYxXgJqBQJkCtScAhsDBQkJZgGABQsJCAcCBhUKC QgLAgQWAgMBAh4BAheAAAoJENhU2lYxXgJqI+QQAML5PTR7KpUFV3SLG60LQJGEOHUfDmJYczxBFb IAq1U4hIbivopu1AdLty7oDDrIjCVoa2/Cy34dd99O7lhLvUmZFB/zDSUtbUg2zhDkU0YSZ11Fdrl Wzky2tFaQRgxpDvWlUP0baa3Pd4dPDRiIUI6AOSR2SL6XANk6sJh56gLVM6G8yyafGsxSyDYg6Z78 EEMFejHwB+KP2DdsahupzM+F97HeC1+bOHYxtqN+2hEkPLtQWizyumPqNg5FvZhwe7yO8V95hF3Rh uDO+9aJT+WLLvcZEb/L1bI04IvZ5FWgCLI7Levd/DuOtZI8gWapHhqGZRbXB2fuJkCoKCl6V67h/7 aWhU3LjFTsC5siJyrxPjapKcIk8a7PqZDswNCKR+24LJ5D59mPgEOnsiCCVpik1WE/kgD+rOu9dQx jpjKwuKowf4EJP1KYNkYtoy1HthzyTOqPwqXC3IUl0GVPO3xw6MhUM3irCVFruC+ecVrv85Rd37vr duT6JvgW92xjWegMsamtkDZH8Ik/cmYhH0K/qEc6OZVNea/4PTEZe0uxODJ6pbMd80AJyGqDPPVeA gWJtEIG4k6IS8XyD5v1QJtlpDVpPwP/bbFnVc3h3Oatfn8Etm1KAqYvNwyO+om2PkF1p732uapDZd LwksVmgc9s79+9pSpeP5DbIeMzhrIK Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-ROMEs70aYfmzhmJO/I+c" User-Agent: Evolution 3.54.2 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 --=-ROMEs70aYfmzhmJO/I+c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2025-01-06 at 13:56 +0000, Peter Wood wrote: > Thanks for the feedback Corvin, and thank you for the hard work you've be= en > putting into GPU passthru. >=20 > I'm reaching the end of my limited knowledge here, and I have no expectat= ion > of any further assistance - but as a status update, BIOS in CSM (with leg= acy > video op rom, so I see the console). >=20 > > If you want to pass the option rom to the guest, you can use the rom op= tion > > of > > passthru devices: > >=20 > > -s 1/2/3,passthru,1/2/3,rom=3D/path/to/rom > >=20 >=20 >=20 > I extracted the option ROM using linux, I was able to use the > /sys/devices/pci*/rom route to extract it, it seems valid at a glance (76= 8k > dump) - but no idea how to really tell. >=20 > Using the patched bhyve executable to bypass gvt-d: -s > 4/0/0,passthru,4/0/0,rom=3D/mnt/vm/intel-arc-a380.bin -s 5/0/0,passthru,5= /0/0 > (5/0/0 is a separate audio device exposing the audio channels of the HDMI > ports). >=20 > Sadly initialization of the GPU in the linux (Ubuntu 24.04 / linux 6.8.0)= VM > still fails: > [ =C2=A0 =C2=A02.508656] i915 0000:00:04.0: enabling device (0000 -> 0002= ) > [ =C2=A0 =C2=A02.520226] i915 0000:00:04.0: [drm] Local memory IO size: > 0x000000017c800000 > [ =C2=A0 =C2=A02.520232] i915 0000:00:04.0: [drm] Local memory available: > 0x000000017c800000 > [ =C2=A0 =C2=A02.540148] i915 0000:00:04.0: vgaarb: VGA decodes changed: > olddecodes=3Dio+mem,decodes=3Dnone:owns=3Dnone > [ =C2=A0 =C2=A02.550829] i915 0000:00:04.0: [drm] Finished loading DMC fi= rmware > i915/dg2_dmc_ver2_08.bin (v2.8) > [ =C2=A0 =C2=A02.564885] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture a= lloc size > changed from 32768 to 36864 > [ =C2=A0 =C2=A02.565855] i915 0000:00:04.0: [drm] GT0: GuC firmware i915/= dg2_guc_70.bin > version 70.20.0 > [ =C2=A0 =C2=A02.565859] i915 0000:00:04.0: [drm] GT0: HuC firmware i915/= dg2_huc_gsc.bin > version 7.10.3 > [ =C2=A0 =C2=A02.565979] i915 0000:00:04.0: [drm] GT0: GUC: ADS capture a= lloc size > changed from 32768 to 36864 > [ =C2=A0 =C2=A02.567001] i915 0000:00:04.0: [drm] GT0: GUC: load failed: = status =3D > 0x40000056, time =3D 0ms, freq =3D 2300MHz, ret =3D 0 > [ =C2=A0 =C2=A02.567006] i915 0000:00:04.0: [drm] GT0: GUC: load failed: = status: Reset =3D > 0, BootROM =3D 0x2B, UKernel =3D 0x00, MIA =3D 0x00, Auth =3D 0x01 > [ =C2=A0 =C2=A02.567009] i915 0000:00:04.0: [drm] GT0: GUC: firmware prod= uction part > check failure > [ =C2=A0 =C2=A02.567077] i915 0000:00:04.0: [drm] *ERROR* GT0: GuC initia= lization failed > -ENOEXEC > [ =C2=A0 =C2=A02.567610] i915 0000:00:04.0: [drm] *ERROR* GT0: Enabling u= c failed (-5) > [ =C2=A0 =C2=A02.567949] i915 0000:00:04.0: [drm] *ERROR* GT0: Failed to = initialize GPU, > declaring it wedged! > [ =C2=A0 =C2=A02.570106] i915 0000:00:04.0: [drm:add_taint_for_CI [i915]]= CI tainted:0x9 > by intel_gt_set_wedged_on_init+0x34/0x50 [i915] > [ =C2=A0 =C2=A02.587048] [drm] Initialized i915 1.6.0 20230929 for 0000:0= 0:04.0 on minor > 1 >=20 If possible, it might be a good idea to check if it's running on a Linux ho= st with QEMU properly. If yes, we may be able to check if QEMU has some specia= l quirks for those devices (don't see one yet). > Interestingly intel_gpu_top will interact with the card to a degree, it s= hows > a render utilization (of 0%), but none of the other card capabilities. Th= ere > are some very similar errors in Google which may suggest it's may not be = a > bhyve/passthru issue, though it could be. I need to spin up a new VM with= more > bleeding edge linux (or maybe even Win11) to see if it can talk to the ca= rd. >=20 > https://github.com/intel-analytics/ipex-llm/issues/12122 >=20 > I'll post if I get any further, but I suspect this is the end for now. >=20 Hmm, the issues you've posted is related to resizable BARs. I'm not familia= r with it but afaik, bhyve isn't able to emulate resizable BARs yet. Btw. resizable BARs are somehow supported by QEMU, so it might be worth giv= ing it a try: https://gitlab.com/qemu-project/qemu/-/commit/b5048a4cbfa0362abc720b5198fe9= a35441bf5fe > Peter. > --=20 > Peter Wood > peter@alastria.net >=20 --=20 Kind regards, Corvin --=-ROMEs70aYfmzhmJO/I+c Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmd763sACgkQ2FTaVjFe Amri8w/8CBq7bGS2fMHQw3P7AEzJLV7nSslm4m6mIdMwwSWHHC7fF5kwPIBw4nUH pQbxhRyns0hlmvJul93La8mpnTCQ1EPozxS/PtK/r36d/j1zPa44JoLWAYrYupkR 8OYEQiBJvU9eE/yT/59v5ebq/s4cV547HhsbZ15ubBA3ZT9Is2kadE376WC7GgD9 xxY19pQM9B3EgW0GUC7AXLj57KCFaPnl15uJ4XssofDetyrOrbTl8TbIu/LmDSsh 7k+pmX8KII9oOsDIdLjL/RFNvNVeapz3TiDjFgSUKLnDfmE2eU5FksgORVu5JH/M fWhHemantla3bJouZ7MqBijRQcjCKeC+FzcMPBBVWH0Q91/PpjMLlyvqHxmu8xHB XT6b7pv1EQ2JfbvkVvhlb42AmvN4pjEq09zsIME445OI2iQV8t2AAMKNMbSY4jbD pr74494PR5u4I+BvNmdJRCxS/rml9oAg50y1sKftBeUf5/rUlkmLUIz5QhjxIU1d kLIPVwn++GILRlw4QXxkj+nc66sgv9f9nFoo3eQ7Wgp/I/9RMsYqspHIdQhWGYgt 9Rj92ZSxLZJk29MJpQaOXLKSwSs9jn0/asVtG0NDq5HyMK87gGjivZTO91Cj4lXw h4X/NCV3uh4GbG68uB79Ojs0fEHW66NcvhbSoV96iAbwVmjZsi4= =eZg9 -----END PGP SIGNATURE----- --=-ROMEs70aYfmzhmJO/I+c--