Date: Mon, 5 Dec 2022 15:13:56 +0000 From: Andrew Turner <andrew@fubar.geek.nz> To: Souradeep Chakrabarti <schakrabarti@microsoft.com> Cc: Warner Losh <imp@bsdimp.com>, Wei Hu <weh@microsoft.com>, "freebsd-hackers@FreeBSD.org" <freebsd-hackers@freebsd.org> Subject: Re: [EXTERNAL] pcib msix allocation in arm64 Message-ID: <877E8C1E-5CEA-44B6-ABC0-0C54D5B69A58@fubar.geek.nz> In-Reply-To: <PSAP153MB0536A022F70C90A6CF0CF249CC149@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> References: <PSAP153MB05363055DF2B4AE3F39C987BCC399@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> <146F5D18-6366-4953-A8D9-61FE7EC67F71@fubar.geek.nz> <PSAP153MB05367C2284B34BD1A3456F61CC399@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> <947DBD67-6443-480F-82DA-2BDDF44C3D03@fubar.geek.nz> <PSAP153MB0536A022F70C90A6CF0CF249CC149@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
> On 1 Dec 2022, at 11:54, Souradeep Chakrabarti =
<schakrabarti@microsoft.com> wrote:
>=20
>=20
>=20
>=20
>> -----Original Message-----
>> From: Andrew Turner <andrew@fubar.geek.nz>
>> Sent: Thursday, November 3, 2022 6:21 PM
>> To: Souradeep Chakrabarti <schakrabarti@microsoft.com>
>> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>; =
freebsd-
>> hackers@FreeBSD.org
>> Subject: Re: [EXTERNAL] pcib msix allocation in arm64
>>=20
>> Hi Souradeep,
>>=20
>> For the vmbus_pcib driver you=E2=80=99ll need to implement the =
pcib_alloc_msi,
>> pcib_release_msi, and the msix versions, and pcib_map_msi.
>>=20
>> For alloc and release you can just call into the same intr_* function =
as
>> pci_host_generic_acpi.c. You=E2=80=99ll need to pass in the xref for =
the interrupt controller.
>> It needs to be the xref the MSI controller registered. For the GICv2m =
it=E2=80=99s
>> ACPI_MSI_XREF.
> [Souradeep]=20
> I am currently specifying gic_mbi_start and gic_mbi_end with SPI start =
and end number, in gic_v3_acpi_attach() and=20
> I can see my intr_alloc_msi() and intr_map_msi() are working. But then =
mlx driver is
> getting stuck at CREATE_EQ .=20
>=20
> mlx5_core0: WARN: wait_func:967:(pid 0): CREATE_EQ(0x301) timeout. =
Will cause a leak of a command resource
> mlx5_core0: WARN: mlx5_start_eqs:588:(pid 0): failed to create async =
EQ -60
> mlx5_core0: ERR: mlx5_load_one:1157:(pid 0): Failed to start pages and =
async EQs
> mlx5_core0: WARN: wait_func:967:(pid 0): MANAGE_PAGES(0x108) timeout. =
Will cause a leak of a command resource
> mlx5_core0: ERR: reclaim_pages:444:(pid 0): failed reclaiming pages
> mlx5_core0: WARN: mlx5_reclaim_startup_pages:574:(pid 0): failed =
reclaiming pages (-60)
> mlx5_core0: ERR: init_one:1644:(pid 0): mlx5_load_one failed -60
>=20
>=20
> gic_v3_acpi_attach( )
> --
> --
> 323 sc->gic_mbi_start =3D 64;
> 324 sc->gic_mbi_end =3D 920;
> 325
> 326 err =3D gic_v3_attach(dev);
> 327 if (err !=3D 0)
> 328 goto error;
> 329
> 330 sc->gic_pic =3D intr_pic_register(dev, ACPI_INTR_XREF);
> 331 if (sc->gic_pic =3D=3D NULL) {
> 332 device_printf(dev, "could not register PIC\n");
> 333 err =3D ENXIO;
> 334 goto error;
> 335 }
> 336 err =3D intr_msi_register(dev, ACPI_MSI_XREF);
> 337 if (err) {
> 338 device_printf(dev, "could not register MSI\n");
> 339 }
> 340 if (intr_pic_claim_root(dev, ACPI_INTR_XREF, arm_gic_v3_intr, =
sc,
> 341 GIC_LAST_SGI - GIC_FIRST_SGI + 1) !=3D 0) {
> 342 err =3D ENXIO;
> 343 goto error;
> 344 }
> We don't have SPI range mentioned in ACPI DSDT, so I need to specify =
it manually in the code.
> It looks like MSI interrupt is not coming to the mlx.=20
> Is there any interrupt remapping blocked by FreeBSD kernel here?
Do you see the expected interrupt in arm_gic_v3_intr? If so it would be =
useful to trace it though intr_isrc_dispatch. If no it may be because =
the interrupt is not being triggered, or is masked.
Andrew=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?877E8C1E-5CEA-44B6-ABC0-0C54D5B69A58>