Date: Fri, 11 Nov 2022 20:10:41 +0000 From: Andrew Turner <andrew@fubar.geek.nz> To: Souradeep Chakrabarti <schakrabarti@microsoft.com> Cc: Warner Losh <imp@bsdimp.com>, Wei Hu <weh@microsoft.com>, "freebsd-hackers@FreeBSD.org" <freebsd-hackers@freebsd.org> Subject: Re: [EXTERNAL] pcib msix allocation in arm64 Message-ID: <DD81C474-8EEA-4B24-A8EC-4B9AE5E9EFED@fubar.geek.nz> In-Reply-To: <PSAP153MB0536273BB0C28ABF24C447B0CC009@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> References: <PSAP153MB05363055DF2B4AE3F39C987BCC399@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> <146F5D18-6366-4953-A8D9-61FE7EC67F71@fubar.geek.nz> <PSAP153MB05367C2284B34BD1A3456F61CC399@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> <947DBD67-6443-480F-82DA-2BDDF44C3D03@fubar.geek.nz> <PSAP153MB05368301E8B0A7AE9133073BCC019@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> <PSAP153MB0536273BB0C28ABF24C447B0CC009@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_C72BF360-FA1E-44BC-ABBC-1D9F0B37BFF7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 11 Nov 2022, at 19:07, Souradeep Chakrabarti = <schakrabarti@microsoft.com> wrote: >=20 >=20 >=20 >=20 >> -----Original Message----- >> From: Souradeep Chakrabarti >> Sent: Thursday, November 10, 2022 3:20 PM >> To: Andrew Turner <andrew@fubar.geek.nz = <mailto:andrew@fubar.geek.nz>> >> Cc: Warner Losh <imp@bsdimp.com <mailto:imp@bsdimp.com>>; Wei Hu = <weh@microsoft.com <mailto:weh@microsoft.com>>; freebsd- >> hackers@FreeBSD.org <mailto:hackers@FreeBSD.org> >> Subject: RE: [EXTERNAL] pcib msix allocation in arm64 >>=20 >>=20 >>=20 >>=20 >>> -----Original Message----- >>> From: Andrew Turner <andrew@fubar.geek.nz> >>> Sent: Thursday, November 3, 2022 6:21 PM >>> To: Souradeep Chakrabarti <schakrabarti@microsoft.com> >>> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>; >> freebsd- >>> hackers@FreeBSD.org >>> Subject: Re: [EXTERNAL] pcib msix allocation in arm64 >>>=20 >>> Hi Souradeep, >>>=20 >>> For the vmbus_pcib driver you=E2=80=99ll need to implement the = pcib_alloc_msi, >>> pcib_release_msi, and the msix versions, and pcib_map_msi. >>>=20 >>> For alloc and release you can just call into the same intr_* = function >>> as pci_host_generic_acpi.c. You=E2=80=99ll need to pass in the xref = for the >>> interrupt controller. It needs to be the xref the MSI controller >>> registered. For the GICv2m it=E2=80=99s ACPI_MSI_XREF. >>>=20 >> [Souradeep] >> I have used following in vmbus_pcib.c : >> ret =3D intr_alloc_msix(pcib, dev, ACPI_MSI_XREF, irq); But it is = failing in >> pic_lookup() with ESRCH by not finding pic with ACPI_MSI_XREF. >> Do I need to do anything for pic_lookup() to be successful? >>=20 > [Souradeep]=20 > I found out the reason here pic_lookup() for ACPI_MSI_XREF is failing, = as > we are not calling intr_msi_register() but intr_pic_register() is = getting called for the PCIB. > This is because we don't have ITS implemented in Hyper-V. Instead, = Hyper-V is presenting the > MSI/MSI-X as SPI because of it's own limitation. > Linux has solved it using custom irqchip driver to handle it.=20 > But I am not sure how to address it in FreeBSD.=20 > Can you please point me out something here? We currently only support SPIs in the GICv3 driver under FDT. To teach = the ACPI attachment to support them you=E2=80=99ll need to set = sc->gic_mbi_start and sc->gic_mbi_end in sys/arm64/arm64/gic_v3_acpi.c = to the correct value from the ACPI tables before calling gic_v3_attach. = After this returns successfully you can then call intr_msi_register. See gic_v3_fdt.c for how we do it under FDT. Andrew= --Apple-Mail=_C72BF360-FA1E-44BC-ABBC-1D9F0B37BFF7 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><br = class=3D""><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On 11 Nov 2022, at 19:07, Souradeep Chakrabarti <<a = href=3D"mailto:schakrabarti@microsoft.com" = class=3D"">schakrabarti@microsoft.com</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><meta = charset=3D"UTF-8" class=3D""><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""><blockquote type=3D"cite" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D"">-----Original Message-----<br = class=3D"">From: Souradeep Chakrabarti<br class=3D"">Sent: Thursday, = November 10, 2022 3:20 PM<br class=3D"">To: Andrew Turner <<a = href=3D"mailto:andrew@fubar.geek.nz" = class=3D"">andrew@fubar.geek.nz</a>><br class=3D"">Cc: Warner Losh = <<a href=3D"mailto:imp@bsdimp.com" class=3D"">imp@bsdimp.com</a>>; = Wei Hu <<a href=3D"mailto:weh@microsoft.com" = class=3D"">weh@microsoft.com</a>>; freebsd-<br class=3D""><a = href=3D"mailto:hackers@FreeBSD.org" class=3D"">hackers@FreeBSD.org</a><br = class=3D"">Subject: RE: [EXTERNAL] pcib msix allocation in arm64<br = class=3D""><br class=3D""><br class=3D""><br class=3D""><br = class=3D""><blockquote type=3D"cite" class=3D"">-----Original = Message-----<br class=3D"">From: Andrew Turner <<a = href=3D"mailto:andrew@fubar.geek.nz" = class=3D"">andrew@fubar.geek.nz</a>><br class=3D"">Sent: Thursday, = November 3, 2022 6:21 PM<br class=3D"">To: Souradeep Chakrabarti <<a = href=3D"mailto:schakrabarti@microsoft.com" = class=3D"">schakrabarti@microsoft.com</a>><br class=3D"">Cc: Warner = Losh <<a href=3D"mailto:imp@bsdimp.com" = class=3D"">imp@bsdimp.com</a>>; Wei Hu <<a = href=3D"mailto:weh@microsoft.com" class=3D"">weh@microsoft.com</a>>;<br= class=3D""></blockquote>freebsd-<br class=3D""><blockquote type=3D"cite" = class=3D""><a href=3D"mailto:hackers@FreeBSD.org" = class=3D"">hackers@FreeBSD.org</a><br class=3D"">Subject: Re: [EXTERNAL] = pcib msix allocation in arm64<br class=3D""><br class=3D"">Hi = Souradeep,<br class=3D""><br class=3D"">For the vmbus_pcib driver = you=E2=80=99ll need to implement the pcib_alloc_msi,<br = class=3D"">pcib_release_msi, and the msix versions, and pcib_map_msi.<br = class=3D""><br class=3D"">For alloc and release you can just call into = the same intr_* function<br class=3D"">as pci_host_generic_acpi.c. = You=E2=80=99ll need to pass in the xref for the<br class=3D"">interrupt = controller. It needs to be the xref the MSI controller<br = class=3D"">registered. For the GICv2m it=E2=80=99s ACPI_MSI_XREF.<br = class=3D""><br class=3D""></blockquote>[Souradeep]<br class=3D"">I have = used following in vmbus_pcib.c :<br class=3D"">ret =3D = intr_alloc_msix(pcib, dev, ACPI_MSI_XREF, irq); But it is failing in<br = class=3D"">pic_lookup() with ESRCH by not finding pic with = ACPI_MSI_XREF.<br class=3D"">Do I need to do anything for pic_lookup() = to be successful?<br class=3D""><br class=3D""></blockquote><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;" class=3D"">[Souradeep]<span = class=3D"Apple-converted-space"> </span></span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;" class=3D"">I found out the reason here = pic_lookup() for ACPI_MSI_XREF is failing, as</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;" class=3D"">we are not calling = intr_msi_register() but intr_pic_register() is getting called for the = PCIB.</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;" class=3D""><span style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;" class=3D"">This is = because we don't have ITS implemented in Hyper-V. Instead, Hyper-V is = presenting the</span><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""><span style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;" = class=3D"">MSI/MSI-X as SPI because of it's own limitation.</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;" class=3D"">Linux has solved it using custom = irqchip driver to handle it.<span = class=3D"Apple-converted-space"> </span></span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;" class=3D"">But I am not sure how to address = it in FreeBSD.<span = class=3D"Apple-converted-space"> </span></span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;" class=3D""><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;" class=3D"">Can you please point me out = something here?</span><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;" class=3D""></div></blockquote><div><br = class=3D""></div><div>We currently only support SPIs in the GICv3 driver = under FDT. To teach the ACPI attachment to support them you=E2=80=99ll = need to set sc->gic_mbi_start and sc->gic_mbi_end in = sys/arm64/arm64/gic_v3_acpi.c to the correct value from the ACPI tables = before calling gic_v3_attach. After this returns successfully you = can then call intr_msi_register.</div><div><br = class=3D""></div><div>See gic_v3_fdt.c for how we do it under = FDT.</div><div><br class=3D""></div><div>Andrew</div></div></body></html>= --Apple-Mail=_C72BF360-FA1E-44BC-ABBC-1D9F0B37BFF7--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?DD81C474-8EEA-4B24-A8EC-4B9AE5E9EFED>