Date: Fri, 11 Nov 2022 20:10:41 +0000 From: Andrew Turner <andrew@fubar.geek.nz> To: Souradeep Chakrabarti <schakrabarti@microsoft.com> Cc: Warner Losh <imp@bsdimp.com>, Wei Hu <weh@microsoft.com>, "freebsd-hackers@FreeBSD.org" <freebsd-hackers@freebsd.org> Subject: Re: [EXTERNAL] pcib msix allocation in arm64 Message-ID: <DD81C474-8EEA-4B24-A8EC-4B9AE5E9EFED@fubar.geek.nz> In-Reply-To: <PSAP153MB0536273BB0C28ABF24C447B0CC009@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> References: <PSAP153MB05363055DF2B4AE3F39C987BCC399@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> <146F5D18-6366-4953-A8D9-61FE7EC67F71@fubar.geek.nz> <PSAP153MB05367C2284B34BD1A3456F61CC399@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> <947DBD67-6443-480F-82DA-2BDDF44C3D03@fubar.geek.nz> <PSAP153MB05368301E8B0A7AE9133073BCC019@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM> <PSAP153MB0536273BB0C28ABF24C447B0CC009@PSAP153MB0536.APCP153.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] > On 11 Nov 2022, at 19:07, Souradeep Chakrabarti <schakrabarti@microsoft.com> wrote: > > > > >> -----Original Message----- >> From: Souradeep Chakrabarti >> Sent: Thursday, November 10, 2022 3:20 PM >> To: Andrew Turner <andrew@fubar.geek.nz <mailto:andrew@fubar.geek.nz>> >> Cc: Warner Losh <imp@bsdimp.com <mailto:imp@bsdimp.com>>; Wei Hu <weh@microsoft.com <mailto:weh@microsoft.com>>; freebsd- >> hackers@FreeBSD.org <mailto:hackers@FreeBSD.org> >> Subject: RE: [EXTERNAL] pcib msix allocation in arm64 >> >> >> >> >>> -----Original Message----- >>> From: Andrew Turner <andrew@fubar.geek.nz> >>> Sent: Thursday, November 3, 2022 6:21 PM >>> To: Souradeep Chakrabarti <schakrabarti@microsoft.com> >>> Cc: Warner Losh <imp@bsdimp.com>; Wei Hu <weh@microsoft.com>; >> freebsd- >>> hackers@FreeBSD.org >>> Subject: Re: [EXTERNAL] pcib msix allocation in arm64 >>> >>> Hi Souradeep, >>> >>> For the vmbus_pcib driver you’ll need to implement the pcib_alloc_msi, >>> pcib_release_msi, and the msix versions, and pcib_map_msi. >>> >>> For alloc and release you can just call into the same intr_* function >>> as pci_host_generic_acpi.c. You’ll need to pass in the xref for the >>> interrupt controller. It needs to be the xref the MSI controller >>> registered. For the GICv2m it’s ACPI_MSI_XREF. >>> >> [Souradeep] >> I have used following in vmbus_pcib.c : >> ret = intr_alloc_msix(pcib, dev, ACPI_MSI_XREF, irq); But it is failing in >> pic_lookup() with ESRCH by not finding pic with ACPI_MSI_XREF. >> Do I need to do anything for pic_lookup() to be successful? >> > [Souradeep] > I found out the reason here pic_lookup() for ACPI_MSI_XREF is failing, as > we are not calling intr_msi_register() but intr_pic_register() is getting called for the PCIB. > This is because we don't have ITS implemented in Hyper-V. Instead, Hyper-V is presenting the > MSI/MSI-X as SPI because of it's own limitation. > Linux has solved it using custom irqchip driver to handle it. > But I am not sure how to address it in FreeBSD. > Can you please point me out something here? We currently only support SPIs in the GICv3 driver under FDT. To teach the ACPI attachment to support them you’ll need to set sc->gic_mbi_start and sc->gic_mbi_end in sys/arm64/arm64/gic_v3_acpi.c to the correct value from the ACPI tables before calling gic_v3_attach. After this returns successfully you can then call intr_msi_register. See gic_v3_fdt.c for how we do it under FDT. Andrew [-- Attachment #2 --] <html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 11 Nov 2022, at 19:07, Souradeep Chakrabarti <<a href="mailto:schakrabarti@microsoft.com" class="">schakrabarti@microsoft.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">-----Original Message-----<br class="">From: Souradeep Chakrabarti<br class="">Sent: Thursday, November 10, 2022 3:20 PM<br class="">To: Andrew Turner <<a href="mailto:andrew@fubar.geek.nz" class="">andrew@fubar.geek.nz</a>><br class="">Cc: Warner Losh <<a href="mailto:imp@bsdimp.com" class="">imp@bsdimp.com</a>>; Wei Hu <<a href="mailto:weh@microsoft.com" class="">weh@microsoft.com</a>>; freebsd-<br class=""><a href="mailto:hackers@FreeBSD.org" class="">hackers@FreeBSD.org</a><br class="">Subject: RE: [EXTERNAL] pcib msix allocation in arm64<br class=""><br class=""><br class=""><br class=""><br class=""><blockquote type="cite" class="">-----Original Message-----<br class="">From: Andrew Turner <<a href="mailto:andrew@fubar.geek.nz" class="">andrew@fubar.geek.nz</a>><br class="">Sent: Thursday, November 3, 2022 6:21 PM<br class="">To: Souradeep Chakrabarti <<a href="mailto:schakrabarti@microsoft.com" class="">schakrabarti@microsoft.com</a>><br class="">Cc: Warner Losh <<a href="mailto:imp@bsdimp.com" class="">imp@bsdimp.com</a>>; Wei Hu <<a href="mailto:weh@microsoft.com" class="">weh@microsoft.com</a>>;<br class=""></blockquote>freebsd-<br class=""><blockquote type="cite" class=""><a href="mailto:hackers@FreeBSD.org" class="">hackers@FreeBSD.org</a><br class="">Subject: Re: [EXTERNAL] pcib msix allocation in arm64<br class=""><br class="">Hi Souradeep,<br class=""><br class="">For the vmbus_pcib driver you’ll need to implement the pcib_alloc_msi,<br class="">pcib_release_msi, and the msix versions, and pcib_map_msi.<br class=""><br class="">For alloc and release you can just call into the same intr_* function<br class="">as pci_host_generic_acpi.c. You’ll need to pass in the xref for the<br class="">interrupt controller. It needs to be the xref the MSI controller<br class="">registered. For the GICv2m it’s ACPI_MSI_XREF.<br class=""><br class=""></blockquote>[Souradeep]<br class="">I have used following in vmbus_pcib.c :<br class="">ret = intr_alloc_msix(pcib, dev, ACPI_MSI_XREF, irq); But it is failing in<br class="">pic_lookup() with ESRCH by not finding pic with ACPI_MSI_XREF.<br class="">Do I need to do anything for pic_lookup() to be successful?<br class=""><br class=""></blockquote><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">[Souradeep]<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I found out the reason here pic_lookup() for ACPI_MSI_XREF is failing, as</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">we are not calling intr_msi_register() but intr_pic_register() is getting called for the PCIB.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">This is because we don't have ITS implemented in Hyper-V. Instead, Hyper-V is presenting the</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">MSI/MSI-X as SPI because of it's own limitation.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Linux has solved it using custom irqchip driver to handle it.<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">But I am not sure how to address it in FreeBSD.<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Can you please point me out something here?</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div></blockquote><div><br class=""></div><div>We currently only support SPIs in the GICv3 driver under FDT. To teach the ACPI attachment to support them you’ll need to set sc->gic_mbi_start and sc->gic_mbi_end in sys/arm64/arm64/gic_v3_acpi.c to the correct value from the ACPI tables before calling gic_v3_attach. After this returns successfully you can then call intr_msi_register.</div><div><br class=""></div><div>See gic_v3_fdt.c for how we do it under FDT.</div><div><br class=""></div><div>Andrew</div></div></body></html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?DD81C474-8EEA-4B24-A8EC-4B9AE5E9EFED>