Date: Wed, 27 Sep 2023 16:11:44 +0200 From: John Hay <john@sanren.ac.za> To: freebsd-hackers@freebsd.org Subject: make msi interrupts available to child devices in a pci card driver Message-ID: <CAGv8uaonG2sE1Ycts=TzQOx4DwpE1zzaTqsz4cot2EiY58us=Q@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--00000000000012f29f060657c634
Content-Type: text/plain; charset="UTF-8"
Hi,
I would like to know how to make msi interrupts available to child devices
in a pci card driver?
I'm writing a driver for the OCP TimeCard. Apart from the timers, it also
has 16650 UARTs and IIC devices. All of them are memory mapped in a single
BAR. It also has 32 MSI interrupts, with each "function/device" connected
to a specific MSI interrupt.
I would like to add a child device for each of these "functions" and let
each of them have their own interrupt, but I'm struggling to figure out the
correct way to make the interrupts available to the child device.
The memory mapping, using rman and creating subregions for the different
devices is working and I can get a UART device connected in polled mode.
The basic process in the card's attach() is much like what the puc(4)
driver does (leaving error checking etc. out):
<snip>
bar->b_rid = PCIR_BAR(0);
bar->b_type = SYS_RES_MEMORY;
bar->b_res = bus_alloc_resource_any(sc->sc_dev, bar->b_type,
&bar->b_rid, RF_ACTIVE);
sc->sc_iomem.rm_type = RMAN_ARRAY;
error = rman_init(&sc->sc_iomem);
error = rman_manage_region(&sc->sc_iomem, start, end);
for (idx = 0; idx < nports; idx++) {
port->p_rres = rman_reserve_resource(&sc->sc_iomem, start,
end, size, 0, NULL);
bsh = rman_get_bushandle(bar->b_res);
bst = rman_get_bustag(bar->b_res);
port->p_rres = bus_space_subregion(bst, bsh, ofs, size,
&bsh);
rman_set_bushandle(port->p_rres, bsh);
rman_set_bustag(port->p_rres, bst);
port->p_dev = device_add_child(dev, NULL, -1);
error = device_probe_and_attach(port->p_dev);
</snip>
I can also allocate MSI interrupts for the driver itself as a test, with no
problem:
<snip>
sc->sc_msi =32;
pci_alloc_msi(dev, &sc->sc_msi);
printf("msi irqs %d\n", sc->sc_msi);
for (idx = 0; idx < sc->sc_msi; idx++) {
iport->pp_irid = idx + 1;
iport->pp_ires = bus_alloc_resource_any(dev, SYS_RES_IRQ,
&iport->pp_irid, RF_ACTIVE);
bus_setup_intr(dev, iport->pp_ires, INTR_TYPE_TTY |
INTR_MPSAFE, timecard_intr, NULL, iport, &iport->p_ihandle);
error = pci_enable_busmaster(dev);
</snip>
But I have not been able to find a way, similar to the memory to make it
available to a child device, that works. It might be because the child is
not a direct child of the pci driver.
One hackish way that I could get to work was if I used my dev instead of
the child's inside the timecard_bus_alloc_resource() and
timecard_bus_setup_intr() methods, something like this:
<snip>
static struct resource *
timecard_bus_release_resource(device_t dev, device_t child, int type, int
rid, struct resource *res)
{
if (type == SYS_RES_IRQ)
return BUS_RELEASE_RESOURCE(device_get_parent(dev), dev, type, rid,
res);
}
static int
timecard_bus_setup_intr(device_t dev, device_t child, struct resource *res,
int flags, driver_filter_t *filt, void (*ihand)(void *), void *arg,
void **cookiep)
{
return bus_setup_intr(dev, res, flags, filt, ihand, arg, cookiep);
}
</snip>
That works, but does not feel quite right and then the interrupt is not
"owned" by the child, but by the timecard driver:
<snip>
# vmstat -ia | grep timecard
irq41: timecard0 10408538 175
</snip>
Regards
John
--00000000000012f29f060657c634
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>Hi,</div><div><br></div><div>I would like to know how=
to make msi interrupts available to child devices in a pci card driver?</d=
iv><div><br></div><div>I'm writing a driver for the OCP TimeCard. Apart=
from the timers, it also has 16650 UARTs and IIC devices. All of them are =
memory mapped in a single BAR. It also has 32 MSI interrupts, with each &qu=
ot;function/device" connected to a specific MSI interrupt.</div><div><=
br></div><div>I would like to add a child device for each of these "fu=
nctions" and let each of them have their own interrupt, but I'm st=
ruggling to figure out the correct way to make the interrupts available to =
the child device. <br></div><div><br></div><div>The memory mapping, using r=
man and creating subregions for the different devices is working and I can =
get a UART device connected in polled mode. The basic process in the card&#=
39;s attach() is much like what the puc(4) driver does (leaving error check=
ing etc. out):</div><div><br></div><div><snip><br></div><div>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 bar->b_rid =3D PCIR_BAR(0);<br>=C2=A0 =C2=A0 =C2=A0=
=C2=A0 bar->b_type =3D SYS_RES_MEMORY;<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 b=
ar->b_res =3D bus_alloc_resource_any(sc->sc_dev, bar->b_type, &=
;bar->b_rid, RF_ACTIVE);</div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 sc->sc_iomem.rm_type =3D RMAN_ARRAY;</div><div>=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 error =3D rman_init(&sc->sc_iomem);</div><d=
iv>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 error =3D rman_manage_region(=
&sc->sc_iomem, start, end);</div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0 for (idx =3D 0; idx < nports; idx++) {</div><div>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 port->p_rre=
s =3D rman_reserve_resource(&sc->sc_iomem, start, end, size, 0, NULL=
);</div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bsh =3D rman_get_bushandle(bar->b_res);</=
div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 bst =3D rman_get_bustag(bar->b_res);</div><div>=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 port->p_rres =3D bus_space_subregion(bst, bsh, ofs, size=
, &bsh);</div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rman_set_bushandle(port->p_rres,=
bsh);</div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 rman_set_bustag(port->p_rres, bst);</d=
iv><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 port->p_dev =3D device_add_child(dev, NULL, -1)=
;</div><div></div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 error =3D device_probe_and_attach(p=
ort->p_dev);</div><div></snip></div><div><br></div><div>I can also=
allocate MSI interrupts for the driver itself as a test, with no problem:<=
/div><div><br></div><div><snip></div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0=
sc->sc_msi =3D32;</div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
pci_alloc_msi(dev, &sc->sc_msi);</div><div>=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0 printf("msi irqs %d\n", sc->sc_msi);</div><=
div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for (idx =3D 0; idx < sc-=
>sc_msi; idx++) {</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 iport->pp_irid =3D idx + 1;<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 iport->pp_ires =3D bus_alloc_resource_any(d=
ev, SYS_RES_IRQ, &iport->pp_irid, RF_ACTIVE);</div><div>=C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 bus_setup_intr(dev, iport->=
;pp_ires, INTR_TYPE_TTY | INTR_MPSAFE, timecard_intr, NULL, iport, &ipo=
rt->p_ihandle);</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 error =3D pci_enab=
le_busmaster(dev);</div><div></snip></div><div><br></div><div>But I h=
ave not been able to find a way, similar to the memory to make it available=
to a child device, that works. It might be because the child is not a dire=
ct child of the pci driver.</div><div><br></div><div>One hackish way that I=
could get to work was if I used my dev instead of the child's inside t=
he timecard_bus_alloc_resource() and timecard_bus_setup_intr() methods, som=
ething like this:</div><div><br></div><div><snip></div><div>static st=
ruct resource *</div><div>timecard_bus_release_resource(device_t dev, devic=
e_t child, int type, int rid, struct resource *res)</div><div>{</div><div>i=
f (type =3D=3D SYS_RES_IRQ)<br></div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 return BUS_RELEASE_RESOURCE(device_get_parent(dev), dev, type, ri=
d, res);</div><div>}</div><div>static int</div><div>timecard_bus_setup_intr=
(device_t dev, device_t child, struct resource *res,</div><div>=C2=A0 =C2=
=A0 int flags, driver_filter_t *filt, void (*ihand)(void *), void *arg, voi=
d **cookiep)<br>{</div><div>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return bus_setup=
_intr(dev, res, flags, filt, ihand, arg, cookiep);</div><div>}</div><div>&l=
t;/snip></div><div><br></div><div>That works, but does not feel quite ri=
ght and then the interrupt is not "owned" by the child, but by th=
e timecard driver:</div><div><snip><br></div><div># vmstat -ia | grep=
timecard<br>irq41: timecard0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A010408538 =C2=A0 =C2=A0 =C2=A0 =C2=A0175</div><div></snip></=
div><div><br></div><div>Regards</div><div><br></div><div>John</div><div><br=
></div></div>
--00000000000012f29f060657c634--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGv8uaonG2sE1Ycts=TzQOx4DwpE1zzaTqsz4cot2EiY58us=Q>