Date: Mon, 7 Jan 2013 10:11:49 -0500 From: John Baldwin <jhb@freebsd.org> To: freebsd-drivers@freebsd.org Subject: Re: How to map device addresses into user space Message-ID: <201301071011.49234.jhb@freebsd.org> In-Reply-To: <033249A9-9C77-426F-BCA2-3A22CC4F9391@cyclaero.com> References: <DFC983B8-91B2-4ED8-89B0-FC4BD4AB2576@cyclaero.com> <033249A9-9C77-426F-BCA2-3A22CC4F9391@cyclaero.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday, January 07, 2013 08:52:01 AM Dr. Rolf Jansen wrote:
> Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen:
> > I am building a loadable kernel module for FreeBSD 9.1-RELEASE x86_64 for
> > a PCI Data Acquisition board from National Instruments.
> >
> > I need to map the Base Address Registers into user space memory, in order
> > to pass the BAR's to the National Instruments Drivers Development Kit
> > (NI-DDK). The DDK is a complex set of C++ classes running in user space,
> > that read/write directly from/into the BAR's.
> >
> > The FreeBSD bus_space_* functions are useless in this respect, because
> > the DDK isn't designed that way, I need the BAR addresses mapped into
> > user space.
> >
> > Having the measurement done by the kernel module is not an option either,
> > because it is math intensive, and kernel modules are build without SSE
> > and with soft float.
> >
> > I got tiny kernel modules/extensions only providing the mapped addresses
> > of the PCI BAR's running together with the Measurement Routines using
> > the NI-DDK on Darwin (Mac OS X) and Linux.
> >
> > So, how can I map device addresses into user space on FreeBSD?
>
> Many Thanks to everybody, who responded.
>
> I got it working. I wrote a PCI device driver, which in addition to
> open/close/read/write responds also to ioctl and mmap requests from user
> space. In its pci_attach routine, it assembles a custom address_space
> struct containing the addresses of the two BAR's of my NI PCI-DAQ board
> and two huge allocated memory regions for DMA.
>
> static struct address_space space;
> ...
> ...
> int bar0_id, bar1_id;
> struct resource *bar0res, *bar1res;
>
> bar0_id = PCIR_BAR(0);
> bar1_id = PCIR_BAR(1);
>
> // these resources must be teared down in pci_detach
> bar0res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &bar0id,
> RF_ACTIVE); bar1res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &bar1id,
> RF_ACTIVE);
>
> // assemble the address_space struct, i.e.
> // virtual addr., size, and offset of its members
> space.bar[0].virtual = rman_get_bushandle(bar0res);
> space.bar[1].virtual = rman_get_bushandle(bar1res);
> space.dma[0].virtual = malloc(dmaAIBufSize, M_MYDAQ, M_WAITOK);
> space.dma[1].virtual = malloc(dmaAOBufSize, M_MYDAQ, M_WAITOK);
>
> space.bar[0].size = rman_get_end(bar0res) - rman_get_start(bar0res) + 1;
> space.bar[1].size = rman_get_end(bar1res) - rman_get_start(bar1res) + 1;
> space.dma[0].size = dmaAIBufSize;
> space.dma[1].size = dmaAOBufSize;
>
> // this is the crucial part -- The Offsets!
> // care must be taken, that the offsets are not negative,
> // therefore, find out the minimum and take this as the base address
> space.base = MIN(MIN(space.bar[0].virt, space.bar[1].virt),
> MIN(space.dma[0].virt, space.dma[1].virt));
> space.bar[0].offs = space.bar[0].virt - space.base;
> space.bar[1].offs = space.bar[1].virt - space.base;
> space.dma[0].offs = space.dma[0].virt - space.base;
> space.dma[1].offs = space.dma[1].virt - space.base;
>
>
> By a call to the ioctl handler of my PCI device driver, this readily
> assembled address_space struct is transferred from the driver to my user
> space measurement controller, and here it must actually be completed by
> subsequent calls to mmap of my device driver:
>
> space.bar[0].physical = mmap(0, space.bar[0].size, PROT_READ|PROT_WRITE,
> MAP_SHARED, fd, space.bar[0].offset); space.bar[1].physical = mmap(0,
> space.bar[1].size, PROT_READ|PROT_WRITE, MAP_SHARED, fd,
> space.bar[1].offset); space.dma[0].physical = mmap(0, space.dma[0].size,
> PROT_READ|PROT_WRITE, MAP_SHARED, fd, space.dma[0].offset);
> space.dma[1].physical = mmap(0, space.dma[1].size, PROT_READ|PROT_WRITE,
> MAP_SHARED, fd, space.dma[1].offset);
>
>
> Because of the coherent offsets, the mmap handler of my PCI device driver
> can be kept quite simple:
>
> static int mydaq_mmap(struct cdev *dev, vm_ooffset_t offset, vm_paddr_t
> *paddr, int nprot, vm_memattr_t *memattr) {
> *paddr = vtophys(space.base + offset);
> return 0;
> }
Note that if your BARs are not contiguous you can allow mapping of things in
the middle. This is a potentital security hole if your driver can be opened
by non-root. A more robust approach is to define a virtual address space for
your device. This is also a lot easier to program against in userland.
For example, you could define offset 0 as mapping to BAR 0 and the next chunk
of virtual address space (the offset) map to BAR 1, etc.:
static int
mydaq_mmap(...)
{
struct mydaq_softc *sc; // you should be storing things like the pointers
// to your BARs in a softc
sc = dev->si_drv1;
if (offset <= rman_get_size(sc->bar0res))
return (rman_get_start(sc->bar0res) + offset);
offset -= rman_get_size(sc->bar0res);
if (offset <= rman_get_size(sc->bar1res))
return (rman_get_start(sc->bar1res) + offset);
offset -= rman_get_size(sc->bar1res);
if (offset <= dmaAIBufSize)
return (vtophys(sc->dma[0]) + offset);
offset -= dmaAIBufSize;
if (offset <= dmaAOBufSize)
return (vtophys(sc->dma[1]) + offset);
/* Invalid offset */
return (-1);
}
Then you can use fixed offsets in userland for your mmap requests. Also,
rman_get_bushandle() is not appropriate to get a virtual address, that is an
implementation detail you should not count on. You could use
rman_get_virtual(), but that is not the best practice. Using rman_get_start()
to get a PA directly is a better approach. On some embedded platforms with
translating bridges it would not work as Warner noted, but it should work fine
for most platforms that FreeBSD supports.
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201301071011.49234.jhb>