From owner-freebsd-drivers@FreeBSD.ORG Mon Jan 7 15:15:00 2013 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3A18750D for ; Mon, 7 Jan 2013 15:15:00 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id EF384FB9 for ; Mon, 7 Jan 2013 15:14:59 +0000 (UTC) Received: from ralph.baldwin.cx (c-68-39-198-164.hsd1.de.comcast.net [68.39.198.164]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 48660B9B0; Mon, 7 Jan 2013 10:14:59 -0500 (EST) From: John Baldwin To: freebsd-drivers@freebsd.org Subject: Re: How to map device addresses into user space Date: Mon, 7 Jan 2013 10:11:49 -0500 User-Agent: KMail/1.13.7 (FreeBSD/9.1-PRERELEASE; KDE/4.8.4; amd64; ; ) References: <033249A9-9C77-426F-BCA2-3A22CC4F9391@cyclaero.com> In-Reply-To: <033249A9-9C77-426F-BCA2-3A22CC4F9391@cyclaero.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201301071011.49234.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 07 Jan 2013 10:14:59 -0500 (EST) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 15:15:00 -0000 On Monday, January 07, 2013 08:52:01 AM Dr. Rolf Jansen wrote: > Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen: > > I am building a loadable kernel module for FreeBSD 9.1-RELEASE x86_64 for > > a PCI Data Acquisition board from National Instruments. > > > > I need to map the Base Address Registers into user space memory, in order > > to pass the BAR's to the National Instruments Drivers Development Kit > > (NI-DDK). The DDK is a complex set of C++ classes running in user space, > > that read/write directly from/into the BAR's. > > > > The FreeBSD bus_space_* functions are useless in this respect, because > > the DDK isn't designed that way, I need the BAR addresses mapped into > > user space. > > > > Having the measurement done by the kernel module is not an option either, > > because it is math intensive, and kernel modules are build without SSE > > and with soft float. > > > > I got tiny kernel modules/extensions only providing the mapped addresses > > of the PCI BAR's running together with the Measurement Routines using > > the NI-DDK on Darwin (Mac OS X) and Linux. > > > > So, how can I map device addresses into user space on FreeBSD? > > Many Thanks to everybody, who responded. > > I got it working. I wrote a PCI device driver, which in addition to > open/close/read/write responds also to ioctl and mmap requests from user > space. In its pci_attach routine, it assembles a custom address_space > struct containing the addresses of the two BAR's of my NI PCI-DAQ board > and two huge allocated memory regions for DMA. > > static struct address_space space; > ... > ... > int bar0_id, bar1_id; > struct resource *bar0res, *bar1res; > > bar0_id = PCIR_BAR(0); > bar1_id = PCIR_BAR(1); > > // these resources must be teared down in pci_detach > bar0res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &bar0id, > RF_ACTIVE); bar1res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &bar1id, > RF_ACTIVE); > > // assemble the address_space struct, i.e. > // virtual addr., size, and offset of its members > space.bar[0].virtual = rman_get_bushandle(bar0res); > space.bar[1].virtual = rman_get_bushandle(bar1res); > space.dma[0].virtual = malloc(dmaAIBufSize, M_MYDAQ, M_WAITOK); > space.dma[1].virtual = malloc(dmaAOBufSize, M_MYDAQ, M_WAITOK); > > space.bar[0].size = rman_get_end(bar0res) - rman_get_start(bar0res) + 1; > space.bar[1].size = rman_get_end(bar1res) - rman_get_start(bar1res) + 1; > space.dma[0].size = dmaAIBufSize; > space.dma[1].size = dmaAOBufSize; > > // this is the crucial part -- The Offsets! > // care must be taken, that the offsets are not negative, > // therefore, find out the minimum and take this as the base address > space.base = MIN(MIN(space.bar[0].virt, space.bar[1].virt), > MIN(space.dma[0].virt, space.dma[1].virt)); > space.bar[0].offs = space.bar[0].virt - space.base; > space.bar[1].offs = space.bar[1].virt - space.base; > space.dma[0].offs = space.dma[0].virt - space.base; > space.dma[1].offs = space.dma[1].virt - space.base; > > > By a call to the ioctl handler of my PCI device driver, this readily > assembled address_space struct is transferred from the driver to my user > space measurement controller, and here it must actually be completed by > subsequent calls to mmap of my device driver: > > space.bar[0].physical = mmap(0, space.bar[0].size, PROT_READ|PROT_WRITE, > MAP_SHARED, fd, space.bar[0].offset); space.bar[1].physical = mmap(0, > space.bar[1].size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, > space.bar[1].offset); space.dma[0].physical = mmap(0, space.dma[0].size, > PROT_READ|PROT_WRITE, MAP_SHARED, fd, space.dma[0].offset); > space.dma[1].physical = mmap(0, space.dma[1].size, PROT_READ|PROT_WRITE, > MAP_SHARED, fd, space.dma[1].offset); > > > Because of the coherent offsets, the mmap handler of my PCI device driver > can be kept quite simple: > > static int mydaq_mmap(struct cdev *dev, vm_ooffset_t offset, vm_paddr_t > *paddr, int nprot, vm_memattr_t *memattr) { > *paddr = vtophys(space.base + offset); > return 0; > } Note that if your BARs are not contiguous you can allow mapping of things in the middle. This is a potentital security hole if your driver can be opened by non-root. A more robust approach is to define a virtual address space for your device. This is also a lot easier to program against in userland. For example, you could define offset 0 as mapping to BAR 0 and the next chunk of virtual address space (the offset) map to BAR 1, etc.: static int mydaq_mmap(...) { struct mydaq_softc *sc; // you should be storing things like the pointers // to your BARs in a softc sc = dev->si_drv1; if (offset <= rman_get_size(sc->bar0res)) return (rman_get_start(sc->bar0res) + offset); offset -= rman_get_size(sc->bar0res); if (offset <= rman_get_size(sc->bar1res)) return (rman_get_start(sc->bar1res) + offset); offset -= rman_get_size(sc->bar1res); if (offset <= dmaAIBufSize) return (vtophys(sc->dma[0]) + offset); offset -= dmaAIBufSize; if (offset <= dmaAOBufSize) return (vtophys(sc->dma[1]) + offset); /* Invalid offset */ return (-1); } Then you can use fixed offsets in userland for your mmap requests. Also, rman_get_bushandle() is not appropriate to get a virtual address, that is an implementation detail you should not count on. You could use rman_get_virtual(), but that is not the best practice. Using rman_get_start() to get a PA directly is a better approach. On some embedded platforms with translating bridges it would not work as Warner noted, but it should work fine for most platforms that FreeBSD supports. -- John Baldwin