From owner-freebsd-drivers@FreeBSD.ORG Mon Jan 7 13:52:10 2013 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8DE7E163 for ; Mon, 7 Jan 2013 13:52:10 +0000 (UTC) (envelope-from rj@cyclaero.com) Received: from mo6-p00-ob.rzone.de (mo6-p00-ob.rzone.de [IPv6:2a01:238:20a:202:5300::1]) by mx1.freebsd.org (Postfix) with ESMTP id 27043AE0 for ; Mon, 7 Jan 2013 13:52:09 +0000 (UTC) X-RZG-AUTH: :O2kGeEG7b/pS1E6gSHOyjPKyNsg/5l1He+DgCy9/8FSej6CwUysqfN3Dd+qW8pifIOtG X-RZG-CLASS-ID: mo00 Received: from rolf.projectworld.net (b150f525.virtua.com.br [177.80.245.37]) by smtp.strato.de (jorabe mo35) (RZmta 31.11 DYNA|AUTH) with (AES128-SHA encrypted) ESMTPA id a06212p07CwTux for ; Mon, 7 Jan 2013 14:52:05 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1283) Subject: Re: How to map device addresses into user space From: "Dr. Rolf Jansen" In-Reply-To: Date: Mon, 7 Jan 2013 11:52:01 -0200 Content-Transfer-Encoding: quoted-printable Message-Id: <033249A9-9C77-426F-BCA2-3A22CC4F9391@cyclaero.com> References: To: freebsd-drivers@freebsd.org X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 13:52:10 -0000 Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen: > I am building a loadable kernel module for FreeBSD 9.1-RELEASE x86_64 = for a PCI Data Acquisition board from National Instruments. >=20 > I need to map the Base Address Registers into user space memory, in = order to pass the BAR's to the National Instruments Drivers Development = Kit (NI-DDK). The DDK is a complex set of C++ classes running in user = space, that read/write directly from/into the BAR's. >=20 > The FreeBSD bus_space_* functions are useless in this respect, because = the DDK isn't designed that way, I need the BAR addresses mapped into = user space. >=20 > Having the measurement done by the kernel module is not an option = either, because it is math intensive, and kernel modules are build = without SSE and with soft float. >=20 > I got tiny kernel modules/extensions only providing the mapped = addresses of the PCI BAR's running together with the Measurement = Routines using the NI-DDK on Darwin (Mac OS X) and Linux. >=20 > So, how can I map device addresses into user space on FreeBSD? Many Thanks to everybody, who responded. I got it working. I wrote a PCI device driver, which in addition to = open/close/read/write responds also to ioctl and mmap requests from user = space. In its pci_attach routine, it assembles a custom address_space = struct containing the addresses of the two BAR's of my NI PCI-DAQ board = and two huge allocated memory regions for DMA. static struct address_space space; ... ... int bar0_id, bar1_id;=20 struct resource *bar0res, *bar1res; bar0_id =3D PCIR_BAR(0); bar1_id =3D PCIR_BAR(1); // these resources must be teared down in pci_detach bar0res =3D bus_alloc_resource_any(dev, SYS_RES_MEMORY, &bar0id, = RF_ACTIVE); bar1res =3D bus_alloc_resource_any(dev, SYS_RES_MEMORY, &bar1id, = RF_ACTIVE); // assemble the address_space struct, i.e. // virtual addr., size, and offset of its members space.bar[0].virtual =3D rman_get_bushandle(bar0res); space.bar[1].virtual =3D rman_get_bushandle(bar1res); space.dma[0].virtual =3D malloc(dmaAIBufSize, M_MYDAQ, M_WAITOK); space.dma[1].virtual =3D malloc(dmaAOBufSize, M_MYDAQ, M_WAITOK); space.bar[0].size =3D rman_get_end(bar0res) - rman_get_start(bar0res) = + 1; space.bar[1].size =3D rman_get_end(bar1res) - rman_get_start(bar1res) = + 1; space.dma[0].size =3D dmaAIBufSize; space.dma[1].size =3D dmaAOBufSize; // this is the crucial part -- The Offsets! // care must be taken, that the offsets are not negative, // therefore, find out the minimum and take this as the base address space.base =3D MIN(MIN(space.bar[0].virt, space.bar[1].virt), MIN(space.dma[0].virt, space.dma[1].virt)); space.bar[0].offs =3D space.bar[0].virt - space.base; space.bar[1].offs =3D space.bar[1].virt - space.base; space.dma[0].offs =3D space.dma[0].virt - space.base; space.dma[1].offs =3D space.dma[1].virt - space.base; By a call to the ioctl handler of my PCI device driver, this readily = assembled address_space struct is transferred from the driver to my user = space measurement controller, and here it must actually be completed by = subsequent calls to mmap of my device driver: space.bar[0].physical =3D mmap(0, space.bar[0].size, = PROT_READ|PROT_WRITE, MAP_SHARED, fd, space.bar[0].offset); space.bar[1].physical =3D mmap(0, space.bar[1].size, = PROT_READ|PROT_WRITE, MAP_SHARED, fd, space.bar[1].offset); space.dma[0].physical =3D mmap(0, space.dma[0].size, = PROT_READ|PROT_WRITE, MAP_SHARED, fd, space.dma[0].offset); space.dma[1].physical =3D mmap(0, space.dma[1].size, = PROT_READ|PROT_WRITE, MAP_SHARED, fd, space.dma[1].offset); Because of the coherent offsets, the mmap handler of my PCI device = driver can be kept quite simple: static int mydaq_mmap(struct cdev *dev, vm_ooffset_t offset, vm_paddr_t = *paddr, int nprot, vm_memattr_t *memattr) { *paddr =3D vtophys(space.base + offset); return 0; } This works for me. Best regards Rolf From owner-freebsd-drivers@FreeBSD.ORG Mon Jan 7 15:15:00 2013 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3A18750D for ; Mon, 7 Jan 2013 15:15:00 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id EF384FB9 for ; Mon, 7 Jan 2013 15:14:59 +0000 (UTC) Received: from ralph.baldwin.cx (c-68-39-198-164.hsd1.de.comcast.net [68.39.198.164]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 48660B9B0; Mon, 7 Jan 2013 10:14:59 -0500 (EST) From: John Baldwin To: freebsd-drivers@freebsd.org Subject: Re: How to map device addresses into user space Date: Mon, 7 Jan 2013 10:11:49 -0500 User-Agent: KMail/1.13.7 (FreeBSD/9.1-PRERELEASE; KDE/4.8.4; amd64; ; ) References: <033249A9-9C77-426F-BCA2-3A22CC4F9391@cyclaero.com> In-Reply-To: <033249A9-9C77-426F-BCA2-3A22CC4F9391@cyclaero.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201301071011.49234.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 07 Jan 2013 10:14:59 -0500 (EST) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 15:15:00 -0000 On Monday, January 07, 2013 08:52:01 AM Dr. Rolf Jansen wrote: > Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen: > > I am building a loadable kernel module for FreeBSD 9.1-RELEASE x86_64 for > > a PCI Data Acquisition board from National Instruments. > > > > I need to map the Base Address Registers into user space memory, in order > > to pass the BAR's to the National Instruments Drivers Development Kit > > (NI-DDK). The DDK is a complex set of C++ classes running in user space, > > that read/write directly from/into the BAR's. > > > > The FreeBSD bus_space_* functions are useless in this respect, because > > the DDK isn't designed that way, I need the BAR addresses mapped into > > user space. > > > > Having the measurement done by the kernel module is not an option either, > > because it is math intensive, and kernel modules are build without SSE > > and with soft float. > > > > I got tiny kernel modules/extensions only providing the mapped addresses > > of the PCI BAR's running together with the Measurement Routines using > > the NI-DDK on Darwin (Mac OS X) and Linux. > > > > So, how can I map device addresses into user space on FreeBSD? > > Many Thanks to everybody, who responded. > > I got it working. I wrote a PCI device driver, which in addition to > open/close/read/write responds also to ioctl and mmap requests from user > space. In its pci_attach routine, it assembles a custom address_space > struct containing the addresses of the two BAR's of my NI PCI-DAQ board > and two huge allocated memory regions for DMA. > > static struct address_space space; > ... > ... > int bar0_id, bar1_id; > struct resource *bar0res, *bar1res; > > bar0_id = PCIR_BAR(0); > bar1_id = PCIR_BAR(1); > > // these resources must be teared down in pci_detach > bar0res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &bar0id, > RF_ACTIVE); bar1res = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &bar1id, > RF_ACTIVE); > > // assemble the address_space struct, i.e. > // virtual addr., size, and offset of its members > space.bar[0].virtual = rman_get_bushandle(bar0res); > space.bar[1].virtual = rman_get_bushandle(bar1res); > space.dma[0].virtual = malloc(dmaAIBufSize, M_MYDAQ, M_WAITOK); > space.dma[1].virtual = malloc(dmaAOBufSize, M_MYDAQ, M_WAITOK); > > space.bar[0].size = rman_get_end(bar0res) - rman_get_start(bar0res) + 1; > space.bar[1].size = rman_get_end(bar1res) - rman_get_start(bar1res) + 1; > space.dma[0].size = dmaAIBufSize; > space.dma[1].size = dmaAOBufSize; > > // this is the crucial part -- The Offsets! > // care must be taken, that the offsets are not negative, > // therefore, find out the minimum and take this as the base address > space.base = MIN(MIN(space.bar[0].virt, space.bar[1].virt), > MIN(space.dma[0].virt, space.dma[1].virt)); > space.bar[0].offs = space.bar[0].virt - space.base; > space.bar[1].offs = space.bar[1].virt - space.base; > space.dma[0].offs = space.dma[0].virt - space.base; > space.dma[1].offs = space.dma[1].virt - space.base; > > > By a call to the ioctl handler of my PCI device driver, this readily > assembled address_space struct is transferred from the driver to my user > space measurement controller, and here it must actually be completed by > subsequent calls to mmap of my device driver: > > space.bar[0].physical = mmap(0, space.bar[0].size, PROT_READ|PROT_WRITE, > MAP_SHARED, fd, space.bar[0].offset); space.bar[1].physical = mmap(0, > space.bar[1].size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, > space.bar[1].offset); space.dma[0].physical = mmap(0, space.dma[0].size, > PROT_READ|PROT_WRITE, MAP_SHARED, fd, space.dma[0].offset); > space.dma[1].physical = mmap(0, space.dma[1].size, PROT_READ|PROT_WRITE, > MAP_SHARED, fd, space.dma[1].offset); > > > Because of the coherent offsets, the mmap handler of my PCI device driver > can be kept quite simple: > > static int mydaq_mmap(struct cdev *dev, vm_ooffset_t offset, vm_paddr_t > *paddr, int nprot, vm_memattr_t *memattr) { > *paddr = vtophys(space.base + offset); > return 0; > } Note that if your BARs are not contiguous you can allow mapping of things in the middle. This is a potentital security hole if your driver can be opened by non-root. A more robust approach is to define a virtual address space for your device. This is also a lot easier to program against in userland. For example, you could define offset 0 as mapping to BAR 0 and the next chunk of virtual address space (the offset) map to BAR 1, etc.: static int mydaq_mmap(...) { struct mydaq_softc *sc; // you should be storing things like the pointers // to your BARs in a softc sc = dev->si_drv1; if (offset <= rman_get_size(sc->bar0res)) return (rman_get_start(sc->bar0res) + offset); offset -= rman_get_size(sc->bar0res); if (offset <= rman_get_size(sc->bar1res)) return (rman_get_start(sc->bar1res) + offset); offset -= rman_get_size(sc->bar1res); if (offset <= dmaAIBufSize) return (vtophys(sc->dma[0]) + offset); offset -= dmaAIBufSize; if (offset <= dmaAOBufSize) return (vtophys(sc->dma[1]) + offset); /* Invalid offset */ return (-1); } Then you can use fixed offsets in userland for your mmap requests. Also, rman_get_bushandle() is not appropriate to get a virtual address, that is an implementation detail you should not count on. You could use rman_get_virtual(), but that is not the best practice. Using rman_get_start() to get a PA directly is a better approach. On some embedded platforms with translating bridges it would not work as Warner noted, but it should work fine for most platforms that FreeBSD supports. -- John Baldwin From owner-freebsd-drivers@FreeBSD.ORG Mon Jan 7 17:25:42 2013 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 05377A46; Mon, 7 Jan 2013 17:25:42 +0000 (UTC) (envelope-from rj@cyclaero.com) Received: from mo6-p00-ob.rzone.de (mo6-p00-ob.rzone.de [IPv6:2a01:238:20a:202:5300::1]) by mx1.freebsd.org (Postfix) with ESMTP id 6DFBF8F7; Mon, 7 Jan 2013 17:25:41 +0000 (UTC) X-RZG-AUTH: :O2kGeEG7b/pS1E6gSHOyjPKyNsg/5l1He+DgCy9/8FSej6CwUysqfN3Dd+qW8pifIOtG X-RZG-CLASS-ID: mo00 Received: from rolf.projectworld.net (b150f525.virtua.com.br [177.80.245.37]) by smtp.strato.de (jored mo31) (RZmta 31.11 DYNA|AUTH) with (AES128-SHA encrypted) ESMTPA id e0713dp07G3WDi ; Mon, 7 Jan 2013 18:25:36 +0100 (CET) Subject: Re: How to map device addresses into user space Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: "Dr. Rolf Jansen" In-Reply-To: <201301071011.49234.jhb@freebsd.org> Date: Mon, 7 Jan 2013 15:25:33 -0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <033249A9-9C77-426F-BCA2-3A22CC4F9391@cyclaero.com> <201301071011.49234.jhb@freebsd.org> To: freebsd-drivers@freebsd.org X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 17:25:42 -0000 Am 07.01.2013 um 13:11 schrieb John Baldwin: > On Monday, January 07, 2013 08:52:01 AM Dr. Rolf Jansen wrote: >=20 >> Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen: >>=20 >>> ... >>>=20 >>> So, how can I map device addresses into user space on FreeBSD? >>=20 >> Many Thanks to everybody, who responded. >>=20 >> I got it working. I wrote a PCI device driver, which in addition to >> open/close/read/write responds also to ioctl and mmap requests from = user >> space. In its pci_attach routine, it assembles a custom address_space >> struct containing the addresses of the two BAR's of my NI PCI-DAQ = board >> and two huge allocated memory regions for DMA... >>=20 >=20 > Note that if your BARs are not contiguous you can allow mapping of = things in=20 > the middle. This is a potentital security hole if your driver can be = opened=20 > by non-root. I can understand this. Actually, on the present model of the DAQ board, = the BARs are contiguous, and in the present incarnation, my driver can only = be addressed by root. Anyway, I want to do it correctly, that means, I = would like to avoid right now potential holes bubbling up in the future, when = using my driver with other DAQ boards, and in different usage scenarios. > A more robust approach is to define a virtual address space for=20 > your device. This is also a lot easier to program against in userland. > For example, you could define offset 0 as mapping to BAR 0 and the = next chunk > of virtual address space (the offset) map to BAR 1, etc.: I did this, but id didn't work out. The 2 BARs are contiguous, and the 2 = DMA regions are also, but there is huge gap between BAR and DMA. And = depending on which mapping is done first, either the DMAs or the BARs receive invalid = paddr. So, I guess, defining a virtual address space is a little bit more = involved than simply getting the offsets straight as you lined out.=20 I am still learning things, so please bear with me, if it is a dumb = question. How do I define a virtual address space for the device? > static int > mydaq_mmap(...) > { > struct mydaq_softc *sc; // you should be storing things like = the pointers > // to your BARs in a softc >=20 > sc =3D dev->si_drv1; > if (offset <=3D rman_get_size(sc->bar0res)) > return (rman_get_start(sc->bar0res) + offset); > offset -=3D rman_get_size(sc->bar0res); > if (offset <=3D rman_get_size(sc->bar1res)) > return (rman_get_start(sc->bar1res) + offset); > offset -=3D rman_get_size(sc->bar1res); > if (offset <=3D dmaAIBufSize) > return (vtophys(sc->dma[0]) + offset); > offset -=3D dmaAIBufSize; > if (offset <=3D dmaAOBufSize) > return (vtophys(sc->dma[1]) + offset); > =09 > /* Invalid offset */ > return (-1); > } I guess, you hacked this quickly together, and I picked the concept, so this is fine. In order to make this working, the addresses should be = assigned to the *paddr parameter, and a status code should be returned. In order = to check things, of course I applied these minor corrections, but it = doesn't work for either the BAR block or the DMA block, depending on which is = mapped first. > Also, rman_get_bushandle() is not appropriate to get a virtual = address, that is an=20 > implementation detail you should not count on. You could use = rman_get_virtual(), > but that is not the best practice. Using rman_get_start()=20 > to get a PA directly is a better approach. Yes, this is much more reasonable than my approach. I was already = concerned about rman_get_bushandle() making a vaddr from a paddr in kernel space, only = to translate it back to a paddr in user space. rman_get_start() is of course much = better, and I will change to this, once I got the definition of a virtual address = space straight. Many thanks for your very helpful response. Best regards Rolf From owner-freebsd-drivers@FreeBSD.ORG Mon Jan 7 20:13:20 2013 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B8478416 for ; Mon, 7 Jan 2013 20:13:20 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 6139B307 for ; Mon, 7 Jan 2013 20:13:20 +0000 (UTC) Received: from ralph.baldwin.cx (c-68-39-198-164.hsd1.de.comcast.net [68.39.198.164]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 87B07B97C; Mon, 7 Jan 2013 15:13:19 -0500 (EST) From: John Baldwin To: "Dr. Rolf Jansen" Subject: Re: How to map device addresses into user space Date: Mon, 7 Jan 2013 14:21:10 -0500 User-Agent: KMail/1.13.7 (FreeBSD/9.1-PRERELEASE; KDE/4.8.4; amd64; ; ) References: <201301071011.49234.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201301071421.10559.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 07 Jan 2013 15:13:19 -0500 (EST) Cc: freebsd-drivers@freebsd.org X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 20:13:20 -0000 On Monday, January 07, 2013 12:25:33 PM Dr. Rolf Jansen wrote: > Am 07.01.2013 um 13:11 schrieb John Baldwin: > > On Monday, January 07, 2013 08:52:01 AM Dr. Rolf Jansen wrote: > >> Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen: > >>> ... > >>> > >>> So, how can I map device addresses into user space on FreeBSD? > >> > >> Many Thanks to everybody, who responded. > >> > >> I got it working. I wrote a PCI device driver, which in addition to > >> open/close/read/write responds also to ioctl and mmap requests from user > >> space. In its pci_attach routine, it assembles a custom address_space > >> struct containing the addresses of the two BAR's of my NI PCI-DAQ board > >> and two huge allocated memory regions for DMA... > > > > Note that if your BARs are not contiguous you can allow mapping of things > > in the middle. This is a potentital security hole if your driver can be > > opened by non-root. > > I can understand this. Actually, on the present model of the DAQ board, the > BARs are contiguous, and in the present incarnation, my driver can only be > addressed by root. Anyway, I want to do it correctly, that means, I would > like to avoid right now potential holes bubbling up in the future, when > using my driver with other DAQ boards, and in different usage scenarios. Note that the BARs being contiguous is a property of your BIOS or OS, not your board. :) The firmware/OS are free to assign whatever ranges to your BARs that they wish. > > A more robust approach is to define a virtual address space for > > your device. This is also a lot easier to program against in userland. > > For example, you could define offset 0 as mapping to BAR 0 and the next > > chunk > > > of virtual address space (the offset) map to BAR 1, etc.: > I did this, but id didn't work out. The 2 BARs are contiguous, and the 2 > DMA regions are also, but there is huge gap between BAR and DMA. And > depending on which mapping is done first, either the DMAs or the BARs > receive invalid paddr. So, I guess, defining a virtual address space is a > little bit more involved than simply getting the offsets straight as you > lined out. > > I am still learning things, so please bear with me, if it is a dumb > question. How do I define a virtual address space for the device? Ah, I'll explain a bit. When you mmap a section of a file, the OS is treating the file as if it were its own section of virtual memory, where files contents were addressed by the offset. That is, virtual address 0 of the file corresponds to the data at file offset 0, and virtual address N corresponds to the data at file offset N. Now, think of your /dev/foo device as a file. When you mmap() your /dev/foo, the OS has assumes it has a similar virtual memory that is addressed by the offset into the file. However, for a character device, the OS needs the driver to describe this virtual address space. That is what you are doing with d_mmap() and the 'offset' parameter. You are telling it what physical page backs each "virtual page". It is important to note that when using d_mmap(), these mappings are immutable. Once you provide a mapping for a given virtual page, the OS will cache it forever (or until reboot). In other words, you can write your d_mmap() to give the appearance that your /dev/foo "file" contains BAR 0 followed by BAR 1, etc. > > static int > > mydaq_mmap(...) > > { > > > > struct mydaq_softc *sc; // you should be storing things like the > > pointers > > > > // to your BARs in a softc > > > > sc = dev->si_drv1; > > if (offset <= rman_get_size(sc->bar0res)) > > > > return (rman_get_start(sc->bar0res) + offset); > > > > offset -= rman_get_size(sc->bar0res); > > if (offset <= rman_get_size(sc->bar1res)) > > > > return (rman_get_start(sc->bar1res) + offset); > > > > offset -= rman_get_size(sc->bar1res); > > if (offset <= dmaAIBufSize) > > > > return (vtophys(sc->dma[0]) + offset); > > > > offset -= dmaAIBufSize; > > if (offset <= dmaAOBufSize) > > > > return (vtophys(sc->dma[1]) + offset); > > > > /* Invalid offset */ > > return (-1); > > > > } > > I guess, you hacked this quickly together, and I picked the concept, so > this is fine. In order to make this working, the addresses should be > assigned to the *paddr parameter, and a status code should be returned. In > order to check things, of course I applied these minor corrections, but it > doesn't work for either the BAR block or the DMA block, depending on which > is mapped first. Yes, it needs the changes you made. Can you post your updated d_mmap() routine along with how you are mapping it in userland? With what I posted above you would do something like this in userland: const char *bar0, *bar1; int fd = open("/dev/foo"); bar0 = mmap(NULL, , PROT_READ, MAP_SHARED, fd, 0); bar1 = mmap(NULL, , PROT_READ, MAP_SHARED, fd, ); etc. -- John Baldwin From owner-freebsd-drivers@FreeBSD.ORG Mon Jan 7 23:40:19 2013 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id ADB92ECA; Mon, 7 Jan 2013 23:40:19 +0000 (UTC) (envelope-from rj@cyclaero.com) Received: from mo6-p00-ob.rzone.de (mo6-p00-ob.rzone.de [IPv6:2a01:238:20a:202:5300::1]) by mx1.freebsd.org (Postfix) with ESMTP id 20AE8D5E; Mon, 7 Jan 2013 23:40:18 +0000 (UTC) X-RZG-AUTH: :O2kGeEG7b/pS1E6gSHOyjPKyNsg/5l1He+DgCy9/8FSej6CwUysqfN3Dd+qW8pifIOtG X-RZG-CLASS-ID: mo00 Received: from rolf.projectworld.net (b150f525.virtua.com.br [177.80.245.37]) by smtp.strato.de (jored mo17) (RZmta 31.11 DYNA|AUTH) with (AES128-SHA encrypted) ESMTPA id w06b3bp07Lo5Px ; Tue, 8 Jan 2013 00:40:14 +0100 (CET) Subject: Re: How to map device addresses into user space Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: "Dr. Rolf Jansen" In-Reply-To: <201301071421.10559.jhb@freebsd.org> Date: Mon, 7 Jan 2013 21:40:11 -0200 Content-Transfer-Encoding: quoted-printable Message-Id: <8BDABFD4-D371-49D4-B21C-EBCD7C86B47C@cyclaero.com> References: <201301071011.49234.jhb@freebsd.org> <201301071421.10559.jhb@freebsd.org> To: freebsd-drivers@freebsd.org X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 23:40:19 -0000 Am 07.01.2013 um 17:21 schrieb John Baldwin: > On Monday, January 07, 2013 12:25:33 PM Dr. Rolf Jansen wrote: >> Am 07.01.2013 um 13:11 schrieb John Baldwin: >>> On Monday, January 07, 2013 08:52:01 AM Dr. Rolf Jansen wrote: >>>> Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen: >>>>> ... >>>>>=20 >>>>> So, how can I map device addresses into user space on FreeBSD? >>>>=20 >>>> Many Thanks to everybody, who responded. >>>>=20 >>>> I got it working. I wrote a PCI device driver, which in addition to >>>> open/close/read/write responds also to ioctl and mmap requests from = user >>>> space. In its pci_attach routine, it assembles a custom = address_space >>>> struct containing the addresses of the two BAR's of my NI PCI-DAQ = board >>>> and two huge allocated memory regions for DMA... >>>=20 >>> Note that if your BARs are not contiguous you can allow mapping of = things >>> in the middle. This is a potentital security hole if your driver = can be >>> opened by non-root. >>=20 >> I can understand this. Actually, on the present model of the DAQ = board, the >> BARs are contiguous, and in the present incarnation, my driver can = only be >> addressed by root. Anyway, I want to do it correctly, that means, I = would >> like to avoid right now potential holes bubbling up in the future, = when >> using my driver with other DAQ boards, and in different usage = scenarios. >=20 > Note that the BARs being contiguous is a property of your BIOS or OS, = not your=20 > board. :) The firmware/OS are free to assign whatever ranges to your = BARs=20 > that they wish. >=20 >>> A more robust approach is to define a virtual address space for >>> your device. This is also a lot easier to program against in = userland. >>> For example, you could define offset 0 as mapping to BAR 0 and the = next >>> chunk >>=20 >>> of virtual address space (the offset) map to BAR 1, etc.: >> I did this, but id didn't work out. The 2 BARs are contiguous, and = the 2 >> DMA regions are also, but there is huge gap between BAR and DMA. And >> depending on which mapping is done first, either the DMAs or the BARs >> receive invalid paddr. So, I guess, defining a virtual address space = is a >> little bit more involved than simply getting the offsets straight as = you >> lined out. >>=20 >> I am still learning things, so please bear with me, if it is a dumb >> question. How do I define a virtual address space for the device? >=20 > Ah, I'll explain a bit. When you mmap a section of a file, the OS is = treating=20 > the file as if it were its own section of virtual memory, where files = contents=20 > were addressed by the offset. That is, virtual address 0 of the file=20= > corresponds to the data at file offset 0, and virtual address N = corresponds to=20 > the data at file offset N. >=20 > Now, think of your /dev/foo device as a file. When you mmap() your = /dev/foo,=20 > the OS has assumes it has a similar virtual memory that is addressed = by the=20 > offset into the file. However, for a character device, the OS needs = the=20 > driver to describe this virtual address space. That is what you are = doing=20 > with d_mmap() and the 'offset' parameter. You are telling it what = physical=20 > page backs each "virtual page". It is important to note that when = using=20 > d_mmap(), these mappings are immutable. Once you provide a mapping = for a=20 > given virtual page, the OS will cache it forever (or until reboot). >=20 > In other words, you can write your d_mmap() to give the appearance = that your=20 > /dev/foo "file" contains BAR 0 followed by BAR 1, etc. >=20 >>> static int >>> mydaq_mmap(...) >>> { >>>=20 >>> struct mydaq_softc *sc; // you should be storing things like = the >>> pointers >>> =09 >>> // to your BARs in a softc >>> =09 >>> sc =3D dev->si_drv1; >>> if (offset <=3D rman_get_size(sc->bar0res)) >>> =09 >>> return (rman_get_start(sc->bar0res) + offset); >>> =09 >>> offset -=3D rman_get_size(sc->bar0res); >>> if (offset <=3D rman_get_size(sc->bar1res)) >>> =09 >>> return (rman_get_start(sc->bar1res) + offset); >>> =09 >>> offset -=3D rman_get_size(sc->bar1res); >>> if (offset <=3D dmaAIBufSize) >>> =09 >>> return (vtophys(sc->dma[0]) + offset); >>> =09 >>> offset -=3D dmaAIBufSize; >>> if (offset <=3D dmaAOBufSize) >>> =09 >>> return (vtophys(sc->dma[1]) + offset); >>> =09 >>> /* Invalid offset */ >>> return (-1); >>>=20 >>> } >>=20 >> I guess, you hacked this quickly together, and I picked the concept, = so >> this is fine. In order to make this working, the addresses should be >> assigned to the *paddr parameter, and a status code should be = returned. In >> order to check things, of course I applied these minor corrections, = but it >> doesn't work for either the BAR block or the DMA block, depending on = which >> is mapped first. >=20 > Yes, it needs the changes you made. Can you post your updated = d_mmap() > routine along with how you are mapping it in userland? With what I = posted > above you would do something like this in userland: >=20 > const char *bar0, *bar1; >=20 > int fd =3D open("/dev/foo"); >=20 > bar0 =3D mmap(NULL, , PROT_READ, MAP_SHARED, fd, 0); > bar1 =3D mmap(NULL, , PROT_READ, MAP_SHARED, fd, > ); >=20 > etc. Yes, now I understood it. I realized, that mydaq_mmap() is not called = only once for each mapping, but once for each page in each mapping. And finally I = got it working. Besides the corrections already mentioned, the comparison = operators needed to be changed from "<=3D" to "<". The working d_mmap() is now: static int mydaq_mmap(struct cdev *dev, vm_ooffset_t offset, vm_paddr_t = *paddr, int nprot, vm_memattr_t *memattr) { mydaq_pci_softc *sc =3D dev->si_drv1; if (offset < rman_get_size(sc->bar0res)) { *paddr =3D rman_get_start(sc->bar0res) + offset; return 0; } offset -=3D rman_get_size(sc->bar0res); if (offset < rman_get_size(sc->bar1res)) { *paddr =3D rman_get_start(sc->bar1res) + offset; return 0; } offset -=3D rman_get_size(sc->bar1res); if (offset < dmaAIBufferSize) { *paddr =3D vtophys(sc->dma[0]) + offset; return 0; } offset -=3D dmaAIBufferSize; if (offset < dmaAOBufferSize) { *paddr =3D vtophys(sc->dma[1]) + offset; return 0; } return -1; } Thank you very much again, for the big help. Best regards Rolf= From owner-freebsd-drivers@FreeBSD.ORG Tue Jan 8 01:37:42 2013 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 52B93A9F for ; Tue, 8 Jan 2013 01:37:42 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-vb0-f46.google.com (mail-vb0-f46.google.com [209.85.212.46]) by mx1.freebsd.org (Postfix) with ESMTP id D52891AA for ; Tue, 8 Jan 2013 01:37:41 +0000 (UTC) Received: by mail-vb0-f46.google.com with SMTP id b13so20320960vby.33 for ; Mon, 07 Jan 2013 17:37:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=EhXwZ2k74PXm2/evf8OdqYos6hTaGOm2SQPiROpV4nw=; b=tkLzZyifXAFoGLT4JAdAHM7Vf4+aaS8oYwXvHbH/mknn7ZXSFZbmcaDkfIqg/5F7ap 1H0cWhtcr4jHn0jH6s8qzZFcGMf1zP2RGC+Kk0EzNV1hv3hyh7+fJTNKnpWskJRguSDR sW/kj7+oPHlae0wPR/LXZJPrtcaeiNUrnxYk8+kzwfurUKQzNEdlQAld+r5ZfnzBaIVU 5wc6JVwKTYUQGwpkTf9gV6U6RV5AoADywG3niTSBPnaKgaApboKR6BdvpYRBwngmcLj4 18VmwPR8J7+T73h6zCxhq709jtsZB0Dbvnccgcex1UjUKnQL5qqUrlRseRhSeIHGstNQ 1kug== MIME-Version: 1.0 Received: by 10.52.93.77 with SMTP id cs13mr52386971vdb.118.1357609060709; Mon, 07 Jan 2013 17:37:40 -0800 (PST) Received: by 10.58.214.226 with HTTP; Mon, 7 Jan 2013 17:37:40 -0800 (PST) In-Reply-To: <8BDABFD4-D371-49D4-B21C-EBCD7C86B47C@cyclaero.com> References: <201301071011.49234.jhb@freebsd.org> <201301071421.10559.jhb@freebsd.org> <8BDABFD4-D371-49D4-B21C-EBCD7C86B47C@cyclaero.com> Date: Mon, 7 Jan 2013 17:37:40 -0800 Message-ID: Subject: Re: How to map device addresses into user space From: Mehmet Erol Sanliturk To: "Dr. Rolf Jansen" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-drivers@freebsd.org X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 01:37:42 -0000 On Mon, Jan 7, 2013 at 3:40 PM, Dr. Rolf Jansen wrote: > Am 07.01.2013 um 17:21 schrieb John Baldwin: > > > On Monday, January 07, 2013 12:25:33 PM Dr. Rolf Jansen wrote: > >> Am 07.01.2013 um 13:11 schrieb John Baldwin: > >>> On Monday, January 07, 2013 08:52:01 AM Dr. Rolf Jansen wrote: > >>>> Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen: > >>>>> ... > >>>>> > >>>>> So, how can I map device addresses into user space on FreeBSD? > >>>> > >>>> Many Thanks to everybody, who responded. > >>>> > >>>> I got it working. I wrote a PCI device driver, which in addition to > >>>> open/close/read/write responds also to ioctl and mmap requests from > user > >>>> space. In its pci_attach routine, it assembles a custom address_space > >>>> struct containing the addresses of the two BAR's of my NI PCI-DAQ > board > >>>> and two huge allocated memory regions for DMA... > >>> > >>> Note that if your BARs are not contiguous you can allow mapping of > things > >>> in the middle. This is a potentital security hole if your driver can > be > >>> opened by non-root. > >> > >> I can understand this. Actually, on the present model of the DAQ board, > the > >> BARs are contiguous, and in the present incarnation, my driver can only > be > >> addressed by root. Anyway, I want to do it correctly, that means, I > would > >> like to avoid right now potential holes bubbling up in the future, when > >> using my driver with other DAQ boards, and in different usage scenarios. > > > > Note that the BARs being contiguous is a property of your BIOS or OS, > not your > > board. :) The firmware/OS are free to assign whatever ranges to your > BARs > > that they wish. > > > >>> A more robust approach is to define a virtual address space for > >>> your device. This is also a lot easier to program against in userland. > >>> For example, you could define offset 0 as mapping to BAR 0 and the next > >>> chunk > >> > >>> of virtual address space (the offset) map to BAR 1, etc.: > >> I did this, but id didn't work out. The 2 BARs are contiguous, and the 2 > >> DMA regions are also, but there is huge gap between BAR and DMA. And > >> depending on which mapping is done first, either the DMAs or the BARs > >> receive invalid paddr. So, I guess, defining a virtual address space is > a > >> little bit more involved than simply getting the offsets straight as you > >> lined out. > >> > >> I am still learning things, so please bear with me, if it is a dumb > >> question. How do I define a virtual address space for the device? > > > > Ah, I'll explain a bit. When you mmap a section of a file, the OS is > treating > > the file as if it were its own section of virtual memory, where files > contents > > were addressed by the offset. That is, virtual address 0 of the file > > corresponds to the data at file offset 0, and virtual address N > corresponds to > > the data at file offset N. > > > > Now, think of your /dev/foo device as a file. When you mmap() your > /dev/foo, > > the OS has assumes it has a similar virtual memory that is addressed by > the > > offset into the file. However, for a character device, the OS needs the > > driver to describe this virtual address space. That is what you are > doing > > with d_mmap() and the 'offset' parameter. You are telling it what > physical > > page backs each "virtual page". It is important to note that when using > > d_mmap(), these mappings are immutable. Once you provide a mapping for a > > given virtual page, the OS will cache it forever (or until reboot). > > > > In other words, you can write your d_mmap() to give the appearance that > your > > /dev/foo "file" contains BAR 0 followed by BAR 1, etc. > > > >>> static int > >>> mydaq_mmap(...) > >>> { > >>> > >>> struct mydaq_softc *sc; // you should be storing things like the > >>> pointers > >>> > >>> // to your BARs in a softc > >>> > >>> sc = dev->si_drv1; > >>> if (offset <= rman_get_size(sc->bar0res)) > >>> > >>> return (rman_get_start(sc->bar0res) + offset); > >>> > >>> offset -= rman_get_size(sc->bar0res); > >>> if (offset <= rman_get_size(sc->bar1res)) > >>> > >>> return (rman_get_start(sc->bar1res) + offset); > >>> > >>> offset -= rman_get_size(sc->bar1res); > >>> if (offset <= dmaAIBufSize) > >>> > >>> return (vtophys(sc->dma[0]) + offset); > >>> > >>> offset -= dmaAIBufSize; > >>> if (offset <= dmaAOBufSize) > >>> > >>> return (vtophys(sc->dma[1]) + offset); > >>> > >>> /* Invalid offset */ > >>> return (-1); > >>> > >>> } > >> > >> I guess, you hacked this quickly together, and I picked the concept, so > >> this is fine. In order to make this working, the addresses should be > >> assigned to the *paddr parameter, and a status code should be returned. > In > >> order to check things, of course I applied these minor corrections, but > it > >> doesn't work for either the BAR block or the DMA block, depending on > which > >> is mapped first. > > > > Yes, it needs the changes you made. Can you post your updated d_mmap() > > routine along with how you are mapping it in userland? With what I > posted > > above you would do something like this in userland: > > > > const char *bar0, *bar1; > > > > int fd = open("/dev/foo"); > > > > bar0 = mmap(NULL, , PROT_READ, MAP_SHARED, fd, 0); > > bar1 = mmap(NULL, , PROT_READ, MAP_SHARED, fd, > > ); > > > > etc. > > > Yes, now I understood it. I realized, that mydaq_mmap() is not called only > once > for each mapping, but once for each page in each mapping. And finally I > got it > working. Besides the corrections already mentioned, the comparison > operators > needed to be changed from "<=" to "<". The working d_mmap() is now: > > > static int mydaq_mmap(struct cdev *dev, vm_ooffset_t offset, vm_paddr_t > *paddr, > int nprot, vm_memattr_t *memattr) > { > mydaq_pci_softc *sc = dev->si_drv1; > > if (offset < rman_get_size(sc->bar0res)) > { > *paddr = rman_get_start(sc->bar0res) + offset; > return 0; > } > > offset -= rman_get_size(sc->bar0res); > if (offset < rman_get_size(sc->bar1res)) > { > *paddr = rman_get_start(sc->bar1res) + offset; > return 0; > } > > offset -= rman_get_size(sc->bar1res); > if (offset < dmaAIBufferSize) > { > *paddr = vtophys(sc->dma[0]) + offset; > return 0; > } > > offset -= dmaAIBufferSize; > if (offset < dmaAOBufferSize) > { > *paddr = vtophys(sc->dma[1]) + offset; > return 0; > } > > return -1; > } > > > Thank you very much again, for the big help. > > Best regards > > Rolf > _______________________________________________ > > If it is suitable for you , is it possible to supply your sources with a permissive license for using them / studying them and learning how to develop such a software ? Thank you very much . Mehmet Erol Sanliturk From owner-freebsd-drivers@FreeBSD.ORG Tue Jan 8 18:42:11 2013 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 9D34872B for ; Tue, 8 Jan 2013 18:42:11 +0000 (UTC) (envelope-from rj@cyclaero.com) Received: from mo6-p00-ob.rzone.de (mo6-p00-ob.rzone.de [IPv6:2a01:238:20a:202:5300::1]) by mx1.freebsd.org (Postfix) with ESMTP id 3787A31D for ; Tue, 8 Jan 2013 18:42:10 +0000 (UTC) X-RZG-AUTH: :O2kGeEG7b/pS1E6gSHOyjPKyNsg/5l1He+DgCy9/8FSej6CwUysqfN3Dd+qW8pifIOtG X-RZG-CLASS-ID: mo00 Received: from rolf.projectworld.net (b150f525.virtua.com.br [177.80.245.37]) by smtp.strato.de (joses mo47) (RZmta 31.12 DYNA|AUTH) with (AES128-SHA encrypted) ESMTPA id K00516p08HIvkM ; Tue, 8 Jan 2013 19:42:06 +0100 (CET) Subject: Re: How to map device addresses into user space Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: "Dr. Rolf Jansen" In-Reply-To: Date: Tue, 8 Jan 2013 16:42:01 -0200 Content-Transfer-Encoding: 7bit Message-Id: <785A9EEB-D670-46CD-80A3-763A9F745131@cyclaero.com> References: <201301071011.49234.jhb@freebsd.org> <201301071421.10559.jhb@freebsd.org> <8BDABFD4-D371-49D4-B21C-EBCD7C86B47C@cyclaero.com> To: freebsd-drivers@freebsd.org X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jan 2013 18:42:11 -0000 Am 07.01.2013 um 23:37 schrieb Mehmet Erol Sanliturk: > On Mon, Jan 7, 2013 at 3:40 PM, Dr. Rolf Jansen wrote: > >> ... And finally I got it working .... > > If it is suitable for you , is it possible to supply your sources > with a permissive license for using them / studying them and learning > how to develop such a software? I took the PCI Device example from the "FreeBSD Architecture Handbook" as the base, and I added the necessary functionality for BAR and other memory mapping. I created also a user-sapce tool, which connects to the PCI device driver by ioctl() and mmap(), and prints some results. I prepared a .zip-Archive containing these files, and a Makefile for the device driver. Enhanced sources for mypci device driver: Original sources for mypci device dirver: If you have questions, then please don't hesitate to send a note to this thread on this list. Best regards Rolf