From owner-freebsd-drivers@FreeBSD.ORG Mon Jan 7 23:40:19 2013 Return-Path: Delivered-To: freebsd-drivers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id ADB92ECA; Mon, 7 Jan 2013 23:40:19 +0000 (UTC) (envelope-from rj@cyclaero.com) Received: from mo6-p00-ob.rzone.de (mo6-p00-ob.rzone.de [IPv6:2a01:238:20a:202:5300::1]) by mx1.freebsd.org (Postfix) with ESMTP id 20AE8D5E; Mon, 7 Jan 2013 23:40:18 +0000 (UTC) X-RZG-AUTH: :O2kGeEG7b/pS1E6gSHOyjPKyNsg/5l1He+DgCy9/8FSej6CwUysqfN3Dd+qW8pifIOtG X-RZG-CLASS-ID: mo00 Received: from rolf.projectworld.net (b150f525.virtua.com.br [177.80.245.37]) by smtp.strato.de (jored mo17) (RZmta 31.11 DYNA|AUTH) with (AES128-SHA encrypted) ESMTPA id w06b3bp07Lo5Px ; Tue, 8 Jan 2013 00:40:14 +0100 (CET) Subject: Re: How to map device addresses into user space Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=iso-8859-1 From: "Dr. Rolf Jansen" In-Reply-To: <201301071421.10559.jhb@freebsd.org> Date: Mon, 7 Jan 2013 21:40:11 -0200 Content-Transfer-Encoding: quoted-printable Message-Id: <8BDABFD4-D371-49D4-B21C-EBCD7C86B47C@cyclaero.com> References: <201301071011.49234.jhb@freebsd.org> <201301071421.10559.jhb@freebsd.org> To: freebsd-drivers@freebsd.org X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 23:40:19 -0000 Am 07.01.2013 um 17:21 schrieb John Baldwin: > On Monday, January 07, 2013 12:25:33 PM Dr. Rolf Jansen wrote: >> Am 07.01.2013 um 13:11 schrieb John Baldwin: >>> On Monday, January 07, 2013 08:52:01 AM Dr. Rolf Jansen wrote: >>>> Am 03.01.2013 um 14:45 schrieb Dr. Rolf Jansen: >>>>> ... >>>>>=20 >>>>> So, how can I map device addresses into user space on FreeBSD? >>>>=20 >>>> Many Thanks to everybody, who responded. >>>>=20 >>>> I got it working. I wrote a PCI device driver, which in addition to >>>> open/close/read/write responds also to ioctl and mmap requests from = user >>>> space. In its pci_attach routine, it assembles a custom = address_space >>>> struct containing the addresses of the two BAR's of my NI PCI-DAQ = board >>>> and two huge allocated memory regions for DMA... >>>=20 >>> Note that if your BARs are not contiguous you can allow mapping of = things >>> in the middle. This is a potentital security hole if your driver = can be >>> opened by non-root. >>=20 >> I can understand this. Actually, on the present model of the DAQ = board, the >> BARs are contiguous, and in the present incarnation, my driver can = only be >> addressed by root. Anyway, I want to do it correctly, that means, I = would >> like to avoid right now potential holes bubbling up in the future, = when >> using my driver with other DAQ boards, and in different usage = scenarios. >=20 > Note that the BARs being contiguous is a property of your BIOS or OS, = not your=20 > board. :) The firmware/OS are free to assign whatever ranges to your = BARs=20 > that they wish. >=20 >>> A more robust approach is to define a virtual address space for >>> your device. This is also a lot easier to program against in = userland. >>> For example, you could define offset 0 as mapping to BAR 0 and the = next >>> chunk >>=20 >>> of virtual address space (the offset) map to BAR 1, etc.: >> I did this, but id didn't work out. The 2 BARs are contiguous, and = the 2 >> DMA regions are also, but there is huge gap between BAR and DMA. And >> depending on which mapping is done first, either the DMAs or the BARs >> receive invalid paddr. So, I guess, defining a virtual address space = is a >> little bit more involved than simply getting the offsets straight as = you >> lined out. >>=20 >> I am still learning things, so please bear with me, if it is a dumb >> question. How do I define a virtual address space for the device? >=20 > Ah, I'll explain a bit. When you mmap a section of a file, the OS is = treating=20 > the file as if it were its own section of virtual memory, where files = contents=20 > were addressed by the offset. That is, virtual address 0 of the file=20= > corresponds to the data at file offset 0, and virtual address N = corresponds to=20 > the data at file offset N. >=20 > Now, think of your /dev/foo device as a file. When you mmap() your = /dev/foo,=20 > the OS has assumes it has a similar virtual memory that is addressed = by the=20 > offset into the file. However, for a character device, the OS needs = the=20 > driver to describe this virtual address space. That is what you are = doing=20 > with d_mmap() and the 'offset' parameter. You are telling it what = physical=20 > page backs each "virtual page". It is important to note that when = using=20 > d_mmap(), these mappings are immutable. Once you provide a mapping = for a=20 > given virtual page, the OS will cache it forever (or until reboot). >=20 > In other words, you can write your d_mmap() to give the appearance = that your=20 > /dev/foo "file" contains BAR 0 followed by BAR 1, etc. >=20 >>> static int >>> mydaq_mmap(...) >>> { >>>=20 >>> struct mydaq_softc *sc; // you should be storing things like = the >>> pointers >>> =09 >>> // to your BARs in a softc >>> =09 >>> sc =3D dev->si_drv1; >>> if (offset <=3D rman_get_size(sc->bar0res)) >>> =09 >>> return (rman_get_start(sc->bar0res) + offset); >>> =09 >>> offset -=3D rman_get_size(sc->bar0res); >>> if (offset <=3D rman_get_size(sc->bar1res)) >>> =09 >>> return (rman_get_start(sc->bar1res) + offset); >>> =09 >>> offset -=3D rman_get_size(sc->bar1res); >>> if (offset <=3D dmaAIBufSize) >>> =09 >>> return (vtophys(sc->dma[0]) + offset); >>> =09 >>> offset -=3D dmaAIBufSize; >>> if (offset <=3D dmaAOBufSize) >>> =09 >>> return (vtophys(sc->dma[1]) + offset); >>> =09 >>> /* Invalid offset */ >>> return (-1); >>>=20 >>> } >>=20 >> I guess, you hacked this quickly together, and I picked the concept, = so >> this is fine. In order to make this working, the addresses should be >> assigned to the *paddr parameter, and a status code should be = returned. In >> order to check things, of course I applied these minor corrections, = but it >> doesn't work for either the BAR block or the DMA block, depending on = which >> is mapped first. >=20 > Yes, it needs the changes you made. Can you post your updated = d_mmap() > routine along with how you are mapping it in userland? With what I = posted > above you would do something like this in userland: >=20 > const char *bar0, *bar1; >=20 > int fd =3D open("/dev/foo"); >=20 > bar0 =3D mmap(NULL, , PROT_READ, MAP_SHARED, fd, 0); > bar1 =3D mmap(NULL, , PROT_READ, MAP_SHARED, fd, > ); >=20 > etc. Yes, now I understood it. I realized, that mydaq_mmap() is not called = only once for each mapping, but once for each page in each mapping. And finally I = got it working. Besides the corrections already mentioned, the comparison = operators needed to be changed from "<=3D" to "<". The working d_mmap() is now: static int mydaq_mmap(struct cdev *dev, vm_ooffset_t offset, vm_paddr_t = *paddr, int nprot, vm_memattr_t *memattr) { mydaq_pci_softc *sc =3D dev->si_drv1; if (offset < rman_get_size(sc->bar0res)) { *paddr =3D rman_get_start(sc->bar0res) + offset; return 0; } offset -=3D rman_get_size(sc->bar0res); if (offset < rman_get_size(sc->bar1res)) { *paddr =3D rman_get_start(sc->bar1res) + offset; return 0; } offset -=3D rman_get_size(sc->bar1res); if (offset < dmaAIBufferSize) { *paddr =3D vtophys(sc->dma[0]) + offset; return 0; } offset -=3D dmaAIBufferSize; if (offset < dmaAOBufferSize) { *paddr =3D vtophys(sc->dma[1]) + offset; return 0; } return -1; } Thank you very much again, for the big help. Best regards Rolf=