Date: Fri, 27 Jun 2003 17:52:29 -0400 (EDT) From: Andrew Gallatin <gallatin@cs.duke.edu> To: Scott Long <scott_long@btc.adaptec.com> Cc: freebsd-arch@freebsd.org Subject: Re: API change for bus_dma Message-ID: <16124.48285.343025.428957@grasshopper.cs.duke.edu> In-Reply-To: <3EFCB725.4060902@btc.adaptec.com> References: <3EF3C12F.9060303@btc.adaptec.com> <16124.39930.142492.356163@grasshopper.cs.duke.edu> <3EFC9F2D.6020908@btc.adaptec.com> <16124.43999.333761.397624@grasshopper.cs.duke.edu> <3EFCAC7A.6060305@btc.adaptec.com> <16124.45051.919899.414795@grasshopper.cs.duke.edu> <3EFCB178.9030207@btc.adaptec.com> <16124.46454.595892.860118@grasshopper.cs.duke.edu> <3EFCB725.4060902@btc.adaptec.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Scott Long writes:
> >
> > Unfortunately, in our application we must assume coherency in some
> > situations. We have kernel memory mmap'ed into user space for
> > zero-copy io of small messages. Doing a system call to force the dma
> > sync would add unacceptable latency. (we're talking sub 10us latencies
> > here, without syscalls).
> >
>
> The bus_dmamap_sync() isn't done from a separate system call. The flow
> is this:
>
>
> bus_dmamap_load();
> driver_callback()
> {
> set up S/G list;
> bus_dmamap_sync(PREWRITE);
> tell hardware that DMA is ready;
> }
>
> The callback gets called immediately as long as conditions are met, as
> we have discuss prior.
>
> Then:
>
> driver_intr()
> {
> see that hardware has DMA'd data to us;
> bus_dmamap_sync(POSTREAD);
> bus_dmamap_unload();
> }
In our application, millions of separate DMAs can happen with no kernel
intervention at all.
We do the bus_dmamap_load() in the context of an ioctl which allocates
some kernel memory, and pokes the DMA addresses for the kernel memory
down onto the board. The user then mmaps the kernel memory, and also
mmaps a page of SRAM on our board.
When the user wants to initate a dma, he writes to the device's SRAM
indicating an offset in the mmaped kernel memory. The board then
initates the transfer (after doing bounds checking..).
When the user is done, or the process exits, the dma addresses are
cleared from the nic, and the kernel memory is freed.
This is OS-bypass networking.
> As I understand it, it is possible to set the pycho bridge to use
> a coherent address range, but FreeBSD doesn't take advantage of that
> yet.
>
Yes, that's what solaris does..
Drew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?16124.48285.343025.428957>