Date: Thu, 17 May 2012 11:01:34 -0500 From: Mark Tinguely <marktinguely@gmail.com> To: Svatopluk Kraus <onwahe@gmail.com> Cc: hackers@freebsd.org Subject: Re: ARM + CACHE_LINE_SIZE + DMA Message-ID: <CAP%2BM-_GbAnAZzJaa=diGACGuYGeSo6zqD-CBbiOL61vw%2B1eJEg@mail.gmail.com> In-Reply-To: <CAFHCsPUdZXGKFvmVGgaEUsfhwd28mNVGaY84ExcJp=ogQxzPJQ@mail.gmail.com> References: <CAFHCsPUdZXGKFvmVGgaEUsfhwd28mNVGaY84ExcJp=ogQxzPJQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, May 17, 2012 at 8:20 AM, Svatopluk Kraus <onwahe@gmail.com> wrote: > Hi, > > I'm working on DMA bus implementation for ARM11mpcore platform. I've > looked at implementation in ARM tree, but IMHO it only works with some > assumptions. There is a problem with DMA on memory block which is not > aligned on CACHE_LINE_SIZE (start and end) if memory is not coherent. > > Let's have a buffer for DMA which is no aligned on CACHE_LINE_SIZE. > Then first cache line associated with the buffer can be divided into > two parts, A and B, where A is a memory we know nothing about it and B > is buffer memory. The same stands for last cache line associatted with > the buffer. We have no problem if a memory is coherent. Otherwise it > depends on memory attributes. > > 1. [no cache] attribute > No problem as memory is coherent. > > 2. [write throught] attribute > The part A can be invalidated without loss of any data. It's not problem too. > > 3. [write back] attribute > In general, there is no way how to keep both parts consistent. At the > start of DMA transaction, the cache line is written back and > invalidated. However, as we know nothing about memory associated with > part A of the cache line, the cache line can be filled again at any > time and messing up DMA transaction if flushed. Even if the cache line > is only filled but not flushed during DMA transaction, we must make it > coherent with memory after that. There is a trick with saving part A > of the line into temporary buffer, invalidating the line, and > restoring part A in current ARM (MIPS) implementation. However, if > somebody is writting to memory associated with part A of the line > during this trick, the part A will be messed up. Moreover, the part A > can be part of another DMA transaction. > > To safely use DMA with no coherent memory, a memory with [no cache] or > [write throught] attributes can be used without problem. A memory with > [write back] attribute must be aligned on CACHE_LINE_SIZE. > > However, for example mbuf, a buffer for DMA can be part of a structure > which can be aligned on CACHE_LINE_SIZE, but not the buffer itself. We > can know that nobody will write to the structure during DMA > transaction, so it's safe to use the buffer event if it's not aligned > on CACHE_LINE_SIZE. > > So, in practice, if DMA buffer is not aligned on CACHE_LINE_SIZE and > we want to avoid bounce pages overhead, we must support additional > information to DMA transaction. It should be easy to support the > information about drivers data buffers. However, what about OS data > buffers like mentioned mbufs? > > The question is following. Is or can be guaranteed for all or at least > well-known OS data buffers which can be part of DMA access that the > not CACHE_LINE_SIZE aligned buffers are surrounded by data which > belongs to the same object as the buffer and the data is not written > by OS when given to a driver? > > Any answer is appreciated. However, 'bounce pages' is not an answer. > > Thanks, Svata Sigh. A several ideas by several people, but a good answer has not been created yet. SMP will make this worse. To make things worse, there are drivers that use memory from the stack as DMA buffers. I was hoping that mbufs are pretty well self-contained, unless you expect to modify them while DMA is happening. This is on my to-do list. --Mark.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAP%2BM-_GbAnAZzJaa=diGACGuYGeSo6zqD-CBbiOL61vw%2B1eJEg>