Date: Thu, 17 May 2012 14:07:28 -0600 From: Ian Lepore <freebsd@damnhippie.dyndns.org> To: Svatopluk Kraus <onwahe@gmail.com> Cc: freebsd-arm@freebsd.org, hackers@freebsd.org Subject: Re: ARM + CACHE_LINE_SIZE + DMA Message-ID: <1337285248.1503.308.camel@revolution.hippie.lan> In-Reply-To: <CAFHCsPUdZXGKFvmVGgaEUsfhwd28mNVGaY84ExcJp=ogQxzPJQ@mail.gmail.com> References: <CAFHCsPUdZXGKFvmVGgaEUsfhwd28mNVGaY84ExcJp=ogQxzPJQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, 2012-05-17 at 15:20 +0200, Svatopluk Kraus wrote:
> Hi,
>
> I'm working on DMA bus implementation for ARM11mpcore platform. I've
> looked at implementation in ARM tree, but IMHO it only works with some
> assumptions. There is a problem with DMA on memory block which is not
> aligned on CACHE_LINE_SIZE (start and end) if memory is not coherent.
>
> Let's have a buffer for DMA which is no aligned on CACHE_LINE_SIZE.
> Then first cache line associated with the buffer can be divided into
> two parts, A and B, where A is a memory we know nothing about it and B
> is buffer memory. The same stands for last cache line associatted with
> the buffer. We have no problem if a memory is coherent. Otherwise it
> depends on memory attributes.
>
> 1. [no cache] attribute
> No problem as memory is coherent.
>
> 2. [write throught] attribute
> The part A can be invalidated without loss of any data. It's not problem too.
>
> 3. [write back] attribute
> In general, there is no way how to keep both parts consistent. At the
> start of DMA transaction, the cache line is written back and
> invalidated. However, as we know nothing about memory associated with
> part A of the cache line, the cache line can be filled again at any
> time and messing up DMA transaction if flushed. Even if the cache line
> is only filled but not flushed during DMA transaction, we must make it
> coherent with memory after that. There is a trick with saving part A
> of the line into temporary buffer, invalidating the line, and
> restoring part A in current ARM (MIPS) implementation. However, if
> somebody is writting to memory associated with part A of the line
> during this trick, the part A will be messed up. Moreover, the part A
> can be part of another DMA transaction.
>
> To safely use DMA with no coherent memory, a memory with [no cache] or
> [write throught] attributes can be used without problem. A memory with
> [write back] attribute must be aligned on CACHE_LINE_SIZE.
>
> However, for example mbuf, a buffer for DMA can be part of a structure
> which can be aligned on CACHE_LINE_SIZE, but not the buffer itself. We
> can know that nobody will write to the structure during DMA
> transaction, so it's safe to use the buffer event if it's not aligned
> on CACHE_LINE_SIZE.
>
> So, in practice, if DMA buffer is not aligned on CACHE_LINE_SIZE and
> we want to avoid bounce pages overhead, we must support additional
> information to DMA transaction. It should be easy to support the
> information about drivers data buffers. However, what about OS data
> buffers like mentioned mbufs?
>
> The question is following. Is or can be guaranteed for all or at least
> well-known OS data buffers which can be part of DMA access that the
> not CACHE_LINE_SIZE aligned buffers are surrounded by data which
> belongs to the same object as the buffer and the data is not written
> by OS when given to a driver?
>
> Any answer is appreciated. However, 'bounce pages' is not an answer.
>
> Thanks, Svata
I'm adding freebsd-arm@ to the CC list; that's where this has been
discussed before.
Your analysis is correct... to the degree that it works at all right
now, it's working by accident. At work we've been making the good
accident a bit more likely by setting the minimum allocation size to
arm_dcache_align in kern_malloc.c. This makes it somewhat less likely
that unrelated objects in the kernel are sharing a cache line, but it
also reduces the effectiveness of the cache somewhat.
Another factor, not mentioned in your analysis, is the size of the IO
operation. Even if the beginning of the DMA buffer is cache-aligned, if
the size isn't exactly a multiple of the cache line size you still have
the partial flush situation and all of its problems.
It's not guaranteed that data surrounding a DMA buffer will be untouched
during the DMA, even when that surrounding data is part of the same
conceptual object as the IO buffer. It's most often true, but certainly
not guaranteed. In addition, as Mark pointed out in a prior reply,
sometimes the DMA buffer is on the stack, and even returning from the
function that starts the IO operation affects the cacheline associated
with the DMA buffer. Consider something like this:
void do_io()
{
int buffer;
start_read(&buffer);
// maybe do other stuff here
wait_for_read_done();
}
start_read() gets some IO going, so before it returns a call has been
made to bus_dmamap_sync(..., BUS_DMASYNC_PREREAD) and an invalidate gets
done on the cacheline containing the variable 'buffer'. The act of
returning from the start_read() function causes that cacheline to get
reloaded, so now the stale pre-DMA value of the variable 'buffer' is in
cache again. Right after that, the DMA completes so that ram has a
newer value that belongs in the buffer variable and the copy in the
cacheline is stale.
Before control gets into the wait_for_read_done() routine that will
attempt to handle the POSTREAD partial cacheline flush, another thread
gets control and begins setting up a new DMA for another device,
different buffer. This time it's a read using a 64K buffer for disk IO.
The busdma sync code calls cpu_dcache_inv_range() for that buffer but
because the range is so large, it gets turned into a
cpu_dcache_wbinv_all() because that's cheaper than looping through an
arbitrarily large range invalidating a line at a time.
Except, ooops, that means we write back to ram the cacheline holding the
stale value of the 'buffer' variable, wiping out the data brought in by
DMA before the partial cachline flush code could do its dance to
preserve it.
There are several variations of the above scenario; it doesn't require a
stack-allocated buffer to trigger a writeback of a stale value. Any
cacheline that gets dirtied after a PREREAD invalidate can end up
overwriting fresh DMA data in ram with stale data from the cacheline at
any time, because any call to cpu_dcache_inv_range() or
cpu_dcache_wbinv_range() can get turned into a cpu_dcache_wbinv_all().
That means any DMA operation and also a context switch which calls
cpu_dcache_wbinv_all().
If you rule out bounce buffers as a solution, then I think that may
leave us with just one option: make the pages uncacheable for the
duration of the DMA. Essentially force a DMA_COHERENT buffer if the
driver didn't already do so. I think doing so blindly may have
performance implications every bit as bad as using bounce buffers. (For
example, turning off cache on a stack page could really hurt.) The code
to do the remapping already exists in pmap.c as part of handling
multiple mappings for VIVT caches. From the busdma code you could
accomplish the remapping by making a temporary writable kernel mapping
for the buffer pages in the PRE handling and undo that mapping in the
POST handling.
It may be that a hybrid approach would work. For an unaligned buffer,
if it isn't already DMA_COHERENT, then if it is below a certain size
bounce it, otherwise remap the buffer's pages. When I was knee-deep in
this problem last summer one of the things I noticed in our systems was
that large DMA operations (1 KByte or larger buffers) tend to be
DMA_COHERENT buffers, and when not they're already cache aligned and a
power of two in size. For us the partial cacheline flush situations are
almost always caused by tiny IO of 1 to 128 bytes length, usually
serial-comms or usb related.
I think pre-allocating a few pages for bouncing small unaligned IO would
be a big win compared to remapping the pages as uncacheable. The
remapping has to take locks and search lists of pages and so on; it
should be way faster to do a small memcpy() instead. Buffers bigger
than some (perhaps tunable) limit would get remapped instead of
bounced.
It also might be nice to have a knob to enable logging when bouncing or
remapping is used to avoid partial cacheline operations, to make it easy
to find drivers that could be tweaked for better performance. If you're
bouncing 2 or 3 operations per second with a 4-byte buffer that's no big
deal. If every network packet is resulting in bouncing/remapping you'd
want to know about that.
-- Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1337285248.1503.308.camel>