Date: Mon, 27 Aug 2012 09:32:37 -0600 From: Ian Lepore <freebsd@damnhippie.dyndns.org> To: Warner Losh <imp@bsdimp.com> Cc: Petter Selasky <hans.petter.selasky@bitfrost.no>, freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org, Hans Subject: Re: Partial cacheline flush problems on ARM and MIPS Message-ID: <1346081557.1140.181.camel@revolution.hippie.lan> In-Reply-To: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <FD8DC82C-AD3B-4EBC-A625-62A37B9ECBF1@bsdimp.com> <1345765503.27688.602.camel@revolution.hippie.lan> <CAJ-VmonOwgR7TNuYGtTOhAbgz-opti_MRJgc8G%2BB9xB3NvPFJQ@mail.gmail.com> <1345766109.27688.606.camel@revolution.hippie.lan> <CAJ-VmomFhqV5rTDf-kKQfbSuW7SSiSnqPEjGPtxWjaHFA046kQ@mail.gmail.com> <F8C9E811-8597-4ED0-9F9D-786EB2301D6F@bsdimp.com> <1346002922.1140.56.camel@revolution.hippie.lan> <CAP%2BM-_HZ4yARwZA2koPJDeJWHT-1LORupjymuVnMtLBzeXe=DA@mail.gmail.com> <1346005507.1140.69.camel@revolution.hippie.lan> <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, 2012-08-26 at 17:13 -0600, Warner Losh wrote:
> On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote:
> > In this regard, it's the busdma implementation that's broken, because it
> > should bounce those IOs through a DMA-safe buffer. There's absolutely
> > no rule that I've ever heard of in FreeBSD that says IO can only take
> > place using memory allocated from busdma.
>
> That's partially true. Since BUSDMA grew up in the storage area, you must allocate the memory from busdma, or it must be page aligned has been the de-facto rule here.
Where does anything say that you must allocate the memory from busdma if
it's not mbuf/page-aligned? I've never seen it in any docs. I
certainly find contrary evidence in existing driver code (and I'm not
talking about USB here).
> The mbuf and uio variants of load were invented to cope with common cases of mbufs and user I/O to properly flag things.
>
What does that mean, "to properly flag things"?
I think with uio we come to the crux of the issue. Userland buffers are
not allocated with knowledge of the busdma constraints. That leads to
some choices:
* We don't support IO with a userland buffer at all.
* We support userland IO if the userland buffers are accidentally
aligned properly for the platform, otherwise the call fails.
* We support userland buffers by making any driver which wants to
do so responsible for always copying the data to aligned buffers
(each individual driver handles bounces).
* We support userland buffers by having the busdma layer handle
the bouncing when required.
The first two seem untenable to me. The third one comes down to "every
driver must always bounce all userland data" because the driver doesn't
actually have access to the info to decide whether a userland buffer is
aligned properly or not for a given platform.
That leaves the option where the busdma layer handles bouncing for
unaligned userland buffers. If that's possible, then the busdma layer
could automatically bounce any unaligned request, whether it came from
userland or a driver reading a status response into an unaligned local
buffer in kernel memory.
> How does busdma know that it is using memory that's not from its allocator?
The busdma code allocates the map along with the buffer, and can record
information in the map that it can use during mapping and sync
operations to know whether it allocated the buffer.
> > The rule is only that the
> > proper sequence of busdma operation must be called, and beyond that it's
> > up to the busdma implementation to make it work.
>
> No. Bouncing is needed due to poor alignment of the underlying device. Not due to cache effects.
Says who? This statement seems to me to be more about opinion and dogma
than about a documented API or technical concerns about how to implement
it. IMO, bouncing is needed when that's the only way to make a
particular busdma sequence work correctly given the parameters of the
hardware and the transfer.
To put it another way, the busdma subsystem is responsible for helping
make DMA transfers work on a platform without every driver needing to
contain platform-specific code, and bouncing is one of the techniques
available to it.
> There's a limited number of things that we support with busdma. Arbitrary data from malloc that might be shared with the CPU isn't on that list.
>
Where are those limitations documented?
This isn't some small oversight in the documention, if the rule is "IO,
any IO at all, can only be performed using buffers allocated from
busdma" that's a pretty major thing that ought to appear in multiple
manpages relating to driver development.
If you don't think "any IO at all" is the right characterization, read
all the way to the end before responding.
> > Our biggest problem, I think, is that we don't have a sufficient
> > definition of "the proper sequence of busdma operations."
>
> I disagree. The sequence has been known for a long time.
>
> > I don't think it will be very hard to make the arm and mips busdma
> > implementations work correctly. It won't even be too hard to make them
> > fairly efficient at bouncing small IOs (my thinking is that we can make
> > small bounces no more expensive than the current partial cacheline flush
> > implementation which copies the data multiple times). Bouncing large IO
> > will never be efficient, but the inefficiency will be a powerful
> > motivator to update drivers that do large IO to work better, such as
> > using buffers allocated from busdma.
>
> I don't think the cache line problem can be solved with bounce buffers. Trying to accommodate broken drivers is what lead us to this spot. We need to fix the broken drivers. If that's impossible, then the best we can do is have the driver set a 'always bounce' flag in the tag it creates and use that to always bounce for operations through that tag.
So you'd be okay with a driver setting a flag that says "always bounce"
but not okay with the busdma layer bouncing only when it's actually
necessary? I'm confused -- do you think the busdma layer will be unable
to detect when it's necessary unless directed from the outside?
Let me pose a question back to you... if it is up to a driver to
allocate DMA buffers using the busdma allocator, how is any given driver
to know whether DMA is going to be involved in the transfer?
Don't think USB or ATA here... think iicbus/foo_temperature_sensor, or
the mmc/sd driver. Those drivers know nothing about the hardware bridge
layers beneath them and whether DMA is involved or not. They just know
"I need to read a 2-byte value from a register on this IIC chip" or "I
need to retrieve the 32-bit CSD data from the SD card" and they
accomplish that by calling a read method of some lower-level bridge
driver. In each of those cases we have both PIO and DMA implementations
of the lower-level drivers that move bits over a wire.
If the concensus is that such drivers should always allocate all IO
buffers using busdma, even though they have no idea whether DMA is going
to be involved, then we certainly have a lot of work ahead of us in
terms of "fixing broken drivers." That also implies that
bus_dmamem_alloc() is misnamed and implemented in the wrong place,
because it's not really about dma at all.
If you push the responsibility down to the layer where it is known
whether DMA is going to be involved, then we're back to the need for
many individual drivers to bounce every request, because they don't have
access to the info needed to know whether bouncing is required or not
for a given buffer they were handed from higher layers. (Remember when
I proposed exporting that info to drivers so they could decide whether a
buffer is dma-aligned or not, the response was "No, keep it all hidden
in the busdma layer," which I think is the right response.)
Or if you push the responsibility down to the busdma layer where all
that info resides, all drivers which use DMA and adhere to the busdma
rules just work, without needing to know anything about the buffers that
were handed to them and without needing to know anything about the
platform requirements.
-- Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1346081557.1140.181.camel>