Date: Fri, 28 Oct 2011 04:28:13 +0800 From: Adrian Chadd <adrian@freebsd.org> To: freebsd-arch@freebsd.org Subject: newbus IO ordering semantics - moving forward Message-ID: <CAJ-VmonFJG3xLn2JvarOUN6o-e7MC%2BA%2B=W9_vocZqY6L3CmTmQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi all,
After Nathan and I diagnosed a rather annoying ath bus bug on PPC, we
discovered that we're likely not the only driver/device combination
that is breaking the documented newbus behaviour of IO ordering.
Specifically, from the bus_space_* manpage:
Because some architectures' memory systems use buffering to improve mem-
ory and device access performance, there is a mechanism which can be used
to create ``barriers'' in the bus space read and write stream. There are
three types of barriers: read, write, and read/write. All reads started
to the region before a read barrier must complete before any reads after
the read barrier are started. (The analogous requirement is true for
write barriers.) Read/write barriers force all reads and writes started
before the barrier to complete before any reads or writes after the bar-
rier are started. Correctly-written drivers will include all appropriate
barriers, and assume only the read/write ordering imposed by the barrier
operations.
..
Read operations done by the bus_space_read_N() functions may be executed
out of order with respect to other pending read and write operations
unless order is enforced by use of the bus_space_barrier() function.
..
Write operations done by the bus_space_write_N() functions may be exe-
cuted out of order with respect to other pending read and write opera-
tions unless order is enforced by use of the bus_space_barrier() func-
tion.
However at least ath(4) doesn't use explicit flushes where needed,
which exposed bugs on PPC. I bet other drivers also have issues but as
they're not tested outside of i386/amd64, these bugs aren't picked up.
(There are endian issues too with some, but I digress.)
So what I'm proposing is:
* Make the bus default to use ordered semantics, much like what Linux
does - ie, all IO read/writes (io or memory) are in-order and flushed
with a barrier;
* Add an option which allows the driver to request a region with
loose-running/lazy semantics, what we're supposed to have now, and
then leave barriers up to the driver;
* Print out something nice and loud if a driver decides to use the
lazy/loose semantics, which may result in unpredictable behaviour on
non-{i386,amd64}.
I'd appreciate some feedback/comments before I go off and code all of this up.
Thanks,
Adrian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmonFJG3xLn2JvarOUN6o-e7MC%2BA%2B=W9_vocZqY6L3CmTmQ>