Date: Sun, 6 Oct 2013 09:31:11 -0700 From: Adrian Chadd <adrian.chadd@gmail.com> To: "M. Warner Losh" <imp@bsdimp.com> Cc: "freebsd-mips@freebsd.org" <freebsd-mips@freebsd.org> Subject: Re: How's bus-space stuff supposed to work with superscalar MIPS? Message-ID: <CAJ-Vmom8FfmoNh2EM4v5CCYcHmpQG0xTLqDmicEhs9%2BA-bNMrg@mail.gmail.com> In-Reply-To: <21AC10EC-BAA6-4F1A-BC17-F781CF77D224@bsdimp.com> References: <CAJ-Vmo=PNSsW0eEAhc9LEDLswsj41VN%2BFX1vakQL=qGGdKqMuw@mail.gmail.com> <5AD9EE93-9D19-4A07-B189-43DA0C4A85E9@FreeBSD.org> <CAJ-Vmoky4Sc6DURPj_YeahUPe8=XurP_j7k1S_6L4gzhCXyPrw@mail.gmail.com> <21AC10EC-BAA6-4F1A-BC17-F781CF77D224@bsdimp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Oct 6, 2013 12:22 AM, "Warner Losh" <imp@bsdimp.com> wrote: > > > On Oct 5, 2013, at 5:51 PM, Adrian Chadd wrote: > > > On 5 October 2013 16:06, Stanislav Sedov <stas@freebsd.org> wrote: > > > >> > >> On Oct 5, 2013, at 10:18 AM, Adrian Chadd <adrian@freebsd.org> wrote: > >> > >>> Hi all, > >>> > >>> I've been bringing up the AR9344 PHY and after a lot of digging, I > >>> discovered that I can fix things by changing ARGE_WRITE() (ie, write to > >> the > >>> ethernet space registers) to: > >>> > >>> bus_write_4(); > >>> bus_read_4(); > >>> > >>> .. to (what I'm guessing here) flush the write out before the next > >>> instruction is run. > >>> > >>> So, given this particular hilarity has shown up, what's the story with > >>> doing IO accesses on a superscalar MIPS CPU? If it's going to kseg1, is > >> it > >>> somehow going to magically enforce ordering? Or am I right in thinking we > >>> will need explicit barriers here? > >>> > >> > >> I don't know specifics of mips74k, but usually one indeed needs memory > >> barriers > >> when performing read of write operation sequences that require ordering on > >> device I/O (e.g changing the ring and writing the new ring index > >> afterwards). I would > >> not be surprised if the cpu reorders i/o bus memory access, especially a > >> multi-issue > >> one. > >> > >> It is a good idea to have barriers where needed regardless. We have > >> special macros > >> for them which are defined to nothing on the in-order platforms. > > > > > > Right. I know this stuff. I really though want to know this kind of stuff: > > > > * What the specifics are for superscalar MIPS CPUs; > > I believe they document that writes can be reordered unless there's an intervening read or memory barrier. I've not looked it up. > > > * What the bus space stuff should be be providing by default (and I've been > > down this path once, with ath(4) bugs, PPC, and the bus space macros not > > enforcing flushes after IO operations, even though the API requires drivers > > do it themselves..); > > It isn't so much flushes as barriers to prevent reordering. By doing the read after write, you are forcing an expensive memory barrier. Drivers that depend on a particular write ordering need to have explicit barriers. > > > * Whether it should be enough to map space COHERENT - then it's up to the > > underlying bus implementation to implement enforcing ordering. > > The question here is whether there should be an implied barrier in write operations. On x86 there is, but as you are discovering on other architectures there isn't. While it would be convenient to force a memory barrier between every write (something trivial to do with an explicit barrier in your driver), it is not very performant to do so, since most writes don't have an explicit ordering... The other thing is how correct the shared driver code is, like pci, usb, etc. I think that allocing bus space coherent means non cached, not non speculative/in order. So, what should we do? And whats the busdma barrier method do? Is it a cache barrier, or did its definition include ordering? Its a stub in mips, with the cache invalidate call commented out. My idea here is to change the definition of coherent, making it imply in order. Then add another flag saying space is potentially non ordered. That puts the onus on drivers to do the right thing if they want the performance boost, but buys us correctness now. I know that ppc modified their bus space to enforce ordered writes. Thanks, > Warner
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-Vmom8FfmoNh2EM4v5CCYcHmpQG0xTLqDmicEhs9%2BA-bNMrg>