From owner-freebsd-mips@FreeBSD.ORG Sun Oct 6 16:31:13 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 5BE1325F; Sun, 6 Oct 2013 16:31:13 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qa0-x229.google.com (mail-qa0-x229.google.com [IPv6:2607:f8b0:400d:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EC6BF230B; Sun, 6 Oct 2013 16:31:12 +0000 (UTC) Received: by mail-qa0-f41.google.com with SMTP id ii20so2397470qab.0 for ; Sun, 06 Oct 2013 09:31:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7QSVFHR7NgHGA6fm0MUKfhlm/z7vXDhbS7y5YdJc0iA=; b=DMCFgdX1spJ9Md/w0ibh+JCBwldgsfXbMHiRRvFH/qtDpzcB5Fj2dpBN1Wtk6kclhi wMyrR+ebnUgXuA/5uS9q8tJDmylwzgeb0tNlavqJU/mzNsD6hX1YAGuveKryITrR58Xc pPNtOfPy2TZk+V42+iXNONCNoIYO0rvt7HfWjgDTVuL7LgIczaaUTS8uZaayrqyqY9tr oWDGqaKAuMhDchZGK1kZx2A0gQV7tH0X+B1xm6LBmNNLYD+BoNZE7dtr5LfK7lOYYON9 H9tsrZAoClQtD1oTIexJU4g4Qy5RHSQzKM5wUFux+VWRXZSebCuCC4dvdol2qfUrY+hy Y0Ng== MIME-Version: 1.0 X-Received: by 10.49.12.14 with SMTP id u14mr3606800qeb.74.1381077072059; Sun, 06 Oct 2013 09:31:12 -0700 (PDT) Received: by 10.224.207.66 with HTTP; Sun, 6 Oct 2013 09:31:11 -0700 (PDT) Received: by 10.224.207.66 with HTTP; Sun, 6 Oct 2013 09:31:11 -0700 (PDT) In-Reply-To: <21AC10EC-BAA6-4F1A-BC17-F781CF77D224@bsdimp.com> References: <5AD9EE93-9D19-4A07-B189-43DA0C4A85E9@FreeBSD.org> <21AC10EC-BAA6-4F1A-BC17-F781CF77D224@bsdimp.com> Date: Sun, 6 Oct 2013 09:31:11 -0700 Message-ID: Subject: Re: How's bus-space stuff supposed to work with superscalar MIPS? From: Adrian Chadd To: "M. Warner Losh" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-mips@freebsd.org" X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Oct 2013 16:31:13 -0000 On Oct 6, 2013 12:22 AM, "Warner Losh" wrote: > > > On Oct 5, 2013, at 5:51 PM, Adrian Chadd wrote: > > > On 5 October 2013 16:06, Stanislav Sedov wrote: > > > >> > >> On Oct 5, 2013, at 10:18 AM, Adrian Chadd wrote: > >> > >>> Hi all, > >>> > >>> I've been bringing up the AR9344 PHY and after a lot of digging, I > >>> discovered that I can fix things by changing ARGE_WRITE() (ie, write to > >> the > >>> ethernet space registers) to: > >>> > >>> bus_write_4(); > >>> bus_read_4(); > >>> > >>> .. to (what I'm guessing here) flush the write out before the next > >>> instruction is run. > >>> > >>> So, given this particular hilarity has shown up, what's the story with > >>> doing IO accesses on a superscalar MIPS CPU? If it's going to kseg1, is > >> it > >>> somehow going to magically enforce ordering? Or am I right in thinking we > >>> will need explicit barriers here? > >>> > >> > >> I don't know specifics of mips74k, but usually one indeed needs memory > >> barriers > >> when performing read of write operation sequences that require ordering on > >> device I/O (e.g changing the ring and writing the new ring index > >> afterwards). I would > >> not be surprised if the cpu reorders i/o bus memory access, especially a > >> multi-issue > >> one. > >> > >> It is a good idea to have barriers where needed regardless. We have > >> special macros > >> for them which are defined to nothing on the in-order platforms. > > > > > > Right. I know this stuff. I really though want to know this kind of stuff: > > > > * What the specifics are for superscalar MIPS CPUs; > > I believe they document that writes can be reordered unless there's an intervening read or memory barrier. I've not looked it up. > > > * What the bus space stuff should be be providing by default (and I've been > > down this path once, with ath(4) bugs, PPC, and the bus space macros not > > enforcing flushes after IO operations, even though the API requires drivers > > do it themselves..); > > It isn't so much flushes as barriers to prevent reordering. By doing the read after write, you are forcing an expensive memory barrier. Drivers that depend on a particular write ordering need to have explicit barriers. > > > * Whether it should be enough to map space COHERENT - then it's up to the > > underlying bus implementation to implement enforcing ordering. > > The question here is whether there should be an implied barrier in write operations. On x86 there is, but as you are discovering on other architectures there isn't. While it would be convenient to force a memory barrier between every write (something trivial to do with an explicit barrier in your driver), it is not very performant to do so, since most writes don't have an explicit ordering... The other thing is how correct the shared driver code is, like pci, usb, etc. I think that allocing bus space coherent means non cached, not non speculative/in order. So, what should we do? And whats the busdma barrier method do? Is it a cache barrier, or did its definition include ordering? Its a stub in mips, with the cache invalidate call commented out. My idea here is to change the definition of coherent, making it imply in order. Then add another flag saying space is potentially non ordered. That puts the onus on drivers to do the right thing if they want the performance boost, but buys us correctness now. I know that ppc modified their bus space to enforce ordered writes. Thanks, > Warner