From owner-freebsd-mips@FreeBSD.ORG Sun Oct 6 04:22:03 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 7939E6F8 for ; Sun, 6 Oct 2013 04:22:03 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ie0-f179.google.com (mail-ie0-f179.google.com [209.85.223.179]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 411F725D9 for ; Sun, 6 Oct 2013 04:22:03 +0000 (UTC) Received: by mail-ie0-f179.google.com with SMTP id e14so12281227iej.24 for ; Sat, 05 Oct 2013 21:22:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=mL6YRwo4quUW966cDq3D/gx2CpcrabK/kT/q+yU0E0g=; b=N3nhh55GTaqAAwNpQoRyWQ7EFgcAyL0InBhYp84ctQeEHdnqaWoSgOSKHxhYH0GsjZ G0rBuyM4diUUSyRQBch7A6nTOy7/IAEdQXRXagT18BJGu+dDKiPTzhCRpmHmZBE/9qWU 3GsnVMqqnrDdCjL3t2BX6V7NN4aDfg0RTRdLYNQdzYuWjkAS2zZVNt97IXTq07B0bgTP xt5AwI0/32GIZYOBdWfvt+Q1gjKO5YOmM2PwZT2LE8W4qHoDhrpT8t8uxl7Vj78IYo7+ nxz/lgQeMcHipuJle4R39EsMiri6XZx0FphHqEQXhXl4CFxEr4w2uQeNkAnEbjp5D/bT XnyQ== X-Gm-Message-State: ALoCoQmjRKR0ljG9cUUfORFFL/ftte87ZuRukhpBVKAqY/Sg/Kg79ktL4c9h9hx82e0WFyt7e/+t X-Received: by 10.43.172.4 with SMTP id nw4mr13729615icc.25.1381033322534; Sat, 05 Oct 2013 21:22:02 -0700 (PDT) Received: from 53.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id ih14sm18254005igb.7.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 05 Oct 2013 21:22:02 -0700 (PDT) Sender: Warner Losh Subject: Re: How's bus-space stuff supposed to work with superscalar MIPS? Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Sat, 5 Oct 2013 22:22:01 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <21AC10EC-BAA6-4F1A-BC17-F781CF77D224@bsdimp.com> References: <5AD9EE93-9D19-4A07-B189-43DA0C4A85E9@FreeBSD.org> To: Adrian Chadd X-Mailer: Apple Mail (2.1085) Cc: "freebsd-mips@freebsd.org" X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Oct 2013 04:22:03 -0000 On Oct 5, 2013, at 5:51 PM, Adrian Chadd wrote: > On 5 October 2013 16:06, Stanislav Sedov wrote: >=20 >>=20 >> On Oct 5, 2013, at 10:18 AM, Adrian Chadd wrote: >>=20 >>> Hi all, >>>=20 >>> I've been bringing up the AR9344 PHY and after a lot of digging, I >>> discovered that I can fix things by changing ARGE_WRITE() (ie, write = to >> the >>> ethernet space registers) to: >>>=20 >>> bus_write_4(); >>> bus_read_4(); >>>=20 >>> .. to (what I'm guessing here) flush the write out before the next >>> instruction is run. >>>=20 >>> So, given this particular hilarity has shown up, what's the story = with >>> doing IO accesses on a superscalar MIPS CPU? If it's going to kseg1, = is >> it >>> somehow going to magically enforce ordering? Or am I right in = thinking we >>> will need explicit barriers here? >>>=20 >>=20 >> I don't know specifics of mips74k, but usually one indeed needs = memory >> barriers >> when performing read of write operation sequences that require = ordering on >> device I/O (e.g changing the ring and writing the new ring index >> afterwards). I would >> not be surprised if the cpu reorders i/o bus memory access, = especially a >> multi-issue >> one. >>=20 >> It is a good idea to have barriers where needed regardless. We have >> special macros >> for them which are defined to nothing on the in-order platforms. >=20 >=20 > Right. I know this stuff. I really though want to know this kind of = stuff: >=20 > * What the specifics are for superscalar MIPS CPUs; I believe they document that writes can be reordered unless there's an = intervening read or memory barrier. I've not looked it up. > * What the bus space stuff should be be providing by default (and I've = been > down this path once, with ath(4) bugs, PPC, and the bus space macros = not > enforcing flushes after IO operations, even though the API requires = drivers > do it themselves..); It isn't so much flushes as barriers to prevent reordering. By doing the = read after write, you are forcing an expensive memory barrier. Drivers = that depend on a particular write ordering need to have explicit = barriers. > * Whether it should be enough to map space COHERENT - then it's up to = the > underlying bus implementation to implement enforcing ordering. The question here is whether there should be an implied barrier in write = operations. On x86 there is, but as you are discovering on other = architectures there isn't. While it would be convenient to force a = memory barrier between every write (something trivial to do with an = explicit barrier in your driver), it is not very performant to do so, = since most writes don't have an explicit ordering... Warner=