From owner-freebsd-arch@FreeBSD.ORG Thu Oct 27 20:28:14 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE4DC106564A for ; Thu, 27 Oct 2011 20:28:14 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9A06D8FC0C for ; Thu, 27 Oct 2011 20:28:14 +0000 (UTC) Received: by vws11 with SMTP id 11so4400942vws.13 for ; Thu, 27 Oct 2011 13:28:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=p3WnUO7yjzW5thln+iAAXI2DpND//JruzM7Xnl8YwqY=; b=OSz8zM8QnDgovVBy20jwkaJcl9g4e3siu5aWo6+F4F7lRTcsR77h9WRLoP9yIWvVD8 id6lQ2dmeM0pfCS3ELLOwVJdGsL8LkYOB92VHho9LVotsAlLFHFJHK9xzYfbhAfv4CGj r75oMGJ+Cr5dXk8EjauJDGIfWVhVJ4UBAbZjY= MIME-Version: 1.0 Received: by 10.52.37.36 with SMTP id v4mr1225913vdj.61.1319747293561; Thu, 27 Oct 2011 13:28:13 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.52.176.1 with HTTP; Thu, 27 Oct 2011 13:28:13 -0700 (PDT) Date: Fri, 28 Oct 2011 04:28:13 +0800 X-Google-Sender-Auth: QdYkDSAqxohMtkALaPDJVSR8C5o Message-ID: From: Adrian Chadd To: freebsd-arch@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: newbus IO ordering semantics - moving forward X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Oct 2011 20:28:14 -0000 Hi all, After Nathan and I diagnosed a rather annoying ath bus bug on PPC, we discovered that we're likely not the only driver/device combination that is breaking the documented newbus behaviour of IO ordering. Specifically, from the bus_space_* manpage: Because some architectures' memory systems use buffering to improve mem- ory and device access performance, there is a mechanism which can be used to create ``barriers'' in the bus space read and write stream. There are three types of barriers: read, write, and read/write. All reads started to the region before a read barrier must complete before any reads after the read barrier are started. (The analogous requirement is true for write barriers.) Read/write barriers force all reads and writes started before the barrier to complete before any reads or writes after the bar- rier are started. Correctly-written drivers will include all appropriate barriers, and assume only the read/write ordering imposed by the barrier operations. .. Read operations done by the bus_space_read_N() functions may be executed out of order with respect to other pending read and write operations unless order is enforced by use of the bus_space_barrier() function. .. Write operations done by the bus_space_write_N() functions may be exe- cuted out of order with respect to other pending read and write opera- tions unless order is enforced by use of the bus_space_barrier() func- tion. However at least ath(4) doesn't use explicit flushes where needed, which exposed bugs on PPC. I bet other drivers also have issues but as they're not tested outside of i386/amd64, these bugs aren't picked up. (There are endian issues too with some, but I digress.) So what I'm proposing is: * Make the bus default to use ordered semantics, much like what Linux does - ie, all IO read/writes (io or memory) are in-order and flushed with a barrier; * Add an option which allows the driver to request a region with loose-running/lazy semantics, what we're supposed to have now, and then leave barriers up to the driver; * Print out something nice and loud if a driver decides to use the lazy/loose semantics, which may result in unpredictable behaviour on non-{i386,amd64}. I'd appreciate some feedback/comments before I go off and code all of this up. Thanks, Adrian