From owner-freebsd-arch@FreeBSD.ORG Sun Aug 26 20:01:55 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB765106564A; Sun, 26 Aug 2012 20:01:55 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 4FF5D8FC1A; Sun, 26 Aug 2012 20:01:55 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7QK1s32069554; Sun, 26 Aug 2012 14:01:54 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7QK1qGi029391; Sun, 26 Aug 2012 14:01:52 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Tim Kientzle In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sun, 26 Aug 2012 14:01:52 -0600 Message-ID: <1346011312.1140.107.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, "arch@" Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 20:01:55 -0000 On Sun, 2012-08-26 at 11:53 -0700, Tim Kientzle wrote: > These rules sound reasonable. Good documentation might > also give examples of what the PRE/POST operations might entail > (e.g., from the preceding discussion, it sounds like PREREAD > and PREWRITE require at least a partial cache flush on ARM). > That helps folks who are coming to the docs with some hardware > background. > I agree, I think it would be good to have something like a RATIONALE section in the manpage that summarizes the issues faced by various categories of platform (hardware coherency, software-assisted coherency, etc) and how they handle it. You don't want people coding to the implementation (some of which is going on now, and I've been guilty of it myself); if there were more info in the docs there'd be less motivation to peek at the implementation. > > * Read and write sync operators may be combined in a single call, > > PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > > allowed, PREREAD|POSTREAD is not. We should note that while > > read and write operations may be combined, on some platforms > > PREREAD|PREWRITE is needlessly expensive when only a read is > > being performed. > > > PREREAD|POSTREAD doesn't sound useful to me, but > why would it be explicitly forbidden? > > Would you also forbid POSTREAD|PREWRITE? > (For a buffer that has just completed a DMA read > and is going to be immediately used for a DMA write?) My thinking on forbidding PREREAD|POSTREAD is at least partly that it removes some temptation to do the wrong thing: treat the busdma API as if it were a general cpu cache manipulation library. With the new definition of the sequences, a PREREAD|POSTREAD operation is nonsensical because it leaves no window during which the DMA hardware has access to memory; it is in effect a no-op. If you think in terms of implementation you might think "This would have to cause a cache invalidation." If you think in terms of API you should be thinking something more like "This marks out a time window during which the DMA hardware has safe access to that memory which has a duration of zero." POSTREAD|PREWRITE is interesting. It is not a no-op in terms of the API. It closes the hardware access window that was opened by an earlier PREREAD, and it opens a new hardware access window with the PREWRITE. Whether it touches hardware or caches or anything in a given implementation isn't the point, in terms of the API it's the right thing to do for a pair of back to back DMA operations, without intervening CPU access, on the same memory. So PREREAD|POSTREAD and PREWRITE|POSTWRITE make no sense, but all other combos should be allowed. Maybe instead of allowing and forbidding specific combos, we should just advise that these two are effectively no-ops. -- Ian