From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 19:01:31 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 89DBC106564A; Mon, 27 Aug 2012 19:01:31 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 1F8C98FC08; Mon, 27 Aug 2012 19:01:19 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7RJ1BMt091960; Mon, 27 Aug 2012 13:01:17 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7RJ0mgZ031745; Mon, 27 Aug 2012 13:00:48 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Adrian Chadd In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <1346083716.1140.212.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Mon, 27 Aug 2012 13:00:47 -0600 Message-ID: <1346094047.1140.264.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 19:01:31 -0000 On Mon, 2012-08-27 at 09:53 -0700, Adrian Chadd wrote: > On 27 August 2012 09:08, Ian Lepore wrote: > > > If two DMAs are going on concurrently in the same buffer, one is going > > to finish before the other, leading to a POSTxxxx sync op happening for > > one DMA operation while the other is still in progress. The unit of > > granularity for sync operations is the mapped region, so now you're > > syncing access to a region which still has active DMA happening within > > it. > > Right. But the enforced idea is "DMA up to this point should be > flushed to memory." > > > While I think it's really an API definition issue, think about it in > > terms of a potential implementation... What if the CPU had to access the > > memory as part of the sync for the first DMA that completes, while the > > second is still running? Now you've got pretty much exactly the same > > situation as when a driver subdivides a buffer without knowing about the > > cache alignment; you end up with the CPU and DMA touching data in the > > same cachline and no sequence of flush/invalidate can be g'teed to > > preserve all data correctly. > > Right. So you realise at that point you can't win and you stick each > of those pieces in a different cache line. > Actually, I think that even discussing cache lines in this context is a mistake (yeah, I'm the one who did so above, in trying to relate an abstract API design concept to a real-world hardware example). Drivers are not supposed to know about interactions between DMA transfers and cache lines or other machine-specific constraints; that info is supposed to be encapsulated and hidden within busdma. I think a driver making the assumption that it can do DMA safely on a buffer as long as that buffer is cacheline-granular is just as flawed as assuming that it can do DMA safely on any arbitrarily sized and aligned buffer. So the right way to "stick each of those pieces in a different cache line" is to allocate two different buffers, one per concurrent DMA transfer. Or, really, to use two separate busdma mappings would be the more rigorous way to say it, since the mapping is the operation at which constraints come into play. Thinking of it that way then drives the need to document that if multiple mappings describe the same area of physical memory, then concurrent operations on those maps yield unpredictable results. -- Ian