From owner-freebsd-arch@FreeBSD.ORG Sun Aug 26 17:42:23 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BA3D91065670; Sun, 26 Aug 2012 17:42:23 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 0634D8FC20; Sun, 26 Aug 2012 17:42:06 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7QHg5RZ067335; Sun, 26 Aug 2012 11:42:05 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7QHg2M1029199; Sun, 26 Aug 2012 11:42:02 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sun, 26 Aug 2012 11:42:02 -0600 Message-ID: <1346002922.1140.56.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Adrian Chadd , Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 17:42:23 -0000 On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: > The bottom line is that you can't mix things like that when cache > lines are involved. The current code that tries is doomed to failure. > Doomed. You just can't control all flushes, as Ian's missive > demonstrates, and trying to accommodate code that does this I don't > think can possibly work. All the interrupt masking, copying in and > out, etc I fear is doomed to utter and abject failure. > Until last weekend I was in the camp that thought the partial cacheline flush problem was solvable with sufficiently clever code. Now I agree that we're doomed to failure and it's time to try another direction. We're going to have some implementation work to do in arm and mips busdma, but I think the larger part of the task is going to be defining more rigorously how a driver must interact with the busdma system to function correctly on all types of platforms, and then update existing drivers to conform. The busdma manpage currently has some vague words about the usage and sequencing of sync ops, such as "If read and write operations are not preceded and followed by the appropriate synchronization operations, behavior is undefined." I think we should more explicitly spell out what the appropriate sequences are. In particular: * The PRE and POST operations must occur in pairs; a PREREAD must be followed eventually by a POSTREAD and a PREWRITE must be followed by a POSTWRITE. * The CPU is not allowed to access the mapped memory after a PRE sync and before the corresponding POST sync. * The DMA hardware is not allowed to access the mapped memory after a POST sync and before the next PRE sync. * Read and write sync operators may be combined in a single call, PRE and POST operators may not be. E.G., PREREAD|PREWRITE is allowed, PREREAD|POSTREAD is not. We should note that while read and write operations may be combined, on some platforms PREREAD|PREWRITE is needlessly expensive when only a read is being performed. We also need some rules about working with buffers obtained from bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). I think the rule should be that a buffer obtained from bus_dmamem_alloc(), or more formally any region of memory mapped by a bus_dmamap_load(), is a single logical object which can only be accessed by one entity at a time. That means that there cannot be two concurrent DMA operations happening in different regions of the same buffer, nor can DMA and CPU access be happening concurrently even if in different parts of the buffer. I've always thought that allocating a dma buffer feels like a big hassle. You sometimes have to create a tag for the sole purpose of setting the maxsize to get the buffer size you need when you call bus_dmamem_alloc(). If bus_dmamem_alloc() took a size parm you could just use your parent tag, or a generic tag appropriate to all the IO you're doing for a given device. If you need a variety of buffers for small control and command and status transfers of different sizes, you end up having to manage up to a dozen tags and maps and buffers. It's all very clunky and inconvenient. It's just the sort of thing that makes you want to allocate a big buffer and subdivide it. Surely we could do something to make it easier? -- Ian