From owner-freebsd-arch@FreeBSD.ORG Sun Aug 26 18:25:23 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D04B1065674; Sun, 26 Aug 2012 18:25:23 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 4881A8FC1E; Sun, 26 Aug 2012 18:25:10 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7QIPAH8068013; Sun, 26 Aug 2012 12:25:10 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7QIP87W029231; Sun, 26 Aug 2012 12:25:08 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Mark Tinguely In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sun, 26 Aug 2012 12:25:07 -0600 Message-ID: <1346005507.1140.69.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, Hans Petter Selasky , freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 18:25:23 -0000 On Sun, 2012-08-26 at 13:05 -0500, Mark Tinguely wrote: > I did a quick look at the drivers last summer. > > Most drivers do the right thing and use memory allocated from > bus_dmamem_alloc(). It is easy for us to give them a cache aligned > buffer. > > Some drivers use mbufs - 256 bytes which cache safe. > > Some drivers directly or indirectly malloc() a buffer and then use it > to dma - rather than try to fix them all, I was okay with making the > smallest malloc() amount equal to the cache line size. It amounts to > getting rid of the 16 byte allocation on some ARM architectures. The > power of 2 allocator will then give us cache line safe allocation. > > A few drivers take a small memory amount from the kernel stack and dma > to it <- broken driver. > > The few drivers that use data from a structure and that memory is not > cached aligned <- broken driver. > I disagree about those last two points -- drivers that choose to use stack memory or malloc'd memory as IO buffers are not broken. Drivers can do IO directly to/from userland buffers, do we say that an application that calls read(2) and passes the address of a stack variable is broken? In this regard, it's the busdma implementation that's broken, because it should bounce those IOs through a DMA-safe buffer. There's absolutely no rule that I've ever heard of in FreeBSD that says IO can only take place using memory allocated from busdma. The rule is only that the proper sequence of busdma operation must be called, and beyond that it's up to the busdma implementation to make it work. Our biggest problem, I think, is that we don't have a sufficient definition of "the proper sequence of busdma operations." I don't think it will be very hard to make the arm and mips busdma implementations work correctly. It won't even be too hard to make them fairly efficient at bouncing small IOs (my thinking is that we can make small bounces no more expensive than the current partial cacheline flush implementation which copies the data multiple times). Bouncing large IO will never be efficient, but the inefficiency will be a powerful motivator to update drivers that do large IO to work better, such as using buffers allocated from busdma. -- Ian