From owner-freebsd-hackers@FreeBSD.ORG Thu May 17 16:01:35 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A9595106566C for ; Thu, 17 May 2012 16:01:35 +0000 (UTC) (envelope-from marktinguely@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7DCFD8FC16 for ; Thu, 17 May 2012 16:01:35 +0000 (UTC) Received: by pbbro2 with SMTP id ro2so3001984pbb.13 for ; Thu, 17 May 2012 09:01:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=dXJSWdiJB+QsS7Bjsf8CjG8+72w7wLCBZDHsILV1rKw=; b=wlef4YGMcAJTGFmEKe+aqhmwpi1M/P21g0eEhwSyWmf9AFHveAQqhuIPSaffTDmqyN +1h8Y4yaWdh8OgO67l3j9iYBrcDnDf5Znn0EZ777YKbSxIUI8JwTEaTaXX2sgKn2iKI5 LqfXC06ls1J5KaK4/RXg6N2cCECumHyTE9AbHOoY+XV7JGK/rGREDMVUv1TsJqFfaLsl 3MUNfbKNK1CnBnY3o7+kLYHnGu7GXEAqsVA8X3tVRNngTkUwShjAGN74QdYyRvZHGceU V1T0ckrLz75p+als/IwWyuOr3OdE9bPHD+W2zhRA+JsHu2LoYafEzyqsVttoma1RITsn 70wQ== MIME-Version: 1.0 Received: by 10.68.203.225 with SMTP id kt1mr28688539pbc.133.1337270495017; Thu, 17 May 2012 09:01:35 -0700 (PDT) Received: by 10.68.49.227 with HTTP; Thu, 17 May 2012 09:01:34 -0700 (PDT) In-Reply-To: References: Date: Thu, 17 May 2012 11:01:34 -0500 Message-ID: From: Mark Tinguely To: Svatopluk Kraus Content-Type: text/plain; charset=ISO-8859-1 Cc: hackers@freebsd.org Subject: Re: ARM + CACHE_LINE_SIZE + DMA X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2012 16:01:35 -0000 On Thu, May 17, 2012 at 8:20 AM, Svatopluk Kraus wrote: > Hi, > > I'm working on DMA bus implementation for ARM11mpcore platform. I've > looked at implementation in ARM tree, but IMHO it only works with some > assumptions. There is a problem with DMA on memory block which is not > aligned on CACHE_LINE_SIZE (start and end) if memory is not coherent. > > Let's have a buffer for DMA which is no aligned on CACHE_LINE_SIZE. > Then first cache line associated with the buffer can be divided into > two parts, A and B, where A is a memory we know nothing about it and B > is buffer memory. The same stands for last cache line associatted with > the buffer. We have no problem if a memory is coherent. Otherwise it > depends on memory attributes. > > 1. [no cache] attribute > No problem as memory is coherent. > > 2. [write throught] attribute > The part A can be invalidated without loss of any data. It's not problem too. > > 3. [write back] attribute > In general, there is no way how to keep both parts consistent. At the > start of DMA transaction, the cache line is written back and > invalidated. However, as we know nothing about memory associated with > part A of the cache line, the cache line can be filled again at any > time and messing up DMA transaction if flushed. Even if the cache line > is only filled but not flushed during DMA transaction, we must make it > coherent with memory after that. There is a trick with saving part A > of the line into temporary buffer, invalidating the line, and > restoring part A in current ARM (MIPS) implementation. However, if > somebody is writting to memory associated with part A of the line > during this trick, the part A will be messed up. Moreover, the part A > can be part of another DMA transaction. > > To safely use DMA with no coherent memory, a memory with [no cache] or > [write throught] attributes can be used without problem. A memory with > [write back] attribute must be aligned on CACHE_LINE_SIZE. > > However, for example mbuf, a buffer for DMA can be part of a structure > which can be aligned on CACHE_LINE_SIZE, but not the buffer itself. We > can know that nobody will write to the structure during DMA > transaction, so it's safe to use the buffer event if it's not aligned > on CACHE_LINE_SIZE. > > So, in practice, if DMA buffer is not aligned on CACHE_LINE_SIZE and > we want to avoid bounce pages overhead, we must support additional > information to DMA transaction. It should be easy to support the > information about drivers data buffers. However, what about OS data > buffers like mentioned mbufs? > > The question is following. Is or can be guaranteed for all or at least > well-known OS data buffers which can be part of DMA access that the > not CACHE_LINE_SIZE aligned buffers are surrounded by data which > belongs to the same object as the buffer and the data is not written > by OS when given to a driver? > > Any answer is appreciated. However, 'bounce pages' is not an answer. > > Thanks, Svata Sigh. A several ideas by several people, but a good answer has not been created yet. SMP will make this worse. To make things worse, there are drivers that use memory from the stack as DMA buffers. I was hoping that mbufs are pretty well self-contained, unless you expect to modify them while DMA is happening. This is on my to-do list. --Mark.