From owner-freebsd-hackers@FreeBSD.ORG Thu May 17 13:20:41 2012 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C006C106564A for ; Thu, 17 May 2012 13:20:41 +0000 (UTC) (envelope-from onwahe@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 42BF38FC12 for ; Thu, 17 May 2012 13:20:40 +0000 (UTC) Received: by lbon10 with SMTP id n10so1792338lbo.13 for ; Thu, 17 May 2012 06:20:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=GRe8eK3TxLwxe9+utMpQSIkFMNpBvBMESoXOp4WaoJw=; b=VhQYKPEkqgE7Zn6ejQy8tiSdqcY6p4CNPo0t409fw7dIM48BLtpeV9Rs1MKpIK452+ 6Svy4FHGURPqRkGdA7gaL+oyvoM2EkHU3hvWsRJVDQZezXx8nXAXkLicQVvWmNpgBWZj rPhPXux3PJ8tJhEhbbYzJffd3+gUlhwgAOIYzJ5rXu87tMkE8P9uL2B5SxejEyOrwxJx PriPYDnhevihERwP+VhjID0CDDkXeaevckeerwzWbLnR5zuHBjOZwyUT5woBJhcw7P8I 64xmO+yEHeJHOYLOZA9mPua3khYpCJo3QnrZ/5YGRDkRLMssshWZCyeFUuG4D3G9H/vg Gx9w== MIME-Version: 1.0 Received: by 10.112.29.131 with SMTP id k3mr3132099lbh.53.1337260839877; Thu, 17 May 2012 06:20:39 -0700 (PDT) Received: by 10.112.60.65 with HTTP; Thu, 17 May 2012 06:20:39 -0700 (PDT) Date: Thu, 17 May 2012 15:20:39 +0200 Message-ID: From: Svatopluk Kraus To: hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: ARM + CACHE_LINE_SIZE + DMA X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2012 13:20:41 -0000 Hi, I'm working on DMA bus implementation for ARM11mpcore platform. I've looked at implementation in ARM tree, but IMHO it only works with some assumptions. There is a problem with DMA on memory block which is not aligned on CACHE_LINE_SIZE (start and end) if memory is not coherent. Let's have a buffer for DMA which is no aligned on CACHE_LINE_SIZE. Then first cache line associated with the buffer can be divided into two parts, A and B, where A is a memory we know nothing about it and B is buffer memory. The same stands for last cache line associatted with the buffer. We have no problem if a memory is coherent. Otherwise it depends on memory attributes. 1. [no cache] attribute No problem as memory is coherent. 2. [write throught] attribute The part A can be invalidated without loss of any data. It's not problem too. 3. [write back] attribute In general, there is no way how to keep both parts consistent. At the start of DMA transaction, the cache line is written back and invalidated. However, as we know nothing about memory associated with part A of the cache line, the cache line can be filled again at any time and messing up DMA transaction if flushed. Even if the cache line is only filled but not flushed during DMA transaction, we must make it coherent with memory after that. There is a trick with saving part A of the line into temporary buffer, invalidating the line, and restoring part A in current ARM (MIPS) implementation. However, if somebody is writting to memory associated with part A of the line during this trick, the part A will be messed up. Moreover, the part A can be part of another DMA transaction. To safely use DMA with no coherent memory, a memory with [no cache] or [write throught] attributes can be used without problem. A memory with [write back] attribute must be aligned on CACHE_LINE_SIZE. However, for example mbuf, a buffer for DMA can be part of a structure which can be aligned on CACHE_LINE_SIZE, but not the buffer itself. We can know that nobody will write to the structure during DMA transaction, so it's safe to use the buffer event if it's not aligned on CACHE_LINE_SIZE. So, in practice, if DMA buffer is not aligned on CACHE_LINE_SIZE and we want to avoid bounce pages overhead, we must support additional information to DMA transaction. It should be easy to support the information about drivers data buffers. However, what about OS data buffers like mentioned mbufs? The question is following. Is or can be guaranteed for all or at least well-known OS data buffers which can be part of DMA access that the not CACHE_LINE_SIZE aligned buffers are surrounded by data which belongs to the same object as the buffer and the data is not written by OS when given to a driver? Any answer is appreciated. However, 'bounce pages' is not an answer. Thanks, Svata