From owner-freebsd-hackers@FreeBSD.ORG Wed Oct 16 20:40:22 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A5330F46 for ; Wed, 16 Oct 2013 20:40:22 +0000 (UTC) (envelope-from ian@FreeBSD.org) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 7B3042971 for ; Wed, 16 Oct 2013 20:40:22 +0000 (UTC) Received: from c-24-8-230-52.hsd1.co.comcast.net ([24.8.230.52] helo=damnhippie.dyndns.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1VWXtF-000ANk-Bi; Wed, 16 Oct 2013 20:40:21 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id r9GKeIJ7028716; Wed, 16 Oct 2013 14:40:18 -0600 (MDT) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 24.8.230.52 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX18dSTG1VQe3SzBJ6jYCW3sA Subject: Re: why does /dev/md call cpu_dcache_flush()? From: Ian Lepore To: Poul-Henning Kamp In-Reply-To: <94783.1381954696@critter.freebsd.dk> References: <1381953459.1168.48.camel@revolution.hippie.lan> <94783.1381954696@critter.freebsd.dk> Content-Type: text/plain; charset="us-ascii" Date: Wed, 16 Oct 2013 14:40:18 -0600 Message-ID: <1381956018.1168.61.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Oct 2013 20:40:22 -0000 On Wed, 2013-10-16 at 20:18 +0000, Poul-Henning Kamp wrote: > In message <1381953459.1168.48.camel@revolution.hippie.lan>, Ian Lepore writes: > >The only caller of cpu_dcache_flush() in the entire system appears to be > >the md device. Does anybody know why it makes the call? > > r192323 | marcel | 2009-05-18 18:37:18 +0000 (Mon, 18 May 2009) | 20 lines > > Add cpu_flush_dcache() for use after non-DMA based I/O so that a > possible future I-cache coherency operation can succeed. On ARM > for example the L1 cache can be (is) virtually mapped, which > means that any I/O that uses temporary mappings will not see the > I-cache made coherent. On ia64 a similar behaviour has been > observed. By flushing the D-cache, execution of binaries backed > by md(4) and/or NFS work reliably. > For Book-E (powerpc), execution over NFS exhibits SIGILL once in > a while as well, though cpu_flush_dcache() hasn't been implemented > yet. > > Doing an explicit D-cache flush as part of the non-DMA based I/O > read operation eliminates the need to do it as part of the > I-cache coherency operation itself and as such avoids pessimizing > the DMA-based I/O read operations for which D-cache are already > flushed/invalidated. It also allows future optimizations whereby > the bcopy() followed by the D-cache flush can be integrated in a > single operation, which could be implemented using on-chips DMA > engines, by-passing the D-cache altogether. > Hrm. Now I feel dumb for not finding that obviously-available info for myself. But I guess I'll get over that and move on to what I would have posted if I had found it... The first part of the comment about temporary mappings on ARM is outdated, although it would have been true at the time that was written. Nowadays if multiple mappings are made to the same memory on VIVT ARM chips, all mappings are changed to non-cacheable to avoid those problems. I think the rest of it is just wrong. Data read in from any source may be in the caches when the read is done, whether the source was memory or anything else. Even if DMA was involved at some point, if bounce buffers were used that involves copying memory->memory after the DMA completes and it's quite likely some of that data is still in the caches. -- Ian