From owner-freebsd-arch@FreeBSD.ORG Thu Apr 30 04:13:46 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B299B530; Thu, 30 Apr 2015 04:13:46 +0000 (UTC) Received: from mail-ig0-x229.google.com (mail-ig0-x229.google.com [IPv6:2607:f8b0:4001:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7427017EE; Thu, 30 Apr 2015 04:13:46 +0000 (UTC) Received: by iget9 with SMTP id t9so3713992ige.1; Wed, 29 Apr 2015 21:13:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=IxMUAtHmz0kI9UGh/ASkTjjQpgipqh5+cPEN50qx2eU=; b=HDW95hATGOmFWppbnYphTjSPXufwYWmx4YKfrtHnSjk4oXwH6HSdPXr/LnE+KNMPjf 9Jp1/op197oMnhfcrnqS/LE5wN5Y0kWEy3p95TkiXjuwrel1pzFCd2zY673y3ZBtuLUp CODHjsfQt0xqaiYd67/sG86m88uIhlBcH7AXvPNEQZXWNJSK35EYGhzWSwVQB4BhKiK9 gxhXvnYO6L+gmELRjQV7UdSMvj3s6F/1rpVnDN8lQKXXCMSH2N0WxjJj9oAshxwVXKyu 869h5rOYKB9SVOhvQUrvngQvPBFYGjbvCARjR1QJxmbVxhnpG+SapK9DSupocuQPuU5g 8EYQ== MIME-Version: 1.0 X-Received: by 10.50.72.8 with SMTP id z8mr1102031igu.36.1430367225931; Wed, 29 Apr 2015 21:13:45 -0700 (PDT) Received: by 10.36.106.70 with HTTP; Wed, 29 Apr 2015 21:13:45 -0700 (PDT) In-Reply-To: References: <38574E63-2D74-4ECB-8D68-09AC76DFB30C@bsdimp.com> <1761247.Bq816CMB8v@ralph.baldwin.cx> <20150429132017.GM2390@kib.kiev.ua> <20150429165432.GN2390@kib.kiev.ua> <20150429185019.GO2390@kib.kiev.ua> <20150429193337.GQ2390@kib.kiev.ua> <1430346204.1157.107.camel@freebsd.org> Date: Wed, 29 Apr 2015 23:13:45 -0500 Message-ID: Subject: Re: bus_dmamap_sync() for bounced client buffers from user address space From: Jason Harmening To: Ian Lepore Cc: Konstantin Belousov , Adrian Chadd , Svatopluk Kraus , freebsd-arch Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Apr 2015 04:13:46 -0000 On Wed, Apr 29, 2015 at 6:10 PM, Jason Harmening wrote: > >> For what we call armv6 (which is mostly armv7)... >> >> The cache maintenance operations require virtual addresses, which means >> it looks a lot like a VIPT cache. Under the hood the implementation >> behaves as if it were a PIPT cache so even in the presence of multiple >> mappings of the same physical page into different virtual addresses, the >> SMP coherency hardware works correctly. >> >> The ARM ARM says... >> >> [Stuff about ARMv6 and page coloring when a cache way exceeds >> 4K.] >> >> ARMv7 does not support page coloring, and requires that all data >> and unified caches behave as Physically Indexed Physically >> Tagged (PIPT) caches. >> >> The only true armv6 chip we support isn't SMP and has a 16K/4-way cache >> that neatly sidesteps the aliasing problem that requires page coloring >> solutions. So modern arm chips we get to act like we've got PIPT data >> caches, but with the quirk that cache ops are initiated by virtual >> address. >> > > Cool, thanks for the explanation! > To satisfy my own curiosity, since it "looks like VIPT", does that mean we > still have to flush the cache on context switch? > > >> >> Basically, when you perform a cache maintainence operation, a >> translation table walk is done on the core that issued the cache op, >> then from that point on the physical address is used within the cache >> hardware and that's what gets broadcast to the other cores by the snoop >> control unit or cache coherency fabric (depending on the chip). > > > So, if we go back to the original problem of wanting to do > bus_dmamap_sync() on userspace buffers from some asynchronous context: > > Say the process that owns the buffer is running on one core and prefetches > some data into a cacheline for the buffer, and bus_dmamap_sync(POSTREAD) is > done by a kernel thread running on another core. Since the core running > the kernel thread is responsible for the TLB lookup to get the physical > address, then since that core has no UVA the cache ops will be treated as > misses and the cacheline on the core that owns the UVA won't be > invalidated, correct? > > That means the panic on !pmap_dmap_iscurrent() in busdma_machdep-v6.c > should stay? > > Sort of the same problem would apply to drivers using > vm_fault_quick_hold_pages + bus_dmamap_load_ma...no cache maintenance, > since there are no VAs to operate on. Indeed, both arm and mips > implementation of _bus_dmamap_load_phys don't do anything with the sync > list. > It occurs to me that one way to deal with both the blocking-sfbuf for physcopy and VIPT cache maintenance might be to have a reserved per-CPU KVA page. For arches that don't have a direct map, the idea would be to grab a critical section, copy the bounce page or do cache maintenance on the synclist entry, then drop the critical section. That brought up a dim memory I had of Linux doing something similar, and in fact it seems to use kmap_atomic for both cache ops and bounce buffers.