From owner-freebsd-embedded@FreeBSD.ORG Wed Dec 31 14:32:17 2008 Return-Path: Delivered-To: embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D3C11065672; Wed, 31 Dec 2008 14:32:17 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (casselton.net [63.165.140.2]) by mx1.freebsd.org (Postfix) with ESMTP id 3740C8FC17; Wed, 31 Dec 2008 14:32:17 +0000 (UTC) (envelope-from tinguely@casselton.net) Received: from casselton.net (localhost [127.0.0.1]) by casselton.net (8.14.2/8.14.2) with ESMTP id mBVEWGGk047500; Wed, 31 Dec 2008 08:32:16 -0600 (CST) (envelope-from tinguely@casselton.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=casselton.net; s=ccnMail; t=1230733936; bh=aaaxbB/uAZe2tbfDF3E6dzb3PSd0MV69SrigECO +HqI=; h=Date:From:Message-Id:To:Subject:Cc:In-Reply-To; b=BtrGC8wg XPdhAhEW7Hw8GWRmLQ1oTbaaHnLFiyylZo5NmjsYexxcj2Ya9CEBZIV5Wau54QXR5QG qDOxujlwhHSB+HZ2ZDaYAVuLpVLOqGCmLipOsLKToWC5HlDaf/mbcqOau0ETwaiw7wZ +wAokuYWQqpRAOOOcACilDkdCRC7M= Received: (from tinguely@localhost) by casselton.net (8.14.2/8.14.2/Submit) id mBVEWF6O047499; Wed, 31 Dec 2008 08:32:15 -0600 (CST) (envelope-from tinguely) Date: Wed, 31 Dec 2008 08:32:15 -0600 (CST) From: Mark Tinguely Message-Id: <200812311432.mBVEWF6O047499@casselton.net> To: gjb@semihalf.com, xcllnt@mac.com In-Reply-To: <495B5531.8040604@semihalf.com> Cc: arm@freebsd.org, embedded@freebsd.org Subject: Re: Multiple virtual mappings considered harmful on ARM X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Dec 2008 14:32:18 -0000 > Marcel Moolenaar wrote: > > > > On Dec 19, 2008, at 6:07 AM, Grzegorz Bernacki wrote: > > > >> 2. Root cause. > >> The root cause of the problem is additional virtual mapping of read/write > >> buffers at cluster read/write (sys/kern/vfs_cluster.c, cluster_rbuild(), > >> cluster_wbuild(). Buffers for sequential read/write operation are > >> concatenated > >> and sent to device as one big buffer. Concatenation of buffers uses > >> pmap_qenter(), which puts *additional* mapping in the KVA for physical > >> area > >> already mapped. For each buffer we extract pages it contains and then > >> all the > >> pages from all the buffers are mapped into new virtual address of new > >> buffer. > >> So we end up with at least two virtual addresses for each page. > > > > Could this also affect I-cache coherency by virtue of not > > flushing the D-cache properly before synchronizing the > > I-cache, as you mention reading? > > > > I am not sure. I can't think of scenario which might lead to I-cache incoherency. > Have you experienced any issues with I-cache which might be related described problem? > > pozdrawiam, > Grzesiek If would be surprised for you to see an I-cache problems. pmap_qenter() writes back any managed cache pages (already under the control of a pv_entry) and then calls pmap_kenter_internal(), which tries to writeback and invalid any previously mapped pages. At the end of the pmap_qenter() the caches are clean. The problem is the new mapping is added with cache turned on, and the old mapping's page table caching is not modified either, so if either side modifies the page, there can be this caching problem. I think that the pmap_kenter_internal() should become a special kind of managed page (controlled by a pv_entry and a special flag, (I called PVF_UNMAN) that runs through the pmap_fix_cache(), so the mappings for the shared page should have the cache turned off. There are special checks The less obvious question is should PG_UNMANAGED pages be simularly managed? Or put another way, could the PG_UNMANAGED page be remapped by the I/O caching programs? My guts says yes. Doing so also forces BUS_DMA_COHERENT pages to turn off the caches on the share page. I have a crude workup of my idea, but I have to admit, I don't have the equipment to even try to compile it. There concerns about pv_entry use increase, there is lock issues, performance, and even could we cause a problem trying get a pv_entry at the wrong time, such as interrupt. Maybe the kernel pv_entrys need to come from a special pool. I have some crude ideas to flush the caches less often in the ARM11's virtual index / physical tag caches and ideas for the multicore physical index / physical tag cache. --Mark Tinguely