From owner-freebsd-arm@FreeBSD.ORG Tue Aug 4 15:01:16 2009 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CB730106566C; Tue, 4 Aug 2009 15:01:16 +0000 (UTC) (envelope-from gjb@semihalf.com) Received: from smtp.semihalf.com (smtp.semihalf.com [213.17.239.109]) by mx1.freebsd.org (Postfix) with ESMTP id 3AD378FC15; Tue, 4 Aug 2009 15:01:16 +0000 (UTC) (envelope-from gjb@semihalf.com) Received: from [10.0.0.75] (cardhu.semihalf.com [213.17.239.108]) by smtp.semihalf.com (Postfix) with ESMTPA id 97A9DC426C; Tue, 4 Aug 2009 16:39:08 +0200 (CEST) Message-ID: <4A7848A0.4080905@semihalf.com> Date: Tue, 04 Aug 2009 16:41:36 +0200 From: Grzegorz Bernacki User-Agent: Thunderbird 2.0.0.16 (X11/20090618) MIME-Version: 1.0 To: Hans Petter Selasky References: <3E1658AF-67C6-4E61-B6E7-BEF528C3FF4D@mac.com> <20090724.233404.-399282844.imp@bsdimp.com> <8DC8C704-F84D-4A60-A11B-2F877EB903C9@semihalf.com> <200908031759.46491.hselasky@c2i.net> In-Reply-To: <200908031759.46491.hselasky@c2i.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arm@freebsd.org, freebsd-current@freebsd.org, usb@freebsd.org Subject: Re: About the "USB Cache and busdma usage in USB" thread X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2009 15:01:17 -0000 Hans Petter Selasky wrote: > > CC'ed current: We have a case on ARM where bus_dmamap_sync() is not suffient > to update the CPU cache. One reason for this is that USB needs to invalidate > the same memory area multiple times. Busdma sync expects paired operation when > using the PRE and POST flags, from what I understand. I do not consider this > an USB issue, hence Semihalf has got the USB stack working by manually > inserting CPU flush/invalidate calls into usb_pc_cpu_invalidate() and > usb_pc_cpu_flush(). Their other solution however which modifies the > bus_dmamap_sync() flags will break on platforms with more than 4 GByte of > memory. > > Maybe Rafal can give a quick summar to new people at the -current list, or see > previous thread on the ARM mailing list. > > USB needs a solution where it can call a function given a busdma mapping, > preferably with an offset and length, which handles the cache sync issue and > works with bounce pages on +4GB systems. > Hi Hans, New USB stack uses busdma in a little unconventional way. As you mentioned in one of previous mails your assumptions are: XXX_PREXXX functions should be used prior to read/write device access. In other words, PRE has to be a flush operation. XXX_POSTXXX functions should be used after read/write device access. In other words, POST has to be an invalidate operation. Generally it is true, but if you look at ARM code you will find out that it is not that simple. You assumed that after bus_dmamap_sync(..,BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD) there will be no data in cache, but it that's not true. Cache operation are performed on cache lines (32 bytes on our ARM device). Let's say you want to invalidate buffer with size 10 bytes. In this case first whole cache line is invalidated ( and now all requirements related to busdma synchronization are fulfilled, old contents of cache is gone). The second step is to restore back into cache 22 bytes of data which were not a part of buffer. After this second step data are loaded into cache line (it is because our device uses write allocate feature). So busdma on ARM "Perform any synchronization required after an update of host memory by the device", but we still end up with not invalidated flush. It is hard to fix it. We cannot just invalidate whole cache line. We cannot also use cpu_dcache_wbinv, because this function is called after buffer was used by device so we dont want to overwrite those data with old cache contents. One possible solution is to call first bus_dmamap_sync(..,BUS_DMASYNC_POSTREAD) and then bus_dmamap_sync(..,BUS_DMASYNC_PREREAD) in usb_pc_cpu_invalidate(), but this is ugly workaround which applies probably only to ARM case. The second problem is that you cannot use cpu_dcache_wb(inv) function directly because you need to handle bounce pages in USB code. I think that duplication of busdma code makes no sense. Probably it takes less work to add bus_dmamap_sync() before/after each transaction. Could you give us a quick overview of buffer handling in USB stack? I want to understand what is the relation between usb_pc_cpu_invalidate/flush() functions and reading/writing to USB device? From yours previous mail I understand that invalidate is called *before* reading and flush *before* writing. Is that true? Can we add a functions which will be called *after* reading/writing? If you have any questions regarding cache operation on ARM. please let me know, I will try to answer them. regards, Grzesiek