From owner-freebsd-arm@FreeBSD.ORG Tue Aug 4 16:54:54 2009 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 72D39106564A for ; Tue, 4 Aug 2009 16:54:54 +0000 (UTC) (envelope-from hselasky@c2i.net) Received: from swip.net (mailfe14.tele2.se [212.247.155.161]) by mx1.freebsd.org (Postfix) with ESMTP id D1B118FC14 for ; Tue, 4 Aug 2009 16:54:53 +0000 (UTC) (envelope-from hselasky@c2i.net) X-Cloudmark-Score: 0.000000 [] X-Cloudmark-Analysis: v=1.0 c=1 a=uXAB73GHeBQA:10 a=MnI1ikcADjEx7bvsp0jZvQ==:17 a=dU7kav3A9PnrunaxqCUA:9 a=nVf9qWreKgvzqu-H-KsA:7 a=gJU65N-bE7egE_QUlsV8oVjiRrwA:4 Received: from [188.126.201.140] (account mc467741@c2i.net HELO laptop.adsl.tele2.no) by mailfe14.swip.net (CommuniGate Pro SMTP 5.2.13) with ESMTPA id 549127578; Tue, 04 Aug 2009 17:54:50 +0200 From: Hans Petter Selasky To: Grzegorz Bernacki Date: Tue, 4 Aug 2009 17:54:48 +0200 User-Agent: KMail/1.11.4 (FreeBSD/8.0-BETA2; KDE/4.2.4; i386; ; ) References: <3E1658AF-67C6-4E61-B6E7-BEF528C3FF4D@mac.com> <200908031759.46491.hselasky@c2i.net> <4A7848A0.4080905@semihalf.com> In-Reply-To: <4A7848A0.4080905@semihalf.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200908041754.50244.hselasky@c2i.net> Cc: arm@freebsd.org, freebsd-current@freebsd.org, usb@freebsd.org Subject: Re: About the "USB Cache and busdma usage in USB" thread X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Aug 2009 16:54:54 -0000 On Tuesday 04 August 2009 16:41:36 Grzegorz Bernacki wrote: > Hans Petter Selasky wrote: > > CC'ed current: We have a case on ARM where bus_dmamap_sync() is not > > suffient to update the CPU cache. One reason for this is that USB needs > > to invalidate the same memory area multiple times. Busdma sync expects > > paired operation when using the PRE and POST flags, from what I > > understand. I do not consider this an USB issue, hence Semihalf has got > > the USB stack working by manually inserting CPU flush/invalidate calls > > into usb_pc_cpu_invalidate() and usb_pc_cpu_flush(). Their other solution > > however which modifies the bus_dmamap_sync() flags will break on > > platforms with more than 4 GByte of memory. > > > > Maybe Rafal can give a quick summar to new people at the -current list, > > or see previous thread on the ARM mailing list. > > > > USB needs a solution where it can call a function given a busdma mapping, > > preferably with an offset and length, which handles the cache sync issue > > and works with bounce pages on +4GB systems. > > Hi Hans, > > New USB stack uses busdma in a little unconventional way. As you > mentioned in one of previous mails your assumptions are: > > XXX_PREXXX functions should be used prior to read/write device access. > In other words, PRE has to be a flush operation. > > XXX_POSTXXX functions should be used after read/write device access. > In other words, POST has to be an invalidate operation. > > Generally it is true, but if you look at ARM code you will find out that > it is not that simple. You assumed that after > bus_dmamap_sync(..,BUS_DMASYNC_POSTWRITE | BUS_DMASYNC_POSTREAD) there > will be no data in cache, but it that's not true. > > Cache operation are performed on cache lines (32 bytes on our ARM > device). Let's say you want to invalidate buffer with size 10 bytes. In > this case first whole cache line is invalidated ( and now all > requirements related to busdma synchronization are fulfilled, old > contents of cache is gone). The second step is to restore back into > cache 22 bytes of data which were not a part of buffer. After this > second step data are loaded into cache line (it is because our device > uses write allocate feature). > So busdma on ARM "Perform any synchronization required after an update > of host memory by the device", but we still end up with not invalidated > flush. > It is hard to fix it. We cannot just invalidate whole cache line. We > cannot also use cpu_dcache_wbinv, because this function is called after > buffer was used by device so we dont want to overwrite those data with > old cache contents. > > One possible solution is to call first > bus_dmamap_sync(..,BUS_DMASYNC_POSTREAD) and then > bus_dmamap_sync(..,BUS_DMASYNC_PREREAD) in usb_pc_cpu_invalidate(), but > this is ugly workaround which applies probably only to ARM case. > > The second problem is that you cannot use cpu_dcache_wb(inv) function > directly because you need to handle bounce pages in USB code. I think > that duplication of busdma code makes no sense. Probably it takes less > work to add bus_dmamap_sync() before/after each transaction. > > Could you give us a quick overview of buffer handling in USB stack? I > want to understand what is the relation between > usb_pc_cpu_invalidate/flush() functions and reading/writing to USB > device? From yours previous mail I understand that invalidate is called > *before* reading and flush *before* writing. Is that true? Can we add a > functions which will be called *after* reading/writing? Hi, There are two kinds of DMA memory in USB regard: 1) Transfer descriptors are allocated in coherent DMA memory. Operation logic: 1.a) Write to descriptor. 1.b.0) Call usb_pc_cpu_flush() to write data to RAM. 1.b.1) Write more fields to descriptor. 1.b.2) Call usb_pc_cpu_flush() to write data to RAM. 1.c) Call usb_pc_cpu_invalidate() to clear cache. 1.d) Read status field. If not complete goto 1.c) 2) Any kernel virtual memory (which might not be coherent) 2.a.0) CPU read case: 2.a.1) Before transfer start usb_pc_cpu_invalidate() is called to clear any data in cache for this buffer. 2.a.2) After transfer completion usb_pc_cpu_invalidate() is called again. 2.b.0) CPU write case: 2.b.1) Before transfer start usb_pc_cpu_flush() is called to to flush any data in cache to RAM for this buffer. 2.b.2) After transfer completion there is no cache operation. Anything unclear? --HPS > > If you have any questions regarding cache operation on ARM. please let > me know, I will try to answer them. > > regards, > Grzesiek