From owner-freebsd-mips@FreeBSD.ORG Mon Aug 20 10:36:58 2012 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B8D0E10656B1 for ; Mon, 20 Aug 2012 10:36:58 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 49B218FC0A for ; Mon, 20 Aug 2012 10:36:58 +0000 (UTC) Received: by weyx56 with SMTP id x56so6145628wey.13 for ; Mon, 20 Aug 2012 03:36:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ENlRCXkXja0lXUH0Dbou7G1k7DYB+U0NFcvAqo4joxU=; b=GCreHrrGKNuZke/PWuaS+PnGAN0cPoTuzZMkGBuNff8KSg0ZRYpw9W3sO+FtsfbDui m2YoQtTVHw1120PtAUiPPvhkFjKds4Cvu93BgJv2ja49mPDtbdGnc0+1nw7QfLfUJYXm 3CBD7Owesm1qDi8CgdH/tZpzgbOxdmgev3XNzqCbauOMl2Rl3gIwGWbxpD1fuJparcpR LR4/BOtYQBA2sYLH+S0XHfrQcxhykd56F1d5fhPN+veWpUv9FVcuBf3Cr+x0aZASsuJB pxJqRs3v8qG0Se9Yb9IXD3eGaKQ9RAk6ZcRZRm5VHMWbnCE1URsXe4JTUOyY7Ue/u1h8 CLqA== MIME-Version: 1.0 Received: by 10.216.139.196 with SMTP id c46mr7148769wej.220.1345459017108; Mon, 20 Aug 2012 03:36:57 -0700 (PDT) Received: by 10.216.115.3 with HTTP; Mon, 20 Aug 2012 03:36:56 -0700 (PDT) In-Reply-To: <502D2271.6080105@rice.edu> References: <50228F5C.1000408@rice.edu> <50269AD4.9050804@rice.edu> <5029635A.4050209@rice.edu> <502D2271.6080105@rice.edu> Date: Mon, 20 Aug 2012 16:06:56 +0530 Message-ID: From: "Jayachandran C." To: Alan Cox Content-Type: text/plain; charset=ISO-8859-1 Cc: mips@freebsd.org Subject: Re: mips pmap patch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2012 10:36:58 -0000 On Thu, Aug 16, 2012 at 10:10 PM, Alan Cox wrote: > On 08/15/2012 17:21, Jayachandran C. wrote: >> >> On Tue, Aug 14, 2012 at 1:58 AM, Alan Cox wrote: >>> >>> On 08/13/2012 11:37, Jayachandran C. wrote: [...] >>>> I could not test for more than an hour on 32-bit due to another >>>> problem (freelist 1 containing direct-mapped pages runs out of pages >>>> after about an hour of compile test). This issue has been there for a >>>> long time, I am planning to look at it when I get a chance. >>>> >>> What exactly happens? panic? deadlock? >> >> The build slows down to a crawl and hangs when it runs out of pages in >> the freelist. > > > I'd like to see the output of "sysctl vm.phys_segs" and "sysctl > vm.phys_free" from this machine. Even better would be running "sysctl > vm.phys_free" every 60 seconds during the buildworld. Finally, I'd like to > know whether or not either "ps" or "top" shows any threads blocked on the > "swwrt" wait channel once things slow to a crawl. I spent some time looking at this issue. I use a very large kernel image with built-in root filesystem, and this takes about 120 MB out of the direct mapped area. The remaining pages (~64 MB) are not enough for the build process. If I increase free memory in this area either by reducing the rootfs size of by adding a few more memory segments to this area, the build goes through fine. I also found that when the build slows down, most of the pages taken from freelist 1 are allocated by the UMA subsystem, which seems to keep quite a few pages allocated. Regards, JC. From owner-freebsd-mips@FreeBSD.ORG Mon Aug 20 11:07:53 2012 Return-Path: Delivered-To: freebsd-mips@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 40DD31065674 for ; Mon, 20 Aug 2012 11:07:53 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 10FAC8FC0A for ; Mon, 20 Aug 2012 11:07:53 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7KB7qH2047868 for ; Mon, 20 Aug 2012 11:07:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7KB7pQc047847 for freebsd-mips@FreeBSD.org; Mon, 20 Aug 2012 11:07:51 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 20 Aug 2012 11:07:51 GMT Message-Id: <201208201107.q7KB7pQc047847@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-mips@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-mips@FreeBSD.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2012 11:07:53 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/165951 mips [ar913x] [ath] DDR flush isn't being done for the WMAC p kern/163670 mips [mips][arge] arge can't allocate ring buffer on multip 2 problems total. From owner-freebsd-mips@FreeBSD.ORG Mon Aug 20 15:54:48 2012 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A6494106585D for ; Mon, 20 Aug 2012 15:54:48 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh10.mail.rice.edu (mh10.mail.rice.edu [128.42.201.30]) by mx1.freebsd.org (Postfix) with ESMTP id 72EBD8FC08 for ; Mon, 20 Aug 2012 15:54:47 +0000 (UTC) Received: from mh10.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh10.mail.rice.edu (Postfix) with ESMTP id 08661604CE; Mon, 20 Aug 2012 10:54:47 -0500 (CDT) Received: from mh10.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh10.mail.rice.edu (Postfix) with ESMTP id 073F9604CA; Mon, 20 Aug 2012 10:54:47 -0500 (CDT) X-Virus-Scanned: by amavis-2.7.0 at mh10.mail.rice.edu, auth channel Received: from mh10.mail.rice.edu ([127.0.0.1]) by mh10.mail.rice.edu (mh10.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id 7mX0oMSi1cJt; Mon, 20 Aug 2012 10:54:46 -0500 (CDT) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh10.mail.rice.edu (Postfix) with ESMTPSA id 8274E6047D; Mon, 20 Aug 2012 10:54:46 -0500 (CDT) Message-ID: <50325DC3.3090201@rice.edu> Date: Mon, 20 Aug 2012 10:54:43 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:8.0) Gecko/20111113 Thunderbird/8.0 MIME-Version: 1.0 To: "Jayachandran C." References: <50228F5C.1000408@rice.edu> <50269AD4.9050804@rice.edu> <5029635A.4050209@rice.edu> <502D2271.6080105@rice.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: mips@freebsd.org Subject: Re: mips pmap patch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2012 15:54:48 -0000 On 08/20/2012 05:36, Jayachandran C. wrote: > On Thu, Aug 16, 2012 at 10:10 PM, Alan Cox wrote: >> On 08/15/2012 17:21, Jayachandran C. wrote: >>> On Tue, Aug 14, 2012 at 1:58 AM, Alan Cox wrote: >>>> On 08/13/2012 11:37, Jayachandran C. wrote: > [...] >>>>> I could not test for more than an hour on 32-bit due to another >>>>> problem (freelist 1 containing direct-mapped pages runs out of pages >>>>> after about an hour of compile test). This issue has been there for a >>>>> long time, I am planning to look at it when I get a chance. >>>>> >>>> What exactly happens? panic? deadlock? >>> The build slows down to a crawl and hangs when it runs out of pages in >>> the freelist. >> >> I'd like to see the output of "sysctl vm.phys_segs" and "sysctl >> vm.phys_free" from this machine. Even better would be running "sysctl >> vm.phys_free" every 60 seconds during the buildworld. Finally, I'd like to >> know whether or not either "ps" or "top" shows any threads blocked on the >> "swwrt" wait channel once things slow to a crawl. > I spent some time looking at this issue. I use a very large kernel > image with built-in root filesystem, and this takes about 120 MB out > of the direct mapped area. The remaining pages (~64 MB) are not enough > for the build process. If I increase free memory in this area either > by reducing the rootfs size of by adding a few more memory segments to > this area, the build goes through fine. I'm still curious to see what "sysctl vm.phys_segs" says. It sounds like roughly half of the direct map region is going to DRAM and half to memory-mapped I/O devices. Is that correct? > I also found that when the build slows down, most of the pages taken > from freelist 1 are allocated by the UMA subsystem, which seems to > keep quite a few pages allocated. At worst, it may be necessary to disable the use of uma_small_alloc() for this machine configuration. At best, uma_small_alloc() could be revised opportunistically use pages in the direct map region, but have the ability to fall back to pages that have to be mapped. Alan From owner-freebsd-mips@FreeBSD.ORG Mon Aug 20 19:01:38 2012 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A15F2106572D; Mon, 20 Aug 2012 19:01:38 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 4892E8FC24; Mon, 20 Aug 2012 19:01:38 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id B53A1B95B; Mon, 20 Aug 2012 15:01:37 -0400 (EDT) From: John Baldwin To: Ian Lepore , scottl@freebsd.org, Justin Gibbs Date: Mon, 20 Aug 2012 14:34:16 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p17; KDE/4.5.5; amd64; ; ) References: <20120703111753.GB72292@server.rulingia.com> <1344355782.1128.186.camel@revolution.hippie.lan> <201208071406.45172.jhb@freebsd.org> In-Reply-To: <201208071406.45172.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201208201434.16538.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 20 Aug 2012 15:01:37 -0400 (EDT) Cc: arm@freebsd.org, mips@freebsd.org, Peter Jeremy Subject: Re: On-stack allocation of DMA S/G lists X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2012 19:01:39 -0000 [ Adding Scott Long and Justin Gibbs for their input ] On Tuesday, August 07, 2012 2:06:44 pm John Baldwin wrote: > On Tuesday, August 07, 2012 12:09:42 pm Ian Lepore wrote: > > On Mon, 2012-08-06 at 10:26 -0400, John Baldwin wrote: > > > On Thursday, July 12, 2012 8:26:05 am John Baldwin wrote: > > > > On Sunday, July 08, 2012 7:05:16 am Peter Jeremy wrote: > > > > > BTW(2): Whilst studying busdma_machdep.c for arm and mips, I've > > > > > noticed they appear to potentially allocate substantial kernel stack > > > > > under some conditions as several bus_dma(9) functions include: > > > > > bus_dma_segment_t dm_segments[dmat->nsegments]; > > > > > What prevents this overflowing the kernel stack? > > > > > > > > That does seem dubious. x86 stores the array in the tag instead. > > > > > > I have an untested patch to change bus-dma on arm and mips to allocate a > > > dynamic S/G list in each DMA tag on first use instead of using on-stack > > > allocation (which I think is rather bogus). Can folks review and test this > > > patch please? Thanks. > > > > > > http://www.FreeBSD.org/~jhb/patches/arm_mips_dynamic_dma_segs.patch > > > > > > > I'm worried about changing a per-mapping-call resource to a per-dma-tag > > resource here. What prevents the situation where you have two > > bus_dmamap_load() calls in progress at the same time using different > > buffers but the same tag? > > > > I can't find anything in the docs that indicates you have to provide > > external locking of the tag for map load/unload calls, or that even > > implies the tag can be modified by a mapping operation. The lockfunc > > stuff related to creating the tag is documented as being used only > > during a deferred callback. > > Actually, I do think it is implicit that you won't do concurrent loads > on a DMA tag, though that may not be obvious. Keep in mind that this > is what x86's bus_dma has always done. For storage drivers you certainly > can't do this or risk completeing I/O requests out-of-order which can > break an upper-layer assumption in a filesystem. Note that all other > platforms do this as well, only arm and mips allocate on the stack. One thing I should note is that it is assumed that the lock specifed by the lockfunc stuff is held when you invoke bus_dmamap_load*(). > > The existing code seems to go out of its way to avoid modifying the tag > > during a mapping operation. For example, it decides at tag creation > > time whether any bounce pages might ever be needed for the tag, and if > > so it pre-sets a bounce zone in the tag, then at mapping time the bounce > > zone is protected with its own lock when it gets modified. To me this > > feels like a way to specifically avoid the need to lock or modify the > > tag during a mapping operation. > > > > Assuming that all of the foregoing is moot for some reason I've > > overlooked, then on a purely implementation level, could all the > > duplicated code to allocate the array when necessary be moved into > > bus_dmamap_load_buffer(), triggered by a NULL 'segs' pointer? > > Nope, bus_dmamap_load() doesn't know which of M_NOWAIT / M_WAITOK is > appropriate to use. > > > And just for the record, looking at the problem from an even more > > distant vantage... is there really a problem with stack-allocating the > > segments? On a 64-bit arch the struct is like 16 bytes. Typical usage > > is to allocate a tag allowing 1 or just a few segments. Is anyone > > really going to create a tag specifying hundreds of segments that would > > overflow the stack? If they try, wouldn't failing the tag create be > > good enough? > > I/O devices can allocate tags with several S/G elements. An mfi(4) tag > on i386 would use a 256 byte segments array (512 on amd64). That's not > entirely trivial. It would be worse if you couldn't depend on > dmat->nsegments and had to always allocate the full size. Presumably > though we require C99 at that point (and it requires that?). Also, we are moving to a model of using larger MAXPHYS (some places already modify FreeBSD to bump MAXPHYS up), and to looking at using direct dispatch for disk I/O rather than always bouncing requests through g_up/g_down to gain greater concurrency in the disk I/O subsystem. Both of these are going to place greater stress on the kernel stack and argue for less kernel stack use, not more. Scott noted that a 1MB I/O request can require up to 256 segments, which is 4k. That would be too much space on the stack. One other option is to move the S/G lists into the dma map structure instead of being in the tag, but to date that has not been done because it would introduce a significant amount of memory overhead. Also, I don't see a great need to allow concurrent loads within a single tag. Typically you allocate a single tag for each "thing" that can accept a stream of DMA requests (so 1 tag for each command queue on a HBA, or for each descriptor ring on a NIC). For example, in the case of a multiq NIC you want to use separate tags so that each tag uses a per-ring lock. But typically that stream of DMA requests requires serialization, so you would never end up wanting to have concurrent loads on the same tag. -- John Baldwin From owner-freebsd-mips@FreeBSD.ORG Wed Aug 22 17:32:08 2012 Return-Path: Delivered-To: freebsd-mips@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38E0A1065672; Wed, 22 Aug 2012 17:32:08 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 0C6FE8FC0C; Wed, 22 Aug 2012 17:32:08 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7MHW7Cr040311; Wed, 22 Aug 2012 17:32:07 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7MHW7bG040307; Wed, 22 Aug 2012 17:32:07 GMT (envelope-from linimon) Date: Wed, 22 Aug 2012 17:32:07 GMT Message-Id: <201208221732.q7MHW7bG040307@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-mips@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/170864: [mips] [patch] Move SPI flash from AR71XX_BASE (kernel) to board specific files X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 17:32:08 -0000 Synopsis: [mips] [patch] Move SPI flash from AR71XX_BASE (kernel) to board specific files Responsible-Changed-From-To: freebsd-bugs->freebsd-mips Responsible-Changed-By: linimon Responsible-Changed-When: Wed Aug 22 17:31:59 UTC 2012 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=170864 From owner-freebsd-mips@FreeBSD.ORG Wed Aug 22 17:39:41 2012 Return-Path: Delivered-To: freebsd-mips@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3DF8D1065670; Wed, 22 Aug 2012 17:39:41 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 11C478FC1A; Wed, 22 Aug 2012 17:39:41 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7MHdeup041684; Wed, 22 Aug 2012 17:39:40 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7MHdeeE041680; Wed, 22 Aug 2012 17:39:40 GMT (envelope-from linimon) Date: Wed, 22 Aug 2012 17:39:40 GMT Message-Id: <201208221739.q7MHdeeE041680@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-mips@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/170859: [patch] [mips] Move MIPS specific options from generic conf/options file X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 17:39:41 -0000 Old Synopsis: [patch] Move MIPS specific options from generic conf/options file New Synopsis: [patch] [mips] Move MIPS specific options from generic conf/options file Responsible-Changed-From-To: freebsd-bugs->freebsd-mips Responsible-Changed-By: linimon Responsible-Changed-When: Wed Aug 22 17:39:18 UTC 2012 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=170859 From owner-freebsd-mips@FreeBSD.ORG Wed Aug 22 23:46:52 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 21270106566C for ; Wed, 22 Aug 2012 23:46:52 +0000 (UTC) (envelope-from vrtuff@yahoo.com) Received: from nm23.bullet.mail.bf1.yahoo.com (nm23.bullet.mail.bf1.yahoo.com [98.139.212.182]) by mx1.freebsd.org (Postfix) with SMTP id BA8DC8FC12 for ; Wed, 22 Aug 2012 23:46:50 +0000 (UTC) Received: from [98.139.212.152] by nm23.bullet.mail.bf1.yahoo.com with NNFMP; 22 Aug 2012 23:46:44 -0000 Received: from [98.139.212.245] by tm9.bullet.mail.bf1.yahoo.com with NNFMP; 22 Aug 2012 23:46:44 -0000 Received: from [127.0.0.1] by omp1054.mail.bf1.yahoo.com with NNFMP; 22 Aug 2012 23:46:44 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 347619.84286.bm@omp1054.mail.bf1.yahoo.com Received: (qmail 65710 invoked by uid 60001); 22 Aug 2012 23:46:44 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1345679204; bh=5Hew+VgQ5p7fe48Lp/eQ3Z2sR4yuFpxV22S9PuyWZl0=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=YW4RQClXOIqA57BmyClFkLMUXMQTXlXHdOaOjGkVgkEb4H/p3joHrDVRcJaN1YyOqCo2W4RskJmd8V0P2WTYQHmhntX5yUPsOTSxVWmIH2H2BD8j5XaxKcAP8kxYhu741/wvyA4HyB7/fVUSJsyxCPlfDOd8TPCsm91o+wAZ1I4= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=u81kVH11TIo8Ie7bbBbSRQ/LFzSEH8JdmsBHfcyWl21lMy6xohdTZV4PoAZDSHD6jWbOIbNGOz8cZzMmeDHYwG/OTg2c0Cis8C8EI1++Xh3BikUdlexEKx0iecffRncv3lmD+3Y2h6r2GrbRxP6jAWHzyg3Fn71Dau9Duu5Jvfk=; X-YMail-OSG: 7oKwu3AVM1nBHBLb7yK_DCAYMkwF1fyQPt0SKNAc3pQa.0U VoqQSI0l1xomDJu8jggxzFCGckFOffomVifHMjCBX.Tg21aa3OXNP2y1X5JO QJtDFaU9t5.qsB0jdNd4.60mmeraj.MYvcOmKOaUztrrmrp.wlJuDYKx4u_w PTN_0oxXJOTW_S5Ew9sI77tzV2Ip7VU9mN62kGWEZr0CDZgJGq9Apqhgj.3O svZuhAXh.pQlsYKdJ27HR.WRPE7EfYcJw0g2gXFh2JI9WgpSYH_JtHWvRFS. GeHxzg3WNHUOjnX9M7QP1GLmAn89j.2jFCOlLjuxI3Sikp9x1sICq57HA1y9 cZu6LBKyT._wO_mHMSMIKRPo7nj3wf3r0qqdK.Fu0TRD45uA6eWl81Nr4pE. SHBSW5v.9XmSPXKE.Bj5D.QtsjaacCxy0CeBSBj2cZdYhf119xluMI4Wg0Gj oup0mbTkTP9Z8FFiGk3KpdI4JWCOb9DJB_p0tUoty2xneeRxCMSEYoLyffLC f_A-- Received: from [68.194.233.166] by web160706.mail.bf1.yahoo.com via HTTP; Wed, 22 Aug 2012 16:46:44 PDT X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.121.416 Message-ID: <1345679204.91728.YahooMailClassic@web160706.mail.bf1.yahoo.com> Date: Wed, 22 Aug 2012 16:46:44 -0700 (PDT) From: Youri Adonis To: freebsd-mips@FreeBSD.org In-Reply-To: <20120216155625.665708d8.ray@dlink.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Subject: Re: Support for mips74KC X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Aug 2012 23:46:52 -0000 I am happy to report that a quick checkout from zrouter build using dir-620(mips24kc) profile allowed me to boot freebsd on a mips74kc soc, RT3883 - (Edimax BR6475nd). Below is the console output with some debug info after it crashed: ## Booting image at bc070000 ... CSYS magic number at 0xBC070000 Image Name: FreeBSD Kernel Image Created: 2012-08-20 4:08:35 UTC Image Type: MIPS Linux Kernel Image (lzma compressed) Data Size: 1041935 Bytes = 1017.5 kB Load Address: 80001000 Entry Point: 80001000 Verifying Checksum ... OK Uncompressing Kernel Image ... OK No initrd ## Transferring control to Linux (at address 80001000) ... ## Giving linux memsize in MB, 64 Starting kernel ... RT3883 runing with 500MHz clockM System bus clock - 166MHz^M System start from NOR flash storage U-Boot args (from 4 args): Environment:^M entry: mips_init() Cache info: picache_stride = 4096^M picache_loopcount = 16^M pdcache_stride = 4096^M pdcache_loopcount = 8^M cpu0: MIPS Technologies processor v76.151^M MMU: Standard TLB, 32 entries^M L1 i-cache: 4 ways of 512 sets, 32 bytes per line^M L1 d-cache: 4 ways of 256 sets, 32 bytes per line^M Config1=0xbee3519e^M Config3=0x2c20^M KDB: debugger backends: ddb^M KDB: current backend: ddb^M Copyright (c) 1992-2012 The FreeBSD Project.^M Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994^M The Regents of the University of California. All rights reserved.^M FreeBSD is a registered trademark of The FreeBSD Foundation.^M FreeBSD 10.0-CURRENT #0: Mon Aug 20 04:06:56 UTC 2012^M root@XXXXXXXX:/usr/src/zrouter/usr/src/zrouter/zrouter/tmp/mips.mipsel/usr/src/zrouter/FreeBSD/head/sys/usr/src/zrouter/usr/src/zrouter/zrouter/conf/D-Link_DIR-620 mips^M real memory = 33554432 (32768K bytes)^M avail memory = 27705344 (26MB)^M random device not loaded; using insecure entropy^M nexus0: ^M nvram2env0: Use NVRAM at 0x1f030000^M nvram2env0: on nexus0^M Trap cause = 2 (TLB miss (load or instr. fetch) - kernel mode)^M [ thread pid 0 tid 100000 ]^M Stopped at strlen+0x10: lb v0,0(v1)^M db> ^M db> help^M alltrace b break bt c call^M capture continue countfreebufsd delete dhwatch^M dump dwatch examine findstack gdb halt^M hwatch kill match next p panic^M print ps reboot reset run s^M script scripts search set show step^M t textdump thread trace unscript until^M w watch watchdog where write x^M db> where^M Tracing pid 0 tid 100000 td 0x8039f870^M 802e8ffc+c (?,?,?,?) ra 803f78b8 sp 803679750000005c sz 0^M 80006f8c+a8 (186a0,?,8036e450,?) ra 20803f78b8 sp 803691e0000002a8 sz 1^M 80006630+30 (?,?,?,?) ra 18803f78d8 sp 80364ccb00000204 sz 0^M 800066f8+ac (?,?,?,?) ra 803f78f0 sp 80364ccb000002cc sz 0^M pid 0^M db> continue^M panic: trap^M KDB: enter: panic^M [ thread pid 0 tid 100000 ]^M Stopped at kdb_enter+0x4c: lui at,0x803b^M db> continue^M Uptime: 1s^M Automatic reboot in 15 seconds - press a key on the console to abort^M --> Press a key on the console to reboot,^M --> or switch off the system now.^M Rebooting...^M Trap cause = 10 (reserved instruction - kernel mode)^M [ thread pid 0 tid 100000 ]^M Stopped at 0xbf001004: lld a2,t0,-12026^M db> continue^M panic: trap^M KDB: enter: panic^M [ thread pid 0 tid 100000 ]^M Stopped at kdb_enter+0x4c: lui at,0x803b^M db> continue^M Uptime: 1s^M Rebooting...^M Trap cause = 10 (reserved instruction - kernel mode)^M [ thread pid 0 tid 100000 ]^M Stopped at 0xbf001004: lld a2,t0,20742^M db> ^M panic: trap^M KDB: enter: panic^M [ thread pid 0 tid 100000 ]^M Stopped at kdb_enter+0x4c: lui at,0x803b^M db> ^M db> ps^M pid ppid pgrp uid state wmesg wchan cmd^M 13 0 0 0 RL [yarrow]^M 12 0 0 0 RL (threaded) [geom]^M 100008 RunQ [g_down]^M 100007 RunQ [g_up]^M 100006 RunQ [g_event]^M 11 0 0 0 WL (threaded) [intr]^M 100014 I [swi6: task queue]^M 100013 I [swi6: Giant taskq]^M 100011 I [swi5: +]^M 100005 I [swi3: vm]^M 100004 I [swi4: clock]^M 100003 I [swi1: netisr 0]^M 10 0 0 0 RL [idle]^M 1 0 0 0 ?L [kernel]^M 0 0 0 0 RLs (threaded) [kernel]^M 100015 RunQ [kqueue taskq]^M 100012 RunQ [thread taskq]^M 100009 RunQ [firmware taskq]^M 100000 Run CPU 0 [swapper]^ --- On Thu, 2/16/12, Aleksandr Rybalko wrote: > From: Aleksandr Rybalko > Subject: Re: Support for mips74KC > To: "Warner Losh" > Cc: "Youri Adonis" , "Adrian Chadd" , freebsd-mips@FreeBSD.org > Date: Thursday, February 16, 2012, 8:56 AM > On Wed, 15 Feb 2012 10:37:12 -0700 > Warner Losh > wrote: > > >> > >> On Feb 15, 2012, at 8:17 AM, Youri Adonis wrote: > >> > I have a RT-N16 which I tried freebsd on > through Rybalko's > >> > broadcom work. I cannot get it to boot yet. It > appears there is > >> > some sort of 74kc support in NetBSD > >> > >> What support is missing in FreeBSD? > > CPU OS support only differ in two things (from NetBSD code) > CPU_MIPS_HAVE_SPECIAL_CCA | (0 << > CPU_MIPS_CACHED_CCA_SHIFT) > MIPS_CP0FL_CONFIG6 > > 74k have "A 15-stage asymmetric dual-issue pipeline", 24k > have only > 8-stage, but: > 1. our gcc know nothing about 74k. > 2. clang seems same. > > I think if we add CPU_MIPS_HAVE_SPECIAL_CCA flag handling, > it may run > code compiled as mips24k, but I dunno how it will affect > performance. > > Don't have such H/W, so can't check yet. > > Hope it will be not so big pain to add 74k to gcc. > > >> > >> Warner > >> > >> _______________________________________________ > >> freebsd-mips@freebsd.org > mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-mips > >> To unsubscribe, send any mail to > >> "freebsd-mips-unsubscribe@freebsd.org" > > > -- > Alexandr Rybalko > aka Alex RAY > From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 09:05:06 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3915E1065670 for ; Thu, 23 Aug 2012 09:05:06 +0000 (UTC) (envelope-from ray@ddteam.net) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id AC70A8FC0C for ; Thu, 23 Aug 2012 09:05:05 +0000 (UTC) Received: by bkcje9 with SMTP id je9so384925bkc.13 for ; Thu, 23 Aug 2012 02:05:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references:x-mailer :mime-version:content-type:content-transfer-encoding :x-gm-message-state; bh=MWK8JusTIUyjTwxff2P0u7bGrXBHl493Q0QzNx6/T78=; b=VAivF/s0T3CZBDam0/kRTLSdXpCF3qN+ffVEzclr93eoK4E4BR9ic8dk/rllMOtPQE ywGMdN9wxjZFeRQ4D/s/3are2FPP8uxB6jpA+61EV26yi7HloI70xROrWrsEyRsJcfM8 0vt2HIVJMapxCSMsN3c0L3c5ZkYSs+0jY+7wE3NO9OMxJdZUOyqOwGDhIVH3haBxWSiS LC5xJafElunAJM/7rXWJhkDNQpssxdhJpR/MiPUmhsQYkE5hJSw4a0rG2N6kQDkbHksS YUOB4K4tswK0T82e3T28tGcQ22Df/+//R9bNR9oyhzKTDDBxYXJpGjGCsKUrDoyletty zZoA== Received: by 10.204.156.10 with SMTP id u10mr191326bkw.11.1345712704485; Thu, 23 Aug 2012 02:05:04 -0700 (PDT) Received: from rnote.ddteam.net (54-198-201-46.pool.ukrtel.net. [46.201.198.54]) by mx.google.com with ESMTPS id g6sm4101809bkg.2.2012.08.23.02.05.01 (version=SSLv3 cipher=OTHER); Thu, 23 Aug 2012 02:05:02 -0700 (PDT) Date: Thu, 23 Aug 2012 12:04:59 +0300 From: Aleksandr Rybalko To: Youri Adonis Message-Id: <20120823120459.edeb2916.ray@ddteam.net> In-Reply-To: <1345679204.91728.YahooMailClassic@web160706.mail.bf1.yahoo.com> References: <20120216155625.665708d8.ray@dlink.ua> <1345679204.91728.YahooMailClassic@web160706.mail.bf1.yahoo.com> X-Mailer: Sylpheed 3.1.2 (GTK+ 2.24.5; amd64-portbld-freebsd9.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Gm-Message-State: ALoCoQkxfukc1VUEfNl8JgATE5hVbJN6u5js0jePDP8EgInoo34iOFdpp6nPQSUk8YIlAxpwVn2T Cc: freebsd-mips@FreeBSD.org Subject: Re: Support for mips74KC X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 09:05:06 -0000 On Wed, 22 Aug 2012 16:46:44 -0700 (PDT) Youri Adonis wrote: > I am happy to report that a quick checkout from zrouter build using > dir-620(mips24kc) profile allowed me to boot freebsd on a mips74kc > soc, RT3883 - (Edimax BR6475nd). > > Below is the console output with some debug info after it crashed: > > > ## Booting image at bc070000 ... > CSYS magic number at 0xBC070000 > Image Name: FreeBSD Kernel Image > Created: 2012-08-20 4:08:35 UTC > Image Type: MIPS Linux Kernel Image (lzma compressed) > Data Size: 1041935 Bytes = 1017.5 kB > Load Address: 80001000 > Entry Point: 80001000 > Verifying Checksum ... OK > Uncompressing Kernel Image ... OK > No initrd > ## Transferring control to Linux (at address 80001000) ... > ## Giving linux memsize in MB, 64 > > Starting kernel ... > > RT3883 runing with 500MHz clockM > System bus clock - 166MHz^M > System start from NOR flash storage > U-Boot args (from 4 args): > Environment:^M > entry: mips_init() > Cache info: > picache_stride = 4096^M > picache_loopcount = 16^M > pdcache_stride = 4096^M > pdcache_loopcount = 8^M > cpu0: MIPS Technologies processor v76.151^M > MMU: Standard TLB, 32 entries^M > L1 i-cache: 4 ways of 512 sets, 32 bytes per line^M > L1 d-cache: 4 ways of 256 sets, 32 bytes per line^M > Config1=0xbee3519e^M > Config3=0x2c20^M > KDB: debugger backends: ddb^M > KDB: current backend: ddb^M > Copyright (c) 1992-2012 The FreeBSD Project.^M > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, > 1994^M The Regents of the University of California. All rights > reserved.^M FreeBSD is a registered trademark of The FreeBSD > Foundation.^M FreeBSD 10.0-CURRENT #0: Mon Aug 20 04:06:56 UTC 2012^M > root@XXXXXXXX:/usr/src/zrouter/usr/src/zrouter/zrouter/tmp/mips.mipsel/usr/src/zrouter/FreeBSD/head/sys/usr/src/zrouter/usr/src/zrouter/zrouter/conf/D-Link_DIR-620 > mips^M real memory = 33554432 (32768K bytes)^M > avail memory = 27705344 (26MB)^M > random device not loaded; using insecure entropy^M > nexus0: ^M > nvram2env0: Use NVRAM at 0x1f030000^M > nvram2env0: on nexus0^M > Trap cause = 2 (TLB miss (load or instr. fetch) - kernel mode)^M > [ thread pid 0 tid 100000 ]^M > Stopped at strlen+0x10: lb v0,0(v1)^M > db> ^M > db> help^M > alltrace b break bt c call^M > capture continue countfreebufsd delete dhwatch^M > dump dwatch examine findstack gdb halt^M > hwatch kill match next p panic^M > print ps reboot reset run s^M > script scripts search set show step^M > t textdump thread trace unscript until^M > w watch watchdog where write x^M > db> where^M > Tracing pid 0 tid 100000 td 0x8039f870^M > 802e8ffc+c (?,?,?,?) ra 803f78b8 sp 803679750000005c sz 0^M > 80006f8c+a8 (186a0,?,8036e450,?) ra 20803f78b8 sp 803691e0000002a8 sz > 1^M 80006630+30 (?,?,?,?) ra 18803f78d8 sp 80364ccb00000204 sz 0^M > 800066f8+ac (?,?,?,?) ra 803f78f0 sp 80364ccb000002cc sz 0^M > pid 0^M > db> continue^M > panic: trap^M > KDB: enter: panic^M > [ thread pid 0 tid 100000 ]^M > Stopped at kdb_enter+0x4c: lui at,0x803b^M > db> continue^M > Uptime: 1s^M > Automatic reboot in 15 seconds - press a key on the console to abort^M > --> Press a key on the console to reboot,^M > --> or switch off the system now.^M > Rebooting...^M > Trap cause = 10 (reserved instruction - kernel mode)^M > [ thread pid 0 tid 100000 ]^M > Stopped at 0xbf001004: lld a2,t0,-12026^M > db> continue^M > panic: trap^M > KDB: enter: panic^M > [ thread pid 0 tid 100000 ]^M > Stopped at kdb_enter+0x4c: lui at,0x803b^M > db> continue^M > Uptime: 1s^M > Rebooting...^M > Trap cause = 10 (reserved instruction - kernel mode)^M > [ thread pid 0 tid 100000 ]^M > Stopped at 0xbf001004: lld a2,t0,20742^M > db> ^M > panic: trap^M > KDB: enter: panic^M > [ thread pid 0 tid 100000 ]^M > Stopped at kdb_enter+0x4c: lui at,0x803b^M > db> ^M > db> ps^M > pid ppid pgrp uid state wmesg wchan cmd^M > 13 0 0 0 RL [yarrow]^M > 12 0 0 0 RL (threaded) [geom]^M > 100008 RunQ [g_down]^M > 100007 RunQ [g_up]^M > 100006 RunQ [g_event]^M > 11 0 0 0 WL (threaded) [intr]^M > 100014 I [swi6: task > queue]^M 100013 I [swi6: > Giant taskq]^M 100011 I > [swi5: +]^M 100005 I > [swi3: vm]^M 100004 I > [swi4: clock]^M 100003 I > [swi1: netisr 0]^M 10 0 0 0 RL > [idle]^M 1 0 0 0 ?L [kernel]^M > 0 0 0 0 RLs (threaded) [kernel]^M > 100015 RunQ [kqueue taskq]^M > 100012 RunQ [thread taskq]^M > 100009 RunQ [firmware > taskq]^M 100000 Run CPU 0 > [swapper]^ > > > > > --- On Thu, 2/16/12, Aleksandr Rybalko wrote: > > > From: Aleksandr Rybalko > > Subject: Re: Support for mips74KC > > To: "Warner Losh" > > Cc: "Youri Adonis" , "Adrian Chadd" > > , freebsd-mips@FreeBSD.org Date: Thursday, > > February 16, 2012, 8:56 AM On Wed, 15 Feb 2012 10:37:12 -0700 > > Warner Losh > > wrote: > > > > >> > > >> On Feb 15, 2012, at 8:17 AM, Youri Adonis wrote: > > >> > I have a RT-N16 which I tried freebsd on > > through Rybalko's > > >> > broadcom work. I cannot get it to boot yet. It > > appears there is > > >> > some sort of 74kc support in NetBSD > > >> > > >> What support is missing in FreeBSD? > > > > CPU OS support only differ in two things (from NetBSD code) > > CPU_MIPS_HAVE_SPECIAL_CCA | (0 << > > CPU_MIPS_CACHED_CCA_SHIFT) > > MIPS_CP0FL_CONFIG6 > > > > 74k have "A 15-stage asymmetric dual-issue pipeline", 24k > > have only > > 8-stage, but: > > 1. our gcc know nothing about 74k. > > 2. clang seems same. > > > > I think if we add CPU_MIPS_HAVE_SPECIAL_CCA flag handling, > > it may run > > code compiled as mips24k, but I dunno how it will affect > > performance. > > > > Don't have such H/W, so can't check yet. > > > > Hope it will be not so big pain to add 74k to gcc. > > > > >> > > >> Warner > > >> > > >> _______________________________________________ > > >> freebsd-mips@freebsd.org > > mailing list > > >> http://lists.freebsd.org/mailman/listinfo/freebsd-mips > > >> To unsubscribe, send any mail to > > >> "freebsd-mips-unsubscribe@freebsd.org" > > > > > > -- > > Alexandr Rybalko > > aka Alex RAY > > Hi Youri, big thanks for report! can't wait to get some 74k device to play with. :) Any plans to submit patch to zrouter.org? :) Thanks again! WBW -- Aleksandr Rybalko From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 18:04:59 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8861C106564A for ; Thu, 23 Aug 2012 18:04:59 +0000 (UTC) (envelope-from vrtuff@yahoo.com) Received: from nm13-vm0.bullet.mail.bf1.yahoo.com (nm13-vm0.bullet.mail.bf1.yahoo.com [98.139.213.79]) by mx1.freebsd.org (Postfix) with SMTP id 0B0778FC0A for ; Thu, 23 Aug 2012 18:04:58 +0000 (UTC) Received: from [98.139.212.153] by nm13.bullet.mail.bf1.yahoo.com with NNFMP; 23 Aug 2012 18:04:51 -0000 Received: from [98.139.215.249] by tm10.bullet.mail.bf1.yahoo.com with NNFMP; 23 Aug 2012 18:04:51 -0000 Received: from [127.0.0.1] by omp1062.mail.bf1.yahoo.com with NNFMP; 23 Aug 2012 18:04:51 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 866574.35851.bm@omp1062.mail.bf1.yahoo.com Received: (qmail 34322 invoked by uid 60001); 23 Aug 2012 18:04:51 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1345745091; bh=+pbYxD9IofJBWVl55YV5Eo4ejMupcw7GJzl9+/lcJ+w=; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=Tyuc83Fdwy3KOqMCyBz2aB3npViG1eSFL/jEcqlOiwe+H7QlwSvcVLMKtI3HhuKkfDUNME2xsjBDe3BB2A6+EA2WW1/F3Ebg3nAeVd8yBPLw0ZNCRYI5DJ3QqpKug/E+tjkpzhxzLlALerOzj91Y5qOIjJ2qSu1SDc7Akv6xqg4= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=jKg24Db384uwjqbBVxW3qKZVxgmAd3UmTZtyMq6xTiGxgoQ4y7EaRwRSsAVROETllK6kek+BJ2Uq6e9DGc2rmJKNWHUYIo88AwdtWzYFyOv/cjdu1PhH6yCYkTCDCsTIYZNWHM1udjZIHk8bJCAajfGB+0Io4WLT7qeEKltB2s8=; X-YMail-OSG: bCDXLxwVM1mWcHNXac_JdZbmHvWhppbp5mmKJvj2RWD9djI tzbng6yOsoCTkQ05BdkTTakuivbl27sK24HpWXlRxylBIQwfG9Q_FkAiwhfA KZVDcuU1X7MUwEDowSRzRdcCh5iw8KTCzdCBLJCqQGF7PkqtO_xR.K3ZO77N OnKp5rdyxvxkkhtMSceBFMLIveCNViiM4cGG9H8AhXA5MiIyfFxD9nZNXxSF _pxFu5moS5F2gW2xykkTHhzJFH_n25etf6cMAYbr5OPaRgZIv51Lxkj0ASLW SXrfpfbTc1adH3HtVj.Kyj8hpt4TwBPTW54H42Y8EXEy20UoXBlYEhZkylg6 xe.iXxgd5grfO5.K7wpcIi0dSF360TA6GbOx.Gt34QQpJcuHZ5U0UVEu6ZnY Zg9lSXqLR3ryW6OKeMbC2AQ.9n5ewd6quiBdTGWPSX0x3ydBPdxkAjv00kDR M_UcfRxMnP9FZD_MkZx_Hk1kCY7a4TiT1Tm5DaNlM8h3mhFiULK_xLqS816w 5QF_joTwsuJO25rx2TF6tJjtzJ4pTvM5RjW3luVz60wYW3yP6xIqV52Rl Received: from [207.198.161.12] by web160701.mail.bf1.yahoo.com via HTTP; Thu, 23 Aug 2012 11:04:51 PDT X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.121.416 Message-ID: <1345745091.28628.YahooMailClassic@web160701.mail.bf1.yahoo.com> Date: Thu, 23 Aug 2012 11:04:51 -0700 (PDT) From: Youri Adonis To: Aleksandr Rybalko In-Reply-To: <20120823120459.edeb2916.ray@ddteam.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-mips@FreeBSD.org Subject: Re: Support for mips74KC X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 18:04:59 -0000 > Hi Youri, > > big thanks for report! can't wait to get some 74k device to > play > with. :) > > Any plans to submit patch to zrouter.org? :) > > Thanks again! > > WBW > -- > Aleksandr Rybalko Hey Aleks, I did not apply any patch. I just compiled it "as is". So mips74kc is not too far off current mips24kc I sure will submit patches once I can get this going Regards, From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 21:24:15 2012 Return-Path: Delivered-To: freebsd-mips@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AE1CB1065670; Thu, 23 Aug 2012 21:24:15 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 816268FC12; Thu, 23 Aug 2012 21:24:15 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7NLOFJ7070533; Thu, 23 Aug 2012 21:24:15 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7NLOFvO070529; Thu, 23 Aug 2012 21:24:15 GMT (envelope-from linimon) Date: Thu, 23 Aug 2012 21:24:15 GMT Message-Id: <201208232124.q7NLOFvO070529@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-mips@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/170931: [mips] [patch] Remove duplicated GEOM_PART_* options from MIPS kernels X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 21:24:15 -0000 Old Synopsis: [patch] Remove duplicated GEOM_PART_* options from MIPS kernels New Synopsis: [mips] [patch] Remove duplicated GEOM_PART_* options from MIPS kernels Responsible-Changed-From-To: freebsd-bugs->freebsd-mips Responsible-Changed-By: linimon Responsible-Changed-When: Thu Aug 23 21:24:04 UTC 2012 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=170931 From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 21:28:29 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 848351065670 for ; Thu, 23 Aug 2012 21:28:29 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [76.96.30.16]) by mx1.freebsd.org (Postfix) with ESMTP id 5D2078FC18 for ; Thu, 23 Aug 2012 21:28:29 +0000 (UTC) Received: from omta10.emeryville.ca.mail.comcast.net ([76.96.30.28]) by qmta01.emeryville.ca.mail.comcast.net with comcast id qCUQ1j00D0cQ2SLA1MUPpT; Thu, 23 Aug 2012 21:28:23 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta10.emeryville.ca.mail.comcast.net with comcast id qMUN1j00P4NgCEG8WMUPgg; Thu, 23 Aug 2012 21:28:23 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7NLSK2M025408; Thu, 23 Aug 2012 15:28:20 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, freebsd-arm@freebsd.org Content-Type: text/plain; charset="us-ascii" Date: Thu, 23 Aug 2012 15:28:20 -0600 Message-ID: <1345757300.27688.535.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Subject: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 21:28:29 -0000 A recent innocuous change to the USB driver code caused intermittant errors in the umass(4) driver on ARM and MIPS platforms, and this message is some background on partial cacheline flushes, and info on what I found while investigating the cause, which is rooted in the DMA cache coherency code. First I need to say that when I say ARM in this message I mean "ARM and MIPS and any other platform where cache coherency around DMA operations is maintained with help from software (as opposed to being implemented purely in hardware as it is in i386/amd64)." I have no experience on MIPS, but I believe that it's similar to ARM in regards to cache coherency. As far as I know, this is not specific to VIVT caches, but rather is specific to software cache coherency, so it probably applies to armv6/v7 as well as v4 and v5 that I'm working with. Over the years, we've had a series of glitches and patches in the partial cacheline flush logic for arm. Even with all the fixes, I've thought that there are two ways that the scheme could fail, but I've never been able to demonstrate it experimentally, and empirically it seems that the failures are rare. Both ways involve the fact that we have several software entities trying to maintain the caches concurrently, without knowledge of each others' actions or needs. The two ways are basically two variations of a situation where a dirty cacheline is flushed by the software while a DMA operation that overlaps that cacheline is in progress: * A cpu_dcache_wbinv_all() call happens after some DMA data has hit main memory, but before bus_dmamap_sync(POSTREAD) is called. * Two independent DMA operations are happening in different parts of the same cacheline and some DMA data from both devices has hit main memory; whichever operation does the POSTREAD sync first can wipe out data in main memory from the other operation that is still in progress. For problems to happen, the CPU has to also modify the same memory location / cacheline as the DMA, so that the cache holds the newest data for part of the cacheline, and main memory holds the newest data for part of the cacheline. Our logic for handling POSTREAD partial cacheline flushes creates this condition even if it doesn't already exist on entry to the sync routine. The wbinv_all() situation seemed to me the most likely to occur. It gets called from a variety of places for a variety of reasons. It is called from a several places in pmap.c; it appears to me that many of those invocations can happen at completely arbitrary points with respect to any IO that's in progress. Another way wbinv_all() can get invoked is during a call to wbinv_range() or even just inv_range(), when the range to be invalidated is so large that it's too inefficient to loop through the range discarding a line at a time on a given platform. In those cases, for some arm platforms, the inv_range() implementation just calls wbinv_all() internally. The implication of the last paragraph is that wbinv_all() can potentially be invoked as part of the busdma sync operations for any IO, PREREAD, PREWRITE, or POSTREAD, from any device at any time. A recent USB driver change moved some things around in memory, such that a small (13 byte) IO buffer became split across two cachelines, and suddenly we had intermittant (but fairly frequent) failures reported by umass(4). Some logging from the usb driver showed that there was stale data from previous IO operations in part of the IO buffer. I added some code to pre-initialize the buffer to various byte patterns before starting the IO, and after the IO, part of the buffer would contain those patterns, and the rest of the buffer (after the cacheline split point) contained newer data from the IO. It looked pretty conclusively as if the partial cacheline flush logic was failing. First I investigated the logic for handling such splits, but it was working correctly. So I moved on to assuming that the cause was one of the two potential problems I've long suspected. I received a helpful clue from Hans that the buffer in question was allocated once at device creation and remained allocated from that point on. That made it easy to save that buffer pointer when it was created, and write wrappers for all the cache writeback and invalidate routines that checked whether the cacheline containing that buffer was part of the cache operation. What I expected to see was that USB would call the busdma sync ops before starting the IO, and then before it called the post-IO sync ops I would see that something else in the system called wbinv_all() or a [wb]inv_range() that included the umass buffer address. What I actually saw was that that never happened. Not even once. Very rarely I would see some other [wb]inv_range() calls happen, but the ranges never involved the umass buffer, and the unit I'm doing these tests on (a DreamPlug) is not one that ever turns an inv_range into wbinv_all. It eventually occurred to me that I had been overlooking the most obvious way a dirty cacheline can get written back to main memory: the cache hardware needs to evict a line to make room for a new incoming line, and the line it has chosen is dirty and has to be written back before being evicted. Unfortunately, there is no way to instrument the software to detect that happening, so now I'm in the position of proving something based on the complete lack of evidence that anything else is the cause. That's a great way to promote a conspiracy theory, not so great for debugging. In addition to showing that no software-triggered flush/invalidate operations are affecting the cacheline, I was able to show that the problem wasn't just that a partial cacheline flush was involved, but that error condition depended on the specific memory addresses (and thus the specific cacheline) involved. At the point in the usb code where that buffer is allocated I changed the code to add 32 bytes to the buffer offset, so that the buffer is still split across two cachelines in exactly the same way as before, but now it's two different cachelines. When doing this, the error doesn't occur. I think that may lend some weight to the theory that it is hardware-based cacheline eviction which is causing a flush of a dirty cacheline while IO into that memory is in progress, but it's just more circumstantial evidence. I think the intermittant-but-frequent nature of the error may also be circumstantial evidence that hardware eviction is the cause. My DreamPlug unit has a 4-way set-associative cache that selects one of the ways at random when it needs to evict a line for refill. That would seem to imply that there's a one in four chance that the cacheline holding the umass status buffer is the one that gets hit, and that seems to match the symptoms I see of "this usb drive kind of works but there are tons of errors spewing on the console about it". Sometimes you get several failures in a row and the drive fails to attach, but most of the time it limps along with lots of errors followed by succesful retries. I considered trying to lock the cacheline in question into the cache as a way of confirming this situation (that should make the error go away). It turns out that's not especially easy to do on this platform, and you can't lock a single cacheline, you have to lock a whole cache way. That's a pretty big change that will perturb system operations in general, it may be hard to draw conclusions from the results. The ARM Architecture Reference Manual mentions the following guidelines as part of the strategy for handling DMA/cache coherency: * marking the memory areas involved in the DMA operation as uncachable and/or unbufferable * cleaning and/or invalidating the data cache, at least with respect to the address range involved in the DMA operation * draining the write buffer * restrictions on processor accesses to the address range involved in the DMA operation until it is known that the DMA operation is complete. Our partial cacheline flush logic is trying to wish away the last bullet item, but now I think we can never successfully do so. Until last week I thought we had a theoretical problem that could eventually be fixed with a sufficiently-clever cache maintenance implementation that somehow avoided having unrelated parts of the OS interfering with each other's operations. Now it appears that hardware operations we have no direct control over can also lead to memory corruption, and no amount of software cleverness is ever going to allow concurrent CPU and DMA access to the same memory without disabling the cache for that memory range. At this point I was going to launch into some "what we can do about it" rambling, but this is long enough already; I think I'll leave this message as a summary of where we've come from and what I learned in the past few days, and leave "what next" for followups. -- Ian From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 21:36:36 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F64A106564A for ; Thu, 23 Aug 2012 21:36:36 +0000 (UTC) (envelope-from ray@ddteam.net) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9EC038FC14 for ; Thu, 23 Aug 2012 21:36:35 +0000 (UTC) Received: by weyx56 with SMTP id x56so5241273wey.13 for ; Thu, 23 Aug 2012 14:36:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references:x-mailer :mime-version:content-type:content-transfer-encoding :x-gm-message-state; bh=ywibnDTJGc16iFAANogxCMkZEEC8RcFCbzWWutSkTe4=; b=kA6rc1sKrNKUb7AHAYkkPirUlOWENMaA0cHWGZMMV9VAXDDhl+wa7HIsDBVQZrwtSr JEGQOJCGzYKICaNADMCme9csdh5w9y3QcDoP0z/5wbjkCrP7lygySZpXslAOM19txTNw T+soafyBg8SEDIuwb9jDYfVc6t1kV+RCTNmHc6nUR9sxkWRfBSbBaWuLk8xEhPcCzBy+ HE68y0JeDDEgfffyv/2Pys1sN9Vr8K/mb8Pvy2UEQNe4G4HFJDsH0+GCZhscq77pt2Ah 4j2U4LHQH/omkEE8SU0SITHqJQhhqSZHaIunb76bsmQidTIw8uCvo+o3IzWzpnE9zWri vh3A== Received: by 10.216.131.22 with SMTP id l22mr1455941wei.96.1345757794457; Thu, 23 Aug 2012 14:36:34 -0700 (PDT) Received: from rnote.ddteam.net (117-238-133-95.pool.ukrtel.net. [95.133.238.117]) by mx.google.com with ESMTPS id l6sm755301wiz.4.2012.08.23.14.36.32 (version=SSLv3 cipher=OTHER); Thu, 23 Aug 2012 14:36:33 -0700 (PDT) Date: Fri, 24 Aug 2012 00:36:30 +0300 From: Aleksandr Rybalko To: Youri Adonis Message-Id: <20120824003630.9ae5fb63.ray@ddteam.net> In-Reply-To: <1345745091.28628.YahooMailClassic@web160701.mail.bf1.yahoo.com> References: <20120823120459.edeb2916.ray@ddteam.net> <1345745091.28628.YahooMailClassic@web160701.mail.bf1.yahoo.com> X-Mailer: Sylpheed 3.1.2 (GTK+ 2.24.5; amd64-portbld-freebsd9.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Gm-Message-State: ALoCoQnnHNrxzaFiUWlMGaUSp2UprAQf/9okjS9ueL1rYDSRaAYuDphimosYC6hhDAh/alH3kpGa Cc: freebsd-mips@FreeBSD.org Subject: Re: Support for mips74KC X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 21:36:36 -0000 On Thu, 23 Aug 2012 11:04:51 -0700 (PDT) Youri Adonis wrote: > > Hi Youri, > > > > big thanks for report! can't wait to get some 74k device to > > play > > with. :) > > > > Any plans to submit patch to zrouter.org? :) > > > > Thanks again! > > > > WBW > > -- > > Aleksandr Rybalko > > Hey Aleks, Hi Youri! > > I did not apply any patch. I just compiled it "as is". So mips74kc is > not too far off current mips24kc Yeah, I understand that, and I think for mips74kc we have to extend toolchain, but not system code. Since there we have biggest pipeline, so MIPS standard: 1. branch 2. move (which will be done before branch) 3. move (after we return from branch) may change to: 1. branch 2. move (which will be done before branch) 3. move (which will be done before branch unexpectedly for 24k compiller) so we must be careful here. > I sure will submit patches once I can get this going Thank you! > > Regards, WBW -- Aleksandr Rybalko From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 21:50:39 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 02A271065673 for ; Thu, 23 Aug 2012 21:50:39 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id C2C8B8FC14 for ; Thu, 23 Aug 2012 21:50:38 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so2300538pbb.13 for ; Thu, 23 Aug 2012 14:50:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=WSYlzD3T5FYG0E17Iqr7sZ9KknY3rYR8v2QRn3PxG8E=; b=mJObbkcJpDJdomY/ZVjDuIv6OD3SZSzFogPoPU7Mk+5N9XbJRQwVAOwMLa4ha46hDg +PvkvG1rr/zCJu8XkB56r7FrtSRB05Ee+u08n10orw/wvUjzAJLWhJ1BNLmIe6TbtCxp KdnUOUCfxy7o3KFFPvMuxW5mzu0bM5ri1tf8fY7DXzYnKCzyFjFtCEr8HlRQP50ea1rC aLIdn4dCum+7yjiLn0s6n5dzn1QKOkcFiqoQvmN4nJIX7dwmGc5pu7ymkjkTWYo1KQf2 MvYDOOpKDo+5Ew8qIPcvVwpxzjjLmSkNoWXp0LFW+UTvdEe9Sm10ceN6/KCd6zWRE+9F rdrw== Received: by 10.66.80.193 with SMTP id t1mr6152901pax.69.1345758638309; Thu, 23 Aug 2012 14:50:38 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id gf3sm6757324pbc.74.2012.08.23.14.50.36 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 23 Aug 2012 14:50:37 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1345757300.27688.535.camel@revolution.hippie.lan> Date: Thu, 23 Aug 2012 15:50:35 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQmvIkSIBKCnzmCgEOQBNrCAesr3WJTkbX7j6tTzR6CewARoKPnK8uNaWSR15EXtRmvE0aTK Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 21:50:39 -0000 On Aug 23, 2012, at 3:28 PM, Ian Lepore wrote: > A recent innocuous change to the USB driver code caused intermittant > errors in the umass(4) driver on ARM and MIPS platforms, and this I think the proper solution is to segregate DMA and non-DMA parts of = structures so that you don't have both sharing a cache line. I also wonder why we don't allocate the DMA memory for these structures = separately from the non-DMA parts. This would eliminate the = USB_CACHE_BYTES kludge (which is CPU dependent, not arch dependent) and = move the knowledge of this junk into busdma layer where it belongs. = =46rom my understanding of the issue, this would completely eliminate = the problem forever! Sharing a cacheline between something that is DMA aware and something = that is just begging for trouble. We're doing more work than we need = to to support this dubious feature and we'd be miles ahead if we could = not share at all. Warner From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 22:30:07 2012 Return-Path: Delivered-To: freebsd-mips@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 93C991065674 for ; Thu, 23 Aug 2012 22:30:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 198DF8FC14 for ; Thu, 23 Aug 2012 22:30:06 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7NMU556071993 for ; Thu, 23 Aug 2012 22:30:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7NMU5at071990; Thu, 23 Aug 2012 22:30:05 GMT (envelope-from gnats) Date: Thu, 23 Aug 2012 22:30:05 GMT Message-Id: <201208232230.q7NMU5at071990@freefall.freebsd.org> To: freebsd-mips@FreeBSD.org From: dfilter@FreeBSD.ORG (dfilter service) Cc: Subject: Re: kern/170931: commit references a PR X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dfilter service List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 22:30:07 -0000 The following reply was made to PR kern/170931; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/170931: commit references a PR Date: Thu, 23 Aug 2012 22:24:09 +0000 (UTC) Author: ray Date: Thu Aug 23 22:23:56 2012 New Revision: 239625 URL: http://svn.freebsd.org/changeset/base/239625 Log: Remove duplicated GEOM_PART_* options. PR: 170931 Approved by: adrian Modified: head/sys/mips/conf/AP93 head/sys/mips/conf/AP96 head/sys/mips/conf/PB47 head/sys/mips/conf/PB92 head/sys/mips/conf/ROUTERSTATION head/sys/mips/conf/RSPRO head/sys/mips/conf/RSPRO_STANDALONE head/sys/mips/conf/TP-WN1043ND Modified: head/sys/mips/conf/AP93 ============================================================================== --- head/sys/mips/conf/AP93 Thu Aug 23 21:32:02 2012 (r239624) +++ head/sys/mips/conf/AP93 Thu Aug 23 22:23:56 2012 (r239625) @@ -108,8 +108,6 @@ options USB_HOST_ALIGN=32 #device da # Read MSDOS formatted disks -options GEOM_PART_BSD -options GEOM_PART_MBR #options MSDOSFS # GPIO Bus Modified: head/sys/mips/conf/AP96 ============================================================================== --- head/sys/mips/conf/AP96 Thu Aug 23 21:32:02 2012 (r239624) +++ head/sys/mips/conf/AP96 Thu Aug 23 22:23:56 2012 (r239625) @@ -22,8 +22,6 @@ options AR71XX_REALMEM=64*1024*1024 options AR71XX_ENV_UBOOT # For DOS - enable if required -options GEOM_PART_BSD -options GEOM_PART_MBR options MSDOSFS # uncompress - to boot read-only lzma natively from flash Modified: head/sys/mips/conf/PB47 ============================================================================== --- head/sys/mips/conf/PB47 Thu Aug 23 21:32:02 2012 (r239624) +++ head/sys/mips/conf/PB47 Thu Aug 23 22:23:56 2012 (r239625) @@ -27,8 +27,6 @@ options AR71XX_ENV_UBOOT options AR71XX_REALMEM=64*1024*1024 # For DOS - enable if required -options GEOM_PART_BSD -options GEOM_PART_MBR options MSDOSFS # uncompress - to boot read-only lzma natively from flash Modified: head/sys/mips/conf/PB92 ============================================================================== --- head/sys/mips/conf/PB92 Thu Aug 23 21:32:02 2012 (r239624) +++ head/sys/mips/conf/PB92 Thu Aug 23 22:23:56 2012 (r239625) @@ -106,8 +106,6 @@ options USB_HOST_ALIGN=32 #device da # Read MSDOS formatted disks -options GEOM_PART_BSD -options GEOM_PART_MBR # options MSDOSFS # GPIO Bus Modified: head/sys/mips/conf/ROUTERSTATION ============================================================================== --- head/sys/mips/conf/ROUTERSTATION Thu Aug 23 21:32:02 2012 (r239624) +++ head/sys/mips/conf/ROUTERSTATION Thu Aug 23 22:23:56 2012 (r239625) @@ -16,8 +16,6 @@ device geom_uzip # compressed in-memory options GEOM_UZIP # For DOS -options GEOM_PART_BSD -options GEOM_PART_MBR options MSDOSFS # Boot path - redboot MFS Modified: head/sys/mips/conf/RSPRO ============================================================================== --- head/sys/mips/conf/RSPRO Thu Aug 23 21:32:02 2012 (r239624) +++ head/sys/mips/conf/RSPRO Thu Aug 23 22:23:56 2012 (r239625) @@ -17,8 +17,6 @@ device geom_uzip # compressed in-memory options GEOM_UZIP # For DOS -options GEOM_PART_BSD -options GEOM_PART_MBR options MSDOSFS # For etherswitch support Modified: head/sys/mips/conf/RSPRO_STANDALONE ============================================================================== --- head/sys/mips/conf/RSPRO_STANDALONE Thu Aug 23 21:32:02 2012 (r239624) +++ head/sys/mips/conf/RSPRO_STANDALONE Thu Aug 23 22:23:56 2012 (r239625) @@ -18,8 +18,6 @@ device geom_uzip # compressed in-memory options GEOM_UZIP # For DOS -options GEOM_PART_BSD -options GEOM_PART_MBR options MSDOSFS # .. first DOS-partitioned, BSD sliced flash disk Modified: head/sys/mips/conf/TP-WN1043ND ============================================================================== --- head/sys/mips/conf/TP-WN1043ND Thu Aug 23 21:32:02 2012 (r239624) +++ head/sys/mips/conf/TP-WN1043ND Thu Aug 23 22:23:56 2012 (r239625) @@ -29,8 +29,6 @@ device rtl8366rb # read MSDOS formatted disks - USB options MSDOSFS -options GEOM_PART_BSD -options GEOM_PART_MBR # Enable the uboot environment stuff rather then the # redboot stuff. _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org" From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 22:39:40 2012 Return-Path: Delivered-To: freebsd-mips@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4AB831065670; Thu, 23 Aug 2012 22:39:40 +0000 (UTC) (envelope-from ray@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1E61F8FC0C; Thu, 23 Aug 2012 22:39:40 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7NMde7U077526; Thu, 23 Aug 2012 22:39:40 GMT (envelope-from ray@freefall.freebsd.org) Received: (from ray@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7NMddfH077522; Thu, 23 Aug 2012 22:39:39 GMT (envelope-from ray) Date: Thu, 23 Aug 2012 22:39:39 GMT Message-Id: <201208232239.q7NMddfH077522@freefall.freebsd.org> To: loos.br@gmail.com, ray@FreeBSD.org, freebsd-mips@FreeBSD.org From: ray@FreeBSD.org Cc: Subject: Re: kern/170931: [mips] [patch] Remove duplicated GEOM_PART_* options from MIPS kernels X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 22:39:40 -0000 Synopsis: [mips] [patch] Remove duplicated GEOM_PART_* options from MIPS kernels State-Changed-From-To: open->closed State-Changed-By: ray State-Changed-When: Thu Aug 23 22:38:20 UTC 2012 State-Changed-Why: Fixed. http://www.freebsd.org/cgi/query-pr.cgi?pr=170931 From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 23:09:55 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EB858106566B; Thu, 23 Aug 2012 23:09:55 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id B2EEE8FC16; Thu, 23 Aug 2012 23:09:55 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so2387126pbb.13 for ; Thu, 23 Aug 2012 16:09:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=lPP5YvtAv3fjOFO37wLOSmHkqwKO0tRK39RRHX3KsdA=; b=iJlel0NdvHa0pb0YIuPboM2W+Rhof6sYq13DQpJWAFdf6lK+rJ5UaFiLI+eYNL7Q1h uqaT7qobOE4KQkVME6sDZV2/NaWe2eFrX6VP1YWOUw41MY5mf4sneuJNr4sscZMIiEGm wRkDb3DXJcY+Uj4x3p4A1gRJHMSw6UQNlIU3CvJ2vvecgXvW5nzwmwRe5Hv2gGsIlwWl L7evO1FlXhUm3hEt/RrblcgQpJmFvp/Kcq2GLYMmIKsSIBFkCijwlha/McepSbcOowMD kMSao4x61/eWBKHA36z7yml0i9E9dHQR+dmw3ddp0iu4WSaD+Ugid52ev7CrW2OsBozE 59ww== MIME-Version: 1.0 Received: by 10.66.78.73 with SMTP id z9mr6742852paw.9.1345763395120; Thu, 23 Aug 2012 16:09:55 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.36.106 with HTTP; Thu, 23 Aug 2012 16:09:55 -0700 (PDT) In-Reply-To: <1345757300.27688.535.camel@revolution.hippie.lan> References: <1345757300.27688.535.camel@revolution.hippie.lan> Date: Thu, 23 Aug 2012 16:09:55 -0700 X-Google-Sender-Auth: knDTxNAgW4qmvzZwfhemm3Ea6BQ Message-ID: From: Adrian Chadd To: Ian Lepore Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:09:56 -0000 [snip] Whoa, there's USB code that has these small buffers straddle cache lines? Why aren't they just allocated to always be in their own separate buffers, so they don't ever have to worry about overlapping cache lines? Adrian From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 23:10:02 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C4061065672 for ; Thu, 23 Aug 2012 23:10:02 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta12.emeryville.ca.mail.comcast.net (qmta12.emeryville.ca.mail.comcast.net [76.96.27.227]) by mx1.freebsd.org (Postfix) with ESMTP id 652C98FC0C for ; Thu, 23 Aug 2012 23:10:02 +0000 (UTC) Received: from omta06.emeryville.ca.mail.comcast.net ([76.96.30.51]) by qmta12.emeryville.ca.mail.comcast.net with comcast id qD6g1j00716AWCUACP9w9L; Thu, 23 Aug 2012 23:09:56 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta06.emeryville.ca.mail.comcast.net with comcast id qP9v1j00g4NgCEG8SP9ws5; Thu, 23 Aug 2012 23:09:56 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7NN9spk025527; Thu, 23 Aug 2012 17:09:54 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: freebsd-arch@freebsd.org, freebsd-arm@freebsd.org, freebsd-mips@freebsd.org In-Reply-To: <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> Content-Type: text/plain; charset="us-ascii" Date: Thu, 23 Aug 2012 17:09:53 -0600 Message-ID: <1345763393.27688.578.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:10:02 -0000 On Thu, 2012-08-23 at 15:50 -0600, Warner Losh wrote: > On Aug 23, 2012, at 3:28 PM, Ian Lepore wrote: > > A recent innocuous change to the USB driver code caused intermittant > > errors in the umass(4) driver on ARM and MIPS platforms, and this > > I think the proper solution is to segregate DMA and non-DMA parts of structures so that you don't have both sharing a cache line. > > I also wonder why we don't allocate the DMA memory for these structures separately from the non-DMA parts. This would eliminate the USB_CACHE_BYTES kludge (which is CPU dependent, not arch dependent) and move the knowledge of this junk into busdma layer where it belongs. From my understanding of the issue, this would completely eliminate the problem forever! > > Sharing a cacheline between something that is DMA aware and something that is just begging for trouble. We're doing more work than we need to to support this dubious feature and we'd be miles ahead if we could not share at all. > > Warner > It seems to me that what we have here is a new type of constraint on DMA operations, and we need a way to communicate that constraint from the part of the platform support code that knows about it to the drivers and driver support code that needs to know. The way we communicate DMA constraints is with a busdma tag, but right now that tag only communicates constraints that were needed for ISA and PCI busses, namely buffer alignment, boundary-crossing restrictions, and exclusion regions. Now we have a new type of constraint, I think of it as "granularity". In effect, we have a DMA system that can only do DMA in cacheline sized chunks. Even when the IO size -- and thus the number of "bits on the wire" -- is less than the cacheline size, at the end of the DMA operation (which includes the software-assisted coherency operations) the number of bytes in memory that may be modified is the size of a cacheline. This is because "the DMA system" is not just the engine that moves bytes around, it's the combination of hardware and software that work together to maintain cache coherency. Ideally we'd find a way to communicate this new constraint using the existing mechanism, the busdma tag, and ideally we'd not have to change every existing call to bus_dma_tag_create() to add a new parm. As I understand it, parent tags are now passed down through the newbus hierarchy consistantly, such that a tag at the nexus level could describe a platform requirement such as granularity, and all devices and the helper code they use will have access to that constraint via inheritance from ancestors' tags. Maybe we could have a new flavor of bus_dma_tag_create() that takes a struct of parms, and existing calls wouldn't have to be changed. Communicating the constraint is only part of the problem; it also has to be easy for drivers to work with that constraint, especially drivers that are not targeted specifically at platforms with granular DMA. I think we can achieve a huge chunk of that purely within the arm/mips implementation of bus_dmamem_alloc(), but even so there would be a new conceptual limitation on using that routine: it is specifically for allocating DMA buffers, and that means that there will be a new a rule that the CPU cannot access any memory within that buffer while an IO operation is in progress. I'd also like to say there's a new rule that you cannot subdivide a buffer obtained from bus_dmamem_alloc() into multiple buffers, or into a combination of DMA and CPU-accessed data. That would be bad news for the USB subsystem, and perhaps other drivers. If this idea is either impossible or particularly contentious, then I guess we'd need some new helper routines so that a driver can subdivide the memory in a way that doesn't violate any constraints implied by the tag used to allocate the buffer. Not all IO occurs using buffers obtained from bus_dmamem_alloc(), and I doubt we can reasonably ever require that it be so. I think the only hope we have of handling that problem is to bounce the requests that don't meet the granularity constraint, just as we'd have to do if the caller-supplied buffer fell into an exclusion zone or violated an alignment or boundary constraint. When I've tossed this idea out in the past there was instant resistance. Yeah, bounce buffers are massively inefficient, but my experience has been that most of the IO that isn't aligned and sized to a multiple of a cacheline is small IO (a few to a few dozen bytes). I've never seen a case of page-sized or larger IO requests that required partial-cacheline handling. I'm sure some examples exist, but they're probably more the exception than the rule. (And the bad performance you'd get from bouncing and copying massive bulk data flow would be a powerful incentive to track down the culprit and improve the driver.) -- Ian From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 23:26:26 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A6831065670 for ; Thu, 23 Aug 2012 23:26:26 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 1292B8FC0A for ; Thu, 23 Aug 2012 23:26:25 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so2404540pbb.13 for ; Thu, 23 Aug 2012 16:26:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=huWGlQ/K+J9q23N2Dy+i8d3okVdSWb7iIk3+N02V218=; b=YfKNPahQCrou2U4Q1U4stX2TJDFxz4BYJ0KvY5YiRw4/auKAcOx9i1gYq1M+2qFl9K 6HGNMnrm2CnP2EJzUeqHpR7BJZzBXcQoGp6HZRR0G/vCt9L76tM366zas//VX+fE6QXO xGw0oYwgRvzXrZL0UhUyubitzzddSUO2kvJixCeFF3npI4xBmteYQSlgy9BkmdzobaU/ nis2VPvdWUN7keQyw2dOiuDllA7Bvtb+EFYRvha6cu/hnfRjUe79oEo2ZZs03eK5GooI J6exfluNdq3sQToZ9Ak0iRdVJk52OCRdwMLVb95CDlCCbv26MwBu72c19I/tyuKosf/2 bs6g== Received: by 10.68.196.193 with SMTP id io1mr8509341pbc.17.1345764385480; Thu, 23 Aug 2012 16:26:25 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id pj10sm6923195pbb.46.2012.08.23.16.26.24 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 23 Aug 2012 16:26:25 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1345763393.27688.578.camel@revolution.hippie.lan> Date: Thu, 23 Aug 2012 17:26:19 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQlS+8VNNlDUQuomr0fYnz3Q0EHVIKHoR3CRrawN4rXayM8iLbnKpNsVNF2p+tIkERBOaY8X Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:26:26 -0000 On Aug 23, 2012, at 5:09 PM, Ian Lepore wrote: > On Thu, 2012-08-23 at 15:50 -0600, Warner Losh wrote:=20 >> On Aug 23, 2012, at 3:28 PM, Ian Lepore wrote: >>> A recent innocuous change to the USB driver code caused intermittant >>> errors in the umass(4) driver on ARM and MIPS platforms, and this >>=20 >> I think the proper solution is to segregate DMA and non-DMA parts of = structures so that you don't have both sharing a cache line. >>=20 >> I also wonder why we don't allocate the DMA memory for these = structures separately from the non-DMA parts. This would eliminate the = USB_CACHE_BYTES kludge (which is CPU dependent, not arch dependent) and = move the knowledge of this junk into busdma layer where it belongs. = =46rom my understanding of the issue, this would completely eliminate = the problem forever! >>=20 >> Sharing a cacheline between something that is DMA aware and something = that is just begging for trouble. We're doing more work than we need = to to support this dubious feature and we'd be miles ahead if we could = not share at all. >>=20 >> Warner >>=20 >=20 > It seems to me that what we have here is a new type of constraint on = DMA > operations, and we need a way to communicate that constraint from the > part of the platform support code that knows about it to the drivers = and > driver support code that needs to know. The way we communicate DMA > constraints is with a busdma tag, but right now that tag only > communicates constraints that were needed for ISA and PCI busses, = namely > buffer alignment, boundary-crossing restrictions, and exclusion > regions. =20 >=20 > Now we have a new type of constraint, I think of it as "granularity". > In effect, we have a DMA system that can only do DMA in cacheline = sized > chunks. Even when the IO size -- and thus the number of "bits on the > wire" -- is less than the cacheline size, at the end of the DMA > operation (which includes the software-assisted coherency operations) > the number of bytes in memory that may be modified is the size of a > cacheline. This is because "the DMA system" is not just the engine = that > moves bytes around, it's the combination of hardware and software that > work together to maintain cache coherency. But this isn't new. It is an alignment requirement, which carries with = it an implicit size requirement. If you enforce the alignment, and = force all 'sub buffers' to have this alignment, you don't need the new = thing. > Ideally we'd find a way to communicate this new constraint using the > existing mechanism, the busdma tag, and ideally we'd not have to = change > every existing call to bus_dma_tag_create() to add a new parm. As I > understand it, parent tags are now passed down through the newbus > hierarchy consistantly, such that a tag at the nexus level could > describe a platform requirement such as granularity, and all devices = and > the helper code they use will have access to that constraint via > inheritance from ancestors' tags. Maybe we could have a new flavor of > bus_dma_tag_create() that takes a struct of parms, and existing calls > wouldn't have to be changed. Wouldn't a simpler solution be to just make this alignment requirement = be part of the global parent tag on these platforms and to make sure all = drivers on those platforms use it and don't cop-out and pass NULL? > Communicating the constraint is only part of the problem; it also has = to > be easy for drivers to work with that constraint, especially drivers > that are not targeted specifically at platforms with granular DMA. I > think we can achieve a huge chunk of that purely within the arm/mips > implementation of bus_dmamem_alloc(), but even so there would be a new > conceptual limitation on using that routine: it is specifically for > allocating DMA buffers, and that means that there will be a new a rule > that the CPU cannot access any memory within that buffer while an IO > operation is in progress. I don't think we should pander to drivers that don't know how to do DMA = properly. We get it almost right in bus_dma now. However, going from = almost right to completely right is hard and we keep uncovering edge = cases that bite us. Wouldn't it be better to eliminate all these weird = edge cases? > I'd also like to say there's a new rule that you cannot subdivide a > buffer obtained from bus_dmamem_alloc() into multiple buffers, or into = a > combination of DMA and CPU-accessed data. That would be bad news for > the USB subsystem, and perhaps other drivers. If this idea is either > impossible or particularly contentious, then I guess we'd need some = new > helper routines so that a driver can subdivide the memory in a way = that > doesn't violate any constraints implied by the tag used to allocate = the > buffer. When the USB subsystem went into the tree, this was one of the = criticisms that was ignored. It has come back to bite us time and time = again. Perhaps it is time to fix it once and for all. > Not all IO occurs using buffers obtained from bus_dmamem_alloc(), and = I > doubt we can reasonably ever require that it be so. =20 True, but the I/O that's not in memory from bus_dmamem_alloc is page = aligned. > I think the only > hope we have of handling that problem is to bounce the requests that > don't meet the granularity constraint, just as we'd have to do if the > caller-supplied buffer fell into an exclusion zone or violated an > alignment or boundary constraint. When I've tossed this idea out in = the > past there was instant resistance. Yeah, bounce buffers are massively > inefficient, but my experience has been that most of the IO that isn't > aligned and sized to a multiple of a cacheline is small IO (a few to a > few dozen bytes). I've never seen a case of page-sized or larger IO > requests that required partial-cacheline handling. I'm sure some > examples exist, but they're probably more the exception than the rule. > (And the bad performance you'd get from bouncing and copying massive > bulk data flow would be a powerful incentive to track down the culprit > and improve the driver.) That's also the underlying idea in the bus_dma stuff. You give the = constraints, you get the buffers and if you have a buffer that's outside = the constraints it gets bounced. That's why the sync operations on on = DMA items, not on cache line items. While cache lines are one issue, = memory placement can be another. Floppy drives, for example, couldn't = DMA past the first 16MB and if you have a buffer passed in that's = outside of that, it bounces. If this bouncing produces slower code, = then the drivers should be updated to have better alignment. The USB subsystem is making assumptions about the underlying cache = mechanisms that aren't really true. Ideally, we could get it to stop = doing that. Warner= From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 23:27:17 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6790F1065670 for ; Thu, 23 Aug 2012 23:27:17 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta07.emeryville.ca.mail.comcast.net (qmta07.emeryville.ca.mail.comcast.net [76.96.30.64]) by mx1.freebsd.org (Postfix) with ESMTP id 451428FC1C for ; Thu, 23 Aug 2012 23:27:17 +0000 (UTC) Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73]) by qmta07.emeryville.ca.mail.comcast.net with comcast id qPMf1j0031afHeLA7PTBzE; Thu, 23 Aug 2012 23:27:11 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta17.emeryville.ca.mail.comcast.net with comcast id qPTA1j00D4NgCEG8dPTAp2; Thu, 23 Aug 2012 23:27:11 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7NNR8Lw025556; Thu, 23 Aug 2012 17:27:08 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Adrian Chadd In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Thu, 23 Aug 2012 17:27:08 -0600 Message-ID: <1345764428.27688.591.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:27:17 -0000 On Thu, 2012-08-23 at 16:09 -0700, Adrian Chadd wrote: > [snip] > > Whoa, there's USB code that has these small buffers straddle cache lines? > > Why aren't they just allocated to always be in their own separate > buffers, so they don't ever have to worry about overlapping cache > lines? Yes, but note that with our current code, a small buffer doesn't have to straddle a cacheline boundary to invoke the partial cacheline flush logic. Its very smallness does so -- if a buffer doesn't start on a cacheline boundary, and/or if its size doesn't exactly equal a multiple of the cacheline size, then it's a partial cacheline flush situation. So even a driver striving to work around this trouble can get into trouble -- it uses bus_dmamem_alloc() to grab a buffer for a 13 byte status message. It gets a 16-byte chunk from the kernel allocator, and the other 16 bytes of that cacheline are given to something else in the kernel, and now you have DMA and CPU access going on in the same cacheline despite your efforts to Do The Right Thing with buffer allocation. (At work we avoid this by setting the kernel minimum allocation size to 32 bytes instead of 16. We did so after being burned by exactly this situation -- the "other 16 bytes" in our case belonged to a struct ithread, and trying to track down a lockup that was caused by a partial cacheline flush corrupting the struct ithread kept me entertained for a couple weeks.) -- Ian From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 23:36:46 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5376E106564A; Thu, 23 Aug 2012 23:36:46 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 17DD18FC15; Thu, 23 Aug 2012 23:36:45 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so2415467pbb.13 for ; Thu, 23 Aug 2012 16:36:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=ZqDIjzxMgU3yZwSHpcg2vnW8k3VCuuGcmucl0nHa5xc=; b=qFD3SNi7TjXoHyZyjZc9Cgz9jPt4O0ldrMYgS8HZgp7xzg9tLQju4GahEViI36qO7G 69jHiCDwYe7Unv2VK5NwuH8/LSHXHoH7+cQYP30Oq/Dy7aUEd5k07rptR1hfkbTbXXsp dp5tVjalA9IKe87/jGIPFa4op6MpTriSAw93In2RDvlV5a29SSzxJ6NEXxz+EQB8QQhH CEh2GWHctCICw98iocvHxn0xjQYiRupsd1hVXuKZ9a+V5dEZfUjGGhdinEC+OjmH9Idi 2x3D/BEge6egfP/6T8/ziCRcpdiu7a2IpNOFwMVrzTq6XF7gcz/OQ/p9M2YZvfAbgbSZ qSPA== MIME-Version: 1.0 Received: by 10.68.136.40 with SMTP id px8mr8215347pbb.153.1345765005641; Thu, 23 Aug 2012 16:36:45 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.36.106 with HTTP; Thu, 23 Aug 2012 16:36:45 -0700 (PDT) In-Reply-To: <1345764428.27688.591.camel@revolution.hippie.lan> References: <1345757300.27688.535.camel@revolution.hippie.lan> <1345764428.27688.591.camel@revolution.hippie.lan> Date: Thu, 23 Aug 2012 16:36:45 -0700 X-Google-Sender-Auth: 8N9-uFx59Shvyoz92HPvSALf2iE Message-ID: From: Adrian Chadd To: Ian Lepore Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:36:46 -0000 Right, that's what Linux does for ARM/MIPS. It just sets the minimum allocation size to be cache line sized. That way they didn't have to fix their USB and network stack code. .. or, we could fix the USB stack code by saying that anything being used as a DMA buffer needs to be minimum cache line size (which can be determined at run time if appropriate) and make the minimum allocation that. Then either it uses a separate allocation for each buffer or it allocates one big set of buffers and chops them up in at least "cache line size" bits. That reminds me, I should do that to the descriptor allocation in the Atheros driver - ie, round up the descriptor allocation size to a multiple of a cache line. That way DMAs don't conflict with the next DMAed buffer.. Adrian From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 23:45:15 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0703A106566C for ; Thu, 23 Aug 2012 23:45:15 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta04.emeryville.ca.mail.comcast.net (qmta04.emeryville.ca.mail.comcast.net [76.96.30.40]) by mx1.freebsd.org (Postfix) with ESMTP id DA4598FC19 for ; Thu, 23 Aug 2012 23:45:14 +0000 (UTC) Received: from omta18.emeryville.ca.mail.comcast.net ([76.96.30.74]) by qmta04.emeryville.ca.mail.comcast.net with comcast id qPd41j0011bwxycA4Pl76s; Thu, 23 Aug 2012 23:45:07 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta18.emeryville.ca.mail.comcast.net with comcast id qPl61j00R4NgCEG8ePl6lZ; Thu, 23 Aug 2012 23:45:07 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7NNj4Mr025577; Thu, 23 Aug 2012 17:45:04 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Thu, 23 Aug 2012 17:45:03 -0600 Message-ID: <1345765503.27688.602.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:45:15 -0000 On Thu, 2012-08-23 at 17:26 -0600, Warner Losh wrote: > > On Aug 23, 2012, at 3:28 PM, Ian Lepore wrote: > > Now we have a new type of constraint, I think of it as "granularity". > > In effect, we have a DMA system that can only do DMA in cacheline sized > > chunks. Even when the IO size -- and thus the number of "bits on the > > wire" -- is less than the cacheline size, at the end of the DMA > > operation (which includes the software-assisted coherency operations) > > the number of bytes in memory that may be modified is the size of a > > cacheline. This is because "the DMA system" is not just the engine that > > moves bytes around, it's the combination of hardware and software that > > work together to maintain cache coherency. > But this isn't new. It is an alignment requirement, which carries > with it an implicit size requirement. If you enforce the alignment, > and force all 'sub buffers' to have this alignment, you don't need the > new thing. So do you think it's safe to assume that any given dma tag that has an alignment constraint also implicitly has a buffer size constraint that the size must be a multiple of the alignment? What if we have a platform with a 32-byte cacheline / DMA granularity, and then we have a builtin device on that SoC which can only do DMA on a 64K alignment (which its tag would reflect), but the hardware can move as little as 1 byte at a time? Children of that bridge device come along and allocate little 16-byte buffers that eat 16 pages each. It doesn't seem all that far-fetched to me. -- Ian From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 23:48:51 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 76D75106564A; Thu, 23 Aug 2012 23:48:51 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 39D918FC17; Thu, 23 Aug 2012 23:48:50 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so2428732pbb.13 for ; Thu, 23 Aug 2012 16:48:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=Mskt1FyQ/HqkFMIEUCOxaM1D7q7pn1393qcB/jAWPnU=; b=Cog3Xq8zSN5Zl7K2a0nXw/INEb20LCEzMNo7+vVAij2/aBmAPh+3U/OeBe74m0dmGu yAdtXDFPqSEqFCe8mcV593Etog3zCrMHGhF47i0iRyX9oV1gNY2rd0uKCrujMR0lghrE xoLWOYXbxIsljKOc7nRsv7VN8fykkPqlryniKOtsuUZIOnQlNS90mNOdbLPIm/EuaavE AhpgCjau9GHafS+R1SB4v3+8ZVMYaMvawTyITBPJqGICI01PtxMnkSh2codWZU5+ffbF RfElqIHdkf2EoS0iy0+zJi5plfdvMjcWRuhux+m1H+QFy7lNg/SAFMIFedzmPkv8L79C IqOw== MIME-Version: 1.0 Received: by 10.68.231.233 with SMTP id tj9mr8442260pbc.39.1345765730704; Thu, 23 Aug 2012 16:48:50 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.36.106 with HTTP; Thu, 23 Aug 2012 16:48:50 -0700 (PDT) In-Reply-To: <1345765503.27688.602.camel@revolution.hippie.lan> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> Date: Thu, 23 Aug 2012 16:48:50 -0700 X-Google-Sender-Auth: xNYhy3PgvZjcI4uwRsLgaOc5SX4 Message-ID: From: Adrian Chadd To: Ian Lepore Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:48:51 -0000 On 23 August 2012 16:45, Ian Lepore wrote: > So do you think it's safe to assume that any given dma tag that has an > alignment constraint also implicitly has a buffer size constraint that > the size must be a multiple of the alignment? > > What if we have a platform with a 32-byte cacheline / DMA granularity, > and then we have a builtin device on that SoC which can only do DMA on a > 64K alignment (which its tag would reflect), but the hardware can move > as little as 1 byte at a time? Children of that bridge device come > along and allocate little 16-byte buffers that eat 16 pages each. It > doesn't seem all that far-fetched to me. That hardware would suck, wouldn't it? In what case though would the hardware say it can only DMA on a 64k alignment BUT move one byte at a time? Then what would the starting address be for each DMA? adrian From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 23:51:13 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A646F1065670; Thu, 23 Aug 2012 23:51:13 +0000 (UTC) (envelope-from kabaev@gmail.com) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 213518FC0C; Thu, 23 Aug 2012 23:51:12 +0000 (UTC) Received: by vcbgb22 with SMTP id gb22so1903080vcb.13 for ; Thu, 23 Aug 2012 16:51:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:in-reply-to:references:x-mailer :mime-version:content-type; bh=JWgh66ZmryhJI/TjUNJMyoMIjSInrwXjCxbw7sDO0VU=; b=PZ8SFjrKUcp9xmS4PNcgWu/uw++fdJfnc3+SvKpzXkSwwVr50wSmz1+tRNT8abbGCF pD7n+mrr2bczBzMck/r5HVeaw2iQ8PY7TsewLVaT9uCez0hpNTwRfgygw5lg2u021j8e tKVDRgLq5mT+G83tr6qqkuMhr/6vUHFB7u1J/UHCAzT9x0KG2xC0FQ6e8xQ4VFs1Csuu hOPedDLB1aqUk7V2gTZKwnf3EQBYy9AxHZAs74HzGQGvCAC1bTyCcC+mGobafrZ7nopb Bxn2nF3M/SvOpaARk+DrRgAK1iSBsAo0uDWyhvn85Jthb90lOKSMv8An6LsuB4wpCerT c5DA== Received: by 10.220.247.137 with SMTP id mc9mr2858779vcb.52.1345765872283; Thu, 23 Aug 2012 16:51:12 -0700 (PDT) Received: from kan.dyndns.org (c-24-63-226-98.hsd1.ma.comcast.net. [24.63.226.98]) by mx.google.com with ESMTPS id s1sm4496996vdi.14.2012.08.23.16.51.10 (version=SSLv3 cipher=OTHER); Thu, 23 Aug 2012 16:51:11 -0700 (PDT) Date: Thu, 23 Aug 2012 19:51:02 -0400 From: Alexander Kabaev To: Warner Losh Message-ID: <20120823195102.2dcc1fcc@kan.dyndns.org> In-Reply-To: <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> X-Mailer: Claws Mail 3.8.0 (GTK+ 2.24.6; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: multipart/signed; micalg=PGP-SHA1; boundary="Sig_/xAKsUfCFhIS7lv_PazDak7K"; protocol="application/pgp-signature" Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:51:13 -0000 --Sig_/xAKsUfCFhIS7lv_PazDak7K Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable On Thu, 23 Aug 2012 15:50:35 -0600 Warner Losh wrote: >=20 > On Aug 23, 2012, at 3:28 PM, Ian Lepore wrote: > > A recent innocuous change to the USB driver code caused intermittant > > errors in the umass(4) driver on ARM and MIPS platforms, and this >=20 > I think the proper solution is to segregate DMA and non-DMA parts of > structures so that you don't have both sharing a cache line. >=20 > I also wonder why we don't allocate the DMA memory for these > structures separately from the non-DMA parts. This would eliminate > the USB_CACHE_BYTES kludge (which is CPU dependent, not arch > dependent) and move the knowledge of this junk into busdma layer > where it belongs. From my understanding of the issue, this would > completely eliminate the problem forever! >=20 > Sharing a cacheline between something that is DMA aware and something > that is just begging for trouble. We're doing more work than we > need to to support this dubious feature and we'd be miles ahead if we > could not share at all. >=20 > Warner I agree with Warner that this should be addressed at busdma level. When asked for DMA buffer, it should contrain its start address size to the cache line boundary. USB is insane allocating DMA buffers inside of own transfer structure and that was stated on a more than one occasion already. Since USB code refused to budge, we came with a horrible USB_CACHE_BYTES hack as as _temporary_ workaround. --=20 Alexander Kabaev --Sig_/xAKsUfCFhIS7lv_PazDak7K Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iD8DBQFQNsHtQ6z1jMm+XZYRAqeaAJsE2dyZ0FISrcOVX3Je3YvLQ1MNZQCfcF+M bBDN5eQWXQgUWXM3/UOthNI= =mcBu -----END PGP SIGNATURE----- --Sig_/xAKsUfCFhIS7lv_PazDak7K-- From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 23:55:12 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 91A20106564A for ; Thu, 23 Aug 2012 23:55:12 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [76.96.30.16]) by mx1.freebsd.org (Postfix) with ESMTP id 641158FC12 for ; Thu, 23 Aug 2012 23:55:12 +0000 (UTC) Received: from omta22.emeryville.ca.mail.comcast.net ([76.96.30.89]) by qmta01.emeryville.ca.mail.comcast.net with comcast id qEbq1j03q1vN32cA1PvCNt; Thu, 23 Aug 2012 23:55:12 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta22.emeryville.ca.mail.comcast.net with comcast id qPvA1j00d4NgCEG8iPvBTY; Thu, 23 Aug 2012 23:55:12 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7NNt9ti025603; Thu, 23 Aug 2012 17:55:09 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Adrian Chadd In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Thu, 23 Aug 2012 17:55:09 -0600 Message-ID: <1345766109.27688.606.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:55:12 -0000 On Thu, 2012-08-23 at 16:48 -0700, Adrian Chadd wrote: > On 23 August 2012 16:45, Ian Lepore wrote: > > > So do you think it's safe to assume that any given dma tag that has an > > alignment constraint also implicitly has a buffer size constraint that > > the size must be a multiple of the alignment? > > > > What if we have a platform with a 32-byte cacheline / DMA granularity, > > and then we have a builtin device on that SoC which can only do DMA on a > > 64K alignment (which its tag would reflect), but the hardware can move > > as little as 1 byte at a time? Children of that bridge device come > > along and allocate little 16-byte buffers that eat 16 pages each. It > > doesn't seem all that far-fetched to me. > > That hardware would suck, wouldn't it? > > In what case though would the hardware say it can only DMA on a 64k > alignment BUT move one byte at a time? Then what would the starting > address be for each DMA? > Maybe the device has a reduced number of address bits in its registers and the low-order bits always start at zero and increment internally in a wider register so that any length dma can happen, but it has to start on a 64k boundary. Maybe the address you pass it has to be a 64k boundary, but then the bytes actually end up in one-of-N slots within that 64k region, based on other parameters of the transfer. I've worked with some strange hardware over the years. -- Ian From owner-freebsd-mips@FreeBSD.ORG Thu Aug 23 23:59:17 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 458DC106566B; Thu, 23 Aug 2012 23:59:17 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0784E8FC15; Thu, 23 Aug 2012 23:59:16 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so2439273pbb.13 for ; Thu, 23 Aug 2012 16:59:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=bVpnJB73bCLefXPcIvjYhYqKlLLMR3NxqYPJzQo36bs=; b=l37Rd9vmQ/ynZvNsTSwDhK9YQVWXfbu3PrJPR6iGT/dxcAkV0v74ykjudVMxy4x9aM WQy/q8uvDFSJNXPiXtsWxCMW+oMzZoHCj+QDu153Vlr7uZs96ZrrcvurLW2Zk8jmM0Qg ynnQeJ3vNToaY8rJOte5Khe+34yMmsXk1j6N1v/1jHpK2TTD4ogFT6O81cvcKkc1J43y EFdZZjIxsDmwwXp+MgBY9UTRqFgcu16kkPXb/S01nH+a7NQqIOkS9cCsOoMTHgcyWuGe Z1ZdsSqGNOEmFWiFaxgY2u8yhwhV7czSYeJiA7uHYMWB8I0evjvDowD0FOmtmkCOrzwe oXyw== MIME-Version: 1.0 Received: by 10.68.138.169 with SMTP id qr9mr8493185pbb.27.1345766356862; Thu, 23 Aug 2012 16:59:16 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.36.106 with HTTP; Thu, 23 Aug 2012 16:59:16 -0700 (PDT) In-Reply-To: <1345766109.27688.606.camel@revolution.hippie.lan> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> Date: Thu, 23 Aug 2012 16:59:16 -0700 X-Google-Sender-Auth: lfQnLTGTsXKt1gXoEwgg8ou0Azs Message-ID: From: Adrian Chadd To: Ian Lepore Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:59:17 -0000 On 23 August 2012 16:55, Ian Lepore wrote: >> In what case though would the hardware say it can only DMA on a 64k >> alignment BUT move one byte at a time? Then what would the starting >> address be for each DMA? >> > > Maybe the device has a reduced number of address bits in its registers > and the low-order bits always start at zero and increment internally in > a wider register so that any length dma can happen, but it has to start > on a 64k boundary. > > Maybe the address you pass it has to be a 64k boundary, but then the > bytes actually end up in one-of-N slots within that 64k region, based on > other parameters of the transfer. > > I've worked with some strange hardware over the years. Right. That's pretty odd though. But now I understand where you're coming from. I still think the short term solution should be "fix the USB stack to not do that!" The longer term problem is likely to figure out what makes sense. Eg, if you're going to allow the allocator to interleave 16 byte chunks (on a 32 byte cache line platform) with some being DMA buffers and others being non-DMA buffers; or whether you enforce that the whole chunk is a DMA buffer for your hardware and you look after it, or something else.. Adrian From owner-freebsd-mips@FreeBSD.ORG Fri Aug 24 00:51:31 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA37D106566C for ; Fri, 24 Aug 2012 00:51:31 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [76.96.30.16]) by mx1.freebsd.org (Postfix) with ESMTP id 821618FC1C for ; Fri, 24 Aug 2012 00:51:31 +0000 (UTC) Received: from omta08.emeryville.ca.mail.comcast.net ([76.96.30.12]) by qmta01.emeryville.ca.mail.comcast.net with comcast id qEbq1j00V0FhH24A1QrXVz; Fri, 24 Aug 2012 00:51:31 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta08.emeryville.ca.mail.comcast.net with comcast id qQrV1j00l4NgCEG8UQrWx7; Fri, 24 Aug 2012 00:51:30 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7O0pSKT025681; Thu, 23 Aug 2012 18:51:28 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Adrian Chadd In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Thu, 23 Aug 2012 18:51:28 -0600 Message-ID: <1345769488.27688.625.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2012 00:51:32 -0000 On Thu, 2012-08-23 at 16:48 -0700, Adrian Chadd wrote: > On 23 August 2012 16:45, Ian Lepore wrote: > > > So do you think it's safe to assume that any given dma tag that has an > > alignment constraint also implicitly has a buffer size constraint that > > the size must be a multiple of the alignment? > > > > What if we have a platform with a 32-byte cacheline / DMA granularity, > > and then we have a builtin device on that SoC which can only do DMA on a > > 64K alignment (which its tag would reflect), but the hardware can move > > as little as 1 byte at a time? Children of that bridge device come > > along and allocate little 16-byte buffers that eat 16 pages each. It > > doesn't seem all that far-fetched to me. > > That hardware would suck, wouldn't it? > Thinking about this some more, I think that at least for now we don't have to communicate a new constraint to bus_dma_tag_create(), nor do we need to assume that a size constraint is the same as an alignment constraint. The size constraint is machine dependant in nature, and the busdma implementation code is also MD, and thus should have some MD way of knowing about this constraint for itself without being told by callers. -- Ian From owner-freebsd-mips@FreeBSD.ORG Fri Aug 24 03:56:25 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 485C7106566C for ; Fri, 24 Aug 2012 03:56:25 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 00D5E8FC08 for ; Fri, 24 Aug 2012 03:56:24 +0000 (UTC) Received: by ialo14 with SMTP id o14so3229653ial.13 for ; Thu, 23 Aug 2012 20:56:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=A8ZRIPSNaFjBDKLw6gxuyEvrrxkJ3KqudBpGidIaZTY=; b=pVCpliFzUAmj29kjdv+US97KVdyi5ImOuqoQrN4DmLuU5i3pQqIwV6wMDSd/gDR4Ja kpCKL4yrcOy3go36RnGR48TmpevR5Cww4fsG+lqJUnGj63GoG4AMqtVMtmAh0A+xME/W dS7bi4MoxP4ZrCaRp7MjGuaSXm2LRd1k6qXS3x5vYNAi+qADMxrDDSUY2vHUgyeO34/J 1YigYznfRoaFiPWHfVYgN2qHhYT3b/VmD68fSJKD6G0cJG6Zhss2kUR5HP0dC96U1+Lh quNvrjQL5hUCUZCcDGuqPLyQG85rehyoEQ5HaK6xRYFJ05JukhMFdQaQ/MBDEiR1/DSc 5SRA== Received: by 10.50.34.131 with SMTP id z3mr649492igi.45.1345780584053; Thu, 23 Aug 2012 20:56:24 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id va9sm1181375igb.17.2012.08.23.20.56.22 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 23 Aug 2012 20:56:22 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1345765503.27688.602.camel@revolution.hippie.lan> Date: Thu, 23 Aug 2012 21:56:21 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQmjnduhkKMiTmQRXljKd62NU3Zue7JhFw0vTwpE8wZzkkzUlSYi7dzKJPN4GcoelXY404uw Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2012 03:56:25 -0000 On Aug 23, 2012, at 5:45 PM, Ian Lepore wrote: > On Thu, 2012-08-23 at 17:26 -0600, Warner Losh wrote: >>> On Aug 23, 2012, at 3:28 PM, Ian Lepore wrote: >>> Now we have a new type of constraint, I think of it as = "granularity". >>> In effect, we have a DMA system that can only do DMA in cacheline = sized >>> chunks. Even when the IO size -- and thus the number of "bits on = the >>> wire" -- is less than the cacheline size, at the end of the DMA >>> operation (which includes the software-assisted coherency = operations) >>> the number of bytes in memory that may be modified is the size of a >>> cacheline. This is because "the DMA system" is not just the engine = that >>> moves bytes around, it's the combination of hardware and software = that >>> work together to maintain cache coherency. >> But this isn't new. It is an alignment requirement, which carries >> with it an implicit size requirement. If you enforce the alignment, >> and force all 'sub buffers' to have this alignment, you don't need = the >> new thing.=20 >=20 > So do you think it's safe to assume that any given dma tag that has an > alignment constraint also implicitly has a buffer size constraint that > the size must be a multiple of the alignment? Yes. If something must be aligned to N bits, chances are it doesn't = decode the lower N bits which implies a size constraint. > What if we have a platform with a 32-byte cacheline / DMA granularity, > and then we have a builtin device on that SoC which can only do DMA on = a > 64K alignment (which its tag would reflect), but the hardware can move > as little as 1 byte at a time? Children of that bridge device come > along and allocate little 16-byte buffers that eat 16 pages each. It > doesn't seem all that far-fetched to me. This would be a very odd hardware. DMA aligned to 64k that can only = move one byte seems far fetched. How useful would such a design be? = How would you do scatter gather on such a design? But this isn't what I'm saying. If the cache line size is 32, then for = DMA we only ever give out chunks of 32 or larger. In that case, the = split cache line situation you gave as an example can't happen.= From owner-freebsd-mips@FreeBSD.ORG Fri Aug 24 04:00:48 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7AA161065670 for ; Fri, 24 Aug 2012 04:00:48 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 641C78FC12 for ; Fri, 24 Aug 2012 04:00:47 +0000 (UTC) Received: by ialo14 with SMTP id o14so3236392ial.13 for ; Thu, 23 Aug 2012 21:00:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=/fumm9ehnwMxdlhILXylmJSdzHQbk9R61mCz3E6uRRA=; b=Q7gSrdwBTUxW2ZTGw/ApLtDSmzZ4y1XaX1sx5tu1F5CYdb601VUoLYMcob99ECDe0N oIwntL5LZjGGemvi7I4/abrEvEzq3VCUvKlDeJ1KeloZmWcNcQkwU2AbukMt7WeVcwPT jv4LrlhhT/1TEN/xSzYc/9AxVo1J2b7RoJ1IVNXJrBa4VYPuApuZCDvsl4HYzNuDK7va 0X0yT/o2SnTH5+JxyovdpuRhBm3PR0gADR7ihZv36qBOw+o2a6drkbTPRk0PND4/sdr2 LGkZoam9Pb57MgkqOJHhVYgXnfSof/VO6nZgh/cvs8LEMx6mBXsynx27fSOboUtJPfrt 1liA== Received: by 10.50.47.161 with SMTP id e1mr702151ign.11.1345780846750; Thu, 23 Aug 2012 21:00:46 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id ua2sm3449366igb.7.2012.08.23.21.00.45 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 23 Aug 2012 21:00:46 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Thu, 23 Aug 2012 22:00:44 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> To: Adrian Chadd X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQmuRKKE8wUWJ8a/DmeFr0jqxf6cENQmm3tB994rTAhbTuDHdj8/QBX1tIPoeDIy109tgu4r Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2012 04:00:48 -0000 On Aug 23, 2012, at 5:59 PM, Adrian Chadd wrote: > On 23 August 2012 16:55, Ian Lepore = wrote: >=20 >>> In what case though would the hardware say it can only DMA on a 64k >>> alignment BUT move one byte at a time? Then what would the starting >>> address be for each DMA? >>>=20 >>=20 >> Maybe the device has a reduced number of address bits in its = registers >> and the low-order bits always start at zero and increment internally = in >> a wider register so that any length dma can happen, but it has to = start >> on a 64k boundary. >>=20 >> Maybe the address you pass it has to be a 64k boundary, but then the >> bytes actually end up in one-of-N slots within that 64k region, based = on >> other parameters of the transfer. >>=20 >> I've worked with some strange hardware over the years. >=20 > Right. That's pretty odd though. But now I understand where you're = coming from. >=20 > I still think the short term solution should be "fix the USB stack to > not do that!" >=20 > The longer term problem is likely to figure out what makes sense. Eg, > if you're going to allow the allocator to interleave 16 byte chunks > (on a 32 byte cache line platform) with some being DMA buffers and > others being non-DMA buffers; or whether you enforce that the whole > chunk is a DMA buffer for your hardware and you look after it, or > something else.. The bottom line is that you can't mix things like that when cache lines = are involved. The current code that tries is doomed to failure. Doomed. = You just can't control all flushes, as Ian's missive demonstrates, and = trying to accommodate code that does this I don't think can possibly = work. All the interrupt masking, copying in and out, etc I fear is = doomed to utter and abject failure. =20 Warner= From owner-freebsd-mips@FreeBSD.ORG Fri Aug 24 04:03:24 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 172CF1065673 for ; Fri, 24 Aug 2012 04:03:24 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 87AF08FC12 for ; Fri, 24 Aug 2012 04:03:23 +0000 (UTC) Received: by ialo14 with SMTP id o14so3240747ial.13 for ; Thu, 23 Aug 2012 21:03:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=eI5+aVAQMNRlLMXxkUR0Gm4Y8aCwg0o1YoPQvX3yxX8=; b=DX5qwQeDD+BYrEtQmAmM3CBYwgKOA/zaUmMGGDRU7r1Af44ig9cXc+2tIXmucUS0ZV INO7+zePR08JTyXt83R/FU2kAn25YXv4ZVQg+9G+RGLgSonSZSK9+r77pcFEsDdUH+hH C9CK7FRY8DO/yOIxVxEfa7aOqbByTXWAoh6SfCmpzEPZtksU1pD0AsYzqvQGsEzGVZDb HatA3mHwtkw3ytBqkSikyGgRvBwzarBPDNA3DIYJDYNzV3FKlXVtAF9izIyA65oZ/N8L jWj59k7T+kdmv5N3cMroQ5757Z96NDRFAd5jGxjfyH5o8eeE7LABLE+lPYuhtqq3eubW 9bBQ== Received: by 10.50.213.106 with SMTP id nr10mr617077igc.58.1345781002439; Thu, 23 Aug 2012 21:03:22 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id ey10sm3397801igb.17.2012.08.23.21.03.20 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 23 Aug 2012 21:03:21 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1345769488.27688.625.camel@revolution.hippie.lan> Date: Thu, 23 Aug 2012 22:03:19 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <6639450F-11BB-4D01-8D0C-CD66B427EF08@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345769488.27688.625.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQmdrse6vSWQK4r6usRp10hobAtGE8Iwi4m/+BdwAsw/wLuDKCT3/IhRF/BNgV8+7bTiAZgG Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2012 04:03:24 -0000 On Aug 23, 2012, at 6:51 PM, Ian Lepore wrote: > On Thu, 2012-08-23 at 16:48 -0700, Adrian Chadd wrote: >> On 23 August 2012 16:45, Ian Lepore = wrote: >>=20 >>> So do you think it's safe to assume that any given dma tag that has = an >>> alignment constraint also implicitly has a buffer size constraint = that >>> the size must be a multiple of the alignment? >>>=20 >>> What if we have a platform with a 32-byte cacheline / DMA = granularity, >>> and then we have a builtin device on that SoC which can only do DMA = on a >>> 64K alignment (which its tag would reflect), but the hardware can = move >>> as little as 1 byte at a time? Children of that bridge device come >>> along and allocate little 16-byte buffers that eat 16 pages each. = It >>> doesn't seem all that far-fetched to me. >>=20 >> That hardware would suck, wouldn't it? >>=20 >=20 > Thinking about this some more, I think that at least for now we don't > have to communicate a new constraint to bus_dma_tag_create(), nor do = we > need to assume that a size constraint is the same as an alignment > constraint. The size constraint is machine dependant in nature, and = the > busdma implementation code is also MD, and thus should have some MD = way > of knowing about this constraint for itself without being told by > callers. And also allow for different cache line sizes for different CPUs within = the same MACHINE/MACHINE_ARCH, which is common in MIPS land, at least. Warner From owner-freebsd-mips@FreeBSD.ORG Sat Aug 25 10:10:31 2012 Return-Path: Delivered-To: freebsd-mips@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 62937106564A for ; Sat, 25 Aug 2012 10:10:31 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 340168FC17 for ; Sat, 25 Aug 2012 10:10:31 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7PAAVMT081689 for ; Sat, 25 Aug 2012 10:10:31 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7PAAVgw081688; Sat, 25 Aug 2012 10:10:31 GMT (envelope-from gnats) Date: Sat, 25 Aug 2012 10:10:31 GMT Message-Id: <201208251010.q7PAAVgw081688@freefall.freebsd.org> To: freebsd-mips@FreeBSD.org From: Luiz Otavio O Souza Cc: Subject: Re: kern/170859: [patch] [mips] Move MIPS specific options from generic conf/options file X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Luiz Otavio O Souza List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2012 10:10:31 -0000 The following reply was made to PR kern/170859; it has been noted by GNATS. From: Luiz Otavio O Souza To: bug-followup@FreeBSD.org, Luiz Otavio O Souza Cc: Subject: Re: kern/170859: [patch] [mips] Move MIPS specific options from generic conf/options file Date: Sat, 25 Aug 2012 07:02:42 -0300 There is a new patch available at: = http://loos.no-ip.org/rb/options.mips.patch This one apply clean on -current. Thanks.=