From owner-freebsd-arm@FreeBSD.ORG Sun Aug 26 17:42:23 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BA3D91065670; Sun, 26 Aug 2012 17:42:23 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 0634D8FC20; Sun, 26 Aug 2012 17:42:06 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7QHg5RZ067335; Sun, 26 Aug 2012 11:42:05 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7QHg2M1029199; Sun, 26 Aug 2012 11:42:02 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sun, 26 Aug 2012 11:42:02 -0600 Message-ID: <1346002922.1140.56.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 17:42:23 -0000 On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: > The bottom line is that you can't mix things like that when cache > lines are involved. The current code that tries is doomed to failure. > Doomed. You just can't control all flushes, as Ian's missive > demonstrates, and trying to accommodate code that does this I don't > think can possibly work. All the interrupt masking, copying in and > out, etc I fear is doomed to utter and abject failure. > Until last weekend I was in the camp that thought the partial cacheline flush problem was solvable with sufficiently clever code. Now I agree that we're doomed to failure and it's time to try another direction. We're going to have some implementation work to do in arm and mips busdma, but I think the larger part of the task is going to be defining more rigorously how a driver must interact with the busdma system to function correctly on all types of platforms, and then update existing drivers to conform. The busdma manpage currently has some vague words about the usage and sequencing of sync ops, such as "If read and write operations are not preceded and followed by the appropriate synchronization operations, behavior is undefined." I think we should more explicitly spell out what the appropriate sequences are. In particular: * The PRE and POST operations must occur in pairs; a PREREAD must be followed eventually by a POSTREAD and a PREWRITE must be followed by a POSTWRITE. * The CPU is not allowed to access the mapped memory after a PRE sync and before the corresponding POST sync. * The DMA hardware is not allowed to access the mapped memory after a POST sync and before the next PRE sync. * Read and write sync operators may be combined in a single call, PRE and POST operators may not be. E.G., PREREAD|PREWRITE is allowed, PREREAD|POSTREAD is not. We should note that while read and write operations may be combined, on some platforms PREREAD|PREWRITE is needlessly expensive when only a read is being performed. We also need some rules about working with buffers obtained from bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). I think the rule should be that a buffer obtained from bus_dmamem_alloc(), or more formally any region of memory mapped by a bus_dmamap_load(), is a single logical object which can only be accessed by one entity at a time. That means that there cannot be two concurrent DMA operations happening in different regions of the same buffer, nor can DMA and CPU access be happening concurrently even if in different parts of the buffer. I've always thought that allocating a dma buffer feels like a big hassle. You sometimes have to create a tag for the sole purpose of setting the maxsize to get the buffer size you need when you call bus_dmamem_alloc(). If bus_dmamem_alloc() took a size parm you could just use your parent tag, or a generic tag appropriate to all the IO you're doing for a given device. If you need a variety of buffers for small control and command and status transfers of different sizes, you end up having to manage up to a dozen tags and maps and buffers. It's all very clunky and inconvenient. It's just the sort of thing that makes you want to allocate a big buffer and subdivide it. Surely we could do something to make it easier? -- Ian From owner-freebsd-arm@FreeBSD.ORG Sun Aug 26 18:05:31 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0553D106566B; Sun, 26 Aug 2012 18:05:31 +0000 (UTC) (envelope-from marktinguely@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id B86078FC08; Sun, 26 Aug 2012 18:05:30 +0000 (UTC) Received: by dadr6 with SMTP id r6so1986829dad.13 for ; Sun, 26 Aug 2012 11:05:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Uw70O9ZjXEv0noKSxgiAWrFGrA7R41RTQBz+00qiyXg=; b=lRMVdUjT02PasXSBwruHEy+P+tM/5iIRceigrGli1+Adb6r+aF6KRIASE2HatMyk9R 0ydHl0W5mtYVWe/O6UM0CRSVtswg5BzKmV/rpcxgCSfl1qORXuv3xn1eRyk+sOjPNYZO UsgNWJwnflgqUBll5iG9sKtc1AeLU+U5T60xqEM1AVlrLAejg3fAJ0VKl6nIr65wZPKz vJwP5oR93/LEJv3bWXYUo7FgQ/NHLIvNydb3nQMXpPpqifD1Z0iBjgEHf5KgLaoJo4x2 DsuoqXjIV95lXHpAIcPj2D1MuD9zcHa9+/pDu30eLi0XrQhFBGxfoOO2HXfgIYMawTcW TgTQ== MIME-Version: 1.0 Received: by 10.68.221.70 with SMTP id qc6mr29050561pbc.92.1346004330201; Sun, 26 Aug 2012 11:05:30 -0700 (PDT) Received: by 10.68.229.227 with HTTP; Sun, 26 Aug 2012 11:05:29 -0700 (PDT) In-Reply-To: <1346002922.1140.56.camel@revolution.hippie.lan> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> Date: Sun, 26 Aug 2012 13:05:29 -0500 Message-ID: From: Mark Tinguely To: Ian Lepore Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 18:05:31 -0000 On Sun, Aug 26, 2012 at 12:42 PM, Ian Lepore wrote: > On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: >> The bottom line is that you can't mix things like that when cache >> lines are involved. The current code that tries is doomed to failure. >> Doomed. You just can't control all flushes, as Ian's missive >> demonstrates, and trying to accommodate code that does this I don't >> think can possibly work. All the interrupt masking, copying in and >> out, etc I fear is doomed to utter and abject failure. >> > Until last weekend I was in the camp that thought the partial cacheline > flush problem was solvable with sufficiently clever code. Now I agree > that we're doomed to failure and it's time to try another direction. > > We're going to have some implementation work to do in arm and mips > busdma, but I think the larger part of the task is going to be defining > more rigorously how a driver must interact with the busdma system to > function correctly on all types of platforms, and then update existing > drivers to conform. > > The busdma manpage currently has some vague words about the usage and > sequencing of sync ops, such as "If read and write operations are not > preceded and followed by the appropriate synchronization operations, > behavior is undefined." I think we should more explicitly spell out > what the appropriate sequences are. In particular: > > * The PRE and POST operations must occur in pairs; a PREREAD must > be followed eventually by a POSTREAD and a PREWRITE must be > followed by a POSTWRITE. > * The CPU is not allowed to access the mapped memory after a PRE > sync and before the corresponding POST sync. > * The DMA hardware is not allowed to access the mapped memory > after a POST sync and before the next PRE sync. > * Read and write sync operators may be combined in a single call, > PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > allowed, PREREAD|POSTREAD is not. We should note that while > read and write operations may be combined, on some platforms > PREREAD|PREWRITE is needlessly expensive when only a read is > being performed. > > We also need some rules about working with buffers obtained from > bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). I > think the rule should be that a buffer obtained from bus_dmamem_alloc(), > or more formally any region of memory mapped by a bus_dmamap_load(), is > a single logical object which can only be accessed by one entity at a > time. That means that there cannot be two concurrent DMA operations > happening in different regions of the same buffer, nor can DMA and CPU > access be happening concurrently even if in different parts of the > buffer. > > I've always thought that allocating a dma buffer feels like a big > hassle. You sometimes have to create a tag for the sole purpose of > setting the maxsize to get the buffer size you need when you call > bus_dmamem_alloc(). If bus_dmamem_alloc() took a size parm you could > just use your parent tag, or a generic tag appropriate to all the IO > you're doing for a given device. If you need a variety of buffers for > small control and command and status transfers of different sizes, you > end up having to manage up to a dozen tags and maps and buffers. It's > all very clunky and inconvenient. It's just the sort of thing that > makes you want to allocate a big buffer and subdivide it. Surely we > could do something to make it easier? > > -- Ian I did a quick look at the drivers last summer. Most drivers do the right thing and use memory allocated from bus_dmamem_alloc(). It is easy for us to give them a cache aligned buffer. Some drivers use mbufs - 256 bytes which cache safe. Some drivers directly or indirectly malloc() a buffer and then use it to dma - rather than try to fix them all, I was okay with making the smallest malloc() amount equal to the cache line size. It amounts to getting rid of the 16 byte allocation on some ARM architectures. The power of 2 allocator will then give us cache line safe allocation. A few drivers take a small memory amount from the kernel stack and dma to it <- broken driver. The few drivers that use data from a structure and that memory is not cached aligned <- broken driver. --Mark Tinguely. From owner-freebsd-arm@FreeBSD.ORG Sun Aug 26 18:25:23 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D04B1065674; Sun, 26 Aug 2012 18:25:23 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 4881A8FC1E; Sun, 26 Aug 2012 18:25:10 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7QIPAH8068013; Sun, 26 Aug 2012 12:25:10 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7QIP87W029231; Sun, 26 Aug 2012 12:25:08 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Mark Tinguely In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sun, 26 Aug 2012 12:25:07 -0600 Message-ID: <1346005507.1140.69.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, Hans Petter Selasky , freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 18:25:23 -0000 On Sun, 2012-08-26 at 13:05 -0500, Mark Tinguely wrote: > I did a quick look at the drivers last summer. > > Most drivers do the right thing and use memory allocated from > bus_dmamem_alloc(). It is easy for us to give them a cache aligned > buffer. > > Some drivers use mbufs - 256 bytes which cache safe. > > Some drivers directly or indirectly malloc() a buffer and then use it > to dma - rather than try to fix them all, I was okay with making the > smallest malloc() amount equal to the cache line size. It amounts to > getting rid of the 16 byte allocation on some ARM architectures. The > power of 2 allocator will then give us cache line safe allocation. > > A few drivers take a small memory amount from the kernel stack and dma > to it <- broken driver. > > The few drivers that use data from a structure and that memory is not > cached aligned <- broken driver. > I disagree about those last two points -- drivers that choose to use stack memory or malloc'd memory as IO buffers are not broken. Drivers can do IO directly to/from userland buffers, do we say that an application that calls read(2) and passes the address of a stack variable is broken? In this regard, it's the busdma implementation that's broken, because it should bounce those IOs through a DMA-safe buffer. There's absolutely no rule that I've ever heard of in FreeBSD that says IO can only take place using memory allocated from busdma. The rule is only that the proper sequence of busdma operation must be called, and beyond that it's up to the busdma implementation to make it work. Our biggest problem, I think, is that we don't have a sufficient definition of "the proper sequence of busdma operations." I don't think it will be very hard to make the arm and mips busdma implementations work correctly. It won't even be too hard to make them fairly efficient at bouncing small IOs (my thinking is that we can make small bounces no more expensive than the current partial cacheline flush implementation which copies the data multiple times). Bouncing large IO will never be efficient, but the inefficiency will be a powerful motivator to update drivers that do large IO to work better, such as using buffers allocated from busdma. -- Ian From owner-freebsd-arm@FreeBSD.ORG Sun Aug 26 18:53:47 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3AE0D106566C; Sun, 26 Aug 2012 18:53:47 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) by mx1.freebsd.org (Postfix) with ESMTP id E46D08FC16; Sun, 26 Aug 2012 18:53:46 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id q7QIrb6b086928; Sun, 26 Aug 2012 18:53:37 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.143] (CiscoE3000 [192.168.1.65]) by kientzle.com with SMTP id y7e92cf8djshe4rhprsvmx2wr2; Sun, 26 Aug 2012 18:53:37 +0000 (UTC) (envelope-from tim@kientzle.com) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=windows-1252 From: Tim Kientzle In-Reply-To: Date: Sun, 26 Aug 2012 11:53:35 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1278) Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, "arch@" Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 18:53:47 -0000 On Sun, Aug 26, 2012 at 12:42 PM, Ian Lepore wrote: > On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: >> The bottom line is that you can't mix things like that when cache >> lines are involved. >>=20 > =85. I think we should more explicitly spell out > what the appropriate sequences are. As someone who is just tinkering with driver code for the first time, I applaud any attempts to better document this area! ;-) > In particular: >=20 > * The PRE and POST operations must occur in pairs; a PREREAD must > be followed eventually by a POSTREAD and a PREWRITE must be > followed by a POSTWRITE. > * The CPU is not allowed to access the mapped memory after a PRE > sync and before the corresponding POST sync. > * The DMA hardware is not allowed to access the mapped memory > after a POST sync and before the next PRE sync. These rules sound reasonable. Good documentation might also give examples of what the PRE/POST operations might entail (e.g., from the preceding discussion, it sounds like PREREAD and PREWRITE require at least a partial cache flush on ARM). That helps folks who are coming to the docs with some hardware background. > * Read and write sync operators may be combined in a single = call, > PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > allowed, PREREAD|POSTREAD is not. We should note that while > read and write operations may be combined, on some platforms > PREREAD|PREWRITE is needlessly expensive when only a read is > being performed. PREREAD|POSTREAD doesn't sound useful to me, but why would it be explicitly forbidden? Would you also forbid POSTREAD|PREWRITE? (For a buffer that has just completed a DMA read and is going to be immediately used for a DMA write?) Tim From owner-freebsd-arm@FreeBSD.ORG Sun Aug 26 20:01:55 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB765106564A; Sun, 26 Aug 2012 20:01:55 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 4FF5D8FC1A; Sun, 26 Aug 2012 20:01:55 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7QK1s32069554; Sun, 26 Aug 2012 14:01:54 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7QK1qGi029391; Sun, 26 Aug 2012 14:01:52 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Tim Kientzle In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sun, 26 Aug 2012 14:01:52 -0600 Message-ID: <1346011312.1140.107.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, "arch@" Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 20:01:55 -0000 On Sun, 2012-08-26 at 11:53 -0700, Tim Kientzle wrote: > These rules sound reasonable. Good documentation might > also give examples of what the PRE/POST operations might entail > (e.g., from the preceding discussion, it sounds like PREREAD > and PREWRITE require at least a partial cache flush on ARM). > That helps folks who are coming to the docs with some hardware > background. > I agree, I think it would be good to have something like a RATIONALE section in the manpage that summarizes the issues faced by various categories of platform (hardware coherency, software-assisted coherency, etc) and how they handle it. You don't want people coding to the implementation (some of which is going on now, and I've been guilty of it myself); if there were more info in the docs there'd be less motivation to peek at the implementation. > > * Read and write sync operators may be combined in a single call, > > PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > > allowed, PREREAD|POSTREAD is not. We should note that while > > read and write operations may be combined, on some platforms > > PREREAD|PREWRITE is needlessly expensive when only a read is > > being performed. > > > PREREAD|POSTREAD doesn't sound useful to me, but > why would it be explicitly forbidden? > > Would you also forbid POSTREAD|PREWRITE? > (For a buffer that has just completed a DMA read > and is going to be immediately used for a DMA write?) My thinking on forbidding PREREAD|POSTREAD is at least partly that it removes some temptation to do the wrong thing: treat the busdma API as if it were a general cpu cache manipulation library. With the new definition of the sequences, a PREREAD|POSTREAD operation is nonsensical because it leaves no window during which the DMA hardware has access to memory; it is in effect a no-op. If you think in terms of implementation you might think "This would have to cause a cache invalidation." If you think in terms of API you should be thinking something more like "This marks out a time window during which the DMA hardware has safe access to that memory which has a duration of zero." POSTREAD|PREWRITE is interesting. It is not a no-op in terms of the API. It closes the hardware access window that was opened by an earlier PREREAD, and it opens a new hardware access window with the PREWRITE. Whether it touches hardware or caches or anything in a given implementation isn't the point, in terms of the API it's the right thing to do for a pair of back to back DMA operations, without intervening CPU access, on the same memory. So PREREAD|POSTREAD and PREWRITE|POSTWRITE make no sense, but all other combos should be allowed. Maybe instead of allowing and forbidding specific combos, we should just advise that these two are effectively no-ops. -- Ian From owner-freebsd-arm@FreeBSD.ORG Sun Aug 26 21:43:38 2012 Return-Path: Delivered-To: freebsd-arm@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31869106566C; Sun, 26 Aug 2012 21:43:38 +0000 (UTC) (envelope-from ray@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 055858FC08; Sun, 26 Aug 2012 21:43:38 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7QLhaoD072027; Sun, 26 Aug 2012 21:43:36 GMT (envelope-from ray@freefall.freebsd.org) Received: (from ray@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7QLhaJJ072023; Sun, 26 Aug 2012 21:43:36 GMT (envelope-from ray) Date: Sun, 26 Aug 2012 21:43:36 GMT Message-Id: <201208262143.q7QLhaJJ072023@freefall.freebsd.org> To: ray@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-arm@FreeBSD.org From: ray@FreeBSD.org Cc: Subject: Re: kern/171096: [arm][xscale][ixp]Allow 16bit access on PCI bus X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 21:43:38 -0000 Synopsis: [arm][xscale][ixp]Allow 16bit access on PCI bus Responsible-Changed-From-To: freebsd-bugs->freebsd-arm Responsible-Changed-By: ray Responsible-Changed-When: Sun Aug 26 21:42:42 UTC 2012 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=171096 From owner-freebsd-arm@FreeBSD.ORG Sun Aug 26 23:03:56 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6456F1065674 for ; Sun, 26 Aug 2012 23:03:56 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 19A0C8FC18 for ; Sun, 26 Aug 2012 23:03:55 +0000 (UTC) Received: by ialo14 with SMTP id o14so8766147ial.13 for ; Sun, 26 Aug 2012 16:03:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=ep5IxGfVukJ1EG6I0J6d7Tuvp5WIjgyi6rZXdyCg3ho=; b=QQ7WmvPU2WodulTQYYKX6hhxhiBv7oV6CG53ZAMEPvw1ubSa6PLcLCoz6zuM20T785 KgQ9P1vToAv5C97ji0xH0Xzl7e33WGkb+HtNWBHHc7LHDlnx68TSNin51hYHIxY1qoC3 yn6N+9cMQxy9ubTDKL4/m22divtu17eX053VtlaB16dHY9LUMMQeB/9AxFEAY4ks0bhQ 9PCzfTTPwm2HR2fi2KuZM8IyYWiai94tFgCUqmkAMkD6ZjbRbBEHTiMNcpr/JIy0ibW9 vwIpPwZwPB/TNTonhMwnasWypaWG7sXwUEzpGsq1KpPsn1uxKbc+svPt132pwvQNyIzW qZbg== Received: by 10.50.184.198 with SMTP id ew6mr8457358igc.27.1346022235175; Sun, 26 Aug 2012 16:03:55 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id ce6sm4966875igb.1.2012.08.26.16.03.48 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 26 Aug 2012 16:03:54 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346002922.1140.56.camel@revolution.hippie.lan> Date: Sun, 26 Aug 2012 17:03:46 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQn/huctD5GlP4QXVfkG3vdjWo3yJHEPUJbjv9819yMvrY5kmThn9afVeZ3Iv1poOA3Sxt3U Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 23:03:56 -0000 On Aug 26, 2012, at 11:42 AM, Ian Lepore wrote: > On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: >> The bottom line is that you can't mix things like that when cache >> lines are involved. The current code that tries is doomed to = failure. >> Doomed. You just can't control all flushes, as Ian's missive >> demonstrates, and trying to accommodate code that does this I don't >> think can possibly work. All the interrupt masking, copying in and >> out, etc I fear is doomed to utter and abject failure. =20 >>=20 > Until last weekend I was in the camp that thought the partial = cacheline > flush problem was solvable with sufficiently clever code. Now I agree > that we're doomed to failure and it's time to try another direction. >=20 > We're going to have some implementation work to do in arm and mips > busdma, but I think the larger part of the task is going to be = defining > more rigorously how a driver must interact with the busdma system to > function correctly on all types of platforms, and then update existing > drivers to conform. >=20 > The busdma manpage currently has some vague words about the usage and > sequencing of sync ops, such as "If read and write operations are not > preceded and followed by the appropriate synchronization operations, > behavior is undefined." I think we should more explicitly spell out > what the appropriate sequences are. In particular: >=20 > * The PRE and POST operations must occur in pairs; a PREREAD must > be followed eventually by a POSTREAD and a PREWRITE must be > followed by a POSTWRITE.=20 PREREAD means "I am about to tell the device to put data here, have = whaterver things might be pending in the CPU complex to get out of the = way." usually this means 'invalidate the cache for that range', but not = always. POSTREAD means 'The device's DMA is done, I'd like to start = accessing it now.' If the memory will be thrown away without being = looked at, then does the driver necessarily need to issue the POSTREAD? = I think so, but I don't know if that's a new requirement. > * The CPU is not allowed to access the mapped memory after a PRE > sync and before the corresponding POST sync. =20 Correct. > * The DMA hardware is not allowed to access the mapped memory > after a POST sync and before the next PRE sync.=20 Correct. > * Read and write sync operators may be combined in a single call, > PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > allowed, PREREAD|POSTREAD is not. We should note that while > read and write operations may be combined, on some platforms > PREREAD|PREWRITE is needlessly expensive when only a read is > being performed. Correct. > We also need some rules about working with buffers obtained from > bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). = I > think the rule should be that a buffer obtained from = bus_dmamem_alloc(), > or more formally any region of memory mapped by a bus_dmamap_load(), = is > a single logical object which can only be accessed by one entity at a > time. That means that there cannot be two concurrent DMA operations > happening in different regions of the same buffer, nor can DMA and CPU > access be happening concurrently even if in different parts of the > buffer. =20 There's something subtle that I'm missing. Why would two DMA operations = be disallowed? The rest makes good sense. > I've always thought that allocating a dma buffer feels like a big > hassle. You sometimes have to create a tag for the sole purpose of > setting the maxsize to get the buffer size you need when you call > bus_dmamem_alloc(). If bus_dmamem_alloc() took a size parm you could > just use your parent tag, or a generic tag appropriate to all the IO > you're doing for a given device. If you need a variety of buffers for > small control and command and status transfers of different sizes, you > end up having to manage up to a dozen tags and maps and buffers. It's > all very clunky and inconvenient. It's just the sort of thing that > makes you want to allocate a big buffer and subdivide it. Surely we > could do something to make it easier? You'd wind up creating a quick tag on the fly for the bus_dmamap_alloc = if you wanted to do this. Cleanup then becomes unclear. Warner From owner-freebsd-arm@FreeBSD.ORG Sun Aug 26 23:13:36 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 134681065676 for ; Sun, 26 Aug 2012 23:13:36 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6B0A08FC1B for ; Sun, 26 Aug 2012 23:13:34 +0000 (UTC) Received: by ialo14 with SMTP id o14so8778792ial.13 for ; Sun, 26 Aug 2012 16:13:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=ljCRbjZ9ASHSXhbZZlvbTJ/BcuQ0M4nJMzidj/rAYzc=; b=V9yNAXLh9FALuKaabVFzzFAongWW8nHoIXD2lWVjn5fuGWvgNxX1Er2CKmQoyqi6F9 zsTsbctJIOm20GHb9V3YvzG+kQOHNyNbM4cMtqkITpy8bqcCSjkKJ6pQQPb7a5npXnYw MJWRV5NJsjwPmM6mmOpkkND8vdbAAfNzzqxd0thIyaE1S1pIGiCvVGDHi4kP/ZB9Dn0U 1XEY8e9pia9djj68sRGkCfV1IA0mafw7JpqOuP20OiC8gwETDueupLVp+3oO3+FNWSpa aoossI+ytgjezU/c5bz9b9Jfog2tSm2r9LVmZYkon+xqWTaH9zomgESHU0iyX3mV0q2d E8zA== Received: by 10.50.187.229 with SMTP id fv5mr2371955igc.57.1346022814665; Sun, 26 Aug 2012 16:13:34 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id df1sm16929663igc.10.2012.08.26.16.13.32 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 26 Aug 2012 16:13:33 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346005507.1140.69.camel@revolution.hippie.lan> Date: Sun, 26 Aug 2012 17:13:31 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQnZ8oqUNtbRa8DFiv6cYpikyJ8hs3UyZXj5lBXEY/3adrqLSfkGY98nFvGoFpzEDJDicvWt Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 23:13:36 -0000 On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: > On Sun, 2012-08-26 at 13:05 -0500, Mark Tinguely wrote: >> I did a quick look at the drivers last summer. >>=20 >> Most drivers do the right thing and use memory allocated from >> bus_dmamem_alloc(). It is easy for us to give them a cache aligned >> buffer. >>=20 >> Some drivers use mbufs - 256 bytes which cache safe. >>=20 >> Some drivers directly or indirectly malloc() a buffer and then use it >> to dma - rather than try to fix them all, I was okay with making the >> smallest malloc() amount equal to the cache line size. It amounts to >> getting rid of the 16 byte allocation on some ARM architectures. The >> power of 2 allocator will then give us cache line safe allocation. >>=20 >> A few drivers take a small memory amount from the kernel stack and = dma >> to it <- broken driver. >>=20 >> The few drivers that use data from a structure and that memory is not >> cached aligned <- broken driver. >>=20 >=20 > I disagree about those last two points -- drivers that choose to use > stack memory or malloc'd memory as IO buffers are not broken. Stack DMA is bad policy, at best, and broken at worst. The reason is = because of alignment of the underlying unit. Since there's no way to = say that something is aligned to a given spot on the stack, you are = asking for random stack corruption. Also, malloced area is similarly problematic: There's no cache line = informing of the allocator, so you can wind up with an allocation of = memory that's corrupted due to cache effects. > Drivers > can do IO directly to/from userland buffers, do we say that an > application that calls read(2) and passes the address of a stack > variable is broken? Yes, if it is smaller than a cache line size, and not aligned to the = cache line. That's the point of the uio load variant. > In this regard, it's the busdma implementation that's broken, because = it > should bounce those IOs through a DMA-safe buffer. There's absolutely > no rule that I've ever heard of in FreeBSD that says IO can only take > place using memory allocated from busdma. That's partially true. Since BUSDMA grew up in the storage area, you = must allocate the memory from busdma, or it must be page aligned has = been the de-facto rule here. The mbuf and uio variants of load were = invented to cope with common cases of mbufs and user I/O to properly = flag things. How does busdma know that it is using memory that's not from its = allocator? > The rule is only that the > proper sequence of busdma operation must be called, and beyond that = it's > up to the busdma implementation to make it work. =20 No. Bouncing is needed due to poor alignment of the underlying device. = Not due to cache effects. There's a limited number of things that we support with busdma. = Arbitrary data from malloc that might be shared with the CPU isn't on = that list. > Our biggest problem, I think, is that we don't have a sufficient > definition of "the proper sequence of busdma operations." I disagree. The sequence has been known for a long time. > I don't think it will be very hard to make the arm and mips busdma > implementations work correctly. It won't even be too hard to make = them > fairly efficient at bouncing small IOs (my thinking is that we can = make > small bounces no more expensive than the current partial cacheline = flush > implementation which copies the data multiple times). Bouncing large = IO > will never be efficient, but the inefficiency will be a powerful > motivator to update drivers that do large IO to work better, such as > using buffers allocated from busdma. I don't think the cache line problem can be solved with bounce buffers. = Trying to accommodate broken drivers is what lead us to this spot. We = need to fix the broken drivers. If that's impossible, then the best we = can do is have the driver set a 'always bounce' flag in the tag it = creates and use that to always bounce for operations through that tag. Warner From owner-freebsd-arm@FreeBSD.ORG Sun Aug 26 23:15:30 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id AFD31106566C for ; Sun, 26 Aug 2012 23:15:30 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 637B48FC14 for ; Sun, 26 Aug 2012 23:15:30 +0000 (UTC) Received: by ialo14 with SMTP id o14so8781101ial.13 for ; Sun, 26 Aug 2012 16:15:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=iYEhywsT8AYLxmmanrymiKw7wLde2oh04bmK21Zf8Ko=; b=OYM7J6zLNad1UxCvDpsQlO9PdZ7Z3AAmkzCE2xzBZWgGRLvuZHe3OshnFVlDJEiR/R JfQ8t3rWgk8PoZdsWVhwUnqAM1WwQR7ANz9mzzG82xcVFy43SNCYgCDZjTvBPyfh7Tdj WJ/WHRWkWjaNRy1yfDKbZ2dL+mneQ72Do3HwdNqHpsE9lotuklnlgRbrpma/qJ49R4AW TRgb0MBKhU8iNRPR5YeLDa0ObDOcElbiOZ4La0QSIigRBA0mcew06ljmjbaZpmBkeFwi jb6+0MGCQVvFdV7ReadgerOrMQlkkiprRAGpb0Ueyy8r3R0pdSzExe9Sc0zhmlFOCpC/ LlRQ== Received: by 10.50.195.169 with SMTP id if9mr2347874igc.62.1346022930007; Sun, 26 Aug 2012 16:15:30 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id s2sm9413489igb.5.2012.08.26.16.15.27 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 26 Aug 2012 16:15:28 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=windows-1252 From: Warner Losh In-Reply-To: Date: Sun, 26 Aug 2012 17:15:26 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> To: Tim Kientzle X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQmYp/rt+haKWFBL0j8G/zwY0Nv/5bIH5uebqgcEnPn1SQJAoQ+TiMu6LVsQo8hQ1SN6IPPw Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, "arch@" Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 23:15:30 -0000 On Aug 26, 2012, at 12:53 PM, Tim Kientzle wrote: >=20 > On Sun, Aug 26, 2012 at 12:42 PM, Ian Lepore > wrote: >> On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: >>> The bottom line is that you can't mix things like that when cache >>> lines are involved. >>>=20 >> =85. I think we should more explicitly spell out >> what the appropriate sequences are. >=20 >=20 > As someone who is just tinkering with driver code for > the first time, I applaud any attempts to better document > this area! ;-) >=20 >> In particular: >>=20 >> * The PRE and POST operations must occur in pairs; a PREREAD must >> be followed eventually by a POSTREAD and a PREWRITE must be >> followed by a POSTWRITE. >> * The CPU is not allowed to access the mapped memory after a PRE >> sync and before the corresponding POST sync. >> * The DMA hardware is not allowed to access the mapped memory >> after a POST sync and before the next PRE sync. >=20 >=20 > These rules sound reasonable. Good documentation might > also give examples of what the PRE/POST operations might entail > (e.g., from the preceding discussion, it sounds like PREREAD > and PREWRITE require at least a partial cache flush on ARM). > That helps folks who are coming to the docs with some hardware > background. >=20 >> * Read and write sync operators may be combined in a single = call, >> PRE and POST operators may not be. E.G., PREREAD|PREWRITE is >> allowed, PREREAD|POSTREAD is not. We should note that while >> read and write operations may be combined, on some platforms >> PREREAD|PREWRITE is needlessly expensive when only a read is >> being performed. >=20 >=20 > PREREAD|POSTREAD doesn't sound useful to me, but > why would it be explicitly forbidden? It could be useful when DMAing from multiple devices. Let's say I have = a buffer that I want to get data from two different DMA operations. = Wouldn't this be how you'd specify it? > Would you also forbid POSTREAD|PREWRITE? > (For a buffer that has just completed a DMA read > and is going to be immediately used for a DMA write?) This could be useful for some zero-copy things, no? Warner > Tim >=20 > _______________________________________________ > freebsd-arm@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" From owner-freebsd-arm@FreeBSD.ORG Sun Aug 26 23:18:57 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 915801065674 for ; Sun, 26 Aug 2012 23:18:57 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 45AD58FC14 for ; Sun, 26 Aug 2012 23:18:57 +0000 (UTC) Received: by ialo14 with SMTP id o14so8785192ial.13 for ; Sun, 26 Aug 2012 16:18:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=vV/55IW1m35Nojy+yoYqvhV4JtFulO/qaNQbLw6wfac=; b=n4QsK9KvEKBGKWv/oLwIq4TF+NDJOHelU4195Iwt3nqWYj6aoOL/A8K1WC7ee9C48b YfvtaSzH3o8LC7txcMrv4NgxJ7qIQM8kzwmSgQA6+w1qyJ/jSnxTCrHare0xOpK+U1SC pAiFk4Ifr5ts0+CyBtV5VLoM1Wyf/8nV2ErtJTRqg6XQTMBe4v653rT7NSBiWP5D19r/ suMr9632pp31xFVpExr4io5pDgfg+9PRDn/sSZkuNhpQlPDdDvU4m0VgdA/ldR3IN64J ZwVUAlDggIbwSTM6iBqdZB60oUcAJtiROfiIh5Cy1Xa1Hw4XCB+xfBtPEBHIi4ALoqhY CvGQ== Received: by 10.50.88.229 with SMTP id bj5mr8287211igb.21.1346023136744; Sun, 26 Aug 2012 16:18:56 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id i2sm4979193igl.8.2012.08.26.16.18.54 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 26 Aug 2012 16:18:55 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346011312.1140.107.camel@revolution.hippie.lan> Date: Sun, 26 Aug 2012 17:18:54 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <26F65164-EC99-483E-9FA8-916AD99CBCBC@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346011312.1140.107.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQk2/yTwiH6zmrYT7JXdZl2/EBWZsfX+iYOSC0N09NmmOHLx2trArJ1N2FNjif808VIbvtkV Cc: Tim Kientzle , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, "arch@" Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 23:18:57 -0000 On Aug 26, 2012, at 2:01 PM, Ian Lepore wrote: > On Sun, 2012-08-26 at 11:53 -0700, Tim Kientzle wrote: >> These rules sound reasonable. Good documentation might >> also give examples of what the PRE/POST operations might entail >> (e.g., from the preceding discussion, it sounds like PREREAD >> and PREWRITE require at least a partial cache flush on ARM). >> That helps folks who are coming to the docs with some hardware >> background. >>=20 >=20 > I agree, I think it would be good to have something like a RATIONALE > section in the manpage that summarizes the issues faced by various > categories of platform (hardware coherency, software-assisted = coherency, > etc) and how they handle it. You don't want people coding to the > implementation (some of which is going on now, and I've been guilty of > it myself); if there were more info in the docs there'd be less > motivation to peek at the implementation. More documentation is good. >>> * Read and write sync operators may be combined in a single = call, >>> PRE and POST operators may not be. E.G., PREREAD|PREWRITE is >>> allowed, PREREAD|POSTREAD is not. We should note that while >>> read and write operations may be combined, on some platforms >>> PREREAD|PREWRITE is needlessly expensive when only a read is >>> being performed. >>=20 >>=20 >> PREREAD|POSTREAD doesn't sound useful to me, but >> why would it be explicitly forbidden? >>=20 >> Would you also forbid POSTREAD|PREWRITE? >> (For a buffer that has just completed a DMA read >> and is going to be immediately used for a DMA write?) >=20 > My thinking on forbidding PREREAD|POSTREAD is at least partly that it > removes some temptation to do the wrong thing: treat the busdma API as > if it were a general cpu cache manipulation library. =20 I think it should be read as POSTREAD + PREREAD, not PREREAD + POSTREAD. > With the new definition of the sequences, a PREREAD|POSTREAD operation > is nonsensical because it leaves no window during which the DMA = hardware > has access to memory; it is in effect a no-op. If you think in terms = of > implementation you might think "This would have to cause a cache > invalidation." If you think in terms of API you should be thinking > something more like "This marks out a time window during which the DMA > hardware has safe access to that memory which has a duration of zero." Acutally, it would allow one to shift the DMA from one device to = another, if read the way I say above. > POSTREAD|PREWRITE is interesting. It is not a no-op in terms of the > API. It closes the hardware access window that was opened by an = earlier > PREREAD, and it opens a new hardware access window with the PREWRITE. > Whether it touches hardware or caches or anything in a given > implementation isn't the point, in terms of the API it's the right = thing > to do for a pair of back to back DMA operations, without intervening = CPU > access, on the same memory. Right. This is useful in many cases. Consider a log structured device = that reads data into memory, and then writes it back to the tail of the = log as a defragging operation. > So PREREAD|POSTREAD and PREWRITE|POSTWRITE make no sense, but all = other > combos should be allowed. Maybe instead of allowing and forbidding > specific combos, we should just advise that these two are effectively > no-ops. I respectfully disagree.. Warner From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 01:26:09 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E6B57106564A; Mon, 27 Aug 2012 01:26:09 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9BEEE8FC16; Mon, 27 Aug 2012 01:26:09 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so6609002pbb.13 for ; Sun, 26 Aug 2012 18:26:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=buRkMCmZXq3F4Cc5OBO5pchRTJfQaZIQI48CqnG/UVM=; b=PAnNo8QWePhAymqZ/mWGbYh8Urc23Jl/DWWf7mLRxsT+q8Oxkez8Ol9YphyhVM3q35 x/ASzYBiRdpuZhiL4yVUHmVmQkRfh5DgoUCSEMxG862EH+bdfqJ/t1lUxC74+V4NwP9X IMQ1/qx0CsZW79aQogmq3fkNjK/c6aBIlf1Rg8vNjATaoGxRqTsGunzWDw50lXV5K+sG 7W9419jOZyDWkUxOu483e5zgItkzYa9JddxiLzDftB90igFeVRO9mKbjHmdA+jT8lxeS LDve5jhPhL6WMxa0canqUNYzriSRFegQcasvJtHFx6zV/HjHn2eqZaRK+Gl9lyoq+Zwp h0DQ== MIME-Version: 1.0 Received: by 10.68.129.131 with SMTP id nw3mr30406099pbb.43.1346030769022; Sun, 26 Aug 2012 18:26:09 -0700 (PDT) Received: by 10.68.36.106 with HTTP; Sun, 26 Aug 2012 18:26:08 -0700 (PDT) Received: by 10.68.36.106 with HTTP; Sun, 26 Aug 2012 18:26:08 -0700 (PDT) In-Reply-To: <26F65164-EC99-483E-9FA8-916AD99CBCBC@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346011312.1140.107.camel@revolution.hippie.lan> <26F65164-EC99-483E-9FA8-916AD99CBCBC@bsdimp.com> Date: Sun, 26 Aug 2012 18:26:08 -0700 Message-ID: From: Adrian Chadd To: Warner Losh Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-arm@freebsd.org, "arch@" , freebsd-mips@freebsd.org, Tim Kientzle Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 01:26:10 -0000 Hm. So perhaps we should document the busdma assumptions with respect to alignment and coherency? So this doesnt happen again? Adrian On Aug 26, 2012 4:19 PM, "Warner Losh" wrote: > > On Aug 26, 2012, at 2:01 PM, Ian Lepore wrote: > > > On Sun, 2012-08-26 at 11:53 -0700, Tim Kientzle wrote: > >> These rules sound reasonable. Good documentation might > >> also give examples of what the PRE/POST operations might entail > >> (e.g., from the preceding discussion, it sounds like PREREAD > >> and PREWRITE require at least a partial cache flush on ARM). > >> That helps folks who are coming to the docs with some hardware > >> background. > >> > > > > I agree, I think it would be good to have something like a RATIONALE > > section in the manpage that summarizes the issues faced by various > > categories of platform (hardware coherency, software-assisted coherency, > > etc) and how they handle it. You don't want people coding to the > > implementation (some of which is going on now, and I've been guilty of > > it myself); if there were more info in the docs there'd be less > > motivation to peek at the implementation. > > More documentation is good. > > >>> * Read and write sync operators may be combined in a single call, > >>> PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > >>> allowed, PREREAD|POSTREAD is not. We should note that while > >>> read and write operations may be combined, on some platforms > >>> PREREAD|PREWRITE is needlessly expensive when only a read is > >>> being performed. > >> > >> > >> PREREAD|POSTREAD doesn't sound useful to me, but > >> why would it be explicitly forbidden? > >> > >> Would you also forbid POSTREAD|PREWRITE? > >> (For a buffer that has just completed a DMA read > >> and is going to be immediately used for a DMA write?) > > > > My thinking on forbidding PREREAD|POSTREAD is at least partly that it > > removes some temptation to do the wrong thing: treat the busdma API as > > if it were a general cpu cache manipulation library. > > I think it should be read as POSTREAD + PREREAD, not PREREAD + POSTREAD. > > > With the new definition of the sequences, a PREREAD|POSTREAD operation > > is nonsensical because it leaves no window during which the DMA hardware > > has access to memory; it is in effect a no-op. If you think in terms of > > implementation you might think "This would have to cause a cache > > invalidation." If you think in terms of API you should be thinking > > something more like "This marks out a time window during which the DMA > > hardware has safe access to that memory which has a duration of zero." > > Acutally, it would allow one to shift the DMA from one device to another, > if read the way I say above. > > > POSTREAD|PREWRITE is interesting. It is not a no-op in terms of the > > API. It closes the hardware access window that was opened by an earlier > > PREREAD, and it opens a new hardware access window with the PREWRITE. > > Whether it touches hardware or caches or anything in a given > > implementation isn't the point, in terms of the API it's the right thing > > to do for a pair of back to back DMA operations, without intervening CPU > > access, on the same memory. > > Right. This is useful in many cases. Consider a log structured device > that reads data into memory, and then writes it back to the tail of the log > as a defragging operation. > > > So PREREAD|POSTREAD and PREWRITE|POSTWRITE make no sense, but all other > > combos should be allowed. Maybe instead of allowing and forbidding > > specific combos, we should just advise that these two are effectively > > no-ops. > > I respectfully disagree.. > > Warner > > _______________________________________________ > freebsd-arm@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" > From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 04:10:06 2012 Return-Path: Delivered-To: freebsd-arm@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B4900106564A for ; Mon, 27 Aug 2012 04:10:06 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9FBAA8FC1B for ; Mon, 27 Aug 2012 04:10:06 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7R4A6GM022828 for ; Mon, 27 Aug 2012 04:10:06 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7R4A65W022827; Mon, 27 Aug 2012 04:10:06 GMT (envelope-from gnats) Date: Mon, 27 Aug 2012 04:10:06 GMT Message-Id: <201208270410.q7R4A65W022827@freefall.freebsd.org> To: freebsd-arm@FreeBSD.org From: dfilter@FreeBSD.ORG (dfilter service) Cc: Subject: Re: arm/155214: commit references a PR X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dfilter service List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 04:10:06 -0000 The following reply was made to PR arm/155214; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: arm/155214: commit references a PR Date: Mon, 27 Aug 2012 04:04:04 +0000 (UTC) Author: imp Date: Mon Aug 27 04:03:49 2012 New Revision: 239719 URL: http://svn.freebsd.org/changeset/base/239719 Log: Don't puprosely overclock the SD bus to 30MHz, make the user explicltly enable that. The driver chose to use 60MHz / 2 (30MHz) most of the time rather than 60MHz / 4 (15MHz) based on the Linux driver of the time. This pushes the spec a little in order to not suffer the penalty of running at 15MHz. However, when other bus masters are active in the system, and the user tries 4-wire mode, the internal bus arbitration would fail with data loss as a result. # Comments from PR were reworked to reflect my historical perspective PR: 155214 (partial) Submitted by: Ian Lepore Modified: head/sys/arm/at91/at91_mci.c Modified: head/sys/arm/at91/at91_mci.c ============================================================================== --- head/sys/arm/at91/at91_mci.c Mon Aug 27 03:09:39 2012 (r239718) +++ head/sys/arm/at91/at91_mci.c Mon Aug 27 04:03:49 2012 (r239719) @@ -67,6 +67,53 @@ __FBSDID("$FreeBSD$"); #include "opt_at91.h" +/* + * About running the MCI bus at 30mhz... + * + * Historically, the MCI bus has been run at 30mhz on systems with a 60mhz + * master clock, due to a bug in the mantissa table in dev/mmc.c making it + * appear that the card's max speed was always 30mhz. Fixing that bug causes + * the mmc driver to request a 25mhz clock (as it should) and the logic in + * at91_mci_update_ios() picks the highest speed that doesn't exceed that limit. + * With a 60mhz MCK that would be 15mhz, and that's a real performance buzzkill + * when you've been getting away with 30mhz all along. + * + * By defining AT91_MCI_USE_30MHZ (or setting the 30mhz=1 device hint or + * sysctl) you can enable logic in at91_mci_update_ios() to overlcock the SD + * bus a little by running it at MCK / 2 when MCK is between greater than + * 50MHz and the requested speed is 25mhz. This appears to work on virtually + * all SD cards, since it is what this driver has been doing prior to the + * introduction of this option, where the overclocking vs underclocking + * decision was automaticly "overclock". Modern SD cards can run at + * 45mhz/1-bit in standard mode (high speed mode enable commands not sent) + * without problems. + * + * Speaking of high-speed mode, the rm9200 manual says the MCI device supports + * the SD v1.0 specification and can run up to 50mhz. This is interesting in + * that the SD v1.0 spec caps the speed at 25mhz; high speed mode was added in + * the v1.10 spec. Furthermore, high speed mode doesn't just crank up the + * clock, it alters the signal timing. The rm9200 MCI device doesn't support + * these altered timings. So while speeds over 25mhz may work, they only work + * in what the SD spec calls "default" speed mode, and it amounts to violating + * the spec by overclocking the bus. + * + * If you also enable 4-wire mode it's possible the 30mhz transfers will fail. + * On the AT91RM9200, due to bugs in the bus contention logic, if you have the + * USB host device and OHCI driver enabled will fail. Even underclocking to + * 15MHz, intermittant overrun and underrun errors occur. Note that you don't + * even need to have usb devices attached to the system, the errors begin to + * occur as soon as the OHCI driver sets the register bit to enable periodic + * transfers. It appears (based on brief investigation) that the usb host + * controller uses so much ASB bandwidth that sometimes the DMA for MCI + * transfers doesn't get a bus grant in time and data gets dropped. Adding + * even a modicum of network activity changes the symptom from intermittant to + * very frequent. Members of the AT91SAM9 family have corrected this problem, or + * are at least better about their use of the bus. + */ +#ifndef AT91_MCI_USE_30MHZ +#define AT91_MCI_USE_30MHZ 1 +#endif + #define BBSZ 512 struct at91_mci_softc { @@ -76,9 +123,10 @@ struct at91_mci_softc { #define CAP_HAS_4WIRE 1 /* Has 4 wire bus */ #define CAP_NEEDS_BYTESWAP 2 /* broken hardware needing bounce */ int flags; - int has_4wire; #define CMD_STARTED 1 #define STOP_STARTED 2 + int has_4wire; + int use_30mhz; struct resource *irq_res; /* IRQ resource */ struct resource *mem_res; /* Memory resource */ struct mtx sc_mtx; @@ -236,16 +284,33 @@ at91_mci_attach(device_t dev) if (sc->has_4wire) sc->sc_cap |= CAP_HAS_4WIRE; - sc->host.f_min = at91_master_clock / 512; +#if defined(AT91_MCI_USE_30MHZ) && AT91_MCI_USE_30MHZ != 0 + sc->use_30mhz = 1; +#endif + resource_int_value(device_get_name(dev), device_get_unit(dev), + "30mhz", &sc->use_30mhz); + SYSCTL_ADD_UINT(sctx, SYSCTL_CHILDREN(soid), OID_AUTO, "30mhz", + CTLFLAG_RW, &sc->use_30mhz, 0, "use 30mhz clock for 25mhz request"); + + /* Our real min freq is master_clock/512, but upper driver layers are + * going to set the min speed during card discovery, and the right speed + * for that is 400khz, so advertise a safe value just under that. + * + * For max speed, while the rm9200 manual says the max is 50mhz, it also + * says it supports only the SD v1.0 spec, which means the real limit is + * 25mhz. On the other hand, historical use has been to slightly violate + * the standard by running the bus at 30mhz. For more information on + * that, see the comments at the top of this file. + */ sc->host.f_min = 375000; sc->host.f_max = at91_master_clock / 2; - if (sc->host.f_max > 50000000) - sc->host.f_max = 50000000; /* Limit to 50MHz */ - + if (sc->host.f_max > 25000000) + sc->host.f_max = 25000000; /* Limit to 25MHz */ sc->host.host_ocr = MMC_OCR_320_330 | MMC_OCR_330_340; sc->host.caps = 0; if (sc->sc_cap & CAP_HAS_4WIRE) sc->host.caps |= MMC_CAP_4_BIT_DATA; + child = device_add_child(dev, "mmc", 0); device_set_ivars(dev, &sc->host); err = bus_generic_attach(dev); @@ -338,23 +403,38 @@ static int at91_mci_update_ios(device_t brdev, device_t reqdev) { struct at91_mci_softc *sc; - struct mmc_host *host; struct mmc_ios *ios; uint32_t clkdiv; sc = device_get_softc(brdev); - host = &sc->host; - ios = &host->ios; - // bus mode? + ios = &sc->host.ios; + + /* + * Calculate our closest available clock speed that doesn't exceed the + * requested speed. + * + * If the master clock is greater than 50MHz and the requested bus + * speed is 25mhz and the use_30mhz flag is on, set clkdiv to zero to + * get a master_clock / 2 (25-30MHz) MMC/SD clock rather than settle for + * the next lower click (12-15MHz). See comments near the top of the + * file for more info. + * + * Whatever we come up with, store it back into ios->clock so that the + * upper layer drivers can report the actual speed of the bus. + */ if (ios->clock == 0) { WR4(sc, MCI_CR, MCI_CR_MCIDIS); clkdiv = 0; } else { - WR4(sc, MCI_CR, MCI_CR_MCIEN); - if ((at91_master_clock % (ios->clock * 2)) == 0) + WR4(sc, MCI_CR, MCI_CR_MCIEN|MCI_CR_PWSEN); + if (sc->use_30mhz && ios->clock == 25000000 && + at91_master_clock > 50000000) + clkdiv = 0; + else if ((at91_master_clock % (ios->clock * 2)) == 0) clkdiv = ((at91_master_clock / ios->clock) / 2) - 1; else clkdiv = (at91_master_clock / ios->clock) / 2; + ios->clock = at91_master_clock / ((clkdiv+1) * 2); } if (ios->bus_width == bus_width_4) WR4(sc, MCI_SDCR, RD4(sc, MCI_SDCR) | MCI_SDCR_SDCBUS); _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org" From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 06:07:05 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D29CD106566C; Mon, 27 Aug 2012 06:07:05 +0000 (UTC) (envelope-from hans.petter.selasky@bitfrost.no) Received: from smtp02-out.isp.tdc.no (smtp02-out.isp.tdc.no [213.236.144.173]) by mx1.freebsd.org (Postfix) with ESMTP id 5C13D8FC0C; Mon, 27 Aug 2012 06:07:05 +0000 (UTC) Received: from mail.bitfrost.no (mail.bitfrost.no [85.19.79.136]) by smtp02-out.isp.tdc.no (Postfix) with ESMTP id 3X52cd1p2pz3CP; Mon, 27 Aug 2012 08:04:49 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at bitfrost.no From: =?windows-1252?Q?Hans_Petter_Selasky?= To: =?windows-1252?Q?Ian_Lepore?= , =?windows-1252?Q?Warner_Losh?= Date: Mon, 27 Aug 2012 08:06:57 +0200 Mime-Version: 1.0 In-Reply-To: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> X-Priority: 3 (Normal) Message-Id: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "=?windows-1252?Q?freebsd-arm=40freebsd.org?=" , "=?windows-1252?Q?freebsd-mips=40freebsd.org?=" , "=?windows-1252?Q?freebsd-arch=40freebsd.org?=" Subject: RE: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 06:07:06 -0000 Hi,=20=0D=0ACorrect.=0D=0A=0D=0A> We also need some rules about working w= ith buffers obtained from=0D=0A> bus_dmamem_alloc() and external buffers = passed to bus_dmamap_load(). =A0I=0D=0A> think the rule should be that a = buffer obtained from bus_dmamem_alloc(),=0D=0A> or more formally any regi= on of memory mapped by a bus_dmamap_load(), is=0D=0A> a single logical ob= ject which can only be accessed by one entity at a=0D=0A> time. =A0That m= eans that there cannot be two concurrent DMA operations=0D=0A> happening = in different regions of the same buffer, nor can DMA and CPU=0D=0A> acces= s be happening concurrently even if in different parts of the=0D=0A> buff= er. =A0=0D=0A=0D=0A=0D=0AIs this something which we can fix using a simpl= e __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to= be DMA loaded.=0D=0A=0D=0A=A0=0D=0A=0D=0AAlso: Why is busdma not complai= ning when loading a non-valid buffer pointer=3F=0D=0A=0D=0A--HPS=0D=0A From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 11:07:03 2012 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 937E7106566B for ; Mon, 27 Aug 2012 11:07:03 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 7D8618FC1F for ; Mon, 27 Aug 2012 11:07:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7RB73Ue085790 for ; Mon, 27 Aug 2012 11:07:03 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7RB723p085788 for freebsd-arm@FreeBSD.org; Mon, 27 Aug 2012 11:07:02 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 27 Aug 2012 11:07:02 GMT Message-Id: <201208271107.q7RB723p085788@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-arm@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-arm@FreeBSD.org X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 11:07:03 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/171096 arm [arm][xscale][ixp]Allow 16bit access on PCI bus o arm/166256 arm build fail in pmap.c o arm/162159 arm [panic] USB errors leading to panic on DockStar 9.0-RC o arm/161110 arm /usr/src/sys/arm/include/signal.h is bad o arm/161044 arm devel/icu does not build on arm o arm/158950 arm arm/sheevaplug fails fsx when mmap operations are enab o arm/155894 arm [patch] Enable at91 booting from SDHC (high capacity) o arm/155214 arm [patch] MMC/SD IO slow on Atmel ARM with modern large o arm/154227 arm [geli] using GELI leads to panic on ARM o arm/153380 arm Panic / translation fault with wlan on ARM o arm/150581 arm [irq] Unknown error generates IRQ address decoding err o arm/149288 arm mail/dovecot causes panic during configure on Sheevapl o arm/134368 arm [patch] nslu2_led driver for the LEDs on the NSLU2 p arm/134338 arm [patch] Lock GPIO accesses on ixp425 14 problems total. From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 13:38:10 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 85AB61065675 for ; Mon, 27 Aug 2012 13:38:10 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0DAB28FC1D for ; Mon, 27 Aug 2012 13:38:10 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so7583092pbb.13 for ; Mon, 27 Aug 2012 06:38:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:x-priority :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer:x-gm-message-state; bh=sDQvljch4pOy5n8q84V4CzIi45qexsJyPkD23yBHMLw=; b=nnKzkW+rG9QlNY8VyltZ5SoUdi0qFHlJsUL4w8i0ZqVbFafeBcfi65qP+Q+gq6/xgp MX20IvHEaUgbxc0240a44Ehoqr/RMd8+9Oh/yPppJErpTV7mlt2ORG4I7cZtMFWgEVLs 3bQW2xn7770eNm3sSy5N7pOsH7YvcsM62TTWP7YII2mR4HOO1azlCGqfV2UyZSx4HhzL DECamopAV5fjdYyn2mr62JRX/z71cqukg0I8qLvO9ylUhrWMWRomkwbjOx+vvqQPjvlT 1UQom0JXqKdfl8Mwj1Mz207ShMS6CrZzoGR4qa+uLQpWsMrBCvMxY0wOT8dXoytZA9E3 zJhA== Received: by 10.66.74.37 with SMTP id q5mr30255409pav.29.1346074689751; Mon, 27 Aug 2012 06:38:09 -0700 (PDT) Received: from [10.0.0.63] (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id pj10sm14721644pbb.46.2012.08.27.06.38.07 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 06:38:09 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh X-Priority: 3 (Normal) In-Reply-To: Date: Mon, 27 Aug 2012 07:38:06 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> To: Hans Petter Selasky X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQl02RriX3gco82hO+ClUEFHwXEgWKWHIoN6iUCMiGMo6IvoqfvQ4EGygrw6zZRNqYBvAxe2 Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 13:38:10 -0000 On Aug 27, 2012, at 12:06 AM, Hans Petter Selasky wrote: > Hi, > Correct. >=20 > > We also need some rules about working with buffers obtained from > > bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). = I > > think the rule should be that a buffer obtained from = bus_dmamem_alloc(), > > or more formally any region of memory mapped by a bus_dmamap_load(), = is > > a single logical object which can only be accessed by one entity at = a > > time. That means that there cannot be two concurrent DMA operations > > happening in different regions of the same buffer, nor can DMA and = CPU > > access be happening concurrently even if in different parts of the > > buffer. =20 >=20 > Is this something which we can fix using a simple = __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to = be DMA loaded. No. I don't think so. the reason is that you can't define USB_DMA_ALGIN = to be a constant on MIPS, at least, or I think ARM because that's = determined at run time. > Also: Why is busdma not complaining when loading a non-valid buffer = pointer? Speed and code simplicity. Warner From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 15:12:11 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6FC591065674 for ; Mon, 27 Aug 2012 15:12:11 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) by mx1.freebsd.org (Postfix) with ESMTP id 44E478FC14 for ; Mon, 27 Aug 2012 15:12:10 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id q7RFC818091514; Mon, 27 Aug 2012 15:12:08 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.143] (CiscoE3000 [192.168.1.65]) by kientzle.com with SMTP id 8wvqwunvxbytgk5wqkfzrzju8s; Mon, 27 Aug 2012 15:12:08 +0000 (UTC) (envelope-from tim@kientzle.com) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=us-ascii From: Tim Kientzle X-Priority: 3 (Normal) In-Reply-To: Date: Mon, 27 Aug 2012 08:12:07 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> To: Warner Losh X-Mailer: Apple Mail (2.1278) Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:12:11 -0000 On Aug 27, 2012, at 6:38 AM, Warner Losh wrote: >=20 > On Aug 27, 2012, at 12:06 AM, Hans Petter Selasky wrote: >=20 >> Hi, >> Correct. >>=20 >>> We also need some rules about working with buffers obtained from >>> bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). = I >>> think the rule should be that a buffer obtained from = bus_dmamem_alloc(), >>> or more formally any region of memory mapped by a bus_dmamap_load(), = is >>> a single logical object which can only be accessed by one entity at = a >>> time. That means that there cannot be two concurrent DMA operations >>> happening in different regions of the same buffer, nor can DMA and = CPU >>> access be happening concurrently even if in different parts of the >>> buffer. =20 >>=20 >> Is this something which we can fix using a simple = __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to = be DMA loaded. >=20 > No. I don't think so. the reason is that you can't define = USB_DMA_ALGIN to be a constant on MIPS, at least, or I think ARM because = that's determined at run time. But don't mbuf structures do pretty much what Hans is suggesting? Why is mbuf okay? Tim From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 15:20:53 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B44911065673; Mon, 27 Aug 2012 15:20:53 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7601A8FC15; Mon, 27 Aug 2012 15:20:53 +0000 (UTC) Received: by dadr6 with SMTP id r6so2589280dad.13 for ; Mon, 27 Aug 2012 08:20:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=lYrRM128qw04nZGkoy9WgB67sDTxSyD1vHe6xtBexvo=; b=ajYXEeB64Q8sgiJvPDcQkhcYLTpxX+4bxS1dk9blsyBAxhjVkMd5u5tSfiEPg7RmSB kc8Sq1JfWh/cI6jIinuo7tNbnfvkeC3SKBjbZXooJrfhgc9uBHpGQGHhpkO5lyCSkCQ2 i2zMmX8uspoG9YByjg9qYniJdsrthFCXhu7Da/k+7t4xFilPiFrwEIM8OhyYqW0NAxve 1z6Ug3KtaAIRG1Kx7fBG39DhO0ixteho3n4n+m1axU6wpj9gq30F7D+cW3RD9mM8F525 BQ+vOw8oW9uVElLkOygvPFmbcJqyX1TOotxbUMEBtAGyzQWZGau6A9Dy41T+0Husyp83 eoQg== MIME-Version: 1.0 Received: by 10.68.129.131 with SMTP id nw3mr35286569pbb.43.1346080853000; Mon, 27 Aug 2012 08:20:53 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.36.106 with HTTP; Mon, 27 Aug 2012 08:20:52 -0700 (PDT) In-Reply-To: References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> Date: Mon, 27 Aug 2012 08:20:52 -0700 X-Google-Sender-Auth: ohyisdNkmhbkzwukiyHFrt829tI Message-ID: From: Adrian Chadd To: Tim Kientzle Content-Type: text/plain; charset=ISO-8859-1 Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:20:53 -0000 On 27 August 2012 08:12, Tim Kientzle wrote: >> No. I don't think so. the reason is that you can't define USB_DMA_ALGIN to be a constant on MIPS, at least, or I think ARM because that's determined at run time. FYI, I wonder if MIPS may need some run time value for that. Right now we hide it behind needing a custom config file for each mips variant, rather than supporting one mips24k kernel for ${LOTS} of variants. > But don't mbuf structures do pretty much what Hans is suggesting? > > Why is mbuf okay? Which part of mbufs? I think the difference here is that there's no concurrent access. Ie, although you may have an mbuf header + data inside the same cache line, you wouldn't go fondling the mbuf once it's handed to the hardware. But I was under the impression that mbuf + mbuf buffers are already correctly aligned. This all honestly looks like a very i386-centric interpretation of the busdma "intention", which I think illustrates a need to better document what assumptions busdma actually makes. That does remind me, I think the ath(4) driver does the same (since it allocates its own descriptor block and then treats it as an array of descriptors for the hardware to access) - I should ensure that sizeof(ath_desc) is aligned on the relevant architecture. It gets slightly scary - AR93xx TX descriptors are "L1 cache == 128 byte aligned" which is an enormous waste of memory compared to a 16 or 32 byte aligned platform. Alas.. Adrian From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 15:33:24 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 748681065686; Mon, 27 Aug 2012 15:33:24 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 8FCB48FC1E; Mon, 27 Aug 2012 15:33:08 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7RFX0vq088953; Mon, 27 Aug 2012 09:33:00 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7RFWbGs031228; Mon, 27 Aug 2012 09:32:37 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> Content-Type: text/plain; charset="us-ascii" Date: Mon, 27 Aug 2012 09:32:37 -0600 Message-ID: <1346081557.1140.181.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org, Hans Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:33:24 -0000 On Sun, 2012-08-26 at 17:13 -0600, Warner Losh wrote: > On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: > > In this regard, it's the busdma implementation that's broken, because it > > should bounce those IOs through a DMA-safe buffer. There's absolutely > > no rule that I've ever heard of in FreeBSD that says IO can only take > > place using memory allocated from busdma. > > That's partially true. Since BUSDMA grew up in the storage area, you must allocate the memory from busdma, or it must be page aligned has been the de-facto rule here. Where does anything say that you must allocate the memory from busdma if it's not mbuf/page-aligned? I've never seen it in any docs. I certainly find contrary evidence in existing driver code (and I'm not talking about USB here). > The mbuf and uio variants of load were invented to cope with common cases of mbufs and user I/O to properly flag things. > What does that mean, "to properly flag things"? I think with uio we come to the crux of the issue. Userland buffers are not allocated with knowledge of the busdma constraints. That leads to some choices: * We don't support IO with a userland buffer at all. * We support userland IO if the userland buffers are accidentally aligned properly for the platform, otherwise the call fails. * We support userland buffers by making any driver which wants to do so responsible for always copying the data to aligned buffers (each individual driver handles bounces). * We support userland buffers by having the busdma layer handle the bouncing when required. The first two seem untenable to me. The third one comes down to "every driver must always bounce all userland data" because the driver doesn't actually have access to the info to decide whether a userland buffer is aligned properly or not for a given platform. That leaves the option where the busdma layer handles bouncing for unaligned userland buffers. If that's possible, then the busdma layer could automatically bounce any unaligned request, whether it came from userland or a driver reading a status response into an unaligned local buffer in kernel memory. > How does busdma know that it is using memory that's not from its allocator? The busdma code allocates the map along with the buffer, and can record information in the map that it can use during mapping and sync operations to know whether it allocated the buffer. > > The rule is only that the > > proper sequence of busdma operation must be called, and beyond that it's > > up to the busdma implementation to make it work. > > No. Bouncing is needed due to poor alignment of the underlying device. Not due to cache effects. Says who? This statement seems to me to be more about opinion and dogma than about a documented API or technical concerns about how to implement it. IMO, bouncing is needed when that's the only way to make a particular busdma sequence work correctly given the parameters of the hardware and the transfer. To put it another way, the busdma subsystem is responsible for helping make DMA transfers work on a platform without every driver needing to contain platform-specific code, and bouncing is one of the techniques available to it. > There's a limited number of things that we support with busdma. Arbitrary data from malloc that might be shared with the CPU isn't on that list. > Where are those limitations documented? This isn't some small oversight in the documention, if the rule is "IO, any IO at all, can only be performed using buffers allocated from busdma" that's a pretty major thing that ought to appear in multiple manpages relating to driver development. If you don't think "any IO at all" is the right characterization, read all the way to the end before responding. > > Our biggest problem, I think, is that we don't have a sufficient > > definition of "the proper sequence of busdma operations." > > I disagree. The sequence has been known for a long time. > > > I don't think it will be very hard to make the arm and mips busdma > > implementations work correctly. It won't even be too hard to make them > > fairly efficient at bouncing small IOs (my thinking is that we can make > > small bounces no more expensive than the current partial cacheline flush > > implementation which copies the data multiple times). Bouncing large IO > > will never be efficient, but the inefficiency will be a powerful > > motivator to update drivers that do large IO to work better, such as > > using buffers allocated from busdma. > > I don't think the cache line problem can be solved with bounce buffers. Trying to accommodate broken drivers is what lead us to this spot. We need to fix the broken drivers. If that's impossible, then the best we can do is have the driver set a 'always bounce' flag in the tag it creates and use that to always bounce for operations through that tag. So you'd be okay with a driver setting a flag that says "always bounce" but not okay with the busdma layer bouncing only when it's actually necessary? I'm confused -- do you think the busdma layer will be unable to detect when it's necessary unless directed from the outside? Let me pose a question back to you... if it is up to a driver to allocate DMA buffers using the busdma allocator, how is any given driver to know whether DMA is going to be involved in the transfer? Don't think USB or ATA here... think iicbus/foo_temperature_sensor, or the mmc/sd driver. Those drivers know nothing about the hardware bridge layers beneath them and whether DMA is involved or not. They just know "I need to read a 2-byte value from a register on this IIC chip" or "I need to retrieve the 32-bit CSD data from the SD card" and they accomplish that by calling a read method of some lower-level bridge driver. In each of those cases we have both PIO and DMA implementations of the lower-level drivers that move bits over a wire. If the concensus is that such drivers should always allocate all IO buffers using busdma, even though they have no idea whether DMA is going to be involved, then we certainly have a lot of work ahead of us in terms of "fixing broken drivers." That also implies that bus_dmamem_alloc() is misnamed and implemented in the wrong place, because it's not really about dma at all. If you push the responsibility down to the layer where it is known whether DMA is going to be involved, then we're back to the need for many individual drivers to bounce every request, because they don't have access to the info needed to know whether bouncing is required or not for a given buffer they were handed from higher layers. (Remember when I proposed exporting that info to drivers so they could decide whether a buffer is dma-aligned or not, the response was "No, keep it all hidden in the busdma layer," which I think is the right response.) Or if you push the responsibility down to the busdma layer where all that info resides, all drivers which use DMA and adhere to the busdma rules just work, without needing to know anything about the buffers that were handed to them and without needing to know anything about the platform requirements. -- Ian From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 15:34:35 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 740EE1065676 for ; Mon, 27 Aug 2012 15:34:35 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 264368FC21 for ; Mon, 27 Aug 2012 15:34:34 +0000 (UTC) Received: by obbun3 with SMTP id un3so11030977obb.13 for ; Mon, 27 Aug 2012 08:34:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:x-priority :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer:x-gm-message-state; bh=sv9ysqQt0d8GIYk9LOiQWK2BMH1BDWf3NslmGnAmKbY=; b=nevx+lSO0w48fvVUWEnA4hlLgT1yqG1XA/ScYgL5Num4bS6Ap/laHY/dsMryj4VzKe 3mMIPbnSVYHQOsVnN0xHP012g3W1QvG+dQFlRUG8PZw4gFZXnRq+rWoADw3ebjxvznrG LYdUfJQ0oF814p6u0GvwjI9mhu8M2HG8K3WNtVF6bLWxixFwh9kD61KwLQMz1C7HNRDn TN78TeWoq81BnBdBWkx2uc7ekHC/DzL05rZkt4mluCPgpoOci+B6yGwUcpWqbeG32cyv 8nW5cW2+fB2N4TzXlLhQ3t4z0NCuHyzYTWXxMVxuMbWNvTNIr4m5qHEI+iOY1gtSA26Z 0ksQ== Received: by 10.182.190.69 with SMTP id go5mr10597608obc.43.1346081674507; Mon, 27 Aug 2012 08:34:34 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id cp8sm17348774obc.23.2012.08.27.08.34.33 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 08:34:33 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh X-Priority: 3 (Normal) In-Reply-To: Date: Mon, 27 Aug 2012 09:34:30 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <45C204C3-D4CE-4B00-886A-A88A0FE7CAD7@bsdimp.com> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> To: Tim Kientzle X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQlErWKHxlDZJYsLEWJnLHGetUu3/i8fiKb9oJqWrH8BfYlJqWdTxXpzYJ9y50s/bUF6x4Wt Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:34:35 -0000 On Aug 27, 2012, at 9:12 AM, Tim Kientzle wrote: >=20 > On Aug 27, 2012, at 6:38 AM, Warner Losh wrote: >=20 >>=20 >> On Aug 27, 2012, at 12:06 AM, Hans Petter Selasky wrote: >>=20 >>> Hi, >>> Correct. >>>=20 >>>> We also need some rules about working with buffers obtained from >>>> bus_dmamem_alloc() and external buffers passed to = bus_dmamap_load(). I >>>> think the rule should be that a buffer obtained from = bus_dmamem_alloc(), >>>> or more formally any region of memory mapped by a = bus_dmamap_load(), is >>>> a single logical object which can only be accessed by one entity at = a >>>> time. That means that there cannot be two concurrent DMA = operations >>>> happening in different regions of the same buffer, nor can DMA and = CPU >>>> access be happening concurrently even if in different parts of the >>>> buffer. =20 >>>=20 >>> Is this something which we can fix using a simple = __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to = be DMA loaded. >>=20 >> No. I don't think so. the reason is that you can't define = USB_DMA_ALGIN to be a constant on MIPS, at least, or I think ARM because = that's determined at run time. >=20 > But don't mbuf structures do pretty much what Hans is suggesting? They kinda do, and kinda don't. > Why is mbuf okay? mbuf is OK because it is never changed while the DMA is pending to the = buffers. That is, m_hdr and m_ext are invariant while the device owns = the memory. In addition, mbuf allocations are so large that no two mbufs = share the same cache line (although it looks like 256 might be too small = to avoid that on some MIPS processors). usb buffers do not obey these = same rules, otherwise none of the stuff we're talking about would = matter. Warner= From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 15:40:24 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA0561065670 for ; Mon, 27 Aug 2012 15:40:24 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1C21F8FC27 for ; Mon, 27 Aug 2012 15:40:23 +0000 (UTC) Received: by obbun3 with SMTP id un3so11051467obb.13 for ; Mon, 27 Aug 2012 08:40:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=t1YrUXxwc6T+C4ku+1nQRySvTX9kp0FRv2Gmd0Bg2co=; b=JOMx0pJ8Xo+albguYR61eWXcWOpyIYgDSd9uYuM6NVJw6bnCIvfFmfJC1bU4c+xfcO dAiU73JJpU31+skWx5uGkjhjcpSnEyveTgMf+ajCxywafG9HFGT39dJlwBUPvSEM9Z3/ WT70aVk4PS8u0dtfzoDq2aoez9ZNtoPDl/ufB9aPuBVjAC+Wr5ZP7m7OfLr+OysY7cNb nfYLE0hjzpb9Ysu3/fdYWcgh7rL0rGSuTUacNkItZvFncGpy+EaAHduG5F8OtUw46mvh sG8SthwZzSqKV8xP2N1aI6irlg5lqq0Jys4kyW1j1l7XaaXJAowyYmMO8LaYeINz0OxL 72ZA== Received: by 10.60.3.35 with SMTP id 3mr10500585oez.139.1346082023466; Mon, 27 Aug 2012 08:40:23 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id hz6sm17394331obb.1.2012.08.27.08.40.21 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 08:40:21 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Mon, 27 Aug 2012 09:40:19 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <9642068B-3C66-42BD-8515-14F734B3FF89@bsdimp.com> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> To: Adrian Chadd X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQlVz+m3V1a8O/eT81qCUlds9sQMIbw2BOHy7Pt4FrGIcnjn/1AH9Ee2A25oOu8c6WO+wHWZ Cc: Tim Kientzle , Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:40:24 -0000 On Aug 27, 2012, at 9:20 AM, Adrian Chadd wrote: > On 27 August 2012 08:12, Tim Kientzle wrote: >> But don't mbuf structures do pretty much what Hans is suggesting? >>=20 >> Why is mbuf okay? >=20 > Which part of mbufs? >=20 > I think the difference here is that there's no concurrent access. Ie, > although you may have an mbuf header + data inside the same cache > line, you wouldn't go fondling the mbuf once it's handed to the > hardware. >=20 > But I was under the impression that mbuf + mbuf buffers are already > correctly aligned. Not quite. mbufs are large enough and aligned enough so that no two = mbufs are in the same cache line. The data inside the mbuf on 32-bit = platforms is 24 bytes from the start of the mbuf, so the mbuf data and = mbuf header do share the same cache line, but as you say, nobody touches = any part of the mbuf will the hardware has it. Recall that it isn't = alignment of where the buffer starts that matters here, but rather = alignment to ensure that there's no sharing of cache lines between = structures. > This all honestly looks like a very i386-centric interpretation of the > busdma "intention", which I think illustrates a need to better > document what assumptions busdma actually makes. Yes. x86 lets you play fast and loose with many of these things. We've = tried to accommodate this for a long time, but that is lame. > That does remind me, I think the ath(4) driver does the same (since it > allocates its own descriptor block and then treats it as an array of > descriptors for the hardware to access) - I should ensure that > sizeof(ath_desc) is aligned on the relevant architecture. It gets > slightly scary - AR93xx TX descriptors are "L1 cache =3D=3D 128 byte > aligned" which is an enormous waste of memory compared to a 16 or 32 > byte aligned platform. Alas.. The problem is with cache line sharing, not necessarily with alignment. = If you are only ever using one of them at a time, or if you have perfect = hygiene, you can cope with this situation without undue waste. The = perfect hygiene might be hard sometimes. Warner From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 15:57:54 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 67F9F1065673 for ; Mon, 27 Aug 2012 15:57:54 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0C5738FC14 for ; Mon, 27 Aug 2012 15:57:53 +0000 (UTC) Received: by obbun3 with SMTP id un3so11111201obb.13 for ; Mon, 27 Aug 2012 08:57:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=+6b5VLtRymOQqJ25JKslOXKH/1wAxmLCqLr7LE+kYFs=; b=Pxzv2SX3IBwvPqtEHWj5G/kPuCtQMvc3OoLE36JGWitV9nBYGHJfR73iUeFvhfa/hP R1hb/UCLyeP+t0j7ZRfwrFGGNsjQzcN7AiKH8YDoEwRGDv6OyVipeyvDDdpLKv9M2jwk ssppQOsdgLXZeT/9K0R6Odh/fFfkDP5nSDsl7+EPoYRWzZlNQzuzbpVVPB+Kl6qMYzjA NmZ632j0Xq72LN6NgPouRoTPs/T7rJAmFNC84sy8StNBvU7mrwDBGhAfM7cri1GPZ2XZ 7p3wvmD9uAzdAXRvDaajBNBVHU7ikBeG/NdMT8yq5XWsXUTWarxiiAurl0cCfr2VGKFe MRwQ== Received: by 10.60.19.34 with SMTP id b2mr10134983oee.41.1346083073414; Mon, 27 Aug 2012 08:57:53 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id ov5sm9250003obc.2.2012.08.27.08.57.51 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 08:57:52 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346081557.1140.181.camel@revolution.hippie.lan> Date: Mon, 27 Aug 2012 09:57:48 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> <1346081557.1140.181.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQn7xsdR/TiJaMnB8KzFil08ZVW1k19ZkMYkdepb/k438KhTO+NXhGG5mLgi7j3C4dhxTTog Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:57:54 -0000 On Aug 27, 2012, at 9:32 AM, Ian Lepore wrote: > On Sun, 2012-08-26 at 17:13 -0600, Warner Losh wrote: >> On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: >>> In this regard, it's the busdma implementation that's broken, = because it >>> should bounce those IOs through a DMA-safe buffer. There's = absolutely >>> no rule that I've ever heard of in FreeBSD that says IO can only = take >>> place using memory allocated from busdma. >>=20 >> That's partially true. Since BUSDMA grew up in the storage area, you = must allocate the memory from busdma, or it must be page aligned has = been the de-facto rule here. =20 >=20 > Where does anything say that you must allocate the memory from busdma = if > it's not mbuf/page-aligned? I've never seen it in any docs. I > certainly find contrary evidence in existing driver code (and I'm not > talking about USB here). Where does it say that you are allowed to not use the routines? >> The mbuf and uio variants of load were invented to cope with common = cases of mbufs and user I/O to properly flag things. >>=20 >=20 > What does that mean, "to properly flag things"? >=20 > I think with uio we come to the crux of the issue. Userland buffers = are > not allocated with knowledge of the busdma constraints. That leads to > some choices: >=20 > * We don't support IO with a userland buffer at all. You may have to bounce here. > * We support userland IO if the userland buffers are accidentally > aligned properly for the platform, otherwise the call fails. You may have to bounce here. > * We support userland buffers by making any driver which wants to > do so responsible for always copying the data to aligned = buffers > (each individual driver handles bounces). This is what most drivers that want to do zero copying do. > * We support userland buffers by having the busdma layer handle > the bouncing when required. I thought that's what the uio variants of the map loading routines were = supposed to do. > The first two seem untenable to me. The third one comes down to = "every > driver must always bounce all userland data" because the driver = doesn't > actually have access to the info to decide whether a userland buffer = is > aligned properly or not for a given platform. >=20 > That leaves the option where the busdma layer handles bouncing for > unaligned userland buffers. If that's possible, then the busdma layer > could automatically bounce any unaligned request, whether it came from > userland or a driver reading a status response into an unaligned local > buffer in kernel memory. Right, the uio load option is supposed to do this. >> How does busdma know that it is using memory that's not from its = allocator? >=20 > The busdma code allocates the map along with the buffer, and can = record > information in the map that it can use during mapping and sync > operations to know whether it allocated the buffer. >=20 >>> The rule is only that the >>> proper sequence of busdma operation must be called, and beyond that = it's >>> up to the busdma implementation to make it work. =20 >>=20 >> No. Bouncing is needed due to poor alignment of the underlying = device. Not due to cache effects. >=20 > Says who? This statement seems to me to be more about opinion and = dogma > than about a documented API or technical concerns about how to = implement > it. IMO, bouncing is needed when that's the only way to make a > particular busdma sequence work correctly given the parameters of the > hardware and the transfer. Says the people that wrote and use busdma? There's no requirement for = cache effects for the transfer, so there's no constraints for the = transfer. The requirement is about having a buffer that's suitable for = DMA. This means either the buffer (not the start of the transfer) be = aligned to a cache line and be an integral number of cache lines in = size, or it means that you have 100% perfect hygiene when it comes to = ensuring the CPU touches the buffer or the device touches it and ever = have cross threading. I'd agree that better docs here would help. > To put it another way, the busdma subsystem is responsible for helping > make DMA transfers work on a platform without every driver needing to > contain platform-specific code, and bouncing is one of the techniques > available to it. Buffers allocated through the busdma system do this. >> There's a limited number of things that we support with busdma. = Arbitrary data from malloc that might be shared with the CPU isn't on = that list. >>=20 >=20 > Where are those limitations documented? =20 Where is it documented that memory returned from malloc may be used for = DMA? I agree that docs could be better here. > This isn't some small oversight in the documention, if the rule is = "IO, > any IO at all, can only be performed using buffers allocated from > busdma" that's a pretty major thing that ought to appear in multiple > manpages relating to driver development. >=20 > If you don't think "any IO at all" is the right characterization, read > all the way to the end before responding. Yes. I'd state it another way. "Buffers returned from the busdma = allocation routines is always safe." Everything else may or may not be = safe. mbufs, for example, happen to be safe because there is a rigid = protocol for sharing between the driver and the device. uio buffers can = be made safe. busdma has routines to ensure that these types of data = are handled properly. >>> Our biggest problem, I think, is that we don't have a sufficient >>> definition of "the proper sequence of busdma operations." >>=20 >> I disagree. The sequence has been known for a long time. >>=20 >>> I don't think it will be very hard to make the arm and mips busdma >>> implementations work correctly. It won't even be too hard to make = them >>> fairly efficient at bouncing small IOs (my thinking is that we can = make >>> small bounces no more expensive than the current partial cacheline = flush >>> implementation which copies the data multiple times). Bouncing = large IO >>> will never be efficient, but the inefficiency will be a powerful >>> motivator to update drivers that do large IO to work better, such as >>> using buffers allocated from busdma. >>=20 >> I don't think the cache line problem can be solved with bounce = buffers. Trying to accommodate broken drivers is what lead us to this = spot. We need to fix the broken drivers. If that's impossible, then = the best we can do is have the driver set a 'always bounce' flag in the = tag it creates and use that to always bounce for operations through that = tag. >=20 > So you'd be okay with a driver setting a flag that says "always = bounce" > but not okay with the busdma layer bouncing only when it's actually > necessary? I'm confused -- do you think the busdma layer will be = unable > to detect when it's necessary unless directed from the outside? Actually, it is good you are confused. I was wrong when I said I'd be = happy with a flag that says always bounce. The reason is that the driver = can do this all the time for those cases like the at91 mci driver does. > Let me pose a question back to you... if it is up to a driver to > allocate DMA buffers using the busdma allocator, how is any given = driver > to know whether DMA is going to be involved in the transfer? =20 because it does the dma? > Don't think USB or ATA here... think iicbus/foo_temperature_sensor, or > the mmc/sd driver. Those drivers know nothing about the hardware = bridge > layers beneath them and whether DMA is involved or not. They just = know > "I need to read a 2-byte value from a register on this IIC chip" or "I > need to retrieve the 32-bit CSD data from the SD card" and they > accomplish that by calling a read method of some lower-level bridge > driver. In each of those cases we have both PIO and DMA = implementations > of the lower-level drivers that move bits over a wire. The bridge here would know and have to cope. However, the real issue in = this case wouldn't be the transfer alignment, but what traffic might be = happening to other parts of the cache line. For small things like this, typically data copies are involved. There's = a few places that break these rules (I think mci might be breaking the = rules for small transfers, for example). > If the concensus is that such drivers should always allocate all IO > buffers using busdma, even though they have no idea whether DMA is = going > to be involved, then we certainly have a lot of work ahead of us in > terms of "fixing broken drivers." That also implies that > bus_dmamem_alloc() is misnamed and implemented in the wrong place, > because it's not really about dma at all. You can't get around the hardware requirement that any I/O going to the = device cannot share cache lines with other data on the system. > If you push the responsibility down to the layer where it is known > whether DMA is going to be involved, then we're back to the need for > many individual drivers to bounce every request, because they don't = have > access to the info needed to know whether bouncing is required or not > for a given buffer they were handed from higher layers. (Remember = when > I proposed exporting that info to drivers so they could decide whether = a > buffer is dma-aligned or not, the response was "No, keep it all hidden > in the busdma layer," which I think is the right response.) Right now we support only a few things doing DMA on the system. We = support pages, mbufs and to a limited extent uio buffers (but to be = honest, we may have exposure on smaller transfers here). > Or if you push the responsibility down to the busdma layer where all > that info resides, all drivers which use DMA and adhere to the busdma > rules just work, without needing to know anything about the buffers = that > were handed to them and without needing to know anything about the > platform requirements. I'm not sure I follow this last bit... Warner From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 16:09:12 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1AA23106566C; Mon, 27 Aug 2012 16:09:12 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id D96788FC15; Mon, 27 Aug 2012 16:08:59 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7RG8xAG089488; Mon, 27 Aug 2012 10:08:59 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7RG8aPI031283; Mon, 27 Aug 2012 10:08:36 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> Content-Type: text/plain; charset="us-ascii" Date: Mon, 27 Aug 2012 10:08:36 -0600 Message-ID: <1346083716.1140.212.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arch@freebsd.org, freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 16:09:12 -0000 On Sun, 2012-08-26 at 17:03 -0600, Warner Losh wrote: > On Aug 26, 2012, at 11:42 AM, Ian Lepore wrote: > > > > The busdma manpage currently has some vague words about the usage and > > sequencing of sync ops, such as "If read and write operations are not > > preceded and followed by the appropriate synchronization operations, > > behavior is undefined." I think we should more explicitly spell out > > what the appropriate sequences are. In particular: > > > > * The PRE and POST operations must occur in pairs; a PREREAD must > > be followed eventually by a POSTREAD and a PREWRITE must be > > followed by a POSTWRITE. > > PREREAD means "I am about to tell the device to put data here, have whaterver things might be pending in the CPU complex to get out of the way." usually this means 'invalidate the cache for that range', but not always. POSTREAD means 'The device's DMA is done, I'd like to start accessing it now.' If the memory will be thrown away without being looked at, then does the driver necessarily need to issue the POSTREAD? I think so, but I don't know if that's a new requirement. > One of the things that scares me most is the idea that driver writers will glance at an existing implementation and think "Oh I have no need to ever call POSTWRITE because it's implemented as a no-op and I can save the call overhead." In fact we have drivers coded like that now. We've got an API here to support arbitrary hardware, some of which may not have been designed yet. I think it's really unsafe to say that a driver can decide that it's safe to elide some calls in some situations just because that's safe with the busdma implementations that exist today. > > We also need some rules about working with buffers obtained from > > bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). I > > think the rule should be that a buffer obtained from bus_dmamem_alloc(), > > or more formally any region of memory mapped by a bus_dmamap_load(), is > > a single logical object which can only be accessed by one entity at a > > time. That means that there cannot be two concurrent DMA operations > > happening in different regions of the same buffer, nor can DMA and CPU > > access be happening concurrently even if in different parts of the > > buffer. > > There's something subtle that I'm missing. Why would two DMA operations be disallowed? The rest makes good sense. > If two DMAs are going on concurrently in the same buffer, one is going to finish before the other, leading to a POSTxxxx sync op happening for one DMA operation while the other is still in progress. The unit of granularity for sync operations is the mapped region, so now you're syncing access to a region which still has active DMA happening within it. While I think it's really an API definition issue, think about it in terms of a potential implementation... What if the CPU had to access the memory as part of the sync for the first DMA that completes, while the second is still running? Now you've got pretty much exactly the same situation as when a driver subdivides a buffer without knowing about the cache alignment; you end up with the CPU and DMA touching data in the same cachline and no sequence of flush/invalidate can be g'teed to preserve all data correctly. > > I've always thought that allocating a dma buffer feels like a big > > hassle. You sometimes have to create a tag for the sole purpose of > > setting the maxsize to get the buffer size you need when you call > > bus_dmamem_alloc(). If bus_dmamem_alloc() took a size parm you could > > just use your parent tag, or a generic tag appropriate to all the IO > > you're doing for a given device. If you need a variety of buffers for > > small control and command and status transfers of different sizes, you > > end up having to manage up to a dozen tags and maps and buffers. It's > > all very clunky and inconvenient. It's just the sort of thing that > > makes you want to allocate a big buffer and subdivide it. Surely we > > could do something to make it easier? > > You'd wind up creating a quick tag on the fly for the bus_dmamap_alloc if you wanted to do this. Cleanup then becomes unclear. > My point is that the only piece of information in the tag that's specific to the allocation is the maxsize. If the allocation size were passed to bus_dmamem_alloc() then you wouldn't need a tag specific to that buffer, a generic tag for the device would work and you could allocate a dozen different-sized buffers all using that one tag, and the allocator would just have to sanity-check the allocation size against the tag's maxsize. -- Ian From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 16:53:31 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9B291106566B; Mon, 27 Aug 2012 16:53:31 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2CDD38FC15; Mon, 27 Aug 2012 16:53:31 +0000 (UTC) Received: by obbun3 with SMTP id un3so11293602obb.13 for ; Mon, 27 Aug 2012 09:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=gJHpHTLGok1mdDMngy2k8DW11dnWPLHSjDIlGfmoy5o=; b=ezk1B4AI/J8/l3RTz+S6U95QsnA6aZcxymJbdAgyM5BlQwkdg/RMBmt4Dgjwh2l70q z/1AXux2Ac7RLCGrdDiGhpsbFyAXLMgynAst11zcozzCEGe2BmW/T+8NgIQ1VIjc8x76 cB8Cqif38GOFiNDnLq5VZIpqzgLUJf7Q9RhpESC5G9idEgjqMRxOUt7ZT7n7bBJUuwna dO6mfIgIAo6YJzw2pa7mLjUDpbH+X4sCC4+wqByxReoiWrIVX+bmntHNLgTRI72lwwnr Uf/WlLRWeWQj6Ri5/BINCkEi2RKCcBJGA0lnFjgFj1ngZxTbdwJXxuFy890xokaJcrBf 3CFQ== MIME-Version: 1.0 Received: by 10.182.0.13 with SMTP id 13mr10283841oba.55.1346086410479; Mon, 27 Aug 2012 09:53:30 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.76.8.98 with HTTP; Mon, 27 Aug 2012 09:53:29 -0700 (PDT) In-Reply-To: <1346083716.1140.212.camel@revolution.hippie.lan> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <1346083716.1140.212.camel@revolution.hippie.lan> Date: Mon, 27 Aug 2012 09:53:29 -0700 X-Google-Sender-Auth: _mn46Ck_2vyos8KJFH_aDJV2o8M Message-ID: From: Adrian Chadd To: Ian Lepore Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 16:53:31 -0000 On 27 August 2012 09:08, Ian Lepore wrote: > If two DMAs are going on concurrently in the same buffer, one is going > to finish before the other, leading to a POSTxxxx sync op happening for > one DMA operation while the other is still in progress. The unit of > granularity for sync operations is the mapped region, so now you're > syncing access to a region which still has active DMA happening within > it. Right. But the enforced idea is "DMA up to this point should be flushed to memory." > While I think it's really an API definition issue, think about it in > terms of a potential implementation... What if the CPU had to access the > memory as part of the sync for the first DMA that completes, while the > second is still running? Now you've got pretty much exactly the same > situation as when a driver subdivides a buffer without knowing about the > cache alignment; you end up with the CPU and DMA touching data in the > same cachline and no sequence of flush/invalidate can be g'teed to > preserve all data correctly. Right. So you realise at that point you can't win and you stick each of those pieces in a different cache line. Adrian From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 17:40:11 2012 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C798F106574D for ; Mon, 27 Aug 2012 17:40:11 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id 9C2B98FC23 for ; Mon, 27 Aug 2012 17:40:11 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id q7RHe48d026586 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 27 Aug 2012 10:40:04 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id q7RHe44A026585 for freebsd-arm@FreeBSD.org; Mon, 27 Aug 2012 10:40:04 -0700 (PDT) (envelope-from jmg) Date: Mon, 27 Aug 2012 10:40:04 -0700 From: John-Mark Gurney To: freebsd-arm@FreeBSD.org Message-ID: <20120827174004.GD58312@funkthat.com> Mail-Followup-To: freebsd-arm@FreeBSD.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="G4iJoqBmSsgzjUCe" Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Mon, 27 Aug 2012 10:40:04 -0700 (PDT) Cc: Subject: Avila GW2348-4 and VLANs not working X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 17:40:11 -0000 --G4iJoqBmSsgzjUCe Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Over the weekend, I tried to update my board to support VLANs, and when I did, things didn't go so well. I have 9.0-PRERELEASE installed on it, but the npe driver hasn't changed much since then, so I'm not sure updating will fix things... When I tried to run VLANs on the board, things didn't work. When I tcpdump'd the base interface, I got a ton of truncated-ip messages... Some of the packets that were suppose to be on a vlan didn't appear to have a vlan tag on the npe0 interface... I previously used the same switch w/ an older x86 box and vlans worked fine, though on a much older FreeBSD release... I've attached the dmesg.boot... iirc, the modifications to the source were some things to make pf usable, but that's about it... Any ideas what needs to be done to make vlans work? P.S. Since I only have one board, and I'm using this as a firewall experimenting is a little difficult. Anyone know of a another good ARM board that I could use (and isn't too expensive)? Thanks. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." --G4iJoqBmSsgzjUCe Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="dmesg.boot" KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-PRERELEASE #3 r228252M: Wed Nov 2 07:00:48 PDT 2011 jmg@pcbsd-779:/usr/obj/arm.armeb/usr/src/sys/gate2 arm CPU: IXP425 533MHz rev 1 (ARMv5TE) (XScale core) DC enabled IC enabled WB enabled LABT branch prediction enabled 32KB/32B 32-way Instruction cache 32KB/32B 32-way write-back-locking Data cache real memory = 67108864 (64 MB) avail memory = 56352768 (53 MB) ixp0: on motherboard ixp0: 37fff pcib0: on ixp0 pci0: on pcib0 ath0: irq 27 at device 2.0 on pci0 ath0: AR9280 mac 128.2 RF5133 phy 13.0 ixpclk0: on ixp0 ixpiic0: on ixp0 iicbb0: on ixpiic0 iicbus0: on iicbb0 master-only iic0: on iicbus0 ad74180: at addr 0x50 on iicbus0 ds1672_rtc0: at addr 0xd0 on iicbus0 ixpwdog0: on ixp0 uart0: on ixp0 uart0: console (115200,n,8,1) uart1: on ixp0 ixpqmgr0: on ixp0 npe0: on ixp0 npe0: MAC at 0xc8009000 npe0: MII at 0xc8009000 npe0: load fw image IXP425.NPE-B Func 0x2 Rev 2.1 miibus0: on npe0 ukphy0: PHY 0 on miibus0 ukphy0: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto npe0: Ethernet address: 00:d0:12:xx:xx:xx npe1: on ixp0 npe1: MAC at 0xc800a000 npe1: MII at 0xc8009000 npe1: load fw image IXP425.NPE-C Func 0x5 Rev 2.1 miibus1: on npe1 ukphy1: PHY 1 on miibus1 ukphy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto npe1: Ethernet address: 00:d0:12:xx:xx:xx cfi0: on ixp0 cfid0 on cfi0 ata_avila0: on ixp0 ata0: on ata_avila0 led_avila0: on ixp0 gpio_avila0: on ixp0 Timecounter "IXP4XX Timer" frequency 66666600 Hz quality 1000 Timecounters tick every 10.000 msec ada0 at ata0 bus 0 scbus0 target 0 lun 0 ada0: CFA-7 device ada0: 16.700MB/s transfers (PIO4, PIO 512bytes) ada0: 7647MB (15662304 512 byte sectors: 16H 63S/T 15538C) ada0: Previously was known as ad0 hwpmc: XSCALE/4/32/0x1ff Trying to mount root from ufs:ada0s1a []... wlan0: Ethernet address: xx:xx:xx:xx:xx:xx npe0: ixpnpe_intr: status 0x60000 npe1: ixpnpe_intr: status 0x60000 --G4iJoqBmSsgzjUCe-- From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 17:55:45 2012 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3E70C106566C for ; Mon, 27 Aug 2012 17:55:45 +0000 (UTC) (envelope-from gnn@neville-neil.com) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) by mx1.freebsd.org (Postfix) with ESMTP id 0E3D88FC1F for ; Mon, 27 Aug 2012 17:55:44 +0000 (UTC) Received: from [87.224.84.90] (port=53125 helo=[10.7.8.93]) by vps.hungerhost.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.77) (envelope-from ) id 1T63XL-00026l-Qh; Mon, 27 Aug 2012 13:55:44 -0400 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1486\)) From: George Neville-Neil In-Reply-To: <20120827174004.GD58312@funkthat.com> Date: Mon, 27 Aug 2012 18:55:40 +0100 Content-Transfer-Encoding: 7bit Message-Id: <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> References: <20120827174004.GD58312@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1486) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com Cc: freebsd-arm@FreeBSD.org Subject: Re: Avila GW2348-4 and VLANs not working X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 17:55:45 -0000 On Aug 27, 2012, at 18:40 , John-Mark Gurney wrote: > Over the weekend, I tried to update my board to support VLANs, and when > I did, things didn't go so well. I have 9.0-PRERELEASE installed on it, > but the npe driver hasn't changed much since then, so I'm not sure updating > will fix things... > > When I tried to run VLANs on the board, things didn't work. When I > tcpdump'd the base interface, I got a ton of truncated-ip messages... > Some of the packets that were suppose to be on a vlan didn't appear to > have a vlan tag on the npe0 interface... I previously used the same > switch w/ an older x86 box and vlans worked fine, though on a much > older FreeBSD release... > > I've attached the dmesg.boot... iirc, the modifications to the source > were some things to make pf usable, but that's about it... > > Any ideas what needs to be done to make vlans work? > > P.S. Since I only have one board, and I'm using this as a firewall > experimenting is a little difficult. Anyone know of a another good > ARM board that I could use (and isn't too expensive)? I'm a fan of the BeagleBone, which works with FreeBSD, and is about $89. http://beagleboard.org/buy Rasberry Pis are cheap but impossible to come by. Best, George From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 18:17:19 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 34B9C1065675 for ; Mon, 27 Aug 2012 18:17:19 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id E584E8FC0C for ; Mon, 27 Aug 2012 18:17:18 +0000 (UTC) Received: by obbun3 with SMTP id un3so11558279obb.13 for ; Mon, 27 Aug 2012 11:17:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=2eVep/OeklNDCcy8ZowDeVCkXf83NugqZpsY+viONsI=; b=pYZfUm8CTG+q6ZW1b7YaYDs8U6EkQnLYmhWcvYu5RFEhaBnMyOlwUbWS5uqajQ0f37 MlNCikjyixxwH1rmFGB3pVQNja1uDF3TqqD8SSylOmkg9QMJ1WRS7e+15MwgZfDc/TdS XMBsQ3yDBwCaU4wzpUgEJySfAVn1RjXn861E8Gy2Qe4nU1mUqO5HPVU9jIKxsvrHy19W J6vtOgSv4ajnDAGJZSh/Wcm/TgDgODPomHDJ2bhhJMId3UMQWggFjk3Gze/hfNZc7KD1 ebaYlsYQM5ZjQOjcbtuzI6ICw82t0MayE9xocnvWhy4/YAZ3fLE4p86l0nWDDkr4B0SA CVKg== MIME-Version: 1.0 Received: by 10.182.188.41 with SMTP id fx9mr10325330obc.92.1346091438310; Mon, 27 Aug 2012 11:17:18 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.76.8.98 with HTTP; Mon, 27 Aug 2012 11:17:18 -0700 (PDT) In-Reply-To: <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> References: <20120827174004.GD58312@funkthat.com> <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> Date: Mon, 27 Aug 2012 11:17:18 -0700 X-Google-Sender-Auth: UA0kX9UJSLxllLSUEMcqOqEp--w Message-ID: From: Adrian Chadd To: George Neville-Neil Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arm@freebsd.org Subject: Re: Avila GW2348-4 and VLANs not working X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 18:17:19 -0000 Would you please try and narrow down which release it broke in? That would be really, really helpful in narrowing it down. I have a big stack of avila/cambria boards here but I won't be able to test until mid-september. adrian From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 18:54:10 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49B9F1065714; Mon, 27 Aug 2012 18:54:10 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id D55CC8FC0A; Mon, 27 Aug 2012 18:54:09 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q7RIs0nK023175; Mon, 27 Aug 2012 21:54:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q7RIrlGG059144; Mon, 27 Aug 2012 21:53:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q7RIrked059143; Mon, 27 Aug 2012 21:53:46 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 27 Aug 2012 21:53:46 +0300 From: Konstantin Belousov To: Warner Losh Message-ID: <20120827185346.GE33100@deviant.kiev.zoral.com.ua> References: <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BwgxZU5iS1dWSTZh" Content-Disposition: inline In-Reply-To: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 18:54:10 -0000 --BwgxZU5iS1dWSTZh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Aug 26, 2012 at 05:13:31PM -0600, Warner Losh wrote: >=20 > On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: > > In this regard, it's the busdma implementation that's broken, because it > > should bounce those IOs through a DMA-safe buffer. There's absolutely > > no rule that I've ever heard of in FreeBSD that says IO can only take > > place using memory allocated from busdma. >=20 > That's partially true. Since BUSDMA grew up in the storage area, you > must allocate the memory from busdma, or it must be page aligned has > been the de-facto rule here. The mbuf and uio variants of load were > invented to cope with common cases of mbufs and user I/O to properly > flag things. I once looked at x86 bus_dmamap_load_uio(), and I was unable to understand how to use it with usermode uio. I think this is a good moment to ask. Most existing users use UIO_SYSSPACE, but several crypto drivers might allow the UIO_USERSPACE for them. For UIO_USERSPACE, if the page is not resident, the pmap_extract() call from _bus_dmamap_load_buffer() returns 0. So the i/o happens to the page located at 0, which contains real mode IVT and other BIOS sensitive tables. Worse, if the page is resident, but it is mapped at the region which requires COW on write, then DMA will be performed to the wrong page which is typically shared with other innocent users. to the COW area which was not yet copied, Am I missing some trick there ? --BwgxZU5iS1dWSTZh Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAlA7wjoACgkQC3+MBN1Mb4hJsACg9rY5d4jEjVBz1gmM0pcHvbwY 1N0AoKmBOQ6HFrErCDyO1ME2rbSRZLt/ =iNrn -----END PGP SIGNATURE----- --BwgxZU5iS1dWSTZh-- From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 18:57:49 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35DBE106564A for ; Mon, 27 Aug 2012 18:57:49 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id E1F288FC15 for ; Mon, 27 Aug 2012 18:57:48 +0000 (UTC) Received: by obbun3 with SMTP id un3so11673301obb.13 for ; Mon, 27 Aug 2012 11:57:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=HJ5wwPEakYvRTSHrqheG85EsV/Ab925LLgDcZSGxAoU=; b=Yzg1F99E1jYQjzQ5ZB2cr4+DAgwwpevqXbZrZCZs8vBjFkG1lPd5254dBGImqLd7Re kOTRw1+6kCkbhZhgdsO5unjPEdz+7dfOj8yqS05HvrkLEYzjXDXRyFtMYfuUFjCMm63r AVbuqXkbxLbkJStbjJZiEeefYGQh425cPNn5bUWBimVqFy5tDghQO6Sx4ZozjxGgsXe5 OhDVpgNbNEAi99qXuzaRqUaGTj9CSu8Lp44c9DDemcPqOjxgEiaBnmVQjGDhffr61FRN ZsaJH8x0JD4MBpOb62Cb0Rlbvs9s5mWk4wJBWL1NZRcljxWXdvM8YN9t3xRmSvss6qDu m8+Q== Received: by 10.60.10.134 with SMTP id i6mr10501701oeb.137.1346093868068; Mon, 27 Aug 2012 11:57:48 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id g8sm17780229obz.16.2012.08.27.11.57.46 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 11:57:47 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <20120827174004.GD58312@funkthat.com> Date: Mon, 27 Aug 2012 12:57:45 -0600 Content-Transfer-Encoding: 7bit Message-Id: <899D2257-66D2-4120-B80B-4BC6DA959D87@bsdimp.com> References: <20120827174004.GD58312@funkthat.com> To: John-Mark Gurney X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQmlMz9HbU3wTZj5N2tIiHGlWg/LLT+XpBM3eAkcjkekQ2lhAbDB0JB/g5QsVSyX/ZYXQSBI Cc: freebsd-arm@FreeBSD.org Subject: Re: Avila GW2348-4 and VLANs not working X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 18:57:49 -0000 Crazy question: Does NFS root boot work? Warner On Aug 27, 2012, at 11:40 AM, John-Mark Gurney wrote: > Over the weekend, I tried to update my board to support VLANs, and when > I did, things didn't go so well. I have 9.0-PRERELEASE installed on it, > but the npe driver hasn't changed much since then, so I'm not sure updating > will fix things... > > When I tried to run VLANs on the board, things didn't work. When I > tcpdump'd the base interface, I got a ton of truncated-ip messages... > Some of the packets that were suppose to be on a vlan didn't appear to > have a vlan tag on the npe0 interface... I previously used the same > switch w/ an older x86 box and vlans worked fine, though on a much > older FreeBSD release... > > I've attached the dmesg.boot... iirc, the modifications to the source > were some things to make pf usable, but that's about it... > > Any ideas what needs to be done to make vlans work? > > P.S. Since I only have one board, and I'm using this as a firewall > experimenting is a little difficult. Anyone know of a another good > ARM board that I could use (and isn't too expensive)? > > Thanks. > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > _______________________________________________ > freebsd-arm@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 18:58:57 2012 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 352BD1065680 for ; Mon, 27 Aug 2012 18:58:57 +0000 (UTC) (envelope-from torfinn.ingolfsen@getmail.no) Received: from smtp.getmail.no (smtp.getmail.no [84.208.15.66]) by mx1.freebsd.org (Postfix) with ESMTP id DBC758FC1F for ; Mon, 27 Aug 2012 18:58:56 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; charset=US-ASCII Received: from get-mta-scan04.get.basefarm.net ([10.5.16.4]) by get-mta-out02.get.basefarm.net (Sun Java(tm) System Messaging Server 7.0-0.04 64bit (built Jun 20 2008)) with ESMTP id <0M9F006Q6GP6H040@get-mta-out02.get.basefarm.net> for freebsd-arm@FreeBSD.org; Mon, 27 Aug 2012 20:58:18 +0200 (MEST) Received: from get-mta-scan04.get.basefarm.net (localhost.localdomain [127.0.0.1]) by localhost (Email Security Appliance) with SMTP id 0F40A1F02F24_3BC4FEB for ; Mon, 27 Aug 2012 19:05:34 +0000 (GMT) Received: from kg-v2.kg4.no (cm-84.215.134.159.getinternet.no [84.215.134.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by get-mta-scan04.get.basefarm.net (Sophos Email Appliance) with ESMTPSA id 97B051F02F12_3BC4FDF for ; Mon, 27 Aug 2012 19:05:33 +0000 (GMT) Date: Mon, 27 Aug 2012 20:58:17 +0200 From: Torfinn Ingolfsen To: freebsd-arm@FreeBSD.org Message-id: <20120827205817.79c6140811ab6e214944f70e@getmail.no> In-reply-to: <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> References: <20120827174004.GD58312@funkthat.com> <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> X-Mailer: Sylpheed 3.2.0 (GTK+ 2.24.6; amd64-portbld-freebsd8.3) Cc: Subject: Re: Avila GW2348-4 and VLANs not working X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 18:58:57 -0000 On Mon, 27 Aug 2012 18:55:40 +0100 George Neville-Neil wrote: > > > Rasberry Pis are cheap but impossible to come by. "impossible" is now "13 weeks" at RS Components: http://raspberrypi.rsdelivers.com/default.aspx?cl=1 HTH -- Torfinn Ingolfsen From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 19:01:07 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5E3741065700 for ; Mon, 27 Aug 2012 19:01:07 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1584C8FC14 for ; Mon, 27 Aug 2012 19:01:06 +0000 (UTC) Received: by obbun3 with SMTP id un3so11682808obb.13 for ; Mon, 27 Aug 2012 12:01:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=bqsIRokWNS7x9FqJtp5R3qrKWxMuddkhn8J3xEo076U=; b=Fq3OL1dNL0fA0M4EELF95MB/JSglm6NTmiH9kB0QZktN2BWEr2Bp6BUzPZIBnw273L nUkS4ao//ncAkTuMj+xlD/311qs4OY7TXcqb4hHcSiz5dzn8+Dv0Zxi3aGnd6wG5Q3ll l6zZCyVksHe40F8KkjR9S7ZOBCGXUTU9i4ztHefMco4NovT55d+jsaDk46R8EGmFX4BZ hD2qdhTHw1uA6NBND/zOM/R9fxs9v13WJnLzT6sbDPWwmvO2mcM+JWVWNVg9YTROOSCM olhNvtfmHWo7Fiv7RvUfgV5IlLVUvb7Nwek9ZUbYlcu6d1LRInVYcJn8MmNxqSR2WoFs 8AbQ== Received: by 10.60.171.69 with SMTP id as5mr10900676oec.100.1346094066388; Mon, 27 Aug 2012 12:01:06 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id ac8sm17796566obc.11.2012.08.27.12.01.05 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 12:01:05 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> Date: Mon, 27 Aug 2012 13:01:02 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <68F9301B-8B03-47D3-A0AA-9F2490D3481B@bsdimp.com> References: <20120827174004.GD58312@funkthat.com> <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> To: George Neville-Neil X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQlK9nbVbd8fcd+AGJICkoYcK5tYt2VFLXoN1DoYbwlBHY2qBy4ug6m05+KzsSDASFpzdgZX Cc: freebsd-arm@FreeBSD.org Subject: Re: Avila GW2348-4 and VLANs not working X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 19:01:07 -0000 On Aug 27, 2012, at 11:55 AM, George Neville-Neil wrote: >=20 > On Aug 27, 2012, at 18:40 , John-Mark Gurney wrote: >=20 >> Over the weekend, I tried to update my board to support VLANs, and = when >> I did, things didn't go so well. I have 9.0-PRERELEASE installed on = it, >> but the npe driver hasn't changed much since then, so I'm not sure = updating >> will fix things... >>=20 >> When I tried to run VLANs on the board, things didn't work. When I >> tcpdump'd the base interface, I got a ton of truncated-ip messages... >> Some of the packets that were suppose to be on a vlan didn't appear = to >> have a vlan tag on the npe0 interface... I previously used the same >> switch w/ an older x86 box and vlans worked fine, though on a much >> older FreeBSD release... >>=20 >> I've attached the dmesg.boot... iirc, the modifications to the = source >> were some things to make pf usable, but that's about it... >>=20 >> Any ideas what needs to be done to make vlans work? >>=20 >> P.S. Since I only have one board, and I'm using this as a firewall >> experimenting is a little difficult. Anyone know of a another good >> ARM board that I could use (and isn't too expensive)? >=20 > I'm a fan of the BeagleBone, which works with FreeBSD, and is about = $89. >=20 > http://beagleboard.org/buy >=20 > Rasberry Pis are cheap but impossible to come by. Doesn't the BealgleBone have only one Ethernet? Warner From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 19:01:31 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 89DBC106564A; Mon, 27 Aug 2012 19:01:31 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 1F8C98FC08; Mon, 27 Aug 2012 19:01:19 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7RJ1BMt091960; Mon, 27 Aug 2012 13:01:17 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7RJ0mgZ031745; Mon, 27 Aug 2012 13:00:48 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Adrian Chadd In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <1346083716.1140.212.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Mon, 27 Aug 2012 13:00:47 -0600 Message-ID: <1346094047.1140.264.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 19:01:31 -0000 On Mon, 2012-08-27 at 09:53 -0700, Adrian Chadd wrote: > On 27 August 2012 09:08, Ian Lepore wrote: > > > If two DMAs are going on concurrently in the same buffer, one is going > > to finish before the other, leading to a POSTxxxx sync op happening for > > one DMA operation while the other is still in progress. The unit of > > granularity for sync operations is the mapped region, so now you're > > syncing access to a region which still has active DMA happening within > > it. > > Right. But the enforced idea is "DMA up to this point should be > flushed to memory." > > > While I think it's really an API definition issue, think about it in > > terms of a potential implementation... What if the CPU had to access the > > memory as part of the sync for the first DMA that completes, while the > > second is still running? Now you've got pretty much exactly the same > > situation as when a driver subdivides a buffer without knowing about the > > cache alignment; you end up with the CPU and DMA touching data in the > > same cachline and no sequence of flush/invalidate can be g'teed to > > preserve all data correctly. > > Right. So you realise at that point you can't win and you stick each > of those pieces in a different cache line. > Actually, I think that even discussing cache lines in this context is a mistake (yeah, I'm the one who did so above, in trying to relate an abstract API design concept to a real-world hardware example). Drivers are not supposed to know about interactions between DMA transfers and cache lines or other machine-specific constraints; that info is supposed to be encapsulated and hidden within busdma. I think a driver making the assumption that it can do DMA safely on a buffer as long as that buffer is cacheline-granular is just as flawed as assuming that it can do DMA safely on any arbitrarily sized and aligned buffer. So the right way to "stick each of those pieces in a different cache line" is to allocate two different buffers, one per concurrent DMA transfer. Or, really, to use two separate busdma mappings would be the more rigorous way to say it, since the mapping is the operation at which constraints come into play. Thinking of it that way then drives the need to document that if multiple mappings describe the same area of physical memory, then concurrent operations on those maps yield unpredictable results. -- Ian From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 19:03:18 2012 Return-Path: Delivered-To: freebsd-arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E2467106566C for ; Mon, 27 Aug 2012 19:03:18 +0000 (UTC) (envelope-from nick@flirble.org) Received: from plum.flirble.org (plum.flirble.org [195.99.220.128]) by mx1.freebsd.org (Postfix) with ESMTP id A15F28FC1E for ; Mon, 27 Aug 2012 19:03:18 +0000 (UTC) Received: from nick by plum.flirble.org with local (Exim 4.75 (FreeBSD)) (envelope-from ) id 1T648h-000I8S-JK; Mon, 27 Aug 2012 19:34:19 +0100 Date: Mon, 27 Aug 2012 19:34:19 +0100 From: Nicholas Clark To: George Neville-Neil Message-ID: <20120827183419.GG9583@plum.flirble.org> References: <20120827174004.GD58312@funkthat.com> <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> X-Organisation: Tetrachloromethane User-Agent: Mutt/1.5.21 (2010-09-15) Sender: Nicholas Clark Cc: freebsd-arm@FreeBSD.org Subject: Re: Avila GW2348-4 and VLANs not working X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 19:03:19 -0000 On Mon, Aug 27, 2012 at 06:55:40PM +0100, George Neville-Neil wrote: > Rasberry Pis are cheap but impossible to come by. Farnell are claiming a delivery estimate of 3 weeks now: http://export.farnell.com/rp/order/?COM=raspberrypi-group I believe RS still have a lot longer delivery estimates. No, this doesn't quite make sense. Nicholas Clark From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 19:18:55 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ABA0A106566B for ; Mon, 27 Aug 2012 19:18:55 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 091A98FC1F for ; Mon, 27 Aug 2012 19:18:53 +0000 (UTC) Received: by obbun3 with SMTP id un3so11731249obb.13 for ; Mon, 27 Aug 2012 12:18:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=NWAxBZraqJ39VsS1Q2psNADkTl2YA8RO4lXjmSt0qtk=; b=Oo6SneAT0Eh4ptRwaawUkSTodEx3ahTcGtKUS1+TojGfMB7/pqxo7jEAEqyFd/3TBI bteCo991V6yA/QgVh1Na0zqrkAXElRx917pZa9jFnpBwOWUbIbZYm4uz2L+k2QCc40Kv TR4j2fqfFRLB4pNOriqNlGBQGJqVM1XJByDBVizQsBbrhrWSdD3HYChFB4glwEPmh/CP oHwCcu2lb6ebKUHGDSZ19jfUogDSrx2U75Y+GW99wt+hzMTPNUynsjJSjaH5eUggbta4 rJyENSUYvR+0fSNmoYWyOpw+EgKFAiFXY7DEHPClCqdH4yM6EZvqARqci7O+1bWuJuF1 3MUA== Received: by 10.182.182.71 with SMTP id ec7mr8049310obc.22.1346095133419; Mon, 27 Aug 2012 12:18:53 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id r1sm13110813oea.4.2012.08.27.12.18.51 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 12:18:52 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346094047.1140.264.camel@revolution.hippie.lan> Date: Mon, 27 Aug 2012 13:18:49 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <1346083716.1140.212.camel@revolution.hippie.lan> <1346094047.1140.264.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQlttZzqkhCfJ0lgiheQOo9AGr9a600oEtPzF1I1U9kRw2ljShs0ihBl1vA9WC5xwrwxiImW Cc: freebsd-arch@freebsd.org, freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 19:18:55 -0000 On Aug 27, 2012, at 1:00 PM, Ian Lepore wrote: > On Mon, 2012-08-27 at 09:53 -0700, Adrian Chadd wrote: >> On 27 August 2012 09:08, Ian Lepore = wrote: >>=20 >>> If two DMAs are going on concurrently in the same buffer, one is = going >>> to finish before the other, leading to a POSTxxxx sync op happening = for >>> one DMA operation while the other is still in progress. The unit of >>> granularity for sync operations is the mapped region, so now you're >>> syncing access to a region which still has active DMA happening = within >>> it. >>=20 >> Right. But the enforced idea is "DMA up to this point should be >> flushed to memory." >>=20 >>> While I think it's really an API definition issue, think about it in >>> terms of a potential implementation... What if the CPU had to access = the >>> memory as part of the sync for the first DMA that completes, while = the >>> second is still running? Now you've got pretty much exactly the = same >>> situation as when a driver subdivides a buffer without knowing about = the >>> cache alignment; you end up with the CPU and DMA touching data in = the >>> same cachline and no sequence of flush/invalidate can be g'teed to >>> preserve all data correctly. >>=20 >> Right. So you realise at that point you can't win and you stick each >> of those pieces in a different cache line. >>=20 >=20 > Actually, I think that even discussing cache lines in this context is = a > mistake (yeah, I'm the one who did so above, in trying to relate an > abstract API design concept to a real-world hardware example). >=20 > Drivers are not supposed to know about interactions between DMA > transfers and cache lines or other machine-specific constraints; that > info is supposed to be encapsulated and hidden within busdma. I think = a > driver making the assumption that it can do DMA safely on a buffer as > long as that buffer is cacheline-granular is just as flawed as = assuming > that it can do DMA safely on any arbitrarily sized and aligned buffer. >=20 > So the right way to "stick each of those pieces in a different cache > line" is to allocate two different buffers, one per concurrent DMA > transfer. Or, really, to use two separate busdma mappings would be = the > more rigorous way to say it, since the mapping is the operation at = which > constraints come into play. Thinking of it that way then drives the > need to document that if multiple mappings describe the same area of > physical memory, then concurrent operations on those maps yield > unpredictable results. Despite what I said earlier, I think this is sane. busdma only should = support one DMA active at a time into a buffer. If the driver wants to = do two different ones, they are on their own. Warner= From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 19:31:53 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0A4EE106568A; Mon, 27 Aug 2012 19:31:53 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id CDF9E8FC15; Mon, 27 Aug 2012 19:31:52 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 0C5F3B983; Mon, 27 Aug 2012 15:31:52 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Date: Mon, 27 Aug 2012 15:31:42 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p17; KDE/4.5.5; amd64; ; ) References: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> <20120827185346.GE33100@deviant.kiev.zoral.com.ua> In-Reply-To: <20120827185346.GE33100@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201208271531.42725.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 27 Aug 2012 15:31:52 -0400 (EDT) Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, Konstantin Belousov Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 19:31:53 -0000 On Monday, August 27, 2012 2:53:46 pm Konstantin Belousov wrote: > On Sun, Aug 26, 2012 at 05:13:31PM -0600, Warner Losh wrote: > > > > On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: > > > In this regard, it's the busdma implementation that's broken, because it > > > should bounce those IOs through a DMA-safe buffer. There's absolutely > > > no rule that I've ever heard of in FreeBSD that says IO can only take > > > place using memory allocated from busdma. > > > > That's partially true. Since BUSDMA grew up in the storage area, you > > must allocate the memory from busdma, or it must be page aligned has > > been the de-facto rule here. The mbuf and uio variants of load were > > invented to cope with common cases of mbufs and user I/O to properly > > flag things. > > I once looked at x86 bus_dmamap_load_uio(), and I was unable to > understand how to use it with usermode uio. I think this is a good > moment to ask. Most existing users use UIO_SYSSPACE, but several crypto > drivers might allow the UIO_USERSPACE for them. > > For UIO_USERSPACE, if the page is not resident, the pmap_extract() call from > _bus_dmamap_load_buffer() returns 0. So the i/o happens to the page > located at 0, which contains real mode IVT and other BIOS sensitive tables. > > Worse, if the page is resident, but it is mapped at the region which > requires COW on write, then DMA will be performed to the wrong page > which is typically shared with other innocent users. to the COW area > which was not yet copied, > > Am I missing some trick there ? No. The caller is required to wire the pages first in some manner. In general bus_dmamap_load_uio() isn't a good idea. I do believe the crypto drivers are careful to wire the buffer first. I think requiring the caller to wire is the only sane way that can be used. Also, doing DMA to a stack variable is absolutely horrible for a related reason since presumably the thread will block while it waits for the DMA to complete, and a sleeping thread can be swapped out (including having it's stack swapped out). -- John Baldwin From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 20:06:00 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 55AF81065674 for ; Mon, 27 Aug 2012 20:06:00 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 12F6E8FC23 for ; Mon, 27 Aug 2012 20:05:59 +0000 (UTC) Received: by obbun3 with SMTP id un3so11861289obb.13 for ; Mon, 27 Aug 2012 13:05:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=+n4sSMIJPIJ1KpjF1l4RL7SHSp5wZqELPnRTrkLw1fE=; b=y2k0SMzQPlxoJsYPjc7Y42UUN9+LMg0KkIMWo6S1ktEEcuSvj4sqR0r+qgikS7hPTx IVKE87NVMnFm3ibCRoaDGc1MG7Lt08pC42qLGdZXe0/sclU6V0SRyCIVEKLCqOrFQvA1 byZiE9JfPXPl/h/nysMZwx+0cm3ld2t6aEq/L7g3rEMT9qQuOQZU7sitxyyrNGOzaWIe NWuM9CIBQ5JjUCqJPwpcUqNieD28vw/hKXzot6y381EYjWrKnNX4fqDgrCTCyiHEpHRa fpTmN+4+76lfEhBfeXYMgPnqjFYHsbnqKOWT//nTXD/F2Lvl0X9IFGmBlYzoJH3nRmor 3G+w== MIME-Version: 1.0 Received: by 10.60.32.136 with SMTP id j8mr11209663oei.0.1346097959304; Mon, 27 Aug 2012 13:05:59 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.76.8.98 with HTTP; Mon, 27 Aug 2012 13:05:59 -0700 (PDT) In-Reply-To: <20120827183419.GG9583@plum.flirble.org> References: <20120827174004.GD58312@funkthat.com> <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> <20120827183419.GG9583@plum.flirble.org> Date: Mon, 27 Aug 2012 13:05:59 -0700 X-Google-Sender-Auth: JLU3qqSQkDrr2Vy93k9jhLf6pSo Message-ID: From: Adrian Chadd To: Nicholas Clark Content-Type: text/plain; charset=ISO-8859-1 Cc: George Neville-Neil , freebsd-arm@freebsd.org Subject: Re: Avila GW2348-4 and VLANs not working X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 20:06:00 -0000 Honestly, the Avila is a much nicer board to do router-y stuff with than a R-Pi. The ethernet is connected via USB. It's not terribly fast. :-) Adrian From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 22:08:17 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E93E4106566C; Mon, 27 Aug 2012 22:08:16 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 6F7F08FC0A; Mon, 27 Aug 2012 22:08:04 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7RM7q0C094077; Mon, 27 Aug 2012 16:07:59 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7RM7Uu6031952; Mon, 27 Aug 2012 16:07:30 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: <9642068B-3C66-42BD-8515-14F734B3FF89@bsdimp.com> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <9642068B-3C66-42BD-8515-14F734B3FF89@bsdimp.com> Content-Type: text/plain; charset="us-ascii" Date: Mon, 27 Aug 2012 16:07:30 -0600 Message-ID: <1346105250.1140.314.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Tim Kientzle , Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 22:08:17 -0000 On Mon, 2012-08-27 at 09:40 -0600, Warner Losh wrote: > On Aug 27, 2012, at 9:20 AM, Adrian Chadd wrote: > > That does remind me, I think the ath(4) driver does the same (since it > > allocates its own descriptor block and then treats it as an array of > > descriptors for the hardware to access) - I should ensure that > > sizeof(ath_desc) is aligned on the relevant architecture. It gets > > slightly scary - AR93xx TX descriptors are "L1 cache == 128 byte > > aligned" which is an enormous waste of memory compared to a 16 or 32 > > byte aligned platform. Alas.. > > The problem is with cache line sharing, not necessarily with alignment. If you are only ever using one of them at a time, or if you have perfect hygiene, you can cope with this situation without undue waste. The perfect hygiene might be hard sometimes. This brings up an interesting tangential issue for this busdma discussion. For some controller hardware you allocate a block of memory which is treated as an array of "descriptors" or some other shared control information, you set a register in the hardware to point to that block of memory, and then there is some degree of concurrent access of that memory by hardware and CPU. The interesting part is that some such hardware cannot operate in phases as anticipated by our busdma model. That is, there's no clear demarkation points between "the CPU has exclusive access to the memory" and "the hardware has exclusive access to the memory." Usually for these schemes to work correctly, the memory has to be mapped as uncached, unbuffered, strongly ordered, or whatever combo of those makes sense for a given platform. We have arm drivers that use bus_dmamem_alloc() with the BUS_DMA_COHERENT flag to obtain such memory, even though that wasn't the intended meaning for that flag. If the armv4 busdma implementation were changed to stop honoring the COHERENT flag (it's supposed to be an optional feature) those drivers would stop working. So we need to track down such mistakes and fix them, but the question is: fix them how? I think it may make sense to let busdma handle it, because you may get some advantage from the allocation being made based upon the constraints encoded in the inherited chain of tags for the driver. On the other hand, drivers doing this sort of thing are usually pretty close to the silicon and have a good idea for themselves what the hardware constraints are. We could just say that drivers with such needs should call kmem_alloc_contig() or kmem_alloc_attr() for themselves. If we say it's a thing that busdma should handle, then I think we need: * A flag that is universal across all platforms that means unambiguously that you need memory that is mapped however device-register memory is mapped on that platform (uncached, unbuffered, strongly ordered; I'm tempted to say "whatever pmap_mapdev() does" but I'm not sure that's rigorously correct). * If the request cannot be honored for some reason it has to return failure, not quietly give you regular cached memory instead (which is what BUS_DMA_COHERENT does). * The busdma sequence of sync operations does not apply to memory allocated with this flag, and indeed you must not call the sync functions on such memory. The x86 busdma code recently grew a BUS_DMA_NOCACHE flag, perhaps that's the name that should be supported across all platforms? -- Ian From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 22:12:56 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C81371065670; Mon, 27 Aug 2012 22:12:56 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id 55C148FC0A; Mon, 27 Aug 2012 22:12:55 +0000 (UTC) Received: from mail.cicely.de ([10.1.1.37]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id q7RMCjUO059568 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 28 Aug 2012 00:12:46 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (cicely7.cicely.de [10.1.1.9]) by mail.cicely.de (8.14.4/8.14.4) with ESMTP id q7RMCWPU082659 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 28 Aug 2012 00:12:32 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.14.2/8.14.2) with ESMTP id q7RMCWaC074506; Tue, 28 Aug 2012 00:12:32 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.14.2/8.14.2/Submit) id q7RMCWMQ074505; Tue, 28 Aug 2012 00:12:32 +0200 (CEST) (envelope-from ticso) Date: Tue, 28 Aug 2012 00:12:32 +0200 From: Bernd Walter To: Warner Losh Message-ID: <20120827221231.GB73992@cicely7.cicely.de> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <45C204C3-D4CE-4B00-886A-A88A0FE7CAD7@bsdimp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <45C204C3-D4CE-4B00-886A-A88A0FE7CAD7@bsdimp.com> X-Operating-System: FreeBSD cicely7.cicely.de 7.0-STABLE i386 User-Agent: Mutt/1.5.11 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED=-1, BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01 autolearn=unavailable version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on spamd.cicely.de Cc: Tim Kientzle , freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 22:12:57 -0000 On Mon, Aug 27, 2012 at 09:34:30AM -0600, Warner Losh wrote: > > On Aug 27, 2012, at 9:12 AM, Tim Kientzle wrote: > > > > > On Aug 27, 2012, at 6:38 AM, Warner Losh wrote: > > > >> > >> On Aug 27, 2012, at 12:06 AM, Hans Petter Selasky wrote: > >> > >>> Hi, > >>> Correct. > >>> > >>>> We also need some rules about working with buffers obtained from > >>>> bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). I > >>>> think the rule should be that a buffer obtained from bus_dmamem_alloc(), > >>>> or more formally any region of memory mapped by a bus_dmamap_load(), is > >>>> a single logical object which can only be accessed by one entity at a > >>>> time. That means that there cannot be two concurrent DMA operations > >>>> happening in different regions of the same buffer, nor can DMA and CPU > >>>> access be happening concurrently even if in different parts of the > >>>> buffer. > >>> > >>> Is this something which we can fix using a simple __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to be DMA loaded. > >> > >> No. I don't think so. the reason is that you can't define USB_DMA_ALGIN to be a constant on MIPS, at least, or I think ARM because that's determined at run time. > > > > But don't mbuf structures do pretty much what Hans is suggesting? > > They kinda do, and kinda don't. > > > Why is mbuf okay? > > mbuf is OK because it is never changed while the DMA is pending to the buffers. That is, m_hdr and m_ext are invariant while the device owns the memory. In addition, mbuf allocations are so large that no two mbufs share the same cache line (although it looks like 256 might be too small to avoid that on some MIPS processors). usb buffers do not obey these same rules, otherwise none of the stuff we're talking about would matter. I've always hated the design of USB controllers with their zillions of tiny DMA buffers. -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. From owner-freebsd-arm@FreeBSD.ORG Tue Aug 28 01:30:11 2012 Return-Path: Delivered-To: freebsd-arm@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 308951065670 for ; Tue, 28 Aug 2012 01:30:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 191788FC1A for ; Tue, 28 Aug 2012 01:30:11 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7S1UAcj057819 for ; Tue, 28 Aug 2012 01:30:10 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7S1UAKP057816; Tue, 28 Aug 2012 01:30:10 GMT (envelope-from gnats) Date: Tue, 28 Aug 2012 01:30:10 GMT Message-Id: <201208280130.q7S1UAKP057816@freefall.freebsd.org> To: freebsd-arm@FreeBSD.org From: dfilter@FreeBSD.ORG (dfilter service) Cc: Subject: Re: arm/155214: commit references a PR X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dfilter service List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2012 01:30:11 -0000 The following reply was made to PR arm/155214; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: arm/155214: commit references a PR Date: Tue, 28 Aug 2012 01:29:10 +0000 (UTC) Author: imp Date: Tue Aug 28 01:28:52 2012 New Revision: 239762 URL: http://svn.freebsd.org/changeset/base/239762 Log: Bring in the multi-block patches for mci. These required extensive restructuring of the driver. I've tried to preserve the other silicon workarounds that we've added over the years, but haven't had a chance to extensively test on other hardware. On my AT91RM9200 with 30MHz/1 wire/64 block transfers, I've been able to go from ~.66MB/s to 2.25MB/s in the simple tests I performed, almost a 3.5x improvement. This cuts the boot time almost in half when everything else goes right (timed from rtc message to login: prompt). PR: 155214 Submitted by: Ian Lapore Modified: head/sys/arm/at91/at91_mci.c Modified: head/sys/arm/at91/at91_mci.c ============================================================================== --- head/sys/arm/at91/at91_mci.c Mon Aug 27 23:27:41 2012 (r239761) +++ head/sys/arm/at91/at91_mci.c Tue Aug 28 01:28:52 2012 (r239762) @@ -114,7 +114,24 @@ __FBSDID("$FreeBSD$"); #define AT91_MCI_USE_30MHZ 1 #endif -#define BBSZ 512 +/* + * Allocate 2 bounce buffers we'll use to endian-swap the data due to the rm9200 + * erratum. We use a pair of buffers because when reading that lets us begin + * endian-swapping the data in the first buffer while the DMA is reading into + * the second buffer. (We can't use the same trick for writing because we might + * not get all the data in the 2nd buffer swapped before the hardware needs it; + * dealing with that would add complexity to the driver.) + * + * The buffers are sized at 16K each due to the way the busdma cache sync + * operations work on arm. A dcache_inv_range() operation on a range larger + * than 16K gets turned into a dcache_wbinv_all(). That needlessly flushes the + * entire data cache, impacting overall system performance. + */ +#define BBCOUNT 2 +#define BBSIZE (16*1024) +#define MAX_BLOCKS ((BBSIZE*BBCOUNT)/512) + +static int mci_debug; struct at91_mci_softc { void *intrhand; /* Interrupt handle */ @@ -123,21 +140,25 @@ struct at91_mci_softc { #define CAP_HAS_4WIRE 1 /* Has 4 wire bus */ #define CAP_NEEDS_BYTESWAP 2 /* broken hardware needing bounce */ int flags; -#define CMD_STARTED 1 -#define STOP_STARTED 2 +#define PENDING_CMD 0x01 +#define PENDING_STOP 0x02 +#define CMD_MULTIREAD 0x10 +#define CMD_MULTIWRITE 0x20 int has_4wire; int use_30mhz; struct resource *irq_res; /* IRQ resource */ struct resource *mem_res; /* Memory resource */ struct mtx sc_mtx; bus_dma_tag_t dmatag; - bus_dmamap_t map; - int mapped; struct mmc_host host; int bus_busy; struct mmc_request *req; struct mmc_command *curcmd; - char bounce_buffer[BBSZ]; + bus_dmamap_t bbuf_map[BBCOUNT]; + char * bbuf_vaddr[BBCOUNT]; /* bounce bufs in KVA space */ + uint32_t bbuf_len[BBCOUNT]; /* len currently queued for bounce buf */ + uint32_t bbuf_curidx; /* which bbuf is the active DMA buffer */ + uint32_t xfer_offset; /* offset so far into caller's buf */ }; static inline uint32_t @@ -172,6 +193,51 @@ static int at91_mci_is_mci1rev2xx(void); #define AT91_MCI_ASSERT_LOCKED(_sc) mtx_assert(&_sc->sc_mtx, MA_OWNED); #define AT91_MCI_ASSERT_UNLOCKED(_sc) mtx_assert(&_sc->sc_mtx, MA_NOTOWNED); +static void +at91_bswap_buf(struct at91_mci_softc *sc, void * dptr, void * sptr, uint32_t memsize) +{ + uint32_t * dst = (uint32_t *)dptr; + uint32_t * src = (uint32_t *)sptr; + uint32_t i; + + /* + * If the hardware doesn't need byte-swapping, let bcopy() do the + * work. Use bounce buffer even if we don't need byteswap, since + * buffer may straddle a page boundry, and we don't handle + * multi-segment transfers in hardware. Seen from 'bsdlabel -w' which + * uses raw geom access to the volume. Greg Ansley (gja (at) + * ansley.com) + */ + if (!(sc->sc_cap & CAP_NEEDS_BYTESWAP)) { + bcopy(dptr, sptr, memsize); + return; + } + + /* + * Nice performance boost for slightly unrolling this loop. + * (But very little extra boost for further unrolling it.) + */ + for (i = 0; i < memsize; i += 16) { + *dst++ = bswap32(*src++); + *dst++ = bswap32(*src++); + *dst++ = bswap32(*src++); + *dst++ = bswap32(*src++); + } + + /* Mop up the last 1-3 words, if any. */ + for (i = 0; i < (memsize & 0x0F); i += 4) { + *dst++ = bswap32(*src++); + } +} + +static void +at91_mci_getaddr(void *arg, bus_dma_segment_t *segs, int nsegs, int error) +{ + if (error != 0) + return; + *(bus_addr_t *)arg = segs[0].ds_addr; +} + static void at91_mci_pdc_disable(struct at91_mci_softc *sc) { @@ -186,13 +252,57 @@ at91_mci_pdc_disable(struct at91_mci_sof WR4(sc, PDC_TNCR, 0); } +/* + * Reset the controller, then restore most of the current state. + * + * This is called after detecting an error. It's also called after stopping a + * multi-block write, to un-wedge the device so that it will handle the NOTBUSY + * signal correctly. See comments in at91_mci_stop_done() for more details. + */ +static void at91_mci_reset(struct at91_mci_softc *sc) +{ + uint32_t mr; + uint32_t sdcr; + uint32_t dtor; + uint32_t imr; + + at91_mci_pdc_disable(sc); + + /* save current state */ + + imr = RD4(sc, MCI_IMR); + mr = RD4(sc, MCI_MR) & 0x7fff; + sdcr = RD4(sc, MCI_SDCR); + dtor = RD4(sc, MCI_DTOR); + + /* reset the controller */ + + WR4(sc, MCI_IDR, 0xffffffff); + WR4(sc, MCI_CR, MCI_CR_MCIDIS | MCI_CR_SWRST); + + /* restore state */ + + WR4(sc, MCI_CR, MCI_CR_MCIEN|MCI_CR_PWSEN); + WR4(sc, MCI_MR, mr); + WR4(sc, MCI_SDCR, sdcr); + WR4(sc, MCI_DTOR, dtor); + WR4(sc, MCI_IER, imr); + + /* + * Make sure sdio interrupts will fire. Not sure why reading + * SR ensures that, but this is in the linux driver. + */ + + RD4(sc, MCI_SR); +} + static void at91_mci_init(device_t dev) { struct at91_mci_softc *sc = device_get_softc(dev); uint32_t val; - WR4(sc, MCI_CR, MCI_CR_MCIEN); /* Enable controller */ + WR4(sc, MCI_CR, MCI_CR_MCIDIS | MCI_CR_SWRST); /* device into reset */ WR4(sc, MCI_IDR, 0xffffffff); /* Turn off interrupts */ WR4(sc, MCI_DTOR, MCI_DTOR_DTOMUL_1M | 1); val = MCI_MR_PDCMODE; @@ -203,10 +313,19 @@ at91_mci_init(device_t dev) #ifndef AT91_MCI_SLOT_B WR4(sc, MCI_SDCR, 0); /* SLOT A, 1 bit bus */ #else - /* XXX Really should add second "unit" but nobody using using - * a two slot card that we know of. -- except they are... XXX */ + /* + * XXX Really should add second "unit" but nobody using using + * a two slot card that we know of. XXX + */ WR4(sc, MCI_SDCR, 1); /* SLOT B, 1 bit bus */ #endif + /* + * Enable controller, including power-save. The slower clock + * of the power-save mode is only in effect when there is no + * transfer in progress, so it can be left in this mode all + * the time. + */ + WR4(sc, MCI_CR, MCI_CR_MCIEN|MCI_CR_PWSEN); } static void @@ -216,7 +335,7 @@ at91_mci_fini(device_t dev) WR4(sc, MCI_IDR, 0xffffffff); /* Turn off interrupts */ at91_mci_pdc_disable(sc); - WR4(sc, MCI_CR, MCI_CR_MCIDIS | MCI_CR_SWRST); /* Put the device into reset */ + WR4(sc, MCI_CR, MCI_CR_MCIDIS | MCI_CR_SWRST); /* device into reset */ } static int @@ -234,7 +353,7 @@ at91_mci_attach(device_t dev) struct sysctl_ctx_list *sctx; struct sysctl_oid *soid; device_t child; - int err; + int err, i; sctx = device_get_sysctl_ctx(dev); soid = device_get_sysctl_tree(dev); @@ -249,21 +368,33 @@ at91_mci_attach(device_t dev) AT91_MCI_LOCK_INIT(sc); + at91_mci_fini(dev); + at91_mci_init(dev); + /* - * Allocate DMA tags and maps + * Allocate DMA tags and maps and bounce buffers. + * + * The parms in the tag_create call cause the dmamem_alloc call to + * create each bounce buffer as a single contiguous buffer of BBSIZE + * bytes aligned to a 4096 byte boundary. + * + * Do not use DMA_COHERENT for these buffers because that maps the + * memory as non-cachable, which prevents cache line burst fills/writes, + * which is something we need since we're trying to overlap the + * byte-swapping with the DMA operations. */ - err = bus_dma_tag_create(bus_get_dma_tag(dev), 1, 0, - BUS_SPACE_MAXADDR_32BIT, BUS_SPACE_MAXADDR, NULL, NULL, MAXPHYS, 1, - MAXPHYS, BUS_DMA_ALLOCNOW, NULL, NULL, &sc->dmatag); + err = bus_dma_tag_create(bus_get_dma_tag(dev), 4096, 0, + BUS_SPACE_MAXADDR_32BIT, BUS_SPACE_MAXADDR, NULL, NULL, + BBSIZE, 1, BBSIZE, 0, NULL, NULL, &sc->dmatag); if (err != 0) goto out; - err = bus_dmamap_create(sc->dmatag, 0, &sc->map); - if (err != 0) - goto out; - - at91_mci_fini(dev); - at91_mci_init(dev); + for (i = 0; i < BBCOUNT; ++i) { + err = bus_dmamem_alloc(sc->dmatag, (void **)&sc->bbuf_vaddr[i], + BUS_DMA_NOWAIT, &sc->bbuf_map[i]); + if (err != 0) + goto out; + } /* * Activate the interrupt @@ -330,8 +461,15 @@ out: static int at91_mci_detach(device_t dev) { + struct at91_mci_softc *sc = device_get_softc(dev); + at91_mci_fini(dev); at91_mci_deactivate(dev); + + bus_dmamem_free(sc->dmatag, sc->bbuf_vaddr[0], sc->bbuf_map[0]); + bus_dmamem_free(sc->dmatag, sc->bbuf_vaddr[1], sc->bbuf_map[1]); + bus_dma_tag_destroy(sc->dmatag); + return (EBUSY); /* XXX */ } @@ -398,14 +536,6 @@ at91_mci_is_mci1rev2xx(void) } } -static void -at91_mci_getaddr(void *arg, bus_dma_segment_t *segs, int nsegs, int error) -{ - if (error != 0) - return; - *(bus_addr_t *)arg = segs[0].ds_addr; -} - static int at91_mci_update_ios(device_t brdev, device_t reqdev) { @@ -437,7 +567,7 @@ at91_mci_update_ios(device_t brdev, devi if (sc->use_30mhz && ios->clock == 25000000 && at91_master_clock > 50000000) clkdiv = 0; - else if ((at91_master_clock % (ios->clock * 2)) == 0) + else if ((at91_master_clock % (ios->clock * 2)) == 0) clkdiv = ((at91_master_clock / ios->clock) / 2) - 1; else clkdiv = (at91_master_clock / ios->clock) / 2; @@ -456,73 +586,182 @@ at91_mci_update_ios(device_t brdev, devi static void at91_mci_start_cmd(struct at91_mci_softc *sc, struct mmc_command *cmd) { - size_t len; - uint32_t cmdr, ier = 0, mr; - uint32_t *src, *dst; - int i; + uint32_t cmdr, mr; struct mmc_data *data; - void *vaddr; - bus_addr_t paddr; sc->curcmd = cmd; data = cmd->data; - cmdr = cmd->opcode; /* XXX Upper layers don't always set this */ cmd->mrq = sc->req; + /* Begin setting up command register. */ + + cmdr = cmd->opcode; + + if (sc->host.ios.bus_mode == opendrain) + cmdr |= MCI_CMDR_OPDCMD; + + /* Set up response handling. Allow max timeout for responses. */ + if (MMC_RSP(cmd->flags) == MMC_RSP_NONE) cmdr |= MCI_CMDR_RSPTYP_NO; else { - /* Allow big timeout for responses */ cmdr |= MCI_CMDR_MAXLAT; if (cmd->flags & MMC_RSP_136) cmdr |= MCI_CMDR_RSPTYP_136; else cmdr |= MCI_CMDR_RSPTYP_48; } - if (cmd->opcode == MMC_STOP_TRANSMISSION) - cmdr |= MCI_CMDR_TRCMD_STOP; - if (sc->host.ios.bus_mode == opendrain) - cmdr |= MCI_CMDR_OPDCMD; - if (!data) { - // The no data case is fairly simple + + /* + * If there is no data transfer, just set up the right interrupt mask + * and start the command. + * + * The interrupt mask needs to be CMDRDY plus all non-data-transfer + * errors. It's important to leave the transfer-related errors out, to + * avoid spurious timeout or crc errors on a STOP command following a + * multiblock read. When a multiblock read is in progress, sending a + * STOP in the middle of a block occasionally triggers such errors, but + * we're totally disinterested in them because we've already gotten all + * the data we wanted without error before sending the STOP command. + */ + + if (data == NULL) { + uint32_t ier = MCI_SR_CMDRDY | + MCI_SR_RTOE | MCI_SR_RENDE | + MCI_SR_RCRCE | MCI_SR_RDIRE | MCI_SR_RINDE; + at91_mci_pdc_disable(sc); -// printf("CMDR %x ARGR %x\n", cmdr, cmd->arg); + + if (cmd->opcode == MMC_STOP_TRANSMISSION) + cmdr |= MCI_CMDR_TRCMD_STOP; + + /* Ignore response CRC on CMD2 and ACMD41, per standard. */ + + if (cmd->opcode == MMC_SEND_OP_COND || + cmd->opcode == ACMD_SD_SEND_OP_COND) + ier &= ~MCI_SR_RCRCE; + + if (mci_debug) + printf("CMDR %x (opcode %d) ARGR %x no data\n", + cmdr, cmd->opcode, cmd->arg); + WR4(sc, MCI_ARGR, cmd->arg); WR4(sc, MCI_CMDR, cmdr); - WR4(sc, MCI_IER, MCI_SR_ERROR | MCI_SR_CMDRDY); + WR4(sc, MCI_IDR, 0xffffffff); + WR4(sc, MCI_IER, ier); return; } + + /* There is data, set up the transfer-related parts of the command. */ + if (data->flags & MMC_DATA_READ) cmdr |= MCI_CMDR_TRDIR; + if (data->flags & (MMC_DATA_READ | MMC_DATA_WRITE)) cmdr |= MCI_CMDR_TRCMD_START; + if (data->flags & MMC_DATA_STREAM) cmdr |= MCI_CMDR_TRTYP_STREAM; - if (data->flags & MMC_DATA_MULTI) + else if (data->flags & MMC_DATA_MULTI) { cmdr |= MCI_CMDR_TRTYP_MULTIPLE; - // Set block size and turn on PDC mode for dma xfer and disable - // PDC until we're ready. - mr = RD4(sc, MCI_MR) & ~MCI_MR_BLKLEN; - WR4(sc, MCI_MR, mr | (data->len << 16) | MCI_MR_PDCMODE); - WR4(sc, PDC_PTCR, PDC_PTCR_RXTDIS | PDC_PTCR_TXTDIS); - if (cmdr & MCI_CMDR_TRCMD_START) { - len = data->len; - if (cmdr & MCI_CMDR_TRDIR) - vaddr = cmd->data->data; - else { - /* Use bounce buffer even if we don't need - * byteswap, since buffer may straddle a page - * boundry, and we don't handle multi-segment - * transfers in hardware. - * (page issues seen from 'bsdlabel -w' which - * uses raw geom access to the volume). - * Greg Ansley (gja (at) ansley.com) - */ - vaddr = sc->bounce_buffer; - src = (uint32_t *)cmd->data->data; - dst = (uint32_t *)vaddr; + sc->flags |= (data->flags & MMC_DATA_READ) ? + CMD_MULTIREAD : CMD_MULTIWRITE; + } + + /* + * Disable PDC until we're ready. + * + * Set block size and turn on PDC mode for dma xfer. + * Note that the block size is the smaller of the amount of data to be + * transferred, or 512 bytes. The 512 size is fixed by the standard; + * smaller blocks are possible, but never larger. + */ + + WR4(sc, PDC_PTCR, PDC_PTCR_RXTDIS | PDC_PTCR_TXTDIS); + + mr = RD4(sc,MCI_MR) & ~MCI_MR_BLKLEN; + mr |= min(data->len, 512) << 16; + WR4(sc, MCI_MR, mr | MCI_MR_PDCMODE|MCI_MR_PDCPADV); + + /* + * Set up DMA. + * + * Use bounce buffers even if we don't need to byteswap, because doing + * multi-block IO with large DMA buffers is way fast (compared to + * single-block IO), even after incurring the overhead of also copying + * from/to the caller's buffers (which may be in non-contiguous physical + * pages). + * + * In an ideal non-byteswap world we could create a dma tag that allows + * for discontiguous segments and do the IO directly from/to the + * caller's buffer(s), using ENDRX/ENDTX interrupts to chain the + * discontiguous buffers through the PDC. Someday. + * + * If a read is bigger than 2k, split it in half so that we can start + * byte-swapping the first half while the second half is on the wire. + * It would be best if we could split it into 8k chunks, but we can't + * always keep up with the byte-swapping due to other system activity, + * and if an RXBUFF interrupt happens while we're still handling the + * byte-swap from the prior buffer (IE, we haven't returned from + * handling the prior interrupt yet), then data will get dropped on the + * floor and we can't easily recover from that. The right fix for that + * would be to have the interrupt handling only keep the DMA flowing and + * enqueue filled buffers to be byte-swapped in a non-interrupt context. + * Even that won't work on the write side of things though; in that + * context we have to have all the data ready to go before starting the + * dma. + * + * XXX what about stream transfers? + */ + sc->xfer_offset = 0; + sc->bbuf_curidx = 0; + + if (data->flags & (MMC_DATA_READ | MMC_DATA_WRITE)) { + uint32_t len; + uint32_t remaining = data->len; + bus_addr_t paddr; + int err; + + if (remaining > (BBCOUNT*BBSIZE)) + panic("IO read size exceeds MAXDATA\n"); + + if (data->flags & MMC_DATA_READ) { + if (remaining > 2048) // XXX + len = remaining / 2; + else + len = remaining; + err = bus_dmamap_load(sc->dmatag, sc->bbuf_map[0], + sc->bbuf_vaddr[0], len, at91_mci_getaddr, + &paddr, BUS_DMA_NOWAIT); + if (err != 0) + panic("IO read dmamap_load failed\n"); + bus_dmamap_sync(sc->dmatag, sc->bbuf_map[0], + BUS_DMASYNC_PREREAD); + WR4(sc, PDC_RPR, paddr); + WR4(sc, PDC_RCR, len / 4); + sc->bbuf_len[0] = len; + remaining -= len; + if (remaining == 0) { + sc->bbuf_len[1] = 0; + } else { + len = remaining; + err = bus_dmamap_load(sc->dmatag, sc->bbuf_map[1], + sc->bbuf_vaddr[1], len, at91_mci_getaddr, + &paddr, BUS_DMA_NOWAIT); + if (err != 0) + panic("IO read dmamap_load failed\n"); + bus_dmamap_sync(sc->dmatag, sc->bbuf_map[1], + BUS_DMASYNC_PREREAD); + WR4(sc, PDC_RNPR, paddr); + WR4(sc, PDC_RNCR, len / 4); + sc->bbuf_len[1] = len; + remaining -= len; + } + WR4(sc, PDC_PTCR, PDC_PTCR_RXTEN); + } else { + len = min(BBSIZE, remaining); /* * If this is MCI1 revision 2xx controller, apply * a work-around for the "Data Write Operation and @@ -530,74 +769,75 @@ at91_mci_start_cmd(struct at91_mci_softc */ if (at91_mci_is_mci1rev2xx() && data->len < 12) { len = 12; - memset(dst, 0, 12); + memset(data->data, 0, 12); } - if (sc->sc_cap & CAP_NEEDS_BYTESWAP) { - for (i = 0; i < data->len / 4; i++) - dst[i] = bswap32(src[i]); - } else - memcpy(dst, src, data->len); - } - data->xfer_len = 0; - if (bus_dmamap_load(sc->dmatag, sc->map, vaddr, len, - at91_mci_getaddr, &paddr, 0) != 0) { - cmd->error = MMC_ERR_NO_MEMORY; - sc->req = NULL; - sc->curcmd = NULL; - cmd->mrq->done(cmd->mrq); - return; - } - sc->mapped++; - if (cmdr & MCI_CMDR_TRDIR) { - bus_dmamap_sync(sc->dmatag, sc->map, BUS_DMASYNC_PREREAD); - WR4(sc, PDC_RPR, paddr); - WR4(sc, PDC_RCR, len / 4); - ier = MCI_SR_ENDRX; - } else { - bus_dmamap_sync(sc->dmatag, sc->map, BUS_DMASYNC_PREWRITE); - WR4(sc, PDC_TPR, paddr); + at91_bswap_buf(sc, sc->bbuf_vaddr[0], data->data, len); + err = bus_dmamap_load(sc->dmatag, sc->bbuf_map[0], + sc->bbuf_vaddr[0], len, at91_mci_getaddr, + &paddr, BUS_DMA_NOWAIT); + if (err != 0) + panic("IO write dmamap_load failed\n"); + bus_dmamap_sync(sc->dmatag, sc->bbuf_map[0], + BUS_DMASYNC_PREWRITE); + WR4(sc, PDC_TPR,paddr); WR4(sc, PDC_TCR, len / 4); - ier = MCI_SR_TXBUFE; + sc->bbuf_len[0] = len; + remaining -= len; + if (remaining == 0) { + sc->bbuf_len[1] = 0; + } else { + len = remaining; + at91_bswap_buf(sc, sc->bbuf_vaddr[1], + ((char *)data->data)+BBSIZE, len); + err = bus_dmamap_load(sc->dmatag, sc->bbuf_map[1], + sc->bbuf_vaddr[1], len, at91_mci_getaddr, + &paddr, BUS_DMA_NOWAIT); + if (err != 0) + panic("IO write dmamap_load failed\n"); + bus_dmamap_sync(sc->dmatag, sc->bbuf_map[1], + BUS_DMASYNC_PREWRITE); + WR4(sc, PDC_TNPR, paddr); + WR4(sc, PDC_TNCR, len / 4); + sc->bbuf_len[1] = len; + remaining -= len; + } + /* do not enable PDC xfer until CMDRDY asserted */ } + data->xfer_len = 0; /* XXX what's this? appears to be unused. */ } -// printf("CMDR %x ARGR %x with data\n", cmdr, cmd->arg); + + if (mci_debug) + printf("CMDR %x (opcode %d) ARGR %x with data len %d\n", + cmdr, cmd->opcode, cmd->arg, cmd->data->len); + WR4(sc, MCI_ARGR, cmd->arg); - if (cmdr & MCI_CMDR_TRCMD_START) { - if (cmdr & MCI_CMDR_TRDIR) { - WR4(sc, PDC_PTCR, PDC_PTCR_RXTEN); - WR4(sc, MCI_CMDR, cmdr); - } else { - WR4(sc, MCI_CMDR, cmdr); - WR4(sc, PDC_PTCR, PDC_PTCR_TXTEN); - } - } - WR4(sc, MCI_IER, MCI_SR_ERROR | ier); + WR4(sc, MCI_CMDR, cmdr); + WR4(sc, MCI_IER, MCI_SR_ERROR | MCI_SR_CMDRDY); } static void -at91_mci_start(struct at91_mci_softc *sc) +at91_mci_next_operation(struct at91_mci_softc *sc) { struct mmc_request *req; req = sc->req; if (req == NULL) return; - // assert locked - if (!(sc->flags & CMD_STARTED)) { - sc->flags |= CMD_STARTED; -// printf("Starting CMD\n"); + + if (sc->flags & PENDING_CMD) { + sc->flags &= ~PENDING_CMD; at91_mci_start_cmd(sc, req->cmd); return; - } - if (!(sc->flags & STOP_STARTED) && req->stop) { -// printf("Starting Stop\n"); - sc->flags |= STOP_STARTED; + } else if (sc->flags & PENDING_STOP) { + sc->flags &= ~PENDING_STOP; at91_mci_start_cmd(sc, req->stop); return; } - /* We must be done -- bad idea to do this while locked? */ + + WR4(sc, MCI_IDR, 0xffffffff); sc->req = NULL; sc->curcmd = NULL; + //printf("req done\n"); req->done(req); } @@ -607,16 +847,16 @@ at91_mci_request(device_t brdev, device_ struct at91_mci_softc *sc = device_get_softc(brdev); AT91_MCI_LOCK(sc); - // XXX do we want to be able to queue up multiple commands? - // XXX sounds like a good idea, but all protocols are sync, so - // XXX maybe the idea is naive... if (sc->req != NULL) { AT91_MCI_UNLOCK(sc); return (EBUSY); } + //printf("new req\n"); sc->req = req; - sc->flags = 0; - at91_mci_start(sc); + sc->flags = PENDING_CMD; + if (sc->req->stop) + sc->flags |= PENDING_STOP; + at91_mci_next_operation(sc); AT91_MCI_UNLOCK(sc); return (0); } @@ -654,120 +894,351 @@ at91_mci_release_host(device_t brdev, de } static void -at91_mci_read_done(struct at91_mci_softc *sc) +at91_mci_read_done(struct at91_mci_softc *sc, uint32_t sr) { - uint32_t *walker; - struct mmc_command *cmd; - int i, len; - - cmd = sc->curcmd; - bus_dmamap_sync(sc->dmatag, sc->map, BUS_DMASYNC_POSTREAD); - bus_dmamap_unload(sc->dmatag, sc->map); - sc->mapped--; - if (sc->sc_cap & CAP_NEEDS_BYTESWAP) { - walker = (uint32_t *)cmd->data->data; - len = cmd->data->len / 4; - for (i = 0; i < len; i++) - walker[i] = bswap32(walker[i]); - } - // Finish up the sequence... - WR4(sc, MCI_IDR, MCI_SR_ENDRX); - WR4(sc, MCI_IER, MCI_SR_RXBUFF); - WR4(sc, PDC_PTCR, PDC_PTCR_RXTDIS | PDC_PTCR_TXTDIS); + struct mmc_command *cmd = sc->curcmd; + char * dataptr = (char *)cmd->data->data; + uint32_t curidx = sc->bbuf_curidx; + uint32_t len = sc->bbuf_len[curidx]; + + /* + * We arrive here when a DMA transfer for a read is done, whether it's + * a single or multi-block read. + * + * We byte-swap the buffer that just completed, and if that is the + * last buffer that's part of this read then we move on to the next + * operation, otherwise we wait for another ENDRX for the next bufer. + */ + + bus_dmamap_sync(sc->dmatag, sc->bbuf_map[curidx], BUS_DMASYNC_POSTREAD); + bus_dmamap_unload(sc->dmatag, sc->bbuf_map[curidx]); + + at91_bswap_buf(sc, dataptr + sc->xfer_offset, sc->bbuf_vaddr[curidx], len); + + if (mci_debug) { + printf("read done sr %x curidx %d len %d xfer_offset %d\n", + sr, curidx, len, sc->xfer_offset); + } + + sc->xfer_offset += len; + sc->bbuf_curidx = !curidx; /* swap buffers */ + + /* + * If we've transferred all the data, move on to the next operation. + * + * If we're still transferring the last buffer, RNCR is already zero but + * we have to write a zero anyway to clear the ENDRX status so we don't + * re-interrupt until the last buffer is done. + */ + if (sc->xfer_offset == cmd->data->len) { + WR4(sc, PDC_PTCR, PDC_PTCR_RXTDIS | PDC_PTCR_TXTDIS); + cmd->error = MMC_ERR_NONE; + at91_mci_next_operation(sc); + } else { + WR4(sc, PDC_RNCR, 0); + WR4(sc, MCI_IER, MCI_SR_ERROR | MCI_SR_ENDRX); + } } static void -at91_mci_xmit_done(struct at91_mci_softc *sc) +at91_mci_write_done(struct at91_mci_softc *sc, uint32_t sr) { - // Finish up the sequence... + struct mmc_command *cmd = sc->curcmd; + + /* + * We arrive here when the entire DMA transfer for a write is done, + * whether it's a single or multi-block write. If it's multi-block we + * have to immediately move on to the next operation which is to send + * the stop command. If it's a single-block transfer we need to wait + * for NOTBUSY, but if that's already asserted we can avoid another + * interrupt and just move on to completing the request right away. + */ + WR4(sc, PDC_PTCR, PDC_PTCR_RXTDIS | PDC_PTCR_TXTDIS); - WR4(sc, MCI_IDR, MCI_SR_TXBUFE); - WR4(sc, MCI_IER, MCI_SR_NOTBUSY); - bus_dmamap_sync(sc->dmatag, sc->map, BUS_DMASYNC_POSTWRITE); - bus_dmamap_unload(sc->dmatag, sc->map); - sc->mapped--; + + bus_dmamap_sync(sc->dmatag, sc->bbuf_map[sc->bbuf_curidx], + BUS_DMASYNC_POSTWRITE); + bus_dmamap_unload(sc->dmatag, sc->bbuf_map[sc->bbuf_curidx]); + + if ((cmd->data->flags & MMC_DATA_MULTI) || (sr & MCI_SR_NOTBUSY)) { + cmd->error = MMC_ERR_NONE; + at91_mci_next_operation(sc); + } else { + WR4(sc, MCI_IER, MCI_SR_ERROR | MCI_SR_NOTBUSY); + } +} + +static void +at91_mci_notbusy(struct at91_mci_softc *sc) +{ + struct mmc_command *cmd = sc->curcmd; + + /* + * We arrive here by either completion of a single-block write, or + * completion of the stop command that ended a multi-block write (and, + * I suppose, after a card-select or erase, but I haven't tested + * those). Anyway, we're done and it's time to move on to the next + * command. + */ + + cmd->error = MMC_ERR_NONE; + at91_mci_next_operation(sc); +} + +static void +at91_mci_stop_done(struct at91_mci_softc *sc, uint32_t sr) +{ + struct mmc_command *cmd = sc->curcmd; + + /* + * We arrive here after receiving CMDRDY for a MMC_STOP_TRANSMISSION + * command. Depending on the operation being stopped, we may have to + * do some unusual things to work around hardware bugs. + */ + + /* + * This is known to be true of at91rm9200 hardware; it may or may not + * apply to more recent chips: + * + * After stopping a multi-block write, the NOTBUSY bit in MCI_SR does + * not properly reflect the actual busy state of the card as signaled + * on the DAT0 line; it always claims the card is not-busy. If we + * believe that and let operations continue, following commands will + * fail with response timeouts (except of course MMC_SEND_STATUS -- it + * indicates the card is busy in the PRG state, which was the smoking + * gun that showed MCI_SR NOTBUSY was not tracking DAT0 correctly). + * + * The atmel docs are emphatic: "This flag [NOTBUSY] must be used only + * for Write Operations." I guess technically since we sent a stop + * it's not a write operation anymore. But then just what did they + * think it meant for the stop command to have "...an optional busy + * signal transmitted on the data line" according to the SD spec? + * + * I tried a variety of things to un-wedge the MCI and get the status + * register to reflect NOTBUSY correctly again, but the only thing + * that worked was a full device reset. It feels like an awfully big + * hammer, but doing a full reset after every multiblock write is + * still faster than doing single-block IO (by almost two orders of + * magnitude: 20KB/sec improves to about 1.8MB/sec best case). + * + * After doing the reset, wait for a NOTBUSY interrupt before + * continuing with the next operation. + */ + if (sc->flags & CMD_MULTIWRITE) { + at91_mci_reset(sc); + WR4(sc, MCI_IER, MCI_SR_ERROR | MCI_SR_NOTBUSY); + return; + } + + /* + * This is known to be true of at91rm9200 hardware; it may or may not + * apply to more recent chips: + * + * After stopping a multi-block read, loop to read and discard any + * data that coasts in after we sent the stop command. The docs don't + * say anything about it, but empirical testing shows that 1-3 + * additional words of data get buffered up in some unmentioned + * internal fifo and if we don't read and discard them here they end + * up on the front of the next read DMA transfer we do. + */ + if (sc->flags & CMD_MULTIREAD) { + uint32_t sr; + int count = 0; + + do { + sr = RD4(sc, MCI_SR); + if (sr & MCI_SR_RXRDY) { + RD4(sc, MCI_RDR); + ++count; + } + } while (sr & MCI_SR_RXRDY); + at91_mci_reset(sc); +// if (count != 0) +// printf("Had to soak up %d words after read\n", count); + } + + cmd->error = MMC_ERR_NONE; + at91_mci_next_operation(sc); + +} + +static void +at91_mci_cmdrdy(struct at91_mci_softc *sc, uint32_t sr) +{ + struct mmc_command *cmd = sc->curcmd; + int i; + + if (cmd == NULL) + return; + + /* + * We get here at the end of EVERY command. We retrieve the command + * response (if any) then decide what to do next based on the command. + */ + + if (cmd->flags & MMC_RSP_PRESENT) { + for (i = 0; i < ((cmd->flags & MMC_RSP_136) ? 4 : 1); i++) { + cmd->resp[i] = RD4(sc, MCI_RSPR + i * 4); + if (mci_debug) + printf("RSPR[%d] = %x sr=%x\n", i, cmd->resp[i], sr); + } + } + + /* + * If this was a stop command, go handle the various special + * conditions (read: bugs) that have to be dealt with following a stop. + */ + if (cmd->opcode == MMC_STOP_TRANSMISSION) { + at91_mci_stop_done(sc, sr); + return; + } + + /* + * If this command can continue to assert BUSY beyond the response then + * we need to wait for NOTBUSY before the command is really done. + * + * Note that this may not work properly on the at91rm9200. It certainly + * doesn't work for the STOP command that follows a multi-block write, + * so post-stop CMDRDY is handled separately; see the special handling + * in at91_mci_stop_done(). + * + * Beside STOP, there are other R1B-type commands that use the busy + * signal after CMDRDY: CMD7 (card select), CMD28-29 (write protect), + * CMD38 (erase). I haven't tested any of them, but I rather expect + * them all to have the same sort of problem with MCI_SR not actually + * reflecting the state of the DAT0-line busy indicator. So this code + * may need to grow some sort of special handling for them too. (This + * just in: CMD7 isn't a problem right now because dev/mmc.c incorrectly + * sets the response flags to R1 rather than R1B.) XXX + */ + if ((cmd->flags & MMC_RSP_BUSY)) { + WR4(sc, MCI_IER, MCI_SR_ERROR | MCI_SR_NOTBUSY); + return; + } + + /* + * If there is a data transfer with this command, then... + * - If it's a read, we need to wait for ENDRX. + * - If it's a write, now is the time to enable the PDC, and we need + * to wait for a BLKE that follows a TXBUFE, because if we're doing + * a split transfer we get a BLKE after the first half (when TPR/TCR + * get loaded from TNPR/TNCR). So first we wait for the TXBUFE, and + * the handling for that interrupt will then invoke the wait for the + * subsequent BLKE which indicates actual completion. + */ + if (cmd->data) { + uint32_t ier; + if (cmd->data->flags & MMC_DATA_READ) { + ier = MCI_SR_ENDRX; + } else { + ier = MCI_SR_TXBUFE; + WR4(sc, PDC_PTCR, PDC_PTCR_TXTEN); + } + WR4(sc, MCI_IER, MCI_SR_ERROR | ier); + return; + } + + /* + * If we made it to here, we don't need to wait for anything more for + * the current command, move on to the next command (will complete the + * request if there is no next command). + */ + cmd->error = MMC_ERR_NONE; + at91_mci_next_operation(sc); } static void at91_mci_intr(void *arg) { struct at91_mci_softc *sc = (struct at91_mci_softc*)arg; - uint32_t sr; - int i, done = 0; - struct mmc_command *cmd; + struct mmc_command *cmd = sc->curcmd; + uint32_t sr, isr; AT91_MCI_LOCK(sc); - sr = RD4(sc, MCI_SR) & RD4(sc, MCI_IMR); -// printf("i 0x%x\n", sr); - cmd = sc->curcmd; - if (sr & MCI_SR_ERROR) { - // Ignore CRC errors on CMD2 and ACMD47, per relevant standards - if ((sr & MCI_SR_RCRCE) && (cmd->opcode == MMC_SEND_OP_COND || - cmd->opcode == ACMD_SD_SEND_OP_COND)) - cmd->error = MMC_ERR_NONE; - else if (sr & (MCI_SR_RTOE | MCI_SR_DTOE)) + + sr = RD4(sc, MCI_SR); + isr = sr & RD4(sc, MCI_IMR); + + if (mci_debug) + printf("i 0x%x sr 0x%x\n", isr, sr); + + /* + * All interrupts are one-shot; disable it now. + * The next operation will re-enable whatever interrupts it wants. + */ + WR4(sc, MCI_IDR, isr); + if (isr & MCI_SR_ERROR) { + if (isr & (MCI_SR_RTOE | MCI_SR_DTOE)) cmd->error = MMC_ERR_TIMEOUT; - else if (sr & (MCI_SR_RCRCE | MCI_SR_DCRCE)) + else if (isr & (MCI_SR_RCRCE | MCI_SR_DCRCE)) cmd->error = MMC_ERR_BADCRC; - else if (sr & (MCI_SR_OVRE | MCI_SR_UNRE)) + else if (isr & (MCI_SR_OVRE | MCI_SR_UNRE)) cmd->error = MMC_ERR_FIFO; else cmd->error = MMC_ERR_FAILED; - done = 1; - if (sc->mapped && cmd->error) { - bus_dmamap_unload(sc->dmatag, sc->map); - sc->mapped--; + /* + * CMD8 is used to probe for SDHC cards, a standard SD card + * will get a response timeout; don't report it because it's a + * normal and expected condition. One might argue that all + * error reporting should be left to higher levels, but when + * they report at all it's always EIO, which isn't very + * helpful. XXX bootverbose? + */ + if (cmd->opcode != 8) { + device_printf(sc->dev, *** DIFF OUTPUT TRUNCATED AT 1000 LINES *** _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org" From owner-freebsd-arm@FreeBSD.ORG Tue Aug 28 03:51:41 2012 Return-Path: Delivered-To: freebsd-arm@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B8E32106566C; Tue, 28 Aug 2012 03:51:41 +0000 (UTC) (envelope-from imp@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8C2048FC08; Tue, 28 Aug 2012 03:51:41 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7S3pfWO074102; Tue, 28 Aug 2012 03:51:41 GMT (envelope-from imp@freefall.freebsd.org) Received: (from imp@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7S3pfrF074098; Mon, 27 Aug 2012 21:51:41 -0600 (MDT) (envelope-from imp) Date: Mon, 27 Aug 2012 21:51:41 -0600 (MDT) Message-Id: <201208280351.q7S3pfrF074098@freefall.freebsd.org> To: freebsd@damnhippie.dyndns.org, imp@FreeBSD.org, freebsd-arm@FreeBSD.org From: imp@FreeBSD.org Cc: Subject: Re: arm/155214: [patch] MMC/SD IO slow on Atmel ARM with modern large SD cards X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2012 03:51:41 -0000 Synopsis: [patch] MMC/SD IO slow on Atmel ARM with modern large SD cards State-Changed-From-To: open->patched State-Changed-By: imp State-Changed-When: Mon Aug 27 21:51:28 MDT 2012 State-Changed-Why: Committed. http://www.freebsd.org/cgi/query-pr.cgi?pr=155214 From owner-freebsd-arm@FreeBSD.ORG Tue Aug 28 04:31:51 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C0B39106566B for ; Tue, 28 Aug 2012 04:31:51 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) by mx1.freebsd.org (Postfix) with ESMTP id 995DE8FC1B for ; Tue, 28 Aug 2012 04:31:51 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id q7S4VgTS095104; Tue, 28 Aug 2012 04:31:42 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.143] (CiscoE3000 [192.168.1.65]) by kientzle.com with SMTP id 8cvyx5b7wznf97t4bz3h8urxfe; Tue, 28 Aug 2012 04:31:42 +0000 (UTC) (envelope-from tim@kientzle.com) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=us-ascii From: Tim Kientzle In-Reply-To: <68F9301B-8B03-47D3-A0AA-9F2490D3481B@bsdimp.com> Date: Mon, 27 Aug 2012 21:31:41 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <716FE9A8-EC5B-41F4-9AE2-5302EEBB86B8@kientzle.com> References: <20120827174004.GD58312@funkthat.com> <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> <68F9301B-8B03-47D3-A0AA-9F2490D3481B@bsdimp.com> To: Warner Losh X-Mailer: Apple Mail (2.1278) Cc: George Neville-Neil , freebsd-arm@freebsd.org Subject: Re: Avila GW2348-4 and VLANs not working X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2012 04:31:51 -0000 On Aug 27, 2012, at 12:01 PM, Warner Losh wrote: >=20 > On Aug 27, 2012, at 11:55 AM, George Neville-Neil wrote: >=20 >>=20 >> On Aug 27, 2012, at 18:40 , John-Mark Gurney = wrote: >>=20 >>> P.S. Since I only have one board, and I'm using this as a firewall >>> experimenting is a little difficult. Anyone know of a another good >>> ARM board that I could use (and isn't too expensive)? >>=20 >> I'm a fan of the BeagleBone, which works with FreeBSD, and is about = $89. >>=20 >> http://beagleboard.org/buy >>=20 >> Rasberry Pis are cheap but impossible to come by. >=20 > Doesn't the BealgleBone have only one Ethernet? 1 Ethernet 1 Host USB 1 client USB 1 micro-SD plus a fair chunk of GPIO and other expansion. The Ethernet controller has a built-in switch, so you could maybe = connect another Ethernet PHY and connector via the expansion headers. = I'm still scratching my head over the right driver model for an Ethernet = controller that has two connectors with independent MACs and a = programmable switch engine between them and the host. Tim From owner-freebsd-arm@FreeBSD.ORG Tue Aug 28 13:48:53 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2CCE4106566C; Tue, 28 Aug 2012 13:48:53 +0000 (UTC) (envelope-from marktinguely@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id DDDF58FC17; Tue, 28 Aug 2012 13:48:51 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so9660554pbb.13 for ; Tue, 28 Aug 2012 06:48:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=P71mtBUeKJqJJ7uppZZUYzebtvvWMylC3ZFDuLWwN8U=; b=bHZLuzFKqvhefHZ1LPS5tIpUsqC75R1yub1u8o2tFUZoD5ipMaiREpoTrKdMT7Emxe dnj+MRbW+060ycWl/z6nYyRNRPZTFa448jZz9i2j6C9d9ucVZAoRZeVWkxflBRdoLPMg +/SnpzKDAqAgwiHQJwZoTo4N0mHy64YtmDaxMpnKRl0TJxFZovCxGuVOnie5nHCH/rE9 OneNADf1F1c1eEp3mNDK/8FrZeRFWghdsCcbVk747yOV5YDPmORpnyqMNlfCbVgNqcJF gDNyB6UKcpKjaNyb/zQeH/WsKEDkXVME05CdzU/QWMEhNo7GT3GF2v/ITheHdTzvYmRc C6yQ== MIME-Version: 1.0 Received: by 10.68.231.130 with SMTP id tg2mr3863853pbc.70.1346161731445; Tue, 28 Aug 2012 06:48:51 -0700 (PDT) Received: by 10.68.229.227 with HTTP; Tue, 28 Aug 2012 06:48:51 -0700 (PDT) In-Reply-To: <201208271531.42725.jhb@freebsd.org> References: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> <20120827185346.GE33100@deviant.kiev.zoral.com.ua> <201208271531.42725.jhb@freebsd.org> Date: Tue, 28 Aug 2012 08:48:51 -0500 Message-ID: From: Mark Tinguely To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org, Konstantin Belousov Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2012 13:48:53 -0000 On Mon, Aug 27, 2012 at 2:31 PM, John Baldwin wrote: > On Monday, August 27, 2012 2:53:46 pm Konstantin Belousov wrote: >> On Sun, Aug 26, 2012 at 05:13:31PM -0600, Warner Losh wrote: >> > >> > On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: >> > > In this regard, it's the busdma implementation that's broken, because it >> > > should bounce those IOs through a DMA-safe buffer. There's absolutely >> > > no rule that I've ever heard of in FreeBSD that says IO can only take >> > > place using memory allocated from busdma. >> > >> > That's partially true. Since BUSDMA grew up in the storage area, you >> > must allocate the memory from busdma, or it must be page aligned has >> > been the de-facto rule here. The mbuf and uio variants of load were >> > invented to cope with common cases of mbufs and user I/O to properly >> > flag things. >> >> I once looked at x86 bus_dmamap_load_uio(), and I was unable to >> understand how to use it with usermode uio. I think this is a good >> moment to ask. Most existing users use UIO_SYSSPACE, but several crypto >> drivers might allow the UIO_USERSPACE for them. >> >> For UIO_USERSPACE, if the page is not resident, the pmap_extract() call from >> _bus_dmamap_load_buffer() returns 0. So the i/o happens to the page >> located at 0, which contains real mode IVT and other BIOS sensitive tables. >> >> Worse, if the page is resident, but it is mapped at the region which >> requires COW on write, then DMA will be performed to the wrong page >> which is typically shared with other innocent users. to the COW area >> which was not yet copied, >> >> Am I missing some trick there ? > > No. The caller is required to wire the pages first in some manner. > In general bus_dmamap_load_uio() isn't a good idea. I do believe the > crypto drivers are careful to wire the buffer first. I think requiring > the caller to wire is the only sane way that can be used. > > Also, doing DMA to a stack variable is absolutely horrible for a related > reason since presumably the thread will block while it waits for the DMA > to complete, and a sleeping thread can be swapped out (including having > it's stack swapped out). > > -- > John Baldwin The POST commands for UIO DMA may be a bit hairy because the address space may not be the same as the caller. Someone once told me they got around that (in a Sparc enviroment?) by shadow mapping the address to KVA. --Mark TInguely. From owner-freebsd-arm@FreeBSD.ORG Tue Aug 28 14:31:27 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA758106566B; Tue, 28 Aug 2012 14:31:27 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 571348FC12; Tue, 28 Aug 2012 14:31:23 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7SEVMEj006002; Tue, 28 Aug 2012 08:31:22 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7SEVJ5w033171; Tue, 28 Aug 2012 08:31:19 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> <1346081557.1140.181.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Tue, 28 Aug 2012 08:31:19 -0600 Message-ID: <1346164279.1140.328.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, Hans Petter Selasky , freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2012 14:31:28 -0000 On Mon, 2012-08-27 at 09:57 -0600, Warner Losh wrote: > >> I don't think the cache line problem can be solved with bounce > buffers. Trying to accommodate broken drivers is what lead us to this > spot. We need to fix the broken drivers. If that's impossible, then > the best we can do is have the driver set a 'always bounce' flag in > the tag it creates and use that to always bounce for operations > through that tag. > > > > So you'd be okay with a driver setting a flag that says "always > bounce" > > but not okay with the busdma layer bouncing only when it's actually > > necessary? I'm confused -- do you think the busdma layer will be > unable > > to detect when it's necessary unless directed from the outside? > > Actually, it is good you are confused. I was wrong when I said I'd be > happy with a flag that says always bounce. The reason is that the > driver can do this all the time for those cases like the at91 mci > driver does. After studying the busdma implementation code and pondering this some more, I now understand why the automatic bouncing that I think is the ideal solution to this is so hard to accomplish. I'm not sure "Impossible" is the right description, but I'm afraid that in the long run "impractical" might turn out to be the case. I'm going to think all this over for a few days, maybe do a bit of actual paying work, before revisiting it. -- Ian From owner-freebsd-arm@FreeBSD.ORG Tue Aug 28 18:49:22 2012 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 055BB106564A for ; Tue, 28 Aug 2012 18:49:22 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh11.mail.rice.edu (mh11.mail.rice.edu [128.42.199.30]) by mx1.freebsd.org (Postfix) with ESMTP id C67388FC18 for ; Tue, 28 Aug 2012 18:49:21 +0000 (UTC) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id AE8CD4C03CF; Tue, 28 Aug 2012 13:49:19 -0500 (CDT) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id ACF2F4C03CE; Tue, 28 Aug 2012 13:49:19 -0500 (CDT) X-Virus-Scanned: by amavis-2.7.0 at mh11.mail.rice.edu, auth channel Received: from mh11.mail.rice.edu ([127.0.0.1]) by mh11.mail.rice.edu (mh11.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id quAHtbF3yR_7; Tue, 28 Aug 2012 13:49:19 -0500 (CDT) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh11.mail.rice.edu (Postfix) with ESMTPSA id 073BC4C03C8; Tue, 28 Aug 2012 13:49:18 -0500 (CDT) Message-ID: <503D12AE.1050705@rice.edu> Date: Tue, 28 Aug 2012 13:49:18 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:8.0) Gecko/20111113 Thunderbird/8.0 MIME-Version: 1.0 To: Ian Lepore References: <502FD67A.7030003@rice.edu> <1345315508.27688.260.camel@revolution.hippie.lan> In-Reply-To: <1345315508.27688.260.camel@revolution.hippie.lan> Content-Type: multipart/mixed; boundary="------------090303090800090407010201" Cc: "arm@freebsd.org" , Alan Cox Subject: Re: arm pmap locking X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2012 18:49:22 -0000 This is a multi-part message in MIME format. --------------090303090800090407010201 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 08/18/2012 13:45, Ian Lepore wrote: > On Sat, 2012-08-18 at 12:52 -0500, Alan Cox wrote: >> Can someone here please test the attached patch? I'm currently working >> my way through the various pmap implementations eliminating the use of >> the page queues lock. Arm is next on my list. >> >> Alan > I get a panic for recursion on my dreamplug, see attached. I applied > the patches over this: > > FreeBSD dpcur 10.0-CURRENT FreeBSD 10.0-CURRENT #4 r239193M: Sat Aug 18 > 12:37:24 MDT 2012 > > The 'M' is some dreamplug-specific changes (dts files, etc). I had a > quick look and it didn't seem like any of the checkins since r239193 > were related to arm pmap. I can try a fresher checkout when I have more > time if you think it's important. > > I tried adding WITNESS to see if it would get you more info, but it > panics for a LOR before getting to the panic in the attached output. > That's not new, it's been there for a while and I haven't had any time > to chase it down. > Can you please retry with the attached patch? For the time being, I decided to address the above problem by simply enabling recursion on the new pmap lock. As I mentioned in my prior message, the lock recursion in the arm pmap is a mistake. However, I'd rather not change two things at once, i.e., replace the page queues lock and fix the lock recursion. I'll take a look at eliminating the lock recursion later this week. Thanks, Alan --------------090303090800090407010201 Content-Type: text/plain; name="arm_pmap3.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="arm_pmap3.patch" Index: arm/arm/pmap.c =================================================================== --- arm/arm/pmap.c (revision 239681) +++ arm/arm/pmap.c (working copy) @@ -150,6 +150,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -408,6 +409,7 @@ static vm_offset_t pmap_kernel_l2ptp_kva; static vm_paddr_t pmap_kernel_l2ptp_phys; static struct vm_object pvzone_obj; static int pv_entry_count=0, pv_entry_max=0, pv_entry_high_water=0; +static struct rwlock pvh_global_lock; /* * This list exists for the benefit of pmap_map_chunk(). It keeps track @@ -866,7 +868,7 @@ pmap_alloc_l2_bucket(pmap_t pm, vm_offset_t va) l1idx = L1_IDX(va); PMAP_ASSERT_LOCKED(pm); - mtx_assert(&vm_page_queue_mtx, MA_OWNED); + rw_assert(&pvh_global_lock, RA_WLOCKED); if ((l2 = pm->pm_l2[L2_IDX(l1idx)]) == NULL) { /* * No mapping at this address, as there is @@ -875,19 +877,19 @@ pmap_alloc_l2_bucket(pmap_t pm, vm_offset_t va) */ again_l2table: PMAP_UNLOCK(pm); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); if ((l2 = pmap_alloc_l2_dtable()) == NULL) { - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); return (NULL); } - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); if (pm->pm_l2[L2_IDX(l1idx)] != NULL) { PMAP_UNLOCK(pm); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); uma_zfree(l2table_zone, l2); - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); l2 = pm->pm_l2[L2_IDX(l1idx)]; if (l2 == NULL) @@ -919,16 +921,16 @@ again_l2table: */ again_ptep: PMAP_UNLOCK(pm); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); ptep = (void*)uma_zalloc(l2zone, M_NOWAIT|M_USE_RESERVE); - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); if (l2b->l2b_kva != 0) { /* We lost the race. */ PMAP_UNLOCK(pm); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); uma_zfree(l2zone, ptep); - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); if (l2b->l2b_kva == 0) goto again_ptep; @@ -1314,7 +1316,7 @@ pmap_fix_cache(struct vm_page *pg, pmap_t pm, vm_o int entries = 0, kentries = 0, uentries = 0; struct pv_entry *pv; - mtx_assert(&vm_page_queue_mtx, MA_OWNED); + rw_assert(&pvh_global_lock, RA_WLOCKED); /* the cache gets written back/invalidated on context switch. * therefore, if a user page shares an entry in the same page or @@ -1426,7 +1428,7 @@ pmap_clearbit(struct vm_page *pg, u_int maskbits) u_int oflags; int count = 0; - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); if (maskbits & PVF_WRITE) maskbits |= PVF_MOD; @@ -1436,7 +1438,7 @@ pmap_clearbit(struct vm_page *pg, u_int maskbits) pg->md.pvh_attrs &= ~(maskbits & (PVF_MOD | PVF_REF)); if (TAILQ_EMPTY(&pg->md.pv_list)) { - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); return (0); } @@ -1572,7 +1574,7 @@ pmap_clearbit(struct vm_page *pg, u_int maskbits) if (maskbits & PVF_WRITE) vm_page_aflag_clear(pg, PGA_WRITEABLE); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); return (count); } @@ -1601,7 +1603,7 @@ pmap_enter_pv(struct vm_page *pg, struct pv_entry int km; - mtx_assert(&vm_page_queue_mtx, MA_OWNED); + rw_assert(&pvh_global_lock, RA_WLOCKED); if (pg->md.pv_kva) { /* PMAP_ASSERT_LOCKED(pmap_kernel()); */ @@ -1615,10 +1617,10 @@ pmap_enter_pv(struct vm_page *pg, struct pv_entry TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list); TAILQ_INSERT_HEAD(&pve->pv_pmap->pm_pvlist, pve, pv_plist); PMAP_UNLOCK(pmap_kernel()); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); if ((pve = pmap_get_pv_entry()) == NULL) panic("pmap_kenter_internal: no pv entries"); - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); if (km) PMAP_LOCK(pmap_kernel()); } @@ -1647,7 +1649,7 @@ pmap_find_pv(struct vm_page *pg, pmap_t pm, vm_off { struct pv_entry *pv; - mtx_assert(&vm_page_queue_mtx, MA_OWNED); + rw_assert(&pvh_global_lock, RA_WLOCKED); TAILQ_FOREACH(pv, &pg->md.pv_list, pv_list) if (pm == pv->pv_pmap && va == pv->pv_va) break; @@ -1691,7 +1693,7 @@ pmap_nuke_pv(struct vm_page *pg, pmap_t pm, struct { struct pv_entry *pv; - mtx_assert(&vm_page_queue_mtx, MA_OWNED); + rw_assert(&pvh_global_lock, RA_WLOCKED); PMAP_ASSERT_LOCKED(pm); TAILQ_REMOVE(&pg->md.pv_list, pve, pv_list); TAILQ_REMOVE(&pm->pm_pvlist, pve, pv_plist); @@ -1737,7 +1739,7 @@ pmap_remove_pv(struct vm_page *pg, pmap_t pm, vm_o { struct pv_entry *pve; - mtx_assert(&vm_page_queue_mtx, MA_OWNED); + rw_assert(&pvh_global_lock, RA_WLOCKED); pve = TAILQ_FIRST(&pg->md.pv_list); while (pve) { @@ -1771,7 +1773,7 @@ pmap_modify_pv(struct vm_page *pg, pmap_t pm, vm_o u_int flags, oflags; PMAP_ASSERT_LOCKED(pm); - mtx_assert(&vm_page_queue_mtx, MA_OWNED); + rw_assert(&pvh_global_lock, RA_WLOCKED); if ((npv = pmap_find_pv(pg, pm, va)) == NULL) return (0); @@ -1878,7 +1880,7 @@ pmap_fault_fixup(pmap_t pm, vm_offset_t va, vm_pro int rv = 0; l1idx = L1_IDX(va); - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); /* @@ -2075,7 +2077,7 @@ pmap_fault_fixup(pmap_t pm, vm_offset_t va, vm_pro rv = 1; out: - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pm); return (rv); } @@ -2400,6 +2402,11 @@ pmap_bootstrap(vm_offset_t firstaddr, vm_offset_t CPU_FILL(&kernel_pmap->pm_active); kernel_pmap->pm_domain = PMAP_DOMAIN_KERNEL; TAILQ_INIT(&kernel_pmap->pm_pvlist); + + /* + * Initialize the global pv list lock. + */ + rw_init_flags(&pvh_global_lock, "pmap pv global", RW_RECURSE); /* * Reserve some special page table entries/VA space for temporary @@ -2672,7 +2679,7 @@ pmap_remove_pages(pmap_t pmap) vm_page_t m; pt_entry_t *pt; - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); cpu_idcache_wbinv_all(); cpu_l2cache_wbinv_all(); @@ -2701,7 +2708,7 @@ pmap_remove_pages(pmap_t pmap) pmap_free_pv_entry(pv); pmap_free_l2_bucket(pmap, l2b, 1); } - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); cpu_tlb_flushID(); cpu_cpwait(); PMAP_UNLOCK(pmap); @@ -2826,13 +2833,13 @@ pmap_kenter_internal(vm_offset_t va, vm_offset_t p * This expects the physical memory to have vm_page_array entry. */ if (pvzone != NULL && (m = vm_phys_paddr_to_vm_page(pa))) { - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); if (!TAILQ_EMPTY(&m->md.pv_list) || m->md.pv_kva) { /* release vm_page lock for pv_entry UMA */ - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); if ((pve = pmap_get_pv_entry()) == NULL) panic("pmap_kenter_internal: no pv entries"); - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap_kernel()); pmap_enter_pv(m, pve, pmap_kernel(), va, PVF_WRITE | PVF_UNMAN); @@ -2841,7 +2848,7 @@ pmap_kenter_internal(vm_offset_t va, vm_offset_t p } else { m->md.pv_kva = va; } - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); } } @@ -2902,13 +2909,13 @@ pmap_kremove(vm_offset_t va) /* note: should never have to remove an allocation * before the pvzone is initialized. */ - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap_kernel()); if (pvzone != NULL && (m = vm_phys_paddr_to_vm_page(pa)) && (pve = pmap_remove_pv(m, pmap_kernel(), va))) pmap_free_pv_entry(pve); PMAP_UNLOCK(pmap_kernel()); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); va = va & ~PAGE_MASK; cpu_dcache_wbinv_range(va, PAGE_SIZE); cpu_l2cache_wbinv_range(va, PAGE_SIZE); @@ -3126,7 +3133,7 @@ pmap_remove_all(vm_page_t m) ("pmap_remove_all: page %p is not managed", m)); if (TAILQ_EMPTY(&m->md.pv_list)) return; - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); pmap_remove_write(m); curpm = vmspace_pmap(curproc->p_vmspace); while ((pv = TAILQ_FIRST(&m->md.pv_list)) != NULL) { @@ -3175,7 +3182,7 @@ pmap_remove_all(vm_page_t m) pmap_tlb_flushD(curpm); } vm_page_aflag_clear(m, PGA_WRITEABLE); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); } @@ -3208,7 +3215,7 @@ pmap_protect(pmap_t pm, vm_offset_t sva, vm_offset return; } - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); /* @@ -3275,7 +3282,7 @@ pmap_protect(pmap_t pm, vm_offset_t sva, vm_offset if (PV_BEEN_REFD(flags)) pmap_tlb_flushD(pm); } - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pm); } @@ -3299,10 +3306,10 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot_t vm_prot_t prot, boolean_t wired) { - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); pmap_enter_locked(pmap, va, m, prot, wired, M_WAITOK); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); } @@ -3322,7 +3329,7 @@ pmap_enter_locked(pmap_t pmap, vm_offset_t va, vm_ vm_paddr_t pa; PMAP_ASSERT_LOCKED(pmap); - mtx_assert(&vm_page_queue_mtx, MA_OWNED); + rw_assert(&pvh_global_lock, RA_WLOCKED); if (va == vector_page) { pa = systempage.pv_pa; m = NULL; @@ -3352,9 +3359,9 @@ do_l2b_alloc: if (l2b == NULL) { if (flags & M_WAITOK) { PMAP_UNLOCK(pmap); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); VM_WAIT; - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); goto do_l2b_alloc; } @@ -3594,14 +3601,14 @@ pmap_enter_object(pmap_t pmap, vm_offset_t start, psize = atop(end - start); m = m_start; - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); while (m != NULL && (diff = m->pindex - m_start->pindex) < psize) { pmap_enter_locked(pmap, start + ptoa(diff), m, prot & (VM_PROT_READ | VM_PROT_EXECUTE), FALSE, M_NOWAIT); m = TAILQ_NEXT(m, listq); } - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); } @@ -3618,11 +3625,11 @@ void pmap_enter_quick(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) { - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); pmap_enter_locked(pmap, va, m, prot & (VM_PROT_READ | VM_PROT_EXECUTE), FALSE, M_NOWAIT); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); } @@ -3640,7 +3647,7 @@ pmap_change_wiring(pmap_t pmap, vm_offset_t va, bo pt_entry_t *ptep, pte; vm_page_t pg; - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); l2b = pmap_get_l2_bucket(pmap, va); KASSERT(l2b, ("No l2b bucket in pmap_change_wiring")); @@ -3649,7 +3656,7 @@ pmap_change_wiring(pmap_t pmap, vm_offset_t va, bo pg = PHYS_TO_VM_PAGE(l2pte_pa(pte)); if (pg) pmap_modify_pv(pg, pmap, va, PVF_WIRED, wired ? PVF_WIRED : 0); - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); } @@ -3895,7 +3902,7 @@ pmap_remove(pmap_t pm, vm_offset_t sva, vm_offset_ * we lock in the pmap => pv_head direction */ - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); total = 0; while (sva < eva) { @@ -3989,7 +3996,7 @@ pmap_remove(pmap_t pm, vm_offset_t sva, vm_offset_ pmap_free_l2_bucket(pm, l2b, mappings); } - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); if (flushall) cpu_tlb_flushID(); PMAP_UNLOCK(pm); @@ -4405,7 +4412,7 @@ pmap_page_exists_quick(pmap_t pmap, vm_page_t m) KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_page_exists_quick: page %p is not managed", m)); rv = FALSE; - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); TAILQ_FOREACH(pv, &m->md.pv_list, pv_list) { if (pv->pv_pmap == pmap) { rv = TRUE; @@ -4415,7 +4422,7 @@ pmap_page_exists_quick(pmap_t pmap, vm_page_t m) if (loops >= 16) break; } - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); return (rv); } @@ -4434,11 +4441,11 @@ pmap_page_wired_mappings(vm_page_t m) count = 0; if ((m->oflags & VPO_UNMANAGED) != 0) return (count); - vm_page_lock_queues(); + rw_wlock(&pvh_global_lock); TAILQ_FOREACH(pv, &m->md.pv_list, pv_list) if ((pv->pv_flags & PVF_WIRED) != 0) count++; - vm_page_unlock_queues(); + rw_wunlock(&pvh_global_lock); return (count); } --------------090303090800090407010201-- From owner-freebsd-arm@FreeBSD.ORG Wed Aug 29 00:02:13 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63243106566C for ; Wed, 29 Aug 2012 00:02:13 +0000 (UTC) (envelope-from damjan.marion@gmail.com) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by mx1.freebsd.org (Postfix) with ESMTP id DE8408FC18 for ; Wed, 29 Aug 2012 00:02:12 +0000 (UTC) Received: by wicr5 with SMTP id r5so4214226wic.13 for ; Tue, 28 Aug 2012 17:02:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=9bVOACxEsqmoIi/2/Xqoz7r0rM2vBzGeGpgfCBHsFFA=; b=sbYSQ3YdISCwj+f7WdrKCRWSsxWSJ2Te5mrjsT16qz+trF6Obs2ZZJ79kF5FeCOE0t Pq8haUpy4OmobG+GYyfrYKAN8AT0MKvIZlN9ujPaFJVPlaI2g9x1zxfBeuDrC5H1P2zw XmAmQAY/zXLBtITqZAt0BhbRxHa0ILmcJQW8+ouS8BE0JL/dXpGW4DK+kyi79t8ZTndd RHZ1rWixVvnyWtecJbTMhtRGTBxODogsPoV0wYQkwTAa5+jXPN5rpZzw1idAYZmrCTdX mH0sJG4KbaxGspyNAoFLjyotAzhMTtLaZ42JirozaBqBZ3gH6jPjBtKhuLC1q2NjtMF6 1qrg== Received: by 10.180.107.103 with SMTP id hb7mr36254414wib.3.1346198531432; Tue, 28 Aug 2012 17:02:11 -0700 (PDT) Received: from damarion-mac-wl.home (cpe-109-60-84-35.zg3.cable.xnet.hr. [109.60.84.35]) by mx.google.com with ESMTPS id j6sm11150994wiy.4.2012.08.28.17.02.09 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 28 Aug 2012 17:02:10 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1486\)) From: Damjan Marion In-Reply-To: <716FE9A8-EC5B-41F4-9AE2-5302EEBB86B8@kientzle.com> Date: Wed, 29 Aug 2012 02:02:08 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20120827174004.GD58312@funkthat.com> <1E1B5E39-A2FC-437D-A759-BF75B680EFC5@neville-neil.com> <68F9301B-8B03-47D3-A0AA-9F2490D3481B@bsdimp.com> <716FE9A8-EC5B-41F4-9AE2-5302EEBB86B8@kientzle.com> To: Tim Kientzle X-Mailer: Apple Mail (2.1486) Cc: George Neville-Neil , freebsd-arm@freebsd.org Subject: Re: Avila GW2348-4 and VLANs not working X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 00:02:13 -0000 On Aug 28, 2012, at 6:31 AM, Tim Kientzle wrote: > The Ethernet controller has a built-in switch, so you could maybe = connect another Ethernet PHY and connector via the expansion headers. = I'm still scratching my head over the right driver model for an Ethernet = controller that has two connectors with independent MACs and a = programmable switch engine between them and the host. Have you take a look into sys/dev/etherswitch/rtl8366 ? I guess we need something similar.... Damjan From owner-freebsd-arm@FreeBSD.ORG Thu Aug 30 18:12:58 2012 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 992E21065675 for ; Thu, 30 Aug 2012 18:12:58 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 4C4288FC15 for ; Thu, 30 Aug 2012 18:12:57 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7UICuOR050842 for ; Thu, 30 Aug 2012 12:12:57 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7UICsIv036066; Thu, 30 Aug 2012 12:12:54 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Alan Cox In-Reply-To: <503D12AE.1050705@rice.edu> References: <502FD67A.7030003@rice.edu> <1345315508.27688.260.camel@revolution.hippie.lan> <503D12AE.1050705@rice.edu> Content-Type: text/plain; charset="us-ascii" Date: Thu, 30 Aug 2012 12:12:54 -0600 Message-ID: <1346350374.1140.525.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: "arm@freebsd.org" Subject: Re: arm pmap locking X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2012 18:12:58 -0000 On Tue, 2012-08-28 at 13:49 -0500, Alan Cox wrote: > Can you please retry with the attached patch? For the time being, I > decided to address the above problem by simply enabling recursion on the > new pmap lock. As I mentioned in my prior message, the lock recursion > in the arm pmap is a mistake. However, I'd rather not change two things > at once, i.e., replace the page queues lock and fix the lock recursion. > I'll take a look at eliminating the lock recursion later this week. > > Thanks, > Alan > Sorry for the delay, I finally got around to trying this today, and it seems to be working well initially -- it boots to multiuser and the only difference in the dmesg.boot with and without the patch is the compile date, and the kernel image is 128 bytes smaller with the patch. I've got DIAGNOSTIC and INVARIANTS enabled; I'll run with the patch in place and let you know if anything glitches. -- Ian From owner-freebsd-arm@FreeBSD.ORG Fri Aug 31 03:06:55 2012 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 16C2E106566C for ; Fri, 31 Aug 2012 03:06:55 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh11.mail.rice.edu (mh11.mail.rice.edu [128.42.199.30]) by mx1.freebsd.org (Postfix) with ESMTP id DC0748FC14 for ; Fri, 31 Aug 2012 03:06:54 +0000 (UTC) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 056064C02A2; Thu, 30 Aug 2012 22:06:53 -0500 (CDT) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 040334C029C; Thu, 30 Aug 2012 22:06:53 -0500 (CDT) X-Virus-Scanned: by amavis-2.7.0 at mh11.mail.rice.edu, auth channel Received: from mh11.mail.rice.edu ([127.0.0.1]) by mh11.mail.rice.edu (mh11.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id NY4HIJsdKPGn; Thu, 30 Aug 2012 22:06:52 -0500 (CDT) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh11.mail.rice.edu (Postfix) with ESMTPSA id AABA24C0272; Thu, 30 Aug 2012 22:06:52 -0500 (CDT) Message-ID: <50402A4A.9070407@rice.edu> Date: Thu, 30 Aug 2012 22:06:50 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:8.0) Gecko/20111113 Thunderbird/8.0 MIME-Version: 1.0 To: Ian Lepore References: <502FD67A.7030003@rice.edu> <1345315508.27688.260.camel@revolution.hippie.lan> <503D12AE.1050705@rice.edu> <1346350374.1140.525.camel@revolution.hippie.lan> In-Reply-To: <1346350374.1140.525.camel@revolution.hippie.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "arm@freebsd.org" Subject: Re: arm pmap locking X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2012 03:06:55 -0000 On 08/30/2012 13:12, Ian Lepore wrote: > On Tue, 2012-08-28 at 13:49 -0500, Alan Cox wrote: >> Can you please retry with the attached patch? For the time being, I >> decided to address the above problem by simply enabling recursion on the >> new pmap lock. As I mentioned in my prior message, the lock recursion >> in the arm pmap is a mistake. However, I'd rather not change two things >> at once, i.e., replace the page queues lock and fix the lock recursion. >> I'll take a look at eliminating the lock recursion later this week. >> >> Thanks, >> Alan >> > Sorry for the delay, I finally got around to trying this today, and it > seems to be working well initially -- it boots to multiuser and the only > difference in the dmesg.boot with and without the patch is the compile > date, and the kernel image is 128 bytes smaller with the patch. I've > got DIAGNOSTIC and INVARIANTS enabled; I'll run with the patch in place > and let you know if anything glitches. > Thanks! Alan From owner-freebsd-arm@FreeBSD.ORG Fri Aug 31 05:53:54 2012 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5F0E2106566B; Fri, 31 Aug 2012 05:53:54 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-legacy2.sentex.ca (freebsd-legacy2.sentex.ca [IPv6:2607:f3e0:0:3::6502:9c]) by mx1.freebsd.org (Postfix) with ESMTP id 1FB188FC12; Fri, 31 Aug 2012 05:53:54 +0000 (UTC) Received: from freebsd-legacy2.sentex.ca (localhost [127.0.0.1]) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5) with ESMTP id q7V5rrbQ093712; Fri, 31 Aug 2012 05:53:53 GMT (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-legacy2.sentex.ca (8.14.5/8.14.5/Submit) id q7V5rrSV093695; Fri, 31 Aug 2012 05:53:53 GMT (envelope-from tinderbox@freebsd.org) Date: Fri, 31 Aug 2012 05:53:53 GMT Message-Id: <201208310553.q7V5rrSV093695@freebsd-legacy2.sentex.ca> X-Authentication-Warning: freebsd-legacy2.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Precedence: bulk Cc: Subject: [releng_8 tinderbox] failure on arm/arm X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2012 05:53:54 -0000 TB --- 2012-08-31 05:22:37 - tinderbox 2.9 running on freebsd-legacy2.sentex.ca TB --- 2012-08-31 05:22:37 - FreeBSD freebsd-legacy2.sentex.ca 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2012-08-31 05:22:37 - starting RELENG_8 tinderbox run for arm/arm TB --- 2012-08-31 05:22:37 - cleaning the object tree TB --- 2012-08-31 05:22:37 - cvsupping the source tree TB --- 2012-08-31 05:22:37 - /usr/bin/csup -z -r 3 -g -L 1 -h cvsup.sentex.ca /tinderbox/RELENG_8/arm/arm/supfile TB --- 2012-08-31 05:22:54 - building world TB --- 2012-08-31 05:22:54 - CROSS_BUILD_TESTING=YES TB --- 2012-08-31 05:22:54 - MAKEOBJDIRPREFIX=/obj TB --- 2012-08-31 05:22:54 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2012-08-31 05:22:54 - SRCCONF=/dev/null TB --- 2012-08-31 05:22:54 - TARGET=arm TB --- 2012-08-31 05:22:54 - TARGET_ARCH=arm TB --- 2012-08-31 05:22:54 - TZ=UTC TB --- 2012-08-31 05:22:54 - __MAKE_CONF=/dev/null TB --- 2012-08-31 05:22:54 - cd /src TB --- 2012-08-31 05:22:54 - /usr/bin/make -B buildworld >>> World build started on Fri Aug 31 05:22:55 UTC 2012 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything [...] : multiple definition of `SHA512_Update' /obj/arm/src/tmp/usr/lib/libmd.a(sha512c.o)(.text+0xb0d4): first defined here /obj/arm/src/tmp/usr/lib/libcrypto.a(sha512.o)(.text+0x55d8): In function `SHA512_Final': : multiple definition of `SHA512_Final' /obj/arm/src/tmp/usr/lib/libmd.a(sha512c.o)(.text+0xb244): first defined here /obj/arm/src/tmp/usr/bin/ld: Warning: size of symbol `SHA512_Final' changed from 228 in /obj/arm/src/tmp/usr/lib/libmd.a(sha512c.o) to 536 in /obj/arm/src/tmp/usr/lib/libmd.a(sha512c.o) nc.lo(.text+0x18e0): In function `$a': : warning: warning: mktemp() possibly used unsafely; consider using mkstemp() *** Error code 1 Stop in /obj/arm/src/rescue/rescue. *** Error code 1 Stop in /src/rescue/rescue. *** Error code 1 Stop in /src/rescue. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. TB --- 2012-08-31 05:53:53 - WARNING: /usr/bin/make returned exit code 1 TB --- 2012-08-31 05:53:53 - ERROR: failed to build world TB --- 2012-08-31 05:53:53 - 1418.01 user 325.89 system 1876.52 real http://tinderbox.freebsd.org/tinderbox-releng_8-RELENG_8-arm-arm.full From owner-freebsd-arm@FreeBSD.ORG Fri Aug 31 08:45:33 2012 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9C5881065673; Fri, 31 Aug 2012 08:45:33 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-stable.sentex.ca (freebsd-stable.sentex.ca [IPv6:2607:f3e0:0:3::6502:9b]) by mx1.freebsd.org (Postfix) with ESMTP id 5D9798FC15; Fri, 31 Aug 2012 08:45:33 +0000 (UTC) Received: from freebsd-stable.sentex.ca (localhost [127.0.0.1]) by freebsd-stable.sentex.ca (8.14.5/8.14.5) with ESMTP id q7V8jW9M065082; Fri, 31 Aug 2012 08:45:32 GMT (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-stable.sentex.ca (8.14.5/8.14.5/Submit) id q7V8jWgD065081; Fri, 31 Aug 2012 08:45:32 GMT (envelope-from tinderbox@freebsd.org) Date: Fri, 31 Aug 2012 08:45:32 GMT Message-Id: <201208310845.q7V8jWgD065081@freebsd-stable.sentex.ca> X-Authentication-Warning: freebsd-stable.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Precedence: bulk Cc: Subject: [releng_9 tinderbox] failure on arm/arm X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2012 08:45:33 -0000 TB --- 2012-08-31 07:45:00 - tinderbox 2.9 running on freebsd-stable.sentex.ca TB --- 2012-08-31 07:45:00 - FreeBSD freebsd-stable.sentex.ca 8.2-STABLE FreeBSD 8.2-STABLE #4: Wed Sep 28 13:48:49 UTC 2011 mdtancsa@freebsd-stable.sentex.ca:/usr/obj/usr/src/sys/server amd64 TB --- 2012-08-31 07:45:00 - starting RELENG_9 tinderbox run for arm/arm TB --- 2012-08-31 07:45:00 - cleaning the object tree TB --- 2012-08-31 07:45:00 - cvsupping the source tree TB --- 2012-08-31 07:45:00 - /usr/bin/csup -z -r 3 -g -L 1 -h cvsup.sentex.ca /tinderbox/RELENG_9/arm/arm/supfile TB --- 2012-08-31 07:46:21 - building world TB --- 2012-08-31 07:46:21 - CROSS_BUILD_TESTING=YES TB --- 2012-08-31 07:46:21 - MAKEOBJDIRPREFIX=/obj TB --- 2012-08-31 07:46:21 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2012-08-31 07:46:21 - SRCCONF=/dev/null TB --- 2012-08-31 07:46:21 - TARGET=arm TB --- 2012-08-31 07:46:21 - TARGET_ARCH=arm TB --- 2012-08-31 07:46:21 - TZ=UTC TB --- 2012-08-31 07:46:21 - __MAKE_CONF=/dev/null TB --- 2012-08-31 07:46:21 - cd /src TB --- 2012-08-31 07:46:21 - /usr/bin/make -B buildworld >>> World build started on Fri Aug 31 07:46:22 UTC 2012 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything [...] /obj/arm.arm/src/tmp/usr/lib/libcrypto.a(sha512.o): In function `SHA512_Update': sha512.c:(.text+0x5454): multiple definition of `SHA512_Update' /obj/arm.arm/src/tmp/usr/lib/libmd.a(sha512c.o):sha512c.c:(.text+0xb0b4): first defined here /obj/arm.arm/src/tmp/usr/lib/libcrypto.a(sha512.o): In function `SHA512_Final': sha512.c:(.text+0x55d8): multiple definition of `SHA512_Final' /obj/arm.arm/src/tmp/usr/lib/libmd.a(sha512c.o):sha512c.c:(.text+0xb224): first defined here nc.lo: In function `_$$hide$$ nc.lo $a': _$$hide$$ nc.lo socks.c:(.text+0x18e0): warning: warning: mktemp() possibly used unsafely; consider using mkstemp() *** Error code 1 Stop in /obj/arm.arm/src/rescue/rescue. *** Error code 1 Stop in /src/rescue/rescue. *** Error code 1 Stop in /src/rescue. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. TB --- 2012-08-31 08:45:32 - WARNING: /usr/bin/make returned exit code 1 TB --- 2012-08-31 08:45:32 - ERROR: failed to build world TB --- 2012-08-31 08:45:32 - 2100.91 user 497.43 system 3632.34 real http://tinderbox.freebsd.org/tinderbox-releng_9-RELENG_9-arm-arm.full