From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 08:49:32 2012 Return-Path: Delivered-To: mips@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09419106566C for ; Sun, 26 Aug 2012 08:49:32 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id A449F8FC08 for ; Sun, 26 Aug 2012 08:49:28 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id 1684B46B0A for ; Sun, 26 Aug 2012 04:49:28 -0400 (EDT) Date: Sun, 26 Aug 2012 09:49:27 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: mips@FreeBSD.org Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: FYI: syscons on MIPS (was: svn commit: r239670 - in head/sys: conf dev/fb dev/syscons mips/mips (fwd)) X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 08:49:32 -0000 Dear all: Just an FYI that I've added the required glue to let syscons work with MIPS (this plus the follow-up commit r239684 adding a header file). We're using it here with a VGA-like text frame buffer pretty happily. There are quite a few loose ends remaining in syscons for non-x86 platforms -- in particular, ones that require busspace be used for memory-mapped device access. Not sure if there are other FreeBSD-supporting MIPS devices around that could use syscons -- gxemul, I expect, but perhaps actual devices? Robert ---------- Forwarded message ---------- Date: Sat, 25 Aug 2012 08:09:38 +0000 (UTC) From: Robert Watson To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r239670 - in head/sys: conf dev/fb dev/syscons mips/mips Author: rwatson Date: Sat Aug 25 08:09:37 2012 New Revision: 239670 URL: http://svn.freebsd.org/changeset/base/239670 Log: Provide basic glue to allow syscons to be used on MIPS, modelled on PowerPC support. This was clearly not something syscons was designed to do (very specific assumptions about the nature of VGA consoles on PCs), but fortunately others have long since blazed the way on making it work regardless of that. Sponsored by: DARPA, AFRL Added: head/sys/mips/mips/sc_machdep.c (contents, props changed) Modified: head/sys/conf/files.mips head/sys/conf/options.mips head/sys/dev/fb/fbreg.h head/sys/dev/syscons/schistory.c head/sys/dev/syscons/scterm-teken.c head/sys/dev/syscons/syscons.c Modified: head/sys/conf/files.mips ============================================================================== --- head/sys/conf/files.mips Sat Aug 25 08:02:46 2012 (r239669) +++ head/sys/conf/files.mips Sat Aug 25 08:09:37 2012 (r239670) @@ -122,3 +122,9 @@ dev/ofw/ofw_fdt.c optional fdt dev/fdt/fdt_mips.c optional fdt +dev/fb/fb.c optional sc +dev/kbd/kbd.c optional sc +dev/syscons/scgfbrndr.c optional sc +dev/syscons/scterm-teken.c optional sc +dev/syscons/scvtb.c optional sc +mips/mips/sc_machdep.c optional sc Modified: head/sys/conf/options.mips ============================================================================== --- head/sys/conf/options.mips Sat Aug 25 08:02:46 2012 (r239669) +++ head/sys/conf/options.mips Sat Aug 25 08:09:37 2012 (r239670) @@ -46,6 +46,10 @@ CFE_CONSOLE opt_global.h CFE_ENV opt_global.h CFE_ENV_SIZE opt_global.h +GFB_DEBUG opt_gfb.h +GFB_NO_FONT_LOADING opt_gfb.h +GFB_NO_MODE_CHANGE opt_gfb.h + NOFPU opt_global.h TICK_USE_YAMON_FREQ opt_global.h Modified: head/sys/dev/fb/fbreg.h ============================================================================== --- head/sys/dev/fb/fbreg.h Sat Aug 25 08:02:46 2012 (r239669) +++ head/sys/dev/fb/fbreg.h Sat Aug 25 08:09:37 2012 (r239670) @@ -92,6 +92,29 @@ void ofwfb_fillw(int pat, void *base, si u_int16_t ofwfb_readw(u_int16_t *addr); void ofwfb_writew(u_int16_t *addr, u_int16_t val); +#elif defined(__mips__) + +/* + * Use amd64/i386-like settings under the assumption that MIPS-based display + * drivers will have to add a level of indirection between a syscons-managed + * frame buffer and the actual video hardware. We are forced to do this + * because syscons doesn't carry around required busspace handles and tags to + * use here. This is only really a problem for true VGA devices hooked up to + * MIPS, as others will be performing a translation anyway. + */ +#define bcopy_io(s, d, c) memcpy((void *)(d), (void *)(s), (c)) +#define bcopy_toio(s, d, c) memcpy((void *)(d), (void *)(s), (c)) +#define bcopy_fromio(s, d, c) memcpy((void *)(d), (void *)(s), (c)) +#define bzero_io(d, c) memset((void *)(d), 0, (c)) +#define fill_io(p, d, c) memset((void *)(d), (p), (c)) +static __inline void +fillw(int val, uint16_t *buf, size_t size) +{ + while (size--) + *buf++ = val; +} +#define fillw_io(p, d, c) fillw((p), (void *)(d), (c)) + #else /* !__i386__ && !__amd64__ && !__ia64__ && !__sparc64__ && !__powerpc__ */ #define bcopy_io(s, d, c) memcpy_io((d), (s), (c)) #define bcopy_toio(s, d, c) memcpy_toio((d), (void *)(s), (c)) Modified: head/sys/dev/syscons/schistory.c ============================================================================== --- head/sys/dev/syscons/schistory.c Sat Aug 25 08:02:46 2012 (r239669) +++ head/sys/dev/syscons/schistory.c Sat Aug 25 08:09:37 2012 (r239670) @@ -42,7 +42,7 @@ __FBSDID("$FreeBSD$"); #include #include -#if defined(__sparc64__) || defined(__powerpc__) +#if defined(__sparc64__) || defined(__powerpc__) || defined(__mips__) #include #else #include Modified: head/sys/dev/syscons/scterm-teken.c ============================================================================== --- head/sys/dev/syscons/scterm-teken.c Sat Aug 25 08:02:46 2012 (r239669) +++ head/sys/dev/syscons/scterm-teken.c Sat Aug 25 08:09:37 2012 (r239670) @@ -40,7 +40,7 @@ __FBSDID("$FreeBSD$"); #include #include -#if defined(__sparc64__) || defined(__powerpc__) +#if defined(__sparc64__) || defined(__powerpc__) || defined(__mips__) #include #else #include Modified: head/sys/dev/syscons/syscons.c ============================================================================== --- head/sys/dev/syscons/syscons.c Sat Aug 25 08:02:46 2012 (r239669) +++ head/sys/dev/syscons/syscons.c Sat Aug 25 08:09:37 2012 (r239670) @@ -62,7 +62,7 @@ __FBSDID("$FreeBSD$"); #include #include -#if defined(__sparc64__) || defined(__powerpc__) +#if defined(__sparc64__) || defined(__powerpc__) || defined(__mips__) #include #else #include Added: head/sys/mips/mips/sc_machdep.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/mips/mips/sc_machdep.c Sat Aug 25 08:09:37 2012 (r239670) @@ -0,0 +1,90 @@ +/*- + * Copyright (c) 2003 Jake Burkholder. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +static sc_softc_t sc_softcs[8]; + +int +sc_get_cons_priority(int *unit, int *flags) +{ + + *unit = 0; + *flags = 0; + return (CN_INTERNAL); +} + +int +sc_max_unit(void) +{ + return (1); +} + +sc_softc_t * +sc_get_softc(int unit, int flags) +{ + sc_softc_t *sc; + + if (unit < 0) + return (NULL); + sc = &sc_softcs[unit]; + sc->unit = unit; + if ((sc->flags & SC_INIT_DONE) == 0) { + sc->keyboard = -1; + sc->adapter = -1; + sc->cursor_char = SC_CURSOR_CHAR; + sc->mouse_char = SC_MOUSE_CHAR; + } + return (sc); +} + +void +sc_get_bios_values(bios_values_t *values) +{ + values->cursor_start = 0; + values->cursor_end = 32; + values->shift_state = 0; +} + +int +sc_tone(int hz) +{ + return (0); +} From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 17:42:23 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BA3D91065670; Sun, 26 Aug 2012 17:42:23 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 0634D8FC20; Sun, 26 Aug 2012 17:42:06 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7QHg5RZ067335; Sun, 26 Aug 2012 11:42:05 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7QHg2M1029199; Sun, 26 Aug 2012 11:42:02 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sun, 26 Aug 2012 11:42:02 -0600 Message-ID: <1346002922.1140.56.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 17:42:23 -0000 On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: > The bottom line is that you can't mix things like that when cache > lines are involved. The current code that tries is doomed to failure. > Doomed. You just can't control all flushes, as Ian's missive > demonstrates, and trying to accommodate code that does this I don't > think can possibly work. All the interrupt masking, copying in and > out, etc I fear is doomed to utter and abject failure. > Until last weekend I was in the camp that thought the partial cacheline flush problem was solvable with sufficiently clever code. Now I agree that we're doomed to failure and it's time to try another direction. We're going to have some implementation work to do in arm and mips busdma, but I think the larger part of the task is going to be defining more rigorously how a driver must interact with the busdma system to function correctly on all types of platforms, and then update existing drivers to conform. The busdma manpage currently has some vague words about the usage and sequencing of sync ops, such as "If read and write operations are not preceded and followed by the appropriate synchronization operations, behavior is undefined." I think we should more explicitly spell out what the appropriate sequences are. In particular: * The PRE and POST operations must occur in pairs; a PREREAD must be followed eventually by a POSTREAD and a PREWRITE must be followed by a POSTWRITE. * The CPU is not allowed to access the mapped memory after a PRE sync and before the corresponding POST sync. * The DMA hardware is not allowed to access the mapped memory after a POST sync and before the next PRE sync. * Read and write sync operators may be combined in a single call, PRE and POST operators may not be. E.G., PREREAD|PREWRITE is allowed, PREREAD|POSTREAD is not. We should note that while read and write operations may be combined, on some platforms PREREAD|PREWRITE is needlessly expensive when only a read is being performed. We also need some rules about working with buffers obtained from bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). I think the rule should be that a buffer obtained from bus_dmamem_alloc(), or more formally any region of memory mapped by a bus_dmamap_load(), is a single logical object which can only be accessed by one entity at a time. That means that there cannot be two concurrent DMA operations happening in different regions of the same buffer, nor can DMA and CPU access be happening concurrently even if in different parts of the buffer. I've always thought that allocating a dma buffer feels like a big hassle. You sometimes have to create a tag for the sole purpose of setting the maxsize to get the buffer size you need when you call bus_dmamem_alloc(). If bus_dmamem_alloc() took a size parm you could just use your parent tag, or a generic tag appropriate to all the IO you're doing for a given device. If you need a variety of buffers for small control and command and status transfers of different sizes, you end up having to manage up to a dozen tags and maps and buffers. It's all very clunky and inconvenient. It's just the sort of thing that makes you want to allocate a big buffer and subdivide it. Surely we could do something to make it easier? -- Ian From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 18:05:31 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0553D106566B; Sun, 26 Aug 2012 18:05:31 +0000 (UTC) (envelope-from marktinguely@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id B86078FC08; Sun, 26 Aug 2012 18:05:30 +0000 (UTC) Received: by dadr6 with SMTP id r6so1986829dad.13 for ; Sun, 26 Aug 2012 11:05:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Uw70O9ZjXEv0noKSxgiAWrFGrA7R41RTQBz+00qiyXg=; b=lRMVdUjT02PasXSBwruHEy+P+tM/5iIRceigrGli1+Adb6r+aF6KRIASE2HatMyk9R 0ydHl0W5mtYVWe/O6UM0CRSVtswg5BzKmV/rpcxgCSfl1qORXuv3xn1eRyk+sOjPNYZO UsgNWJwnflgqUBll5iG9sKtc1AeLU+U5T60xqEM1AVlrLAejg3fAJ0VKl6nIr65wZPKz vJwP5oR93/LEJv3bWXYUo7FgQ/NHLIvNydb3nQMXpPpqifD1Z0iBjgEHf5KgLaoJo4x2 DsuoqXjIV95lXHpAIcPj2D1MuD9zcHa9+/pDu30eLi0XrQhFBGxfoOO2HXfgIYMawTcW TgTQ== MIME-Version: 1.0 Received: by 10.68.221.70 with SMTP id qc6mr29050561pbc.92.1346004330201; Sun, 26 Aug 2012 11:05:30 -0700 (PDT) Received: by 10.68.229.227 with HTTP; Sun, 26 Aug 2012 11:05:29 -0700 (PDT) In-Reply-To: <1346002922.1140.56.camel@revolution.hippie.lan> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> Date: Sun, 26 Aug 2012 13:05:29 -0500 Message-ID: From: Mark Tinguely To: Ian Lepore Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 18:05:31 -0000 On Sun, Aug 26, 2012 at 12:42 PM, Ian Lepore wrote: > On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: >> The bottom line is that you can't mix things like that when cache >> lines are involved. The current code that tries is doomed to failure. >> Doomed. You just can't control all flushes, as Ian's missive >> demonstrates, and trying to accommodate code that does this I don't >> think can possibly work. All the interrupt masking, copying in and >> out, etc I fear is doomed to utter and abject failure. >> > Until last weekend I was in the camp that thought the partial cacheline > flush problem was solvable with sufficiently clever code. Now I agree > that we're doomed to failure and it's time to try another direction. > > We're going to have some implementation work to do in arm and mips > busdma, but I think the larger part of the task is going to be defining > more rigorously how a driver must interact with the busdma system to > function correctly on all types of platforms, and then update existing > drivers to conform. > > The busdma manpage currently has some vague words about the usage and > sequencing of sync ops, such as "If read and write operations are not > preceded and followed by the appropriate synchronization operations, > behavior is undefined." I think we should more explicitly spell out > what the appropriate sequences are. In particular: > > * The PRE and POST operations must occur in pairs; a PREREAD must > be followed eventually by a POSTREAD and a PREWRITE must be > followed by a POSTWRITE. > * The CPU is not allowed to access the mapped memory after a PRE > sync and before the corresponding POST sync. > * The DMA hardware is not allowed to access the mapped memory > after a POST sync and before the next PRE sync. > * Read and write sync operators may be combined in a single call, > PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > allowed, PREREAD|POSTREAD is not. We should note that while > read and write operations may be combined, on some platforms > PREREAD|PREWRITE is needlessly expensive when only a read is > being performed. > > We also need some rules about working with buffers obtained from > bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). I > think the rule should be that a buffer obtained from bus_dmamem_alloc(), > or more formally any region of memory mapped by a bus_dmamap_load(), is > a single logical object which can only be accessed by one entity at a > time. That means that there cannot be two concurrent DMA operations > happening in different regions of the same buffer, nor can DMA and CPU > access be happening concurrently even if in different parts of the > buffer. > > I've always thought that allocating a dma buffer feels like a big > hassle. You sometimes have to create a tag for the sole purpose of > setting the maxsize to get the buffer size you need when you call > bus_dmamem_alloc(). If bus_dmamem_alloc() took a size parm you could > just use your parent tag, or a generic tag appropriate to all the IO > you're doing for a given device. If you need a variety of buffers for > small control and command and status transfers of different sizes, you > end up having to manage up to a dozen tags and maps and buffers. It's > all very clunky and inconvenient. It's just the sort of thing that > makes you want to allocate a big buffer and subdivide it. Surely we > could do something to make it easier? > > -- Ian I did a quick look at the drivers last summer. Most drivers do the right thing and use memory allocated from bus_dmamem_alloc(). It is easy for us to give them a cache aligned buffer. Some drivers use mbufs - 256 bytes which cache safe. Some drivers directly or indirectly malloc() a buffer and then use it to dma - rather than try to fix them all, I was okay with making the smallest malloc() amount equal to the cache line size. It amounts to getting rid of the 16 byte allocation on some ARM architectures. The power of 2 allocator will then give us cache line safe allocation. A few drivers take a small memory amount from the kernel stack and dma to it <- broken driver. The few drivers that use data from a structure and that memory is not cached aligned <- broken driver. --Mark Tinguely. From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 18:25:23 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D04B1065674; Sun, 26 Aug 2012 18:25:23 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 4881A8FC1E; Sun, 26 Aug 2012 18:25:10 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7QIPAH8068013; Sun, 26 Aug 2012 12:25:10 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7QIP87W029231; Sun, 26 Aug 2012 12:25:08 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Mark Tinguely In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sun, 26 Aug 2012 12:25:07 -0600 Message-ID: <1346005507.1140.69.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, Hans Petter Selasky , freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 18:25:23 -0000 On Sun, 2012-08-26 at 13:05 -0500, Mark Tinguely wrote: > I did a quick look at the drivers last summer. > > Most drivers do the right thing and use memory allocated from > bus_dmamem_alloc(). It is easy for us to give them a cache aligned > buffer. > > Some drivers use mbufs - 256 bytes which cache safe. > > Some drivers directly or indirectly malloc() a buffer and then use it > to dma - rather than try to fix them all, I was okay with making the > smallest malloc() amount equal to the cache line size. It amounts to > getting rid of the 16 byte allocation on some ARM architectures. The > power of 2 allocator will then give us cache line safe allocation. > > A few drivers take a small memory amount from the kernel stack and dma > to it <- broken driver. > > The few drivers that use data from a structure and that memory is not > cached aligned <- broken driver. > I disagree about those last two points -- drivers that choose to use stack memory or malloc'd memory as IO buffers are not broken. Drivers can do IO directly to/from userland buffers, do we say that an application that calls read(2) and passes the address of a stack variable is broken? In this regard, it's the busdma implementation that's broken, because it should bounce those IOs through a DMA-safe buffer. There's absolutely no rule that I've ever heard of in FreeBSD that says IO can only take place using memory allocated from busdma. The rule is only that the proper sequence of busdma operation must be called, and beyond that it's up to the busdma implementation to make it work. Our biggest problem, I think, is that we don't have a sufficient definition of "the proper sequence of busdma operations." I don't think it will be very hard to make the arm and mips busdma implementations work correctly. It won't even be too hard to make them fairly efficient at bouncing small IOs (my thinking is that we can make small bounces no more expensive than the current partial cacheline flush implementation which copies the data multiple times). Bouncing large IO will never be efficient, but the inefficiency will be a powerful motivator to update drivers that do large IO to work better, such as using buffers allocated from busdma. -- Ian From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 18:53:47 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3AE0D106566C; Sun, 26 Aug 2012 18:53:47 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) by mx1.freebsd.org (Postfix) with ESMTP id E46D08FC16; Sun, 26 Aug 2012 18:53:46 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id q7QIrb6b086928; Sun, 26 Aug 2012 18:53:37 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.143] (CiscoE3000 [192.168.1.65]) by kientzle.com with SMTP id y7e92cf8djshe4rhprsvmx2wr2; Sun, 26 Aug 2012 18:53:37 +0000 (UTC) (envelope-from tim@kientzle.com) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=windows-1252 From: Tim Kientzle In-Reply-To: Date: Sun, 26 Aug 2012 11:53:35 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1278) Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, "arch@" Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 18:53:47 -0000 On Sun, Aug 26, 2012 at 12:42 PM, Ian Lepore wrote: > On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: >> The bottom line is that you can't mix things like that when cache >> lines are involved. >>=20 > =85. I think we should more explicitly spell out > what the appropriate sequences are. As someone who is just tinkering with driver code for the first time, I applaud any attempts to better document this area! ;-) > In particular: >=20 > * The PRE and POST operations must occur in pairs; a PREREAD must > be followed eventually by a POSTREAD and a PREWRITE must be > followed by a POSTWRITE. > * The CPU is not allowed to access the mapped memory after a PRE > sync and before the corresponding POST sync. > * The DMA hardware is not allowed to access the mapped memory > after a POST sync and before the next PRE sync. These rules sound reasonable. Good documentation might also give examples of what the PRE/POST operations might entail (e.g., from the preceding discussion, it sounds like PREREAD and PREWRITE require at least a partial cache flush on ARM). That helps folks who are coming to the docs with some hardware background. > * Read and write sync operators may be combined in a single = call, > PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > allowed, PREREAD|POSTREAD is not. We should note that while > read and write operations may be combined, on some platforms > PREREAD|PREWRITE is needlessly expensive when only a read is > being performed. PREREAD|POSTREAD doesn't sound useful to me, but why would it be explicitly forbidden? Would you also forbid POSTREAD|PREWRITE? (For a buffer that has just completed a DMA read and is going to be immediately used for a DMA write?) Tim From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 20:01:55 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AB765106564A; Sun, 26 Aug 2012 20:01:55 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 4FF5D8FC1A; Sun, 26 Aug 2012 20:01:55 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7QK1s32069554; Sun, 26 Aug 2012 14:01:54 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7QK1qGi029391; Sun, 26 Aug 2012 14:01:52 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Tim Kientzle In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Sun, 26 Aug 2012 14:01:52 -0600 Message-ID: <1346011312.1140.107.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, "arch@" Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 20:01:55 -0000 On Sun, 2012-08-26 at 11:53 -0700, Tim Kientzle wrote: > These rules sound reasonable. Good documentation might > also give examples of what the PRE/POST operations might entail > (e.g., from the preceding discussion, it sounds like PREREAD > and PREWRITE require at least a partial cache flush on ARM). > That helps folks who are coming to the docs with some hardware > background. > I agree, I think it would be good to have something like a RATIONALE section in the manpage that summarizes the issues faced by various categories of platform (hardware coherency, software-assisted coherency, etc) and how they handle it. You don't want people coding to the implementation (some of which is going on now, and I've been guilty of it myself); if there were more info in the docs there'd be less motivation to peek at the implementation. > > * Read and write sync operators may be combined in a single call, > > PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > > allowed, PREREAD|POSTREAD is not. We should note that while > > read and write operations may be combined, on some platforms > > PREREAD|PREWRITE is needlessly expensive when only a read is > > being performed. > > > PREREAD|POSTREAD doesn't sound useful to me, but > why would it be explicitly forbidden? > > Would you also forbid POSTREAD|PREWRITE? > (For a buffer that has just completed a DMA read > and is going to be immediately used for a DMA write?) My thinking on forbidding PREREAD|POSTREAD is at least partly that it removes some temptation to do the wrong thing: treat the busdma API as if it were a general cpu cache manipulation library. With the new definition of the sequences, a PREREAD|POSTREAD operation is nonsensical because it leaves no window during which the DMA hardware has access to memory; it is in effect a no-op. If you think in terms of implementation you might think "This would have to cause a cache invalidation." If you think in terms of API you should be thinking something more like "This marks out a time window during which the DMA hardware has safe access to that memory which has a duration of zero." POSTREAD|PREWRITE is interesting. It is not a no-op in terms of the API. It closes the hardware access window that was opened by an earlier PREREAD, and it opens a new hardware access window with the PREWRITE. Whether it touches hardware or caches or anything in a given implementation isn't the point, in terms of the API it's the right thing to do for a pair of back to back DMA operations, without intervening CPU access, on the same memory. So PREREAD|POSTREAD and PREWRITE|POSTWRITE make no sense, but all other combos should be allowed. Maybe instead of allowing and forbidding specific combos, we should just advise that these two are effectively no-ops. -- Ian From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 20:28:59 2012 Return-Path: Delivered-To: freebsd-mips@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D0A36106566B; Sun, 26 Aug 2012 20:28:59 +0000 (UTC) (envelope-from ray@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A34088FC12; Sun, 26 Aug 2012 20:28:59 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7QKSxjD060351; Sun, 26 Aug 2012 20:28:59 GMT (envelope-from ray@freefall.freebsd.org) Received: (from ray@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7QKSxOJ060347; Sun, 26 Aug 2012 20:28:59 GMT (envelope-from ray) Date: Sun, 26 Aug 2012 20:28:59 GMT Message-Id: <201208262028.q7QKSxOJ060347@freefall.freebsd.org> To: loos.br@gmail.com, ray@FreeBSD.org, freebsd-mips@FreeBSD.org From: ray@FreeBSD.org Cc: Subject: Re: kern/170859: [patch] [mips] Move MIPS specific options from generic conf/options file X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 20:28:59 -0000 Synopsis: [patch] [mips] Move MIPS specific options from generic conf/options file State-Changed-From-To: open->closed State-Changed-By: ray State-Changed-When: Sun Aug 26 20:27:13 UTC 2012 State-Changed-Why: Fixed. http://www.freebsd.org/cgi/query-pr.cgi?pr=170859 From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 20:30:06 2012 Return-Path: Delivered-To: freebsd-mips@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id F21431065674 for ; Sun, 26 Aug 2012 20:30:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BD0848FC0A for ; Sun, 26 Aug 2012 20:30:05 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7QKU54o060592 for ; Sun, 26 Aug 2012 20:30:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7QKU5Zs060591; Sun, 26 Aug 2012 20:30:05 GMT (envelope-from gnats) Date: Sun, 26 Aug 2012 20:30:05 GMT Message-Id: <201208262030.q7QKU5Zs060591@freefall.freebsd.org> To: freebsd-mips@FreeBSD.org From: dfilter@FreeBSD.ORG (dfilter service) Cc: Subject: Re: kern/170859: commit references a PR X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dfilter service List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 20:30:06 -0000 The following reply was made to PR kern/170859; it has been noted by GNATS. From: dfilter@FreeBSD.ORG (dfilter service) To: bug-followup@FreeBSD.org Cc: Subject: Re: kern/170859: commit references a PR Date: Sun, 26 Aug 2012 20:22:55 +0000 (UTC) Author: ray Date: Sun Aug 26 20:22:43 2012 New Revision: 239716 URL: http://svn.freebsd.org/changeset/base/239716 Log: Move AR71XX (MIPS SoCs family) options to options.mips file. PR: 170859 Submitted by: Luiz Otavio O Souza Approved by: adrian (mentor) Modified: head/sys/conf/options head/sys/conf/options.mips Modified: head/sys/conf/options ============================================================================== --- head/sys/conf/options Sun Aug 26 14:30:14 2012 (r239715) +++ head/sys/conf/options Sun Aug 26 20:22:43 2012 (r239716) @@ -906,11 +906,3 @@ RACCT opt_global.h # Resource Limits RCTL opt_global.h - -# At least one of the AR71XX ubiquiti boards has a Redboot configuration -# that "lies" about the amount of RAM it has. Until a cleaner method is -# defined, this option will suffice in overriding what Redboot says. -AR71XX_REALMEM opt_ar71xx.h -AR71XX_ENV_UBOOT opt_ar71xx.h -AR71XX_ENV_REDBOOT opt_ar71xx.h -AR71XX_ATH_EEPROM opt_ar71xx.h Modified: head/sys/conf/options.mips ============================================================================== --- head/sys/conf/options.mips Sun Aug 26 14:30:14 2012 (r239715) +++ head/sys/conf/options.mips Sun Aug 26 20:22:43 2012 (r239716) @@ -74,6 +74,16 @@ ARGE_DEBUG opt_arge.h ARGE_MDIO opt_arge.h # +# At least one of the AR71XX ubiquiti boards has a Redboot configuration +# that "lies" about the amount of RAM it has. Until a cleaner method is +# defined, this option will suffice in overriding what Redboot says. +# +AR71XX_REALMEM opt_ar71xx.h +AR71XX_ENV_UBOOT opt_ar71xx.h +AR71XX_ENV_REDBOOT opt_ar71xx.h +AR71XX_ATH_EEPROM opt_ar71xx.h + +# # Options that control the Ralink RT305xF Etherenet MAC. # IF_RT_DEBUG opt_if_rt.h _______________________________________________ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to "svn-src-all-unsubscribe@freebsd.org" From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 23:03:56 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 724A31065675 for ; Sun, 26 Aug 2012 23:03:56 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 272408FC19 for ; Sun, 26 Aug 2012 23:03:55 +0000 (UTC) Received: by ialo14 with SMTP id o14so8766148ial.13 for ; Sun, 26 Aug 2012 16:03:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=ep5IxGfVukJ1EG6I0J6d7Tuvp5WIjgyi6rZXdyCg3ho=; b=P39MrkTvnx4+dhdMgb7uZtgfLLvJBg/YAihgdzNC+4om83wOM+ZSa9kXI26O7rbDHg w3wWr3NFqtPrxUvxP5CgUtiHkNT9cB8nywbHZQy//i75m/nItV1ag+RdIb3PFVJJnfyT Vy1ZaZYl14LpYMedVasSNh9O/ai0OD2qgyC1va2EHbZJweVWLwt42ahhx0TlXDL9I+sl s1vODlagy0Zg2+ge6y/XzHH+3OeTpl2xhw00tddF0BoMaiPIomKwhyDukuMGYNh9NlAB pvbbQLbN7fuEqNudw6cQvGG4UPHWJvHl4mWEm5qd/cU3VVa9HPoYPyOil+rvIBnb6W/0 Re1Q== Received: by 10.50.184.198 with SMTP id ew6mr8457358igc.27.1346022235175; Sun, 26 Aug 2012 16:03:55 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id ce6sm4966875igb.1.2012.08.26.16.03.48 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 26 Aug 2012 16:03:54 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346002922.1140.56.camel@revolution.hippie.lan> Date: Sun, 26 Aug 2012 17:03:46 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQm1C1DcdP/VOas8BUQ44o7+JHNenbs2XLYWjzx7Fe2YGiMFvDZW8OhQXbNprwcwdnDyqk0s Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 23:03:56 -0000 On Aug 26, 2012, at 11:42 AM, Ian Lepore wrote: > On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: >> The bottom line is that you can't mix things like that when cache >> lines are involved. The current code that tries is doomed to = failure. >> Doomed. You just can't control all flushes, as Ian's missive >> demonstrates, and trying to accommodate code that does this I don't >> think can possibly work. All the interrupt masking, copying in and >> out, etc I fear is doomed to utter and abject failure. =20 >>=20 > Until last weekend I was in the camp that thought the partial = cacheline > flush problem was solvable with sufficiently clever code. Now I agree > that we're doomed to failure and it's time to try another direction. >=20 > We're going to have some implementation work to do in arm and mips > busdma, but I think the larger part of the task is going to be = defining > more rigorously how a driver must interact with the busdma system to > function correctly on all types of platforms, and then update existing > drivers to conform. >=20 > The busdma manpage currently has some vague words about the usage and > sequencing of sync ops, such as "If read and write operations are not > preceded and followed by the appropriate synchronization operations, > behavior is undefined." I think we should more explicitly spell out > what the appropriate sequences are. In particular: >=20 > * The PRE and POST operations must occur in pairs; a PREREAD must > be followed eventually by a POSTREAD and a PREWRITE must be > followed by a POSTWRITE.=20 PREREAD means "I am about to tell the device to put data here, have = whaterver things might be pending in the CPU complex to get out of the = way." usually this means 'invalidate the cache for that range', but not = always. POSTREAD means 'The device's DMA is done, I'd like to start = accessing it now.' If the memory will be thrown away without being = looked at, then does the driver necessarily need to issue the POSTREAD? = I think so, but I don't know if that's a new requirement. > * The CPU is not allowed to access the mapped memory after a PRE > sync and before the corresponding POST sync. =20 Correct. > * The DMA hardware is not allowed to access the mapped memory > after a POST sync and before the next PRE sync.=20 Correct. > * Read and write sync operators may be combined in a single call, > PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > allowed, PREREAD|POSTREAD is not. We should note that while > read and write operations may be combined, on some platforms > PREREAD|PREWRITE is needlessly expensive when only a read is > being performed. Correct. > We also need some rules about working with buffers obtained from > bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). = I > think the rule should be that a buffer obtained from = bus_dmamem_alloc(), > or more formally any region of memory mapped by a bus_dmamap_load(), = is > a single logical object which can only be accessed by one entity at a > time. That means that there cannot be two concurrent DMA operations > happening in different regions of the same buffer, nor can DMA and CPU > access be happening concurrently even if in different parts of the > buffer. =20 There's something subtle that I'm missing. Why would two DMA operations = be disallowed? The rest makes good sense. > I've always thought that allocating a dma buffer feels like a big > hassle. You sometimes have to create a tag for the sole purpose of > setting the maxsize to get the buffer size you need when you call > bus_dmamem_alloc(). If bus_dmamem_alloc() took a size parm you could > just use your parent tag, or a generic tag appropriate to all the IO > you're doing for a given device. If you need a variety of buffers for > small control and command and status transfers of different sizes, you > end up having to manage up to a dozen tags and maps and buffers. It's > all very clunky and inconvenient. It's just the sort of thing that > makes you want to allocate a big buffer and subdivide it. Surely we > could do something to make it easier? You'd wind up creating a quick tag on the fly for the bus_dmamap_alloc = if you wanted to do this. Cleanup then becomes unclear. Warner From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 23:13:35 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC00D106566B for ; Sun, 26 Aug 2012 23:13:35 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6B0108FC1A for ; Sun, 26 Aug 2012 23:13:34 +0000 (UTC) Received: by ialo14 with SMTP id o14so8778791ial.13 for ; Sun, 26 Aug 2012 16:13:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=ljCRbjZ9ASHSXhbZZlvbTJ/BcuQ0M4nJMzidj/rAYzc=; b=jsstuIxBZMAuLkMoxTQML03uF/nmgm0B3rvNotAfuqnorsILE34Uva1KT5zAbUAVdx 9puE/4pdUW4/p0lvVhMohP7ZpQyUMFUEj2iT+tWCCu0f9+TldMFHAewbuH05qgM1OMBb +fUndJZqv7euVmaJQT/oal20WQlNvRTA3Kk0eaCVYlgGthOrpDFItr4kV/CKjffqA5iZ ydU369xzoJGl3HI2vbRfZbC/IA0Uu0pTtLbuF+WNdqXZRHXUuOKaTdjY7NnzHGpW9MT/ YXFKI9nSKVFO7XVI+oWAJK+6Oadv8KjE7kiKSVaIw0wEr4wGMa4nyVOpSgS35qgQoL8K 7djg== Received: by 10.50.187.229 with SMTP id fv5mr2371955igc.57.1346022814665; Sun, 26 Aug 2012 16:13:34 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id df1sm16929663igc.10.2012.08.26.16.13.32 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 26 Aug 2012 16:13:33 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346005507.1140.69.camel@revolution.hippie.lan> Date: Sun, 26 Aug 2012 17:13:31 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQl6r0UZl0GeAjkkFHoJXxXGBHkKfgxVCw97pE8y1plBIWVD6262md027dKGmL88dDnC2UXS Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, Mark Tinguely , freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 23:13:35 -0000 On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: > On Sun, 2012-08-26 at 13:05 -0500, Mark Tinguely wrote: >> I did a quick look at the drivers last summer. >>=20 >> Most drivers do the right thing and use memory allocated from >> bus_dmamem_alloc(). It is easy for us to give them a cache aligned >> buffer. >>=20 >> Some drivers use mbufs - 256 bytes which cache safe. >>=20 >> Some drivers directly or indirectly malloc() a buffer and then use it >> to dma - rather than try to fix them all, I was okay with making the >> smallest malloc() amount equal to the cache line size. It amounts to >> getting rid of the 16 byte allocation on some ARM architectures. The >> power of 2 allocator will then give us cache line safe allocation. >>=20 >> A few drivers take a small memory amount from the kernel stack and = dma >> to it <- broken driver. >>=20 >> The few drivers that use data from a structure and that memory is not >> cached aligned <- broken driver. >>=20 >=20 > I disagree about those last two points -- drivers that choose to use > stack memory or malloc'd memory as IO buffers are not broken. Stack DMA is bad policy, at best, and broken at worst. The reason is = because of alignment of the underlying unit. Since there's no way to = say that something is aligned to a given spot on the stack, you are = asking for random stack corruption. Also, malloced area is similarly problematic: There's no cache line = informing of the allocator, so you can wind up with an allocation of = memory that's corrupted due to cache effects. > Drivers > can do IO directly to/from userland buffers, do we say that an > application that calls read(2) and passes the address of a stack > variable is broken? Yes, if it is smaller than a cache line size, and not aligned to the = cache line. That's the point of the uio load variant. > In this regard, it's the busdma implementation that's broken, because = it > should bounce those IOs through a DMA-safe buffer. There's absolutely > no rule that I've ever heard of in FreeBSD that says IO can only take > place using memory allocated from busdma. That's partially true. Since BUSDMA grew up in the storage area, you = must allocate the memory from busdma, or it must be page aligned has = been the de-facto rule here. The mbuf and uio variants of load were = invented to cope with common cases of mbufs and user I/O to properly = flag things. How does busdma know that it is using memory that's not from its = allocator? > The rule is only that the > proper sequence of busdma operation must be called, and beyond that = it's > up to the busdma implementation to make it work. =20 No. Bouncing is needed due to poor alignment of the underlying device. = Not due to cache effects. There's a limited number of things that we support with busdma. = Arbitrary data from malloc that might be shared with the CPU isn't on = that list. > Our biggest problem, I think, is that we don't have a sufficient > definition of "the proper sequence of busdma operations." I disagree. The sequence has been known for a long time. > I don't think it will be very hard to make the arm and mips busdma > implementations work correctly. It won't even be too hard to make = them > fairly efficient at bouncing small IOs (my thinking is that we can = make > small bounces no more expensive than the current partial cacheline = flush > implementation which copies the data multiple times). Bouncing large = IO > will never be efficient, but the inefficiency will be a powerful > motivator to update drivers that do large IO to work better, such as > using buffers allocated from busdma. I don't think the cache line problem can be solved with bounce buffers. = Trying to accommodate broken drivers is what lead us to this spot. We = need to fix the broken drivers. If that's impossible, then the best we = can do is have the driver set a 'always bounce' flag in the tag it = creates and use that to always bounce for operations through that tag. Warner From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 23:15:30 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AFF291065670 for ; Sun, 26 Aug 2012 23:15:30 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 637438FC12 for ; Sun, 26 Aug 2012 23:15:30 +0000 (UTC) Received: by ialo14 with SMTP id o14so8781100ial.13 for ; Sun, 26 Aug 2012 16:15:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=iYEhywsT8AYLxmmanrymiKw7wLde2oh04bmK21Zf8Ko=; b=N9JGq+hS2L+QUV6SIhGPcUBfFd326O3Uso9z0jXWUFdo3cCXteGsV64LkC8In0Fxhg BlZO8wk474f24/YhCOnm2IBWXG/RgiTOmRdm2YC5DfMDnoEmLs+RYhycxeSkCMw6Jra1 ywdXdQFOwVADEBkJAPOUs7ZM7T+JV/BjgCE7E2E4LBjLYwEZ3fVXueTraGiET6HiJoZJ rtdzTnNLusN+1bMT2UH2PZzStWvT9ClNtjit7DEL5UcFI0Gk0b8D3nM6/0FC5Xpz7zJf 4u2DWxPMMq5na+RNx0l6G5bRDrDF+xBfL2TdYHoxrIRtKTp1XaLbLOgDspKhP6tjc9bH Aw2A== Received: by 10.50.195.169 with SMTP id if9mr2347874igc.62.1346022930007; Sun, 26 Aug 2012 16:15:30 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id s2sm9413489igb.5.2012.08.26.16.15.27 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 26 Aug 2012 16:15:28 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=windows-1252 From: Warner Losh In-Reply-To: Date: Sun, 26 Aug 2012 17:15:26 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> To: Tim Kientzle X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQlRJnkl6+SWuV2JJhKTohXlcqDGseazaCaCoETwoCdVvXj2x47oLFqT4TtUl1gWJW8sFhum Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, "arch@" Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 23:15:30 -0000 On Aug 26, 2012, at 12:53 PM, Tim Kientzle wrote: >=20 > On Sun, Aug 26, 2012 at 12:42 PM, Ian Lepore > wrote: >> On Thu, 2012-08-23 at 22:00 -0600, Warner Losh wrote: >>> The bottom line is that you can't mix things like that when cache >>> lines are involved. >>>=20 >> =85. I think we should more explicitly spell out >> what the appropriate sequences are. >=20 >=20 > As someone who is just tinkering with driver code for > the first time, I applaud any attempts to better document > this area! ;-) >=20 >> In particular: >>=20 >> * The PRE and POST operations must occur in pairs; a PREREAD must >> be followed eventually by a POSTREAD and a PREWRITE must be >> followed by a POSTWRITE. >> * The CPU is not allowed to access the mapped memory after a PRE >> sync and before the corresponding POST sync. >> * The DMA hardware is not allowed to access the mapped memory >> after a POST sync and before the next PRE sync. >=20 >=20 > These rules sound reasonable. Good documentation might > also give examples of what the PRE/POST operations might entail > (e.g., from the preceding discussion, it sounds like PREREAD > and PREWRITE require at least a partial cache flush on ARM). > That helps folks who are coming to the docs with some hardware > background. >=20 >> * Read and write sync operators may be combined in a single = call, >> PRE and POST operators may not be. E.G., PREREAD|PREWRITE is >> allowed, PREREAD|POSTREAD is not. We should note that while >> read and write operations may be combined, on some platforms >> PREREAD|PREWRITE is needlessly expensive when only a read is >> being performed. >=20 >=20 > PREREAD|POSTREAD doesn't sound useful to me, but > why would it be explicitly forbidden? It could be useful when DMAing from multiple devices. Let's say I have = a buffer that I want to get data from two different DMA operations. = Wouldn't this be how you'd specify it? > Would you also forbid POSTREAD|PREWRITE? > (For a buffer that has just completed a DMA read > and is going to be immediately used for a DMA write?) This could be useful for some zero-copy things, no? Warner > Tim >=20 > _______________________________________________ > freebsd-arm@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" From owner-freebsd-mips@FreeBSD.ORG Sun Aug 26 23:18:57 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D10751065675 for ; Sun, 26 Aug 2012 23:18:57 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-iy0-f182.google.com (mail-iy0-f182.google.com [209.85.210.182]) by mx1.freebsd.org (Postfix) with ESMTP id 813D78FC16 for ; Sun, 26 Aug 2012 23:18:57 +0000 (UTC) Received: by ialo14 with SMTP id o14so8785190ial.13 for ; Sun, 26 Aug 2012 16:18:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=vV/55IW1m35Nojy+yoYqvhV4JtFulO/qaNQbLw6wfac=; b=Rz3kyYiP8rnLr4XnTxE4N8jp+Z+t7mQOJus40IdW6rXsMTmdU2K2rtcwIDfYLZPLnX VFHJ+SfgnXPFtnflF8OmiBMpUcJeR8qh13lWWEKpTBZBaprV3C7VPAeKOgqeEZ7g55SW mzzMsJpJK2MKBNWWbAyk4/fA3TE7UlhB81xFNF0Q0suQcEphb+Dp9rYBerZWYxWqepD5 5uF3HfeK8BehI0DnpCQw3O3WlxSRL0HaOekCevz8mp7sNjHs8JVPDOpovtl6ZPjS6nFH 5YMqgfhD+WWkZiMuMOuW5rrh60eXlj3/HytNfYzz5iHzSOsjx0Fnh66580su1ZKfZNk1 gC2w== Received: by 10.50.88.229 with SMTP id bj5mr8287211igb.21.1346023136744; Sun, 26 Aug 2012 16:18:56 -0700 (PDT) Received: from 63.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id i2sm4979193igl.8.2012.08.26.16.18.54 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 26 Aug 2012 16:18:55 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346011312.1140.107.camel@revolution.hippie.lan> Date: Sun, 26 Aug 2012 17:18:54 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <26F65164-EC99-483E-9FA8-916AD99CBCBC@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346011312.1140.107.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQmoQfPiY+ZmwnZeJpx8rYO33U7vFyAoBUJAGUcAEMABkvSB7toJ/9nhlYP3gY2qmC1SBavI Cc: Tim Kientzle , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, "arch@" Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Aug 2012 23:18:57 -0000 On Aug 26, 2012, at 2:01 PM, Ian Lepore wrote: > On Sun, 2012-08-26 at 11:53 -0700, Tim Kientzle wrote: >> These rules sound reasonable. Good documentation might >> also give examples of what the PRE/POST operations might entail >> (e.g., from the preceding discussion, it sounds like PREREAD >> and PREWRITE require at least a partial cache flush on ARM). >> That helps folks who are coming to the docs with some hardware >> background. >>=20 >=20 > I agree, I think it would be good to have something like a RATIONALE > section in the manpage that summarizes the issues faced by various > categories of platform (hardware coherency, software-assisted = coherency, > etc) and how they handle it. You don't want people coding to the > implementation (some of which is going on now, and I've been guilty of > it myself); if there were more info in the docs there'd be less > motivation to peek at the implementation. More documentation is good. >>> * Read and write sync operators may be combined in a single = call, >>> PRE and POST operators may not be. E.G., PREREAD|PREWRITE is >>> allowed, PREREAD|POSTREAD is not. We should note that while >>> read and write operations may be combined, on some platforms >>> PREREAD|PREWRITE is needlessly expensive when only a read is >>> being performed. >>=20 >>=20 >> PREREAD|POSTREAD doesn't sound useful to me, but >> why would it be explicitly forbidden? >>=20 >> Would you also forbid POSTREAD|PREWRITE? >> (For a buffer that has just completed a DMA read >> and is going to be immediately used for a DMA write?) >=20 > My thinking on forbidding PREREAD|POSTREAD is at least partly that it > removes some temptation to do the wrong thing: treat the busdma API as > if it were a general cpu cache manipulation library. =20 I think it should be read as POSTREAD + PREREAD, not PREREAD + POSTREAD. > With the new definition of the sequences, a PREREAD|POSTREAD operation > is nonsensical because it leaves no window during which the DMA = hardware > has access to memory; it is in effect a no-op. If you think in terms = of > implementation you might think "This would have to cause a cache > invalidation." If you think in terms of API you should be thinking > something more like "This marks out a time window during which the DMA > hardware has safe access to that memory which has a duration of zero." Acutally, it would allow one to shift the DMA from one device to = another, if read the way I say above. > POSTREAD|PREWRITE is interesting. It is not a no-op in terms of the > API. It closes the hardware access window that was opened by an = earlier > PREREAD, and it opens a new hardware access window with the PREWRITE. > Whether it touches hardware or caches or anything in a given > implementation isn't the point, in terms of the API it's the right = thing > to do for a pair of back to back DMA operations, without intervening = CPU > access, on the same memory. Right. This is useful in many cases. Consider a log structured device = that reads data into memory, and then writes it back to the tail of the = log as a defragging operation. > So PREREAD|POSTREAD and PREWRITE|POSTWRITE make no sense, but all = other > combos should be allowed. Maybe instead of allowing and forbidding > specific combos, we should just advise that these two are effectively > no-ops. I respectfully disagree.. Warner From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 01:26:09 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E6B57106564A; Mon, 27 Aug 2012 01:26:09 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9BEEE8FC16; Mon, 27 Aug 2012 01:26:09 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so6609002pbb.13 for ; Sun, 26 Aug 2012 18:26:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=buRkMCmZXq3F4Cc5OBO5pchRTJfQaZIQI48CqnG/UVM=; b=PAnNo8QWePhAymqZ/mWGbYh8Urc23Jl/DWWf7mLRxsT+q8Oxkez8Ol9YphyhVM3q35 x/ASzYBiRdpuZhiL4yVUHmVmQkRfh5DgoUCSEMxG862EH+bdfqJ/t1lUxC74+V4NwP9X IMQ1/qx0CsZW79aQogmq3fkNjK/c6aBIlf1Rg8vNjATaoGxRqTsGunzWDw50lXV5K+sG 7W9419jOZyDWkUxOu483e5zgItkzYa9JddxiLzDftB90igFeVRO9mKbjHmdA+jT8lxeS LDve5jhPhL6WMxa0canqUNYzriSRFegQcasvJtHFx6zV/HjHn2eqZaRK+Gl9lyoq+Zwp h0DQ== MIME-Version: 1.0 Received: by 10.68.129.131 with SMTP id nw3mr30406099pbb.43.1346030769022; Sun, 26 Aug 2012 18:26:09 -0700 (PDT) Received: by 10.68.36.106 with HTTP; Sun, 26 Aug 2012 18:26:08 -0700 (PDT) Received: by 10.68.36.106 with HTTP; Sun, 26 Aug 2012 18:26:08 -0700 (PDT) In-Reply-To: <26F65164-EC99-483E-9FA8-916AD99CBCBC@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346011312.1140.107.camel@revolution.hippie.lan> <26F65164-EC99-483E-9FA8-916AD99CBCBC@bsdimp.com> Date: Sun, 26 Aug 2012 18:26:08 -0700 Message-ID: From: Adrian Chadd To: Warner Losh Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-arm@freebsd.org, "arch@" , freebsd-mips@freebsd.org, Tim Kientzle Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 01:26:10 -0000 Hm. So perhaps we should document the busdma assumptions with respect to alignment and coherency? So this doesnt happen again? Adrian On Aug 26, 2012 4:19 PM, "Warner Losh" wrote: > > On Aug 26, 2012, at 2:01 PM, Ian Lepore wrote: > > > On Sun, 2012-08-26 at 11:53 -0700, Tim Kientzle wrote: > >> These rules sound reasonable. Good documentation might > >> also give examples of what the PRE/POST operations might entail > >> (e.g., from the preceding discussion, it sounds like PREREAD > >> and PREWRITE require at least a partial cache flush on ARM). > >> That helps folks who are coming to the docs with some hardware > >> background. > >> > > > > I agree, I think it would be good to have something like a RATIONALE > > section in the manpage that summarizes the issues faced by various > > categories of platform (hardware coherency, software-assisted coherency, > > etc) and how they handle it. You don't want people coding to the > > implementation (some of which is going on now, and I've been guilty of > > it myself); if there were more info in the docs there'd be less > > motivation to peek at the implementation. > > More documentation is good. > > >>> * Read and write sync operators may be combined in a single call, > >>> PRE and POST operators may not be. E.G., PREREAD|PREWRITE is > >>> allowed, PREREAD|POSTREAD is not. We should note that while > >>> read and write operations may be combined, on some platforms > >>> PREREAD|PREWRITE is needlessly expensive when only a read is > >>> being performed. > >> > >> > >> PREREAD|POSTREAD doesn't sound useful to me, but > >> why would it be explicitly forbidden? > >> > >> Would you also forbid POSTREAD|PREWRITE? > >> (For a buffer that has just completed a DMA read > >> and is going to be immediately used for a DMA write?) > > > > My thinking on forbidding PREREAD|POSTREAD is at least partly that it > > removes some temptation to do the wrong thing: treat the busdma API as > > if it were a general cpu cache manipulation library. > > I think it should be read as POSTREAD + PREREAD, not PREREAD + POSTREAD. > > > With the new definition of the sequences, a PREREAD|POSTREAD operation > > is nonsensical because it leaves no window during which the DMA hardware > > has access to memory; it is in effect a no-op. If you think in terms of > > implementation you might think "This would have to cause a cache > > invalidation." If you think in terms of API you should be thinking > > something more like "This marks out a time window during which the DMA > > hardware has safe access to that memory which has a duration of zero." > > Acutally, it would allow one to shift the DMA from one device to another, > if read the way I say above. > > > POSTREAD|PREWRITE is interesting. It is not a no-op in terms of the > > API. It closes the hardware access window that was opened by an earlier > > PREREAD, and it opens a new hardware access window with the PREWRITE. > > Whether it touches hardware or caches or anything in a given > > implementation isn't the point, in terms of the API it's the right thing > > to do for a pair of back to back DMA operations, without intervening CPU > > access, on the same memory. > > Right. This is useful in many cases. Consider a log structured device > that reads data into memory, and then writes it back to the tail of the log > as a defragging operation. > > > So PREREAD|POSTREAD and PREWRITE|POSTWRITE make no sense, but all other > > combos should be allowed. Maybe instead of allowing and forbidding > > specific combos, we should just advise that these two are effectively > > no-ops. > > I respectfully disagree.. > > Warner > > _______________________________________________ > freebsd-arm@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" > From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 04:24:56 2012 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 12195106564A for ; Mon, 27 Aug 2012 04:24:56 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh11.mail.rice.edu (mh11.mail.rice.edu [128.42.199.30]) by mx1.freebsd.org (Postfix) with ESMTP id CF8158FC0C for ; Mon, 27 Aug 2012 04:24:55 +0000 (UTC) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 24D284C02AE; Sun, 26 Aug 2012 23:24:55 -0500 (CDT) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 237434C02AC; Sun, 26 Aug 2012 23:24:55 -0500 (CDT) X-Virus-Scanned: by amavis-2.7.0 at mh11.mail.rice.edu, auth channel Received: from mh11.mail.rice.edu ([127.0.0.1]) by mh11.mail.rice.edu (mh11.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id V5tX-SFotO2b; Sun, 26 Aug 2012 23:24:55 -0500 (CDT) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh11.mail.rice.edu (Postfix) with ESMTPSA id 960664C02A3; Sun, 26 Aug 2012 23:24:54 -0500 (CDT) Message-ID: <503AF695.9060405@rice.edu> Date: Sun, 26 Aug 2012 23:24:53 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:8.0) Gecko/20111113 Thunderbird/8.0 MIME-Version: 1.0 To: "Jayachandran C." References: <50228F5C.1000408@rice.edu> <50269AD4.9050804@rice.edu> <5029635A.4050209@rice.edu> <502D2271.6080105@rice.edu> In-Reply-To: Content-Type: multipart/mixed; boundary="------------050807020503050406030206" Cc: mips@freebsd.org Subject: Re: mips pmap patch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 04:24:56 -0000 This is a multi-part message in MIME format. --------------050807020503050406030206 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Could you please test the attached patch? For a change, this patch is an optimization (as opposed to a bug fix). It carves off a third high-order bit from the PTE in order to implement another software flag. For 64-bit systems, this bit was previously unused. For 32-bit systems, this patch effectively reduces the size of the PTE's page frame number (PFN) field by 1 bit. However, the PFN still has more than enough bits to support the 32-bit vm_paddr_t. This patch needs both 32-bit and 64-bit testing. I'm fairly confident of the correctness of the 64-bit case, but less so of the 32-bit case. Going forward, I expect to claim a fourth bit from the PTE in order to implement a proper access bit emulation. Thanks, Alan --------------050807020503050406030206 Content-Type: text/plain; name="mips_pmap22.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="mips_pmap22.patch" Index: mips/mips/pmap.c =================================================================== --- mips/mips/pmap.c (revision 239681) +++ mips/mips/pmap.c (working copy) @@ -1691,9 +1691,9 @@ pmap_remove_pte(struct pmap *pmap, pt_entry_t *ptq pmap->pm_stats.wired_count -= 1; pmap->pm_stats.resident_count -= 1; - pa = TLBLO_PTE_TO_PA(oldpte); - if (page_is_managed(pa)) { + if (pte_test(&oldpte, PTE_MANAGED)) { + pa = TLBLO_PTE_TO_PA(oldpte); m = PHYS_TO_VM_PAGE(pa); if (pte_test(&oldpte, PTE_D)) { KASSERT(!pte_test(&oldpte, PTE_RO), @@ -1930,8 +1930,8 @@ pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offs if (!pte_test(pte, PTE_V)) continue; pbits = *pte; - pa = TLBLO_PTE_TO_PA(pbits); - if (page_is_managed(pa) && pte_test(&pbits, PTE_D)) { + if (pte_test(&pbits, PTE_MANAGED | PTE_D)) { + pa = TLBLO_PTE_TO_PA(pbits); m = PHYS_TO_VM_PAGE(pa); vm_page_dirty(m); } @@ -1975,6 +1975,8 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot_t KASSERT(va <= VM_MAX_KERNEL_ADDRESS, ("pmap_enter: toobig")); KASSERT((m->oflags & (VPO_UNMANAGED | VPO_BUSY)) != 0, ("pmap_enter: page %p is not busy", m)); + pa = VM_PAGE_TO_PHYS(m); + newpte = TLBLO_PA_TO_PFN(pa) | PTE_V; mpte = NULL; @@ -1997,7 +1999,6 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot_t panic("pmap_enter: invalid page directory, pdir=%p, va=%p", (void *)pmap->pm_segtab, (void *)va); } - pa = VM_PAGE_TO_PHYS(m); om = NULL; origpte = *pte; opa = TLBLO_PTE_TO_PA(origpte); @@ -2027,8 +2028,9 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot_t if (mpte) mpte->wire_count--; - if (page_is_managed(opa)) { + if (pte_test(&origpte, PTE_MANAGED)) { om = m; + newpte |= PTE_MANAGED; } goto validate; } @@ -2043,7 +2045,7 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot_t if (pte_test(&origpte, PTE_W)) pmap->pm_stats.wired_count--; - if (page_is_managed(opa)) { + if (pte_test(&origpte, PTE_MANAGED)) { om = PHYS_TO_VM_PAGE(opa); pv = pmap_pvh_remove(&om->md, pmap, va); } @@ -2068,6 +2070,7 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot_t pv = get_pv_entry(pmap, FALSE); pv->pv_va = va; TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_list); + newpte |= PTE_MANAGED; } else if (pv != NULL) free_pv_entry(pmap, pv); @@ -2088,7 +2091,7 @@ validate: /* * Now validate mapping with desired protection/wiring. */ - newpte = TLBLO_PA_TO_PFN(pa) | rw | PTE_V; + newpte |= rw; if (is_cacheable_mem(pa)) newpte |= PTE_C_CACHE; @@ -2108,7 +2111,7 @@ validate: if (origpte != newpte) { if (pte_test(&origpte, PTE_V)) { *pte = newpte; - if (page_is_managed(opa) && (opa != pa)) { + if (pte_test(&origpte, PTE_MANAGED) && opa != pa) { if (om->md.pv_flags & PV_TABLE_REF) vm_page_aflag_set(om, PGA_REFERENCED); om->md.pv_flags &= ~PV_TABLE_REF; @@ -2117,10 +2120,10 @@ validate: KASSERT(!pte_test(&origpte, PTE_RO), ("pmap_enter: modified page not writable:" " va: %p, pte: %#jx", (void *)va, (uintmax_t)origpte)); - if (page_is_managed(opa)) + if (pte_test(&origpte, PTE_MANAGED)) vm_page_dirty(om); } - if (page_is_managed(opa) && + if (pte_test(&origpte, PTE_MANAGED) && TAILQ_EMPTY(&om->md.pv_list)) vm_page_aflag_clear(om, PGA_WRITEABLE); } else { @@ -2247,6 +2250,8 @@ pmap_enter_quick_locked(pmap_t pmap, vm_offset_t v * Now validate mapping with RO protection */ *pte = TLBLO_PA_TO_PFN(pa) | PTE_V; + if ((m->oflags & VPO_UNMANAGED) == 0) + *pte |= PTE_MANAGED; if (is_cacheable_mem(pa)) *pte |= PTE_C_CACHE; @@ -2982,7 +2987,6 @@ pmap_mincore(pmap_t pmap, vm_offset_t addr, vm_pad vm_paddr_t pa; vm_page_t m; int val; - boolean_t managed; PMAP_LOCK(pmap); retry: @@ -2996,8 +3000,7 @@ retry: if (pte_test(&pte, PTE_D)) val |= MINCORE_MODIFIED | MINCORE_MODIFIED_OTHER; pa = TLBLO_PTE_TO_PA(pte); - managed = page_is_managed(pa); - if (managed) { + if (pte_test(&pte, PTE_MANAGED)) { /* * This may falsely report the given address as * MINCORE_REFERENCED. Unfortunately, due to the lack of @@ -3009,7 +3012,8 @@ retry: val |= MINCORE_REFERENCED | MINCORE_REFERENCED_OTHER; } if ((val & (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER)) != - (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER) && managed) { + (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER) && + pte_test(&pte, PTE_MANAGED)) { /* Ensure that "PHYS_TO_VM_PAGE(pa)->object" doesn't change. */ if (vm_page_pa_tryrelock(pmap, pa, locked_pa)) goto retry; @@ -3232,23 +3236,6 @@ pmap_asid_alloc(pmap) } } -int -page_is_managed(vm_paddr_t pa) -{ - vm_offset_t pgnum = atop(pa); - - if (pgnum >= first_page) { - vm_page_t m; - - m = PHYS_TO_VM_PAGE(pa); - if (m == NULL) - return (0); - if ((m->oflags & VPO_UNMANAGED) == 0) - return (1); - } - return (0); -} - static pt_entry_t init_pte_prot(vm_page_t m, vm_prot_t access, vm_prot_t prot) { @@ -3305,9 +3292,9 @@ pmap_emulate_modified(pmap_t pmap, vm_offset_t va) } pte_set(pte, PTE_D); tlb_update(pmap, va, *pte); + if (!pte_test(pte, PTE_MANAGED)) + panic("pmap_emulate_modified: unmanaged page"); pa = TLBLO_PTE_TO_PA(*pte); - if (!page_is_managed(pa)) - panic("pmap_emulate_modified: unmanaged page"); m = PHYS_TO_VM_PAGE(pa); m->md.pv_flags |= PV_TABLE_REF; PMAP_UNLOCK(pmap); Index: mips/include/pte.h =================================================================== --- mips/include/pte.h (revision 239681) +++ mips/include/pte.h (working copy) @@ -57,8 +57,8 @@ typedef pt_entry_t *pd_entry_t; /* * PFN for EntryLo register. Upper bits are 0, which is to say that - * bit 29 is the last hardware bit; Bits 30 and upwards (EntryLo is - * 64 bit though it can be referred to in 32-bits providing 2 software + * bit 28 is the last hardware bit; Bits 29 and upwards (EntryLo is + * 64 bit though it can be referred to in 32-bits providing 3 software * bits safely. We use it as 64 bits to get many software bits, and * god knows what else.) are unacknowledged by hardware. They may be * written as anything, but otherwise they have as much meaning as @@ -68,11 +68,11 @@ typedef pt_entry_t *pd_entry_t; #define TLBLO_SWBITS_SHIFT (34) #define TLBLO_PFN_MASK 0x3FFFFFFC0ULL #else -#define TLBLO_SWBITS_SHIFT (30) -#define TLBLO_PFN_MASK (0x3FFFFFC0) +#define TLBLO_SWBITS_SHIFT (29) +#define TLBLO_PFN_MASK (0x1FFFFFC0) #endif #define TLBLO_PFN_SHIFT (6) -#define TLBLO_SWBITS_MASK ((pt_entry_t)0x3 << TLBLO_SWBITS_SHIFT) +#define TLBLO_SWBITS_MASK ((pt_entry_t)0x7 << TLBLO_SWBITS_SHIFT) #define TLBLO_PA_TO_PFN(pa) ((((pa) >> TLB_PAGE_SHIFT) << TLBLO_PFN_SHIFT) & TLBLO_PFN_MASK) #define TLBLO_PFN_TO_PA(pfn) ((vm_paddr_t)((pfn) >> TLBLO_PFN_SHIFT) << TLB_PAGE_SHIFT) #define TLBLO_PTE_TO_PFN(pte) ((pte) & TLBLO_PFN_MASK) @@ -132,9 +132,11 @@ typedef pt_entry_t *pd_entry_t; * RO: Read only. Never set PTE_D on this page, and don't * listen to requests to write to it. * W: Wired. ??? + * MANAGED:Managed. This PTE maps a managed page. */ #define PTE_RO ((pt_entry_t)0x01 << TLBLO_SWBITS_SHIFT) #define PTE_W ((pt_entry_t)0x02 << TLBLO_SWBITS_SHIFT) +#define PTE_MANAGED ((pt_entry_t)0x04 << TLBLO_SWBITS_SHIFT) /* * PTE management functions for bits defined above. @@ -160,7 +162,7 @@ typedef pt_entry_t *pd_entry_t; #define PTESIZE 4 #define PTE_L lw #define PTE_MTC0 mtc0 -#define CLEAR_PTE_SWBITS(r) sll r, 2; srl r, 2 /* remove 2 high bits */ +#define CLEAR_PTE_SWBITS(r) sll r, 3; srl r, 3 /* remove 3 high bits */ #endif /* defined(__mips_n64) || defined(__mips_n32) */ #if defined(__mips_n64) Index: mips/include/pmap.h =================================================================== --- mips/include/pmap.h (revision 239681) +++ mips/include/pmap.h (working copy) @@ -171,7 +171,6 @@ void pmap_bootstrap(void); void *pmap_mapdev(vm_paddr_t, vm_size_t); void pmap_unmapdev(vm_offset_t, vm_size_t); vm_offset_t pmap_steal_memory(vm_size_t size); -int page_is_managed(vm_paddr_t pa); void pmap_kenter(vm_offset_t va, vm_paddr_t pa); void pmap_kenter_attr(vm_offset_t va, vm_paddr_t pa, int attr); void pmap_kremove(vm_offset_t va); --------------050807020503050406030206-- From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 06:07:05 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D29CD106566C; Mon, 27 Aug 2012 06:07:05 +0000 (UTC) (envelope-from hans.petter.selasky@bitfrost.no) Received: from smtp02-out.isp.tdc.no (smtp02-out.isp.tdc.no [213.236.144.173]) by mx1.freebsd.org (Postfix) with ESMTP id 5C13D8FC0C; Mon, 27 Aug 2012 06:07:05 +0000 (UTC) Received: from mail.bitfrost.no (mail.bitfrost.no [85.19.79.136]) by smtp02-out.isp.tdc.no (Postfix) with ESMTP id 3X52cd1p2pz3CP; Mon, 27 Aug 2012 08:04:49 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at bitfrost.no From: =?windows-1252?Q?Hans_Petter_Selasky?= To: =?windows-1252?Q?Ian_Lepore?= , =?windows-1252?Q?Warner_Losh?= Date: Mon, 27 Aug 2012 08:06:57 +0200 Mime-Version: 1.0 In-Reply-To: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> X-Priority: 3 (Normal) Message-Id: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: "=?windows-1252?Q?freebsd-arm=40freebsd.org?=" , "=?windows-1252?Q?freebsd-mips=40freebsd.org?=" , "=?windows-1252?Q?freebsd-arch=40freebsd.org?=" Subject: RE: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 06:07:06 -0000 Hi,=20=0D=0ACorrect.=0D=0A=0D=0A> We also need some rules about working w= ith buffers obtained from=0D=0A> bus_dmamem_alloc() and external buffers = passed to bus_dmamap_load(). =A0I=0D=0A> think the rule should be that a = buffer obtained from bus_dmamem_alloc(),=0D=0A> or more formally any regi= on of memory mapped by a bus_dmamap_load(), is=0D=0A> a single logical ob= ject which can only be accessed by one entity at a=0D=0A> time. =A0That m= eans that there cannot be two concurrent DMA operations=0D=0A> happening = in different regions of the same buffer, nor can DMA and CPU=0D=0A> acces= s be happening concurrently even if in different parts of the=0D=0A> buff= er. =A0=0D=0A=0D=0A=0D=0AIs this something which we can fix using a simpl= e __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to= be DMA loaded.=0D=0A=0D=0A=A0=0D=0A=0D=0AAlso: Why is busdma not complai= ning when loading a non-valid buffer pointer=3F=0D=0A=0D=0A--HPS=0D=0A From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 11:07:15 2012 Return-Path: Delivered-To: freebsd-mips@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 442BC106566B for ; Mon, 27 Aug 2012 11:07:15 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 2EB3A8FC25 for ; Mon, 27 Aug 2012 11:07:15 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7RB7FFW085909 for ; Mon, 27 Aug 2012 11:07:15 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7RB7E7b085907 for freebsd-mips@FreeBSD.org; Mon, 27 Aug 2012 11:07:14 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 27 Aug 2012 11:07:14 GMT Message-Id: <201208271107.q7RB7E7b085907@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-mips@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-mips@FreeBSD.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 11:07:15 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/170864 mips [mips] [patch] Move SPI flash from AR71XX_BASE (kernel o kern/165951 mips [ar913x] [ath] DDR flush isn't being done for the WMAC p kern/163670 mips [mips][arge] arge can't allocate ring buffer on multip 3 problems total. From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 13:38:10 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 48F47106566C for ; Mon, 27 Aug 2012 13:38:10 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0D9DB8FC1C for ; Mon, 27 Aug 2012 13:38:10 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so7583091pbb.13 for ; Mon, 27 Aug 2012 06:38:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:x-priority :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer:x-gm-message-state; bh=sDQvljch4pOy5n8q84V4CzIi45qexsJyPkD23yBHMLw=; b=dhf3PLnMuVY95I3/vvCKcBhIMJQCBn5YQ/ynElC5v49Ll2lUsJJB1yXpiygy+eJhEo /a9+OWXPcvnzmDuhQOHsYGaL+u9SBamAeGemUlmoXYC0GOMFHemN3SYp7vL0AwxqIf5w +0kKqpLGJPZrsVws13OqExgA0LtWacgQ1DStsYD7OzlVHMEmeo3TsCYOMSUaeSnzHWA0 OYpbh9QmNy/kodI0FOsc0hTgNSKDPap2Vy53g3Ww/2OmNJUMS+Z+UD06DFZs5Zq/7Heh P+EIlysiK084jZ3CkmZOrTQSBl4AQTWyC1xen7zBtpYYTOxNZn750UJzmeXRck7Zupr/ fRYA== Received: by 10.66.74.37 with SMTP id q5mr30255409pav.29.1346074689751; Mon, 27 Aug 2012 06:38:09 -0700 (PDT) Received: from [10.0.0.63] (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id pj10sm14721644pbb.46.2012.08.27.06.38.07 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 06:38:09 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh X-Priority: 3 (Normal) In-Reply-To: Date: Mon, 27 Aug 2012 07:38:06 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> To: Hans Petter Selasky X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQkhuRqjyhLYt/lHdgBMXtY4AqULZyIlGBnzAiiljq2/VRXdassevCGo2qSC03fJkM7X1Hyf Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 13:38:10 -0000 On Aug 27, 2012, at 12:06 AM, Hans Petter Selasky wrote: > Hi, > Correct. >=20 > > We also need some rules about working with buffers obtained from > > bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). = I > > think the rule should be that a buffer obtained from = bus_dmamem_alloc(), > > or more formally any region of memory mapped by a bus_dmamap_load(), = is > > a single logical object which can only be accessed by one entity at = a > > time. That means that there cannot be two concurrent DMA operations > > happening in different regions of the same buffer, nor can DMA and = CPU > > access be happening concurrently even if in different parts of the > > buffer. =20 >=20 > Is this something which we can fix using a simple = __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to = be DMA loaded. No. I don't think so. the reason is that you can't define USB_DMA_ALGIN = to be a constant on MIPS, at least, or I think ARM because that's = determined at run time. > Also: Why is busdma not complaining when loading a non-valid buffer = pointer? Speed and code simplicity. Warner From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 14:12:14 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7215110656F7 for ; Mon, 27 Aug 2012 14:12:14 +0000 (UTC) (envelope-from bounce-4389-freebsd-mips=freebsd.org@mbmsend.com) Received: from mta056.mbmsend.com (mta056.mbmsend.com [66.85.161.20]) by mx1.freebsd.org (Postfix) with ESMTP id 410808FC14 for ; Mon, 27 Aug 2012 14:12:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=sendmail; d=mbmsend.com; h=Date:To:From:Reply-To:Subject:List-Unsubscribe:Message-ID:MIME-Version:Sender:Content-Type; i=killersportslive=3Daol.com@mbmsend.com; bh=l+iuGe06HpsQdB1gA3orcUUeM7c=; b=pQ/D5uVbSMgP+BhldGCMTCz1xDZU84ZNvoEXZbXUbRYW9C4Ot9tOQM71+SLhg60+vVCFgI9x39/g XDY76Euleg== DomainKey-Signature: a=rsa-sha1; c=nofws; q=dns; s=sendmail; d=mbmsend.com; b=PA5uYjyguZfapVIXzZic7RQEi+doxRE9FdTi+NSOOgWX9ONR7by5NzpOccv6kcj7WDQ8MBTfOVn1 ku6izLHPhg==; Received: from mbmsend.com (127.0.0.1) by mta056.mbmsend.com id h7e03m19uisl for ; Mon, 27 Aug 2012 10:11:33 -0400 (envelope-from ) Date: Mon, 27 Aug 2012 10:11:20 -0400 To: "" From: "Chase Diamond" Message-ID: X-Mailer: mybizmailer v1.0 (MyBizMailer.com) MIME-Version: 1.0 X-Report-Abuse: Please report abuse for this campaign here: http://mybizmailer.com/abuse.aspx?aid=00a1i0h3&uid=003d4e7df373944a0db22662eee96062&e=dc19288a531843b87012d3d774854a46&b=1 Sender: "Chase Diamond" X-CampaignID: dc19288a531843b87012d3d774854a46 X-AccountID: 00a1i0h3 X-FBL: dc19288a531843b87012d3d774854a46 Content-Type: text/plain; charset="utf-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: $100.00 in free capper cash X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Chase Diamond List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 14:12:14 -0000 SIMPLY PUT GET 100.00 IN FREE HANDICAPPER CASH TO USE ON ANY HANDICAPPER AT LASVEGASLINECHANGERS.COM FOR FOOTBALL SEASON. GO TO WWW.LASVEGASLINECHANGERS.COM LOOK FOR A HANDICAPPER UD LIKE SIGHN UP FOR THERE FREE PICKS NEWSLETER SIMPLY BYE ENTERING UR EMAIL IN TOP LEFT OF WEBSITE AND (REPLY BACK TO THIS EMAIL SAYIN WHICH CAPPER UD LIKE UR FREE 100.00 PUT TOWARDS AND YOU WILL HAVE UR FREE TRIAL DONT WAIT FOOTBALL STARTS IN 3 DAYS!!!! BELOW IS A LIST OF HANDICAPPERS TO PICK FROM... Tony Corleone Vernon Croy Chase Diamond Chaz Diamond Matt Fargo Simon Green Tony Karpinski Ross King Steve Merril Stephen Nover TJ Pemberton RJ Robbins Carolina Sports Doc's Sports Kevin Thomas Craig Trapp Paul Wynns >>> Click here to read about our guarantee policy and loyalty program <<< TONY CORLEONE Tony's MLB Play of the Day!! Tony Corleone continues his great success with a strong 2nd half of the MLB season. He was 5-1 with his plays last night and is now 55-36 on the season with his MLB Plays of the Day. He has another one locked and loaded for today. Grab this guaranteed winner now and continue to cash in with Tony!! Tony's 10 Unit MLB Grand Slam!! (21-11 last 32) Tony continues to pick winners. He has been crushing the books all season long. He currently holds down the #1 MLB spot as well as the #1 overall spot. He has his 10 Unit MLB Grand Slam ready for today. Make sure to grab this winner now and absolutely shell your bookie with Tony!! Tony's 10 Unit Thursday NCAAF Enforcer!! Tony is locked and loaded with his 10 Unit Thursday NCAAF Enforcer between Vanderbilt and South Carolina. Last season Tony absolutely shelled the books in college football, as he managed to net his dime bettor cliets over $43,000. Grab this winner now and start the college football season off on a winning note with Tony!! Click here to see all of Tony Corleone's picks CHASE DIAMOND NCAFF KICK OFF 3 PACK 8/30 100/70/50 DIMER CHASE DIAMOND IS READY TO START THE NCAFF SEASON WITH A SWEEP AND HE HAS 3 WINNERS SET FOR THURSDAY 8/30 FOR JUST 29.99 YOU CAN CASH WITH YOUR BOOKS WITH EASE BE SHARP BUY CHASE DIAMONDS NCAFF PLAYS LAST SEASON HE CASHED OVER 64% IN NCAFF. 200 DIME NCAFF GAME OF THE WEEK 51-24 LIFFETIME THIS PLAY IS FOR 9/1 OPENING SATURDAY DONT MISS CHASE DIAMOND COLLEGE FOOTBALL GAME OF THE WEEK. CHASE DIAMOND HAS GOT HIS HANDS OF A GAME THAT WILL CASH WITH EASE. LAST YEAR CHASE DIAMOND CASHED OVER 60% ON HIS COLLEGE FOOTBALL PLAYS THIS YEAR CHASE LOOKS TO IMPROVE ON THAT NUMBER BE HERE OR BE SQUARE.(10.00 OFF NORMAL PRICE) 200 DIME NFL KICK OFF ULTIMATE WINNER 72% LIFETIME CHASE DIAMOND THE BIG PLAY KING HAS LOCKED AND LOADED ON HIS TAKE ON THE COWBOYS AT GIANTS AND YOU ALL KNOW HOW CHASE DOES WITH 200 DIMERS TRY 72% LIFETIME THIS GAME IS BACKED BY 4 SYSTEMS AND SHOULD BE HIT AND HIT HARD WIN BIG GAME 1 GUARANTEED. Click here to see all of Chase Diamond's picks CHAZ DIAMOND 50 DIME KILLER SPORTS KNOCK OUT WINNER CHAZ DIAMOND HAS BEEN VERY SELECTIVE LATELY AND HITTING HIS PLAYS DONT MISS THIS 50 DIME DOG WINNER TODAY AND FOR JUST 14.99 WHO CAN AFFORD NOT TO BE ON IT? Click here to see all of Chaz Diamond's picks MATT FARGO Fargo's MLB BIG BITE BEATDOWN (HUGE 59-35 LIFETIME Matt is coming off a Sunday split on the bases and brings in a POTENT 60% run in baseball into the new week! He comes back in a huge way on Monday with a Big Bite Beatdown and these reports have been OUTSTANDING, going a BLISTERING 59-35 (62.8%) lifetime in MLB! This large chalk UNLEASHES another MASSIVE DESTRUCTION! Hammer it! Do not even think about missing this! Fargo's 10* CFB THURSDAY TOTALS DOMINATOR (53-34) Matt is locked and loaded for a huge CFB year as he carries over what was an EXCEPTIONAL ending last year! He closed on a PERFECT 4-0 run and the numbers were OUTSTANDING going back further as he finished on a SCORCHING 53-34-1 (61%) run! He WINS Thursday with an opening night Totals Dominator that is a SHOOTOUT in the making! This one flies way over the total! Fargo's 10* CFB THURSDAY ENFORCER (53-34 CFB RUN) Matt closed the college football season on a PERFECT 4-0 run and the numbers were OUTSTANDING going back further as he finished on a SCORCHING 53-34-1 (61%) run! He gets things started right where he left off with a MASSIVE 10* Enforcer on opening night! Last year he went a COMMANDING 7-1 (87.5%) in Week One! More of the same! Do not even think about missing this! Click here to see all of Matt Fargo's picks STEVE MERRIL MLB Grand Slam - (PERFECT 5-0 MLB RUN)! Steve Merril has SWEPT his L5 MLB plays and he has uncovered a powerful MLB GRAND SLAM for Monday that is backed by solid winning angles - Grab this value play right now! Guaranteed Run Line that will WIN BIG! Click here to see all of Steve Merril's picks STEPHEN NOVER Stephen Nover's Underdog Play of the Month A wrong favorite is one of many factors making this matchup such a strong play that it rates as Vegas wiseguy Stephen Nover's August Underdog Play of the Month. Don't miss this sure winner and cash big. Read what makes this play so special. Click here to see all of Stephen Nover's picks KEVIN THOMAS KEVIN THURSDAY NITE KICKOFF THREE PACK OF COLLEGE FOOTBALL THIS THURSDAY NAILING OVER 62 PERCENT LIFETIME COLLEGE AND READY TO CASH Click here to see all of Kevin Thomas's picks From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 15:12:19 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EDAAC1065690; Mon, 27 Aug 2012 15:12:18 +0000 (UTC) (envelope-from tim@kientzle.com) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) by mx1.freebsd.org (Postfix) with ESMTP id BC06E8FC08; Mon, 27 Aug 2012 15:12:18 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id q7RFC818091514; Mon, 27 Aug 2012 15:12:08 GMT (envelope-from tim@kientzle.com) Received: from [192.168.2.143] (CiscoE3000 [192.168.1.65]) by kientzle.com with SMTP id 8wvqwunvxbytgk5wqkfzrzju8s; Mon, 27 Aug 2012 15:12:08 +0000 (UTC) (envelope-from tim@kientzle.com) Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: text/plain; charset=us-ascii From: Tim Kientzle X-Priority: 3 (Normal) In-Reply-To: Date: Mon, 27 Aug 2012 08:12:07 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> To: Warner Losh X-Mailer: Apple Mail (2.1278) Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:12:19 -0000 On Aug 27, 2012, at 6:38 AM, Warner Losh wrote: >=20 > On Aug 27, 2012, at 12:06 AM, Hans Petter Selasky wrote: >=20 >> Hi, >> Correct. >>=20 >>> We also need some rules about working with buffers obtained from >>> bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). = I >>> think the rule should be that a buffer obtained from = bus_dmamem_alloc(), >>> or more formally any region of memory mapped by a bus_dmamap_load(), = is >>> a single logical object which can only be accessed by one entity at = a >>> time. That means that there cannot be two concurrent DMA operations >>> happening in different regions of the same buffer, nor can DMA and = CPU >>> access be happening concurrently even if in different parts of the >>> buffer. =20 >>=20 >> Is this something which we can fix using a simple = __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to = be DMA loaded. >=20 > No. I don't think so. the reason is that you can't define = USB_DMA_ALGIN to be a constant on MIPS, at least, or I think ARM because = that's determined at run time. But don't mbuf structures do pretty much what Hans is suggesting? Why is mbuf okay? Tim From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 15:20:53 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B44911065673; Mon, 27 Aug 2012 15:20:53 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 7601A8FC15; Mon, 27 Aug 2012 15:20:53 +0000 (UTC) Received: by dadr6 with SMTP id r6so2589280dad.13 for ; Mon, 27 Aug 2012 08:20:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=lYrRM128qw04nZGkoy9WgB67sDTxSyD1vHe6xtBexvo=; b=ajYXEeB64Q8sgiJvPDcQkhcYLTpxX+4bxS1dk9blsyBAxhjVkMd5u5tSfiEPg7RmSB kc8Sq1JfWh/cI6jIinuo7tNbnfvkeC3SKBjbZXooJrfhgc9uBHpGQGHhpkO5lyCSkCQ2 i2zMmX8uspoG9YByjg9qYniJdsrthFCXhu7Da/k+7t4xFilPiFrwEIM8OhyYqW0NAxve 1z6Ug3KtaAIRG1Kx7fBG39DhO0ixteho3n4n+m1axU6wpj9gq30F7D+cW3RD9mM8F525 BQ+vOw8oW9uVElLkOygvPFmbcJqyX1TOotxbUMEBtAGyzQWZGau6A9Dy41T+0Husyp83 eoQg== MIME-Version: 1.0 Received: by 10.68.129.131 with SMTP id nw3mr35286569pbb.43.1346080853000; Mon, 27 Aug 2012 08:20:53 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.36.106 with HTTP; Mon, 27 Aug 2012 08:20:52 -0700 (PDT) In-Reply-To: References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> Date: Mon, 27 Aug 2012 08:20:52 -0700 X-Google-Sender-Auth: ohyisdNkmhbkzwukiyHFrt829tI Message-ID: From: Adrian Chadd To: Tim Kientzle Content-Type: text/plain; charset=ISO-8859-1 Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:20:53 -0000 On 27 August 2012 08:12, Tim Kientzle wrote: >> No. I don't think so. the reason is that you can't define USB_DMA_ALGIN to be a constant on MIPS, at least, or I think ARM because that's determined at run time. FYI, I wonder if MIPS may need some run time value for that. Right now we hide it behind needing a custom config file for each mips variant, rather than supporting one mips24k kernel for ${LOTS} of variants. > But don't mbuf structures do pretty much what Hans is suggesting? > > Why is mbuf okay? Which part of mbufs? I think the difference here is that there's no concurrent access. Ie, although you may have an mbuf header + data inside the same cache line, you wouldn't go fondling the mbuf once it's handed to the hardware. But I was under the impression that mbuf + mbuf buffers are already correctly aligned. This all honestly looks like a very i386-centric interpretation of the busdma "intention", which I think illustrates a need to better document what assumptions busdma actually makes. That does remind me, I think the ath(4) driver does the same (since it allocates its own descriptor block and then treats it as an array of descriptors for the hardware to access) - I should ensure that sizeof(ath_desc) is aligned on the relevant architecture. It gets slightly scary - AR93xx TX descriptors are "L1 cache == 128 byte aligned" which is an enormous waste of memory compared to a 16 or 32 byte aligned platform. Alas.. Adrian From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 15:24:25 2012 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 87C721065674 for ; Mon, 27 Aug 2012 15:24:25 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by mx1.freebsd.org (Postfix) with ESMTP id 14BAC8FC1F for ; Mon, 27 Aug 2012 15:24:24 +0000 (UTC) Received: by wicr5 with SMTP id r5so2747495wic.13 for ; Mon, 27 Aug 2012 08:24:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=2CRqMP01W2tz4cd+r3ibZ/i6BWeDoplZc3mxoCt4u8Q=; b=JVmO7rsaBNXEfExILtthhar+wPQvHzZBRFg72u4Rywz4PFXdCnrF/i3zb2kDXpmOiZ m5x26FPS6G2wcON0YFwLwPy57RuM44ZiH1YEM78YT79oF2LcDRUZLLfOgke6uquzmpLK 3mY6Xq6yh7Q2xPUeSuJ6MmAFsBv/ZAofRr3IvwjPS/KAVAeHXHUwH0HuVaDfcDadA9aj qaeGLWbu2/PNBm1DLoMiWuPRCKEVb7xWR9jo5kk8qm55WcEavAoP25hKyo3iMKn/Y/0R IG6GVSL04ickV2cl79SIzFxVLOAAFgUVnpUZ3EA3SUq04IXP8LGTMYw4sKCQ63D6iDcV ZSqw== MIME-Version: 1.0 Received: by 10.216.139.196 with SMTP id c46mr3464921wej.220.1346081063556; Mon, 27 Aug 2012 08:24:23 -0700 (PDT) Received: by 10.216.115.3 with HTTP; Mon, 27 Aug 2012 08:24:23 -0700 (PDT) In-Reply-To: <50325DC3.3090201@rice.edu> References: <50228F5C.1000408@rice.edu> <50269AD4.9050804@rice.edu> <5029635A.4050209@rice.edu> <502D2271.6080105@rice.edu> <50325DC3.3090201@rice.edu> Date: Mon, 27 Aug 2012 20:54:23 +0530 Message-ID: From: "Jayachandran C." To: Alan Cox Content-Type: text/plain; charset=ISO-8859-1 Cc: mips@freebsd.org Subject: Re: mips pmap patch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:24:25 -0000 On Mon, Aug 20, 2012 at 9:24 PM, Alan Cox wrote: > On 08/20/2012 05:36, Jayachandran C. wrote: >> >> On Thu, Aug 16, 2012 at 10:10 PM, Alan Cox wrote: >>> >>> On 08/15/2012 17:21, Jayachandran C. wrote: >>>> >>>> On Tue, Aug 14, 2012 at 1:58 AM, Alan Cox wrote: >>>>> >>>>> On 08/13/2012 11:37, Jayachandran C. wrote: >> >> [...] >>>>>> >>>>>> I could not test for more than an hour on 32-bit due to another >>>>>> problem (freelist 1 containing direct-mapped pages runs out of pages >>>>>> after about an hour of compile test). This issue has been there for a >>>>>> long time, I am planning to look at it when I get a chance. >>>>>> >>>>> What exactly happens? panic? deadlock? >>>> >>>> The build slows down to a crawl and hangs when it runs out of pages in >>>> the freelist. >>> >>> >>> I'd like to see the output of "sysctl vm.phys_segs" and "sysctl >>> vm.phys_free" from this machine. Even better would be running "sysctl >>> vm.phys_free" every 60 seconds during the buildworld. Finally, I'd like >>> to >>> know whether or not either "ps" or "top" shows any threads blocked on the >>> "swwrt" wait channel once things slow to a crawl. >> >> I spent some time looking at this issue. I use a very large kernel >> image with built-in root filesystem, and this takes about 120 MB out >> of the direct mapped area. The remaining pages (~64 MB) are not enough >> for the build process. If I increase free memory in this area either >> by reducing the rootfs size of by adding a few more memory segments to >> this area, the build goes through fine. > > > I'm still curious to see what "sysctl vm.phys_segs" says. It sounds like > roughly half of the direct map region is going to DRAM and half to > memory-mapped I/O devices. Is that correct? Yes, about half of the direct mapped region in 32-bit is taken by flash, PCIe and other memory mapped IO. I also made the problem even worse by not reclaiming some bootloader areas in the direct mapped region, which reduced the available direct mapped memory. Here's the output of sysctls: root@testboard:/root # sysctl vm.phys_segs vm.phys_segs: SEGMENT 0: start: 0x887e000 end: 0xc000000 domain: 0 free list: 0x887a407c SEGMENT 1: start: 0x1d000000 end: 0x1fc00000 domain: 0 free list: 0x887a407c SEGMENT 2: start: 0x20000000 end: 0xbc0b3000 domain: 0 free list: 0x887a3f38 SEGMENT 3: start: 0xe0000000 end: 0xfffff000 domain: 0 free list: 0x887a3f38 root@testboard:/root # sysctl vm.phys_free vm.phys_free: FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 | POOL 2 -- -- -- -- -- -- -- -- 8 ( 1024K) | 2877 | 0 | 0 7 ( 512K) | 0 | 1 | 0 6 ( 256K) | 1 | 0 | 0 5 ( 128K) | 0 | 1 | 0 4 ( 64K) | 0 | 1 | 0 3 ( 32K) | 0 | 1 | 0 2 ( 16K) | 0 | 1 | 0 1 ( 8K) | 0 | 0 | 0 0 ( 4K) | 0 | 0 | 0 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 | POOL 2 -- -- -- -- -- -- -- -- 8 ( 1024K) | 66 | 0 | 0 7 ( 512K) | 1 | 1 | 0 6 ( 256K) | 0 | 0 | 0 5 ( 128K) | 0 | 0 | 0 4 ( 64K) | 0 | 1 | 0 3 ( 32K) | 0 | 0 | 0 2 ( 16K) | 0 | 0 | 0 1 ( 8K) | 1 | 1 | 0 0 ( 4K) | 0 | 1 | 0 >> I also found that when the build slows down, most of the pages taken >> from freelist 1 are allocated by the UMA subsystem, which seems to >> keep quite a few pages allocated. > > > At worst, it may be necessary to disable the use of uma_small_alloc() for > this machine configuration. At best, uma_small_alloc() could be revised > opportunistically use pages in the direct map region, but have the ability > to fall back to pages that have to be mapped. I think this probably is not a bug, but a configuration problem (we cannot have such a huge built-in root filesystem when the direct mapped area is at this low). Anyway, I have checked in code to recover more areas from the bootloader, and this mostly solves the issue for me. The above output is taken before the check-in. Regards, JC. From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 15:33:24 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 748681065686; Mon, 27 Aug 2012 15:33:24 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 8FCB48FC1E; Mon, 27 Aug 2012 15:33:08 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7RFX0vq088953; Mon, 27 Aug 2012 09:33:00 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7RFWbGs031228; Mon, 27 Aug 2012 09:32:37 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> Content-Type: text/plain; charset="us-ascii" Date: Mon, 27 Aug 2012 09:32:37 -0600 Message-ID: <1346081557.1140.181.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Mark Tinguely , Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org, Hans Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:33:24 -0000 On Sun, 2012-08-26 at 17:13 -0600, Warner Losh wrote: > On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: > > In this regard, it's the busdma implementation that's broken, because it > > should bounce those IOs through a DMA-safe buffer. There's absolutely > > no rule that I've ever heard of in FreeBSD that says IO can only take > > place using memory allocated from busdma. > > That's partially true. Since BUSDMA grew up in the storage area, you must allocate the memory from busdma, or it must be page aligned has been the de-facto rule here. Where does anything say that you must allocate the memory from busdma if it's not mbuf/page-aligned? I've never seen it in any docs. I certainly find contrary evidence in existing driver code (and I'm not talking about USB here). > The mbuf and uio variants of load were invented to cope with common cases of mbufs and user I/O to properly flag things. > What does that mean, "to properly flag things"? I think with uio we come to the crux of the issue. Userland buffers are not allocated with knowledge of the busdma constraints. That leads to some choices: * We don't support IO with a userland buffer at all. * We support userland IO if the userland buffers are accidentally aligned properly for the platform, otherwise the call fails. * We support userland buffers by making any driver which wants to do so responsible for always copying the data to aligned buffers (each individual driver handles bounces). * We support userland buffers by having the busdma layer handle the bouncing when required. The first two seem untenable to me. The third one comes down to "every driver must always bounce all userland data" because the driver doesn't actually have access to the info to decide whether a userland buffer is aligned properly or not for a given platform. That leaves the option where the busdma layer handles bouncing for unaligned userland buffers. If that's possible, then the busdma layer could automatically bounce any unaligned request, whether it came from userland or a driver reading a status response into an unaligned local buffer in kernel memory. > How does busdma know that it is using memory that's not from its allocator? The busdma code allocates the map along with the buffer, and can record information in the map that it can use during mapping and sync operations to know whether it allocated the buffer. > > The rule is only that the > > proper sequence of busdma operation must be called, and beyond that it's > > up to the busdma implementation to make it work. > > No. Bouncing is needed due to poor alignment of the underlying device. Not due to cache effects. Says who? This statement seems to me to be more about opinion and dogma than about a documented API or technical concerns about how to implement it. IMO, bouncing is needed when that's the only way to make a particular busdma sequence work correctly given the parameters of the hardware and the transfer. To put it another way, the busdma subsystem is responsible for helping make DMA transfers work on a platform without every driver needing to contain platform-specific code, and bouncing is one of the techniques available to it. > There's a limited number of things that we support with busdma. Arbitrary data from malloc that might be shared with the CPU isn't on that list. > Where are those limitations documented? This isn't some small oversight in the documention, if the rule is "IO, any IO at all, can only be performed using buffers allocated from busdma" that's a pretty major thing that ought to appear in multiple manpages relating to driver development. If you don't think "any IO at all" is the right characterization, read all the way to the end before responding. > > Our biggest problem, I think, is that we don't have a sufficient > > definition of "the proper sequence of busdma operations." > > I disagree. The sequence has been known for a long time. > > > I don't think it will be very hard to make the arm and mips busdma > > implementations work correctly. It won't even be too hard to make them > > fairly efficient at bouncing small IOs (my thinking is that we can make > > small bounces no more expensive than the current partial cacheline flush > > implementation which copies the data multiple times). Bouncing large IO > > will never be efficient, but the inefficiency will be a powerful > > motivator to update drivers that do large IO to work better, such as > > using buffers allocated from busdma. > > I don't think the cache line problem can be solved with bounce buffers. Trying to accommodate broken drivers is what lead us to this spot. We need to fix the broken drivers. If that's impossible, then the best we can do is have the driver set a 'always bounce' flag in the tag it creates and use that to always bounce for operations through that tag. So you'd be okay with a driver setting a flag that says "always bounce" but not okay with the busdma layer bouncing only when it's actually necessary? I'm confused -- do you think the busdma layer will be unable to detect when it's necessary unless directed from the outside? Let me pose a question back to you... if it is up to a driver to allocate DMA buffers using the busdma allocator, how is any given driver to know whether DMA is going to be involved in the transfer? Don't think USB or ATA here... think iicbus/foo_temperature_sensor, or the mmc/sd driver. Those drivers know nothing about the hardware bridge layers beneath them and whether DMA is involved or not. They just know "I need to read a 2-byte value from a register on this IIC chip" or "I need to retrieve the 32-bit CSD data from the SD card" and they accomplish that by calling a read method of some lower-level bridge driver. In each of those cases we have both PIO and DMA implementations of the lower-level drivers that move bits over a wire. If the concensus is that such drivers should always allocate all IO buffers using busdma, even though they have no idea whether DMA is going to be involved, then we certainly have a lot of work ahead of us in terms of "fixing broken drivers." That also implies that bus_dmamem_alloc() is misnamed and implemented in the wrong place, because it's not really about dma at all. If you push the responsibility down to the layer where it is known whether DMA is going to be involved, then we're back to the need for many individual drivers to bounce every request, because they don't have access to the info needed to know whether bouncing is required or not for a given buffer they were handed from higher layers. (Remember when I proposed exporting that info to drivers so they could decide whether a buffer is dma-aligned or not, the response was "No, keep it all hidden in the busdma layer," which I think is the right response.) Or if you push the responsibility down to the busdma layer where all that info resides, all drivers which use DMA and adhere to the busdma rules just work, without needing to know anything about the buffers that were handed to them and without needing to know anything about the platform requirements. -- Ian From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 15:34:35 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CEA921065673 for ; Mon, 27 Aug 2012 15:34:35 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 729E88FC30 for ; Mon, 27 Aug 2012 15:34:34 +0000 (UTC) Received: by obbun3 with SMTP id un3so11030980obb.13 for ; Mon, 27 Aug 2012 08:34:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:x-priority :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer:x-gm-message-state; bh=sv9ysqQt0d8GIYk9LOiQWK2BMH1BDWf3NslmGnAmKbY=; b=cZ8Ht/mweFYsCol0lJedZgreCEzY6L0oBvlErTwfIO9JFx2LEhB6m5XJpVKbMxiQmm RcWYlPtUJuIgHnjkiXX+8YxK8xSJYNOvOvwVGEacgxvUBZr1fgLun/hChSrMGJV9uu12 HV+CoBDv8pgTVZWtDhMmTdcONZ2RBpdv4c8MO0jj8aV53r72B2KVfsWEe6j2obOXQsQS g9MhVqmnL0lmX9DFRJhBDAe2B5h5xiwf4adv8g0svMQhWq3c3CD3Stv2LBlPpfzC8ALq 0AZDIH09eunZKZ3tjoJ8iEDanDk3iL1SC6Tokcmb9Wy0EU/2yqSQ+nSLQA8Y0pD25855 CqgA== Received: by 10.182.190.69 with SMTP id go5mr10597608obc.43.1346081674507; Mon, 27 Aug 2012 08:34:34 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id cp8sm17348774obc.23.2012.08.27.08.34.33 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 08:34:33 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh X-Priority: 3 (Normal) In-Reply-To: Date: Mon, 27 Aug 2012 09:34:30 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <45C204C3-D4CE-4B00-886A-A88A0FE7CAD7@bsdimp.com> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> To: Tim Kientzle X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQlETiLG3gXUe5Tae8Y9mvIwhHVdBloBau9e4PawB5ZpzsBhoE0tPLcsyxldsc45IPjhWCZl Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:34:35 -0000 On Aug 27, 2012, at 9:12 AM, Tim Kientzle wrote: >=20 > On Aug 27, 2012, at 6:38 AM, Warner Losh wrote: >=20 >>=20 >> On Aug 27, 2012, at 12:06 AM, Hans Petter Selasky wrote: >>=20 >>> Hi, >>> Correct. >>>=20 >>>> We also need some rules about working with buffers obtained from >>>> bus_dmamem_alloc() and external buffers passed to = bus_dmamap_load(). I >>>> think the rule should be that a buffer obtained from = bus_dmamem_alloc(), >>>> or more formally any region of memory mapped by a = bus_dmamap_load(), is >>>> a single logical object which can only be accessed by one entity at = a >>>> time. That means that there cannot be two concurrent DMA = operations >>>> happening in different regions of the same buffer, nor can DMA and = CPU >>>> access be happening concurrently even if in different parts of the >>>> buffer. =20 >>>=20 >>> Is this something which we can fix using a simple = __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to = be DMA loaded. >>=20 >> No. I don't think so. the reason is that you can't define = USB_DMA_ALGIN to be a constant on MIPS, at least, or I think ARM because = that's determined at run time. >=20 > But don't mbuf structures do pretty much what Hans is suggesting? They kinda do, and kinda don't. > Why is mbuf okay? mbuf is OK because it is never changed while the DMA is pending to the = buffers. That is, m_hdr and m_ext are invariant while the device owns = the memory. In addition, mbuf allocations are so large that no two mbufs = share the same cache line (although it looks like 256 might be too small = to avoid that on some MIPS processors). usb buffers do not obey these = same rules, otherwise none of the stuff we're talking about would = matter. Warner= From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 15:40:24 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8E9A71065679 for ; Mon, 27 Aug 2012 15:40:24 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1C1648FC26 for ; Mon, 27 Aug 2012 15:40:23 +0000 (UTC) Received: by obbun3 with SMTP id un3so11051468obb.13 for ; Mon, 27 Aug 2012 08:40:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=t1YrUXxwc6T+C4ku+1nQRySvTX9kp0FRv2Gmd0Bg2co=; b=a1mCsdFrHrOgO6CVnsUzJ/bQ5l2PdH4hQzVpxCAMFP71oRYZJfQSJZEcrHdz1miCb1 /1HIuPzO9btcE8ygCgoiXEly76S+9A2/ZzwumpBOCvXLZlpaNBakJyJfy1AwUD4trruj LqjTjjjKQFh8132Tqf2rg0L9ktRLNdyg7qJ5Q9uLovUryfenkjpRbxXsgqfT5oxpNKgX WoFzxY/W97EF7Q5Ji/ARRT3haPaA9JxK0pC8EatyI8UpGP38viz3t0T2tJyJlnSeQGqU LvB8nKRxWYk4/hZDrXzJQrOSWA3AKPXF74+bSXtkpFhDr69R7t+BgJD7qiVdCEZ1Zv9F BywQ== Received: by 10.60.3.35 with SMTP id 3mr10500585oez.139.1346082023466; Mon, 27 Aug 2012 08:40:23 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id hz6sm17394331obb.1.2012.08.27.08.40.21 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 08:40:21 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Mon, 27 Aug 2012 09:40:19 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <9642068B-3C66-42BD-8515-14F734B3FF89@bsdimp.com> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> To: Adrian Chadd X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQnV7kuwDh98NncF+eDHzMT5XGC25yNMWbW20dT4+ep96VudHChGlVhuHxD8mwOx8oIcEBA9 Cc: Tim Kientzle , Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:40:24 -0000 On Aug 27, 2012, at 9:20 AM, Adrian Chadd wrote: > On 27 August 2012 08:12, Tim Kientzle wrote: >> But don't mbuf structures do pretty much what Hans is suggesting? >>=20 >> Why is mbuf okay? >=20 > Which part of mbufs? >=20 > I think the difference here is that there's no concurrent access. Ie, > although you may have an mbuf header + data inside the same cache > line, you wouldn't go fondling the mbuf once it's handed to the > hardware. >=20 > But I was under the impression that mbuf + mbuf buffers are already > correctly aligned. Not quite. mbufs are large enough and aligned enough so that no two = mbufs are in the same cache line. The data inside the mbuf on 32-bit = platforms is 24 bytes from the start of the mbuf, so the mbuf data and = mbuf header do share the same cache line, but as you say, nobody touches = any part of the mbuf will the hardware has it. Recall that it isn't = alignment of where the buffer starts that matters here, but rather = alignment to ensure that there's no sharing of cache lines between = structures. > This all honestly looks like a very i386-centric interpretation of the > busdma "intention", which I think illustrates a need to better > document what assumptions busdma actually makes. Yes. x86 lets you play fast and loose with many of these things. We've = tried to accommodate this for a long time, but that is lame. > That does remind me, I think the ath(4) driver does the same (since it > allocates its own descriptor block and then treats it as an array of > descriptors for the hardware to access) - I should ensure that > sizeof(ath_desc) is aligned on the relevant architecture. It gets > slightly scary - AR93xx TX descriptors are "L1 cache =3D=3D 128 byte > aligned" which is an enormous waste of memory compared to a 16 or 32 > byte aligned platform. Alas.. The problem is with cache line sharing, not necessarily with alignment. = If you are only ever using one of them at a time, or if you have perfect = hygiene, you can cope with this situation without undue waste. The = perfect hygiene might be hard sometimes. Warner From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 15:57:54 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6FC881065674 for ; Mon, 27 Aug 2012 15:57:54 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0C5D78FC1B for ; Mon, 27 Aug 2012 15:57:53 +0000 (UTC) Received: by obbun3 with SMTP id un3so11111199obb.13 for ; Mon, 27 Aug 2012 08:57:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=+6b5VLtRymOQqJ25JKslOXKH/1wAxmLCqLr7LE+kYFs=; b=NSclt61jnZO04FAwl5KjPLIABwnBX55To/0oEr71M2ZoALg0KQLeu2tS4Dn+tjfb9s BWq1Wn/MZosGg/ifJDeZmT6byFV//d+M+v/WuqKMoMBdKemwIyH+mxQaESDHgMoclT5u QAIAUknhqlYfRxCb+Tv/QwhPl6obQ6Ai7z1XzPhYHLYtyOvegEZY1OKskhxYV2gcunMD nqhiw3kx0Uwch8B3LRC43mojblNoM+xY7q/eqPxl7vjzZ82fndEdbonU/sFXxbxhS576 pZiDzNk63+B2NgGxClqNphTLHOkcCYGNQ8sX9EhrWd67UC3Tc3APPv1FMDRUlzPvOvep Wjfw== Received: by 10.60.19.34 with SMTP id b2mr10134983oee.41.1346083073414; Mon, 27 Aug 2012 08:57:53 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id ov5sm9250003obc.2.2012.08.27.08.57.51 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 08:57:52 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346081557.1140.181.camel@revolution.hippie.lan> Date: Mon, 27 Aug 2012 09:57:48 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> <1346081557.1140.181.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQnS0YcQMow+94SRN7e2OyYxziB6S6OR7s57L1ecaT439x90QGHPOHRU33rVz+4y6/vV5qI2 Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, Mark Tinguely , freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:57:54 -0000 On Aug 27, 2012, at 9:32 AM, Ian Lepore wrote: > On Sun, 2012-08-26 at 17:13 -0600, Warner Losh wrote: >> On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: >>> In this regard, it's the busdma implementation that's broken, = because it >>> should bounce those IOs through a DMA-safe buffer. There's = absolutely >>> no rule that I've ever heard of in FreeBSD that says IO can only = take >>> place using memory allocated from busdma. >>=20 >> That's partially true. Since BUSDMA grew up in the storage area, you = must allocate the memory from busdma, or it must be page aligned has = been the de-facto rule here. =20 >=20 > Where does anything say that you must allocate the memory from busdma = if > it's not mbuf/page-aligned? I've never seen it in any docs. I > certainly find contrary evidence in existing driver code (and I'm not > talking about USB here). Where does it say that you are allowed to not use the routines? >> The mbuf and uio variants of load were invented to cope with common = cases of mbufs and user I/O to properly flag things. >>=20 >=20 > What does that mean, "to properly flag things"? >=20 > I think with uio we come to the crux of the issue. Userland buffers = are > not allocated with knowledge of the busdma constraints. That leads to > some choices: >=20 > * We don't support IO with a userland buffer at all. You may have to bounce here. > * We support userland IO if the userland buffers are accidentally > aligned properly for the platform, otherwise the call fails. You may have to bounce here. > * We support userland buffers by making any driver which wants to > do so responsible for always copying the data to aligned = buffers > (each individual driver handles bounces). This is what most drivers that want to do zero copying do. > * We support userland buffers by having the busdma layer handle > the bouncing when required. I thought that's what the uio variants of the map loading routines were = supposed to do. > The first two seem untenable to me. The third one comes down to = "every > driver must always bounce all userland data" because the driver = doesn't > actually have access to the info to decide whether a userland buffer = is > aligned properly or not for a given platform. >=20 > That leaves the option where the busdma layer handles bouncing for > unaligned userland buffers. If that's possible, then the busdma layer > could automatically bounce any unaligned request, whether it came from > userland or a driver reading a status response into an unaligned local > buffer in kernel memory. Right, the uio load option is supposed to do this. >> How does busdma know that it is using memory that's not from its = allocator? >=20 > The busdma code allocates the map along with the buffer, and can = record > information in the map that it can use during mapping and sync > operations to know whether it allocated the buffer. >=20 >>> The rule is only that the >>> proper sequence of busdma operation must be called, and beyond that = it's >>> up to the busdma implementation to make it work. =20 >>=20 >> No. Bouncing is needed due to poor alignment of the underlying = device. Not due to cache effects. >=20 > Says who? This statement seems to me to be more about opinion and = dogma > than about a documented API or technical concerns about how to = implement > it. IMO, bouncing is needed when that's the only way to make a > particular busdma sequence work correctly given the parameters of the > hardware and the transfer. Says the people that wrote and use busdma? There's no requirement for = cache effects for the transfer, so there's no constraints for the = transfer. The requirement is about having a buffer that's suitable for = DMA. This means either the buffer (not the start of the transfer) be = aligned to a cache line and be an integral number of cache lines in = size, or it means that you have 100% perfect hygiene when it comes to = ensuring the CPU touches the buffer or the device touches it and ever = have cross threading. I'd agree that better docs here would help. > To put it another way, the busdma subsystem is responsible for helping > make DMA transfers work on a platform without every driver needing to > contain platform-specific code, and bouncing is one of the techniques > available to it. Buffers allocated through the busdma system do this. >> There's a limited number of things that we support with busdma. = Arbitrary data from malloc that might be shared with the CPU isn't on = that list. >>=20 >=20 > Where are those limitations documented? =20 Where is it documented that memory returned from malloc may be used for = DMA? I agree that docs could be better here. > This isn't some small oversight in the documention, if the rule is = "IO, > any IO at all, can only be performed using buffers allocated from > busdma" that's a pretty major thing that ought to appear in multiple > manpages relating to driver development. >=20 > If you don't think "any IO at all" is the right characterization, read > all the way to the end before responding. Yes. I'd state it another way. "Buffers returned from the busdma = allocation routines is always safe." Everything else may or may not be = safe. mbufs, for example, happen to be safe because there is a rigid = protocol for sharing between the driver and the device. uio buffers can = be made safe. busdma has routines to ensure that these types of data = are handled properly. >>> Our biggest problem, I think, is that we don't have a sufficient >>> definition of "the proper sequence of busdma operations." >>=20 >> I disagree. The sequence has been known for a long time. >>=20 >>> I don't think it will be very hard to make the arm and mips busdma >>> implementations work correctly. It won't even be too hard to make = them >>> fairly efficient at bouncing small IOs (my thinking is that we can = make >>> small bounces no more expensive than the current partial cacheline = flush >>> implementation which copies the data multiple times). Bouncing = large IO >>> will never be efficient, but the inefficiency will be a powerful >>> motivator to update drivers that do large IO to work better, such as >>> using buffers allocated from busdma. >>=20 >> I don't think the cache line problem can be solved with bounce = buffers. Trying to accommodate broken drivers is what lead us to this = spot. We need to fix the broken drivers. If that's impossible, then = the best we can do is have the driver set a 'always bounce' flag in the = tag it creates and use that to always bounce for operations through that = tag. >=20 > So you'd be okay with a driver setting a flag that says "always = bounce" > but not okay with the busdma layer bouncing only when it's actually > necessary? I'm confused -- do you think the busdma layer will be = unable > to detect when it's necessary unless directed from the outside? Actually, it is good you are confused. I was wrong when I said I'd be = happy with a flag that says always bounce. The reason is that the driver = can do this all the time for those cases like the at91 mci driver does. > Let me pose a question back to you... if it is up to a driver to > allocate DMA buffers using the busdma allocator, how is any given = driver > to know whether DMA is going to be involved in the transfer? =20 because it does the dma? > Don't think USB or ATA here... think iicbus/foo_temperature_sensor, or > the mmc/sd driver. Those drivers know nothing about the hardware = bridge > layers beneath them and whether DMA is involved or not. They just = know > "I need to read a 2-byte value from a register on this IIC chip" or "I > need to retrieve the 32-bit CSD data from the SD card" and they > accomplish that by calling a read method of some lower-level bridge > driver. In each of those cases we have both PIO and DMA = implementations > of the lower-level drivers that move bits over a wire. The bridge here would know and have to cope. However, the real issue in = this case wouldn't be the transfer alignment, but what traffic might be = happening to other parts of the cache line. For small things like this, typically data copies are involved. There's = a few places that break these rules (I think mci might be breaking the = rules for small transfers, for example). > If the concensus is that such drivers should always allocate all IO > buffers using busdma, even though they have no idea whether DMA is = going > to be involved, then we certainly have a lot of work ahead of us in > terms of "fixing broken drivers." That also implies that > bus_dmamem_alloc() is misnamed and implemented in the wrong place, > because it's not really about dma at all. You can't get around the hardware requirement that any I/O going to the = device cannot share cache lines with other data on the system. > If you push the responsibility down to the layer where it is known > whether DMA is going to be involved, then we're back to the need for > many individual drivers to bounce every request, because they don't = have > access to the info needed to know whether bouncing is required or not > for a given buffer they were handed from higher layers. (Remember = when > I proposed exporting that info to drivers so they could decide whether = a > buffer is dma-aligned or not, the response was "No, keep it all hidden > in the busdma layer," which I think is the right response.) Right now we support only a few things doing DMA on the system. We = support pages, mbufs and to a limited extent uio buffers (but to be = honest, we may have exposure on smaller transfers here). > Or if you push the responsibility down to the busdma layer where all > that info resides, all drivers which use DMA and adhere to the busdma > rules just work, without needing to know anything about the buffers = that > were handed to them and without needing to know anything about the > platform requirements. I'm not sure I follow this last bit... Warner From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 16:09:12 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1AA23106566C; Mon, 27 Aug 2012 16:09:12 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id D96788FC15; Mon, 27 Aug 2012 16:08:59 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7RG8xAG089488; Mon, 27 Aug 2012 10:08:59 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7RG8aPI031283; Mon, 27 Aug 2012 10:08:36 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> Content-Type: text/plain; charset="us-ascii" Date: Mon, 27 Aug 2012 10:08:36 -0600 Message-ID: <1346083716.1140.212.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arch@freebsd.org, freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 16:09:12 -0000 On Sun, 2012-08-26 at 17:03 -0600, Warner Losh wrote: > On Aug 26, 2012, at 11:42 AM, Ian Lepore wrote: > > > > The busdma manpage currently has some vague words about the usage and > > sequencing of sync ops, such as "If read and write operations are not > > preceded and followed by the appropriate synchronization operations, > > behavior is undefined." I think we should more explicitly spell out > > what the appropriate sequences are. In particular: > > > > * The PRE and POST operations must occur in pairs; a PREREAD must > > be followed eventually by a POSTREAD and a PREWRITE must be > > followed by a POSTWRITE. > > PREREAD means "I am about to tell the device to put data here, have whaterver things might be pending in the CPU complex to get out of the way." usually this means 'invalidate the cache for that range', but not always. POSTREAD means 'The device's DMA is done, I'd like to start accessing it now.' If the memory will be thrown away without being looked at, then does the driver necessarily need to issue the POSTREAD? I think so, but I don't know if that's a new requirement. > One of the things that scares me most is the idea that driver writers will glance at an existing implementation and think "Oh I have no need to ever call POSTWRITE because it's implemented as a no-op and I can save the call overhead." In fact we have drivers coded like that now. We've got an API here to support arbitrary hardware, some of which may not have been designed yet. I think it's really unsafe to say that a driver can decide that it's safe to elide some calls in some situations just because that's safe with the busdma implementations that exist today. > > We also need some rules about working with buffers obtained from > > bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). I > > think the rule should be that a buffer obtained from bus_dmamem_alloc(), > > or more formally any region of memory mapped by a bus_dmamap_load(), is > > a single logical object which can only be accessed by one entity at a > > time. That means that there cannot be two concurrent DMA operations > > happening in different regions of the same buffer, nor can DMA and CPU > > access be happening concurrently even if in different parts of the > > buffer. > > There's something subtle that I'm missing. Why would two DMA operations be disallowed? The rest makes good sense. > If two DMAs are going on concurrently in the same buffer, one is going to finish before the other, leading to a POSTxxxx sync op happening for one DMA operation while the other is still in progress. The unit of granularity for sync operations is the mapped region, so now you're syncing access to a region which still has active DMA happening within it. While I think it's really an API definition issue, think about it in terms of a potential implementation... What if the CPU had to access the memory as part of the sync for the first DMA that completes, while the second is still running? Now you've got pretty much exactly the same situation as when a driver subdivides a buffer without knowing about the cache alignment; you end up with the CPU and DMA touching data in the same cachline and no sequence of flush/invalidate can be g'teed to preserve all data correctly. > > I've always thought that allocating a dma buffer feels like a big > > hassle. You sometimes have to create a tag for the sole purpose of > > setting the maxsize to get the buffer size you need when you call > > bus_dmamem_alloc(). If bus_dmamem_alloc() took a size parm you could > > just use your parent tag, or a generic tag appropriate to all the IO > > you're doing for a given device. If you need a variety of buffers for > > small control and command and status transfers of different sizes, you > > end up having to manage up to a dozen tags and maps and buffers. It's > > all very clunky and inconvenient. It's just the sort of thing that > > makes you want to allocate a big buffer and subdivide it. Surely we > > could do something to make it easier? > > You'd wind up creating a quick tag on the fly for the bus_dmamap_alloc if you wanted to do this. Cleanup then becomes unclear. > My point is that the only piece of information in the tag that's specific to the allocation is the maxsize. If the allocation size were passed to bus_dmamem_alloc() then you wouldn't need a tag specific to that buffer, a generic tag for the device would work and you could allocate a dozen different-sized buffers all using that one tag, and the allocator would just have to sanity-check the allocation size against the tag's maxsize. -- Ian From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 16:53:31 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9B291106566B; Mon, 27 Aug 2012 16:53:31 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2CDD38FC15; Mon, 27 Aug 2012 16:53:31 +0000 (UTC) Received: by obbun3 with SMTP id un3so11293602obb.13 for ; Mon, 27 Aug 2012 09:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=gJHpHTLGok1mdDMngy2k8DW11dnWPLHSjDIlGfmoy5o=; b=ezk1B4AI/J8/l3RTz+S6U95QsnA6aZcxymJbdAgyM5BlQwkdg/RMBmt4Dgjwh2l70q z/1AXux2Ac7RLCGrdDiGhpsbFyAXLMgynAst11zcozzCEGe2BmW/T+8NgIQ1VIjc8x76 cB8Cqif38GOFiNDnLq5VZIpqzgLUJf7Q9RhpESC5G9idEgjqMRxOUt7ZT7n7bBJUuwna dO6mfIgIAo6YJzw2pa7mLjUDpbH+X4sCC4+wqByxReoiWrIVX+bmntHNLgTRI72lwwnr Uf/WlLRWeWQj6Ri5/BINCkEi2RKCcBJGA0lnFjgFj1ngZxTbdwJXxuFy890xokaJcrBf 3CFQ== MIME-Version: 1.0 Received: by 10.182.0.13 with SMTP id 13mr10283841oba.55.1346086410479; Mon, 27 Aug 2012 09:53:30 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.76.8.98 with HTTP; Mon, 27 Aug 2012 09:53:29 -0700 (PDT) In-Reply-To: <1346083716.1140.212.camel@revolution.hippie.lan> References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <1346083716.1140.212.camel@revolution.hippie.lan> Date: Mon, 27 Aug 2012 09:53:29 -0700 X-Google-Sender-Auth: _mn46Ck_2vyos8KJFH_aDJV2o8M Message-ID: From: Adrian Chadd To: Ian Lepore Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 16:53:31 -0000 On 27 August 2012 09:08, Ian Lepore wrote: > If two DMAs are going on concurrently in the same buffer, one is going > to finish before the other, leading to a POSTxxxx sync op happening for > one DMA operation while the other is still in progress. The unit of > granularity for sync operations is the mapped region, so now you're > syncing access to a region which still has active DMA happening within > it. Right. But the enforced idea is "DMA up to this point should be flushed to memory." > While I think it's really an API definition issue, think about it in > terms of a potential implementation... What if the CPU had to access the > memory as part of the sync for the first DMA that completes, while the > second is still running? Now you've got pretty much exactly the same > situation as when a driver subdivides a buffer without knowing about the > cache alignment; you end up with the CPU and DMA touching data in the > same cachline and no sequence of flush/invalidate can be g'teed to > preserve all data correctly. Right. So you realise at that point you can't win and you stick each of those pieces in a different cache line. Adrian From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 18:54:10 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49B9F1065714; Mon, 27 Aug 2012 18:54:10 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id D55CC8FC0A; Mon, 27 Aug 2012 18:54:09 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q7RIs0nK023175; Mon, 27 Aug 2012 21:54:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q7RIrlGG059144; Mon, 27 Aug 2012 21:53:47 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q7RIrked059143; Mon, 27 Aug 2012 21:53:46 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 27 Aug 2012 21:53:46 +0300 From: Konstantin Belousov To: Warner Losh Message-ID: <20120827185346.GE33100@deviant.kiev.zoral.com.ua> References: <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BwgxZU5iS1dWSTZh" Content-Disposition: inline In-Reply-To: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Mark Tinguely , Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 18:54:10 -0000 --BwgxZU5iS1dWSTZh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Aug 26, 2012 at 05:13:31PM -0600, Warner Losh wrote: >=20 > On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: > > In this regard, it's the busdma implementation that's broken, because it > > should bounce those IOs through a DMA-safe buffer. There's absolutely > > no rule that I've ever heard of in FreeBSD that says IO can only take > > place using memory allocated from busdma. >=20 > That's partially true. Since BUSDMA grew up in the storage area, you > must allocate the memory from busdma, or it must be page aligned has > been the de-facto rule here. The mbuf and uio variants of load were > invented to cope with common cases of mbufs and user I/O to properly > flag things. I once looked at x86 bus_dmamap_load_uio(), and I was unable to understand how to use it with usermode uio. I think this is a good moment to ask. Most existing users use UIO_SYSSPACE, but several crypto drivers might allow the UIO_USERSPACE for them. For UIO_USERSPACE, if the page is not resident, the pmap_extract() call from _bus_dmamap_load_buffer() returns 0. So the i/o happens to the page located at 0, which contains real mode IVT and other BIOS sensitive tables. Worse, if the page is resident, but it is mapped at the region which requires COW on write, then DMA will be performed to the wrong page which is typically shared with other innocent users. to the COW area which was not yet copied, Am I missing some trick there ? --BwgxZU5iS1dWSTZh Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAlA7wjoACgkQC3+MBN1Mb4hJsACg9rY5d4jEjVBz1gmM0pcHvbwY 1N0AoKmBOQ6HFrErCDyO1ME2rbSRZLt/ =iNrn -----END PGP SIGNATURE----- --BwgxZU5iS1dWSTZh-- From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 19:01:31 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 89DBC106564A; Mon, 27 Aug 2012 19:01:31 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 1F8C98FC08; Mon, 27 Aug 2012 19:01:19 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7RJ1BMt091960; Mon, 27 Aug 2012 13:01:17 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7RJ0mgZ031745; Mon, 27 Aug 2012 13:00:48 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Adrian Chadd In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <1346083716.1140.212.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Mon, 27 Aug 2012 13:00:47 -0600 Message-ID: <1346094047.1140.264.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 19:01:31 -0000 On Mon, 2012-08-27 at 09:53 -0700, Adrian Chadd wrote: > On 27 August 2012 09:08, Ian Lepore wrote: > > > If two DMAs are going on concurrently in the same buffer, one is going > > to finish before the other, leading to a POSTxxxx sync op happening for > > one DMA operation while the other is still in progress. The unit of > > granularity for sync operations is the mapped region, so now you're > > syncing access to a region which still has active DMA happening within > > it. > > Right. But the enforced idea is "DMA up to this point should be > flushed to memory." > > > While I think it's really an API definition issue, think about it in > > terms of a potential implementation... What if the CPU had to access the > > memory as part of the sync for the first DMA that completes, while the > > second is still running? Now you've got pretty much exactly the same > > situation as when a driver subdivides a buffer without knowing about the > > cache alignment; you end up with the CPU and DMA touching data in the > > same cachline and no sequence of flush/invalidate can be g'teed to > > preserve all data correctly. > > Right. So you realise at that point you can't win and you stick each > of those pieces in a different cache line. > Actually, I think that even discussing cache lines in this context is a mistake (yeah, I'm the one who did so above, in trying to relate an abstract API design concept to a real-world hardware example). Drivers are not supposed to know about interactions between DMA transfers and cache lines or other machine-specific constraints; that info is supposed to be encapsulated and hidden within busdma. I think a driver making the assumption that it can do DMA safely on a buffer as long as that buffer is cacheline-granular is just as flawed as assuming that it can do DMA safely on any arbitrarily sized and aligned buffer. So the right way to "stick each of those pieces in a different cache line" is to allocate two different buffers, one per concurrent DMA transfer. Or, really, to use two separate busdma mappings would be the more rigorous way to say it, since the mapping is the operation at which constraints come into play. Thinking of it that way then drives the need to document that if multiple mappings describe the same area of physical memory, then concurrent operations on those maps yield unpredictable results. -- Ian From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 19:18:55 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ABBDE106566C for ; Mon, 27 Aug 2012 19:18:55 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0923A8FC23 for ; Mon, 27 Aug 2012 19:18:53 +0000 (UTC) Received: by obbun3 with SMTP id un3so11731250obb.13 for ; Mon, 27 Aug 2012 12:18:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=NWAxBZraqJ39VsS1Q2psNADkTl2YA8RO4lXjmSt0qtk=; b=KOo8ElXPBc93WpHOrIxtR0I2uTWfkCaHbnaA9IFb8/FAiD5cR3AFLWbarossWPVE3A YjpO5UhK5j5hAgX1gEZeieAafxRpjqFcxIJlN3bGXub41JyKgZCD08OARL7pEQAO5y+1 uXTCRYhzGa7NoDI6Nxdh4N3acAnaAClQfhvMMxitpo1+XOvtfgy9a5IBkm3uWET2cxZb vQm6t43Bf7y5VkU9Qkd26/hCHPVbrS9D1tyEN2ibu+oN8ahk5eS4VskRnFtBQOhHN2Sf aLPOty2HGbt6qmBL56LQXe1II5d/tcZtWh3zFVhrhe31FMwKTff+MaIBv+0mlf5mUJFr nioQ== Received: by 10.182.182.71 with SMTP id ec7mr8049310obc.22.1346095133419; Mon, 27 Aug 2012 12:18:53 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id r1sm13110813oea.4.2012.08.27.12.18.51 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 12:18:52 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346094047.1140.264.camel@revolution.hippie.lan> Date: Mon, 27 Aug 2012 13:18:49 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <1346083716.1140.212.camel@revolution.hippie.lan> <1346094047.1140.264.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQmJUzd466VOfQbLC6BAloBAOBaSbVBH7/9WI6TDatah8sIVwLHIf/j5CztWnCYG3jwh/EMo Cc: freebsd-arch@freebsd.org, freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 19:18:55 -0000 On Aug 27, 2012, at 1:00 PM, Ian Lepore wrote: > On Mon, 2012-08-27 at 09:53 -0700, Adrian Chadd wrote: >> On 27 August 2012 09:08, Ian Lepore = wrote: >>=20 >>> If two DMAs are going on concurrently in the same buffer, one is = going >>> to finish before the other, leading to a POSTxxxx sync op happening = for >>> one DMA operation while the other is still in progress. The unit of >>> granularity for sync operations is the mapped region, so now you're >>> syncing access to a region which still has active DMA happening = within >>> it. >>=20 >> Right. But the enforced idea is "DMA up to this point should be >> flushed to memory." >>=20 >>> While I think it's really an API definition issue, think about it in >>> terms of a potential implementation... What if the CPU had to access = the >>> memory as part of the sync for the first DMA that completes, while = the >>> second is still running? Now you've got pretty much exactly the = same >>> situation as when a driver subdivides a buffer without knowing about = the >>> cache alignment; you end up with the CPU and DMA touching data in = the >>> same cachline and no sequence of flush/invalidate can be g'teed to >>> preserve all data correctly. >>=20 >> Right. So you realise at that point you can't win and you stick each >> of those pieces in a different cache line. >>=20 >=20 > Actually, I think that even discussing cache lines in this context is = a > mistake (yeah, I'm the one who did so above, in trying to relate an > abstract API design concept to a real-world hardware example). >=20 > Drivers are not supposed to know about interactions between DMA > transfers and cache lines or other machine-specific constraints; that > info is supposed to be encapsulated and hidden within busdma. I think = a > driver making the assumption that it can do DMA safely on a buffer as > long as that buffer is cacheline-granular is just as flawed as = assuming > that it can do DMA safely on any arbitrarily sized and aligned buffer. >=20 > So the right way to "stick each of those pieces in a different cache > line" is to allocate two different buffers, one per concurrent DMA > transfer. Or, really, to use two separate busdma mappings would be = the > more rigorous way to say it, since the mapping is the operation at = which > constraints come into play. Thinking of it that way then drives the > need to document that if multiple mappings describe the same area of > physical memory, then concurrent operations on those maps yield > unpredictable results. Despite what I said earlier, I think this is sane. busdma only should = support one DMA active at a time into a buffer. If the driver wants to = do two different ones, they are on their own. Warner= From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 19:31:53 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0A4EE106568A; Mon, 27 Aug 2012 19:31:53 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id CDF9E8FC15; Mon, 27 Aug 2012 19:31:52 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 0C5F3B983; Mon, 27 Aug 2012 15:31:52 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Date: Mon, 27 Aug 2012 15:31:42 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p17; KDE/4.5.5; amd64; ; ) References: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> <20120827185346.GE33100@deviant.kiev.zoral.com.ua> In-Reply-To: <20120827185346.GE33100@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201208271531.42725.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 27 Aug 2012 15:31:52 -0400 (EDT) Cc: Mark Tinguely , Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, Konstantin Belousov Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 19:31:53 -0000 On Monday, August 27, 2012 2:53:46 pm Konstantin Belousov wrote: > On Sun, Aug 26, 2012 at 05:13:31PM -0600, Warner Losh wrote: > > > > On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: > > > In this regard, it's the busdma implementation that's broken, because it > > > should bounce those IOs through a DMA-safe buffer. There's absolutely > > > no rule that I've ever heard of in FreeBSD that says IO can only take > > > place using memory allocated from busdma. > > > > That's partially true. Since BUSDMA grew up in the storage area, you > > must allocate the memory from busdma, or it must be page aligned has > > been the de-facto rule here. The mbuf and uio variants of load were > > invented to cope with common cases of mbufs and user I/O to properly > > flag things. > > I once looked at x86 bus_dmamap_load_uio(), and I was unable to > understand how to use it with usermode uio. I think this is a good > moment to ask. Most existing users use UIO_SYSSPACE, but several crypto > drivers might allow the UIO_USERSPACE for them. > > For UIO_USERSPACE, if the page is not resident, the pmap_extract() call from > _bus_dmamap_load_buffer() returns 0. So the i/o happens to the page > located at 0, which contains real mode IVT and other BIOS sensitive tables. > > Worse, if the page is resident, but it is mapped at the region which > requires COW on write, then DMA will be performed to the wrong page > which is typically shared with other innocent users. to the COW area > which was not yet copied, > > Am I missing some trick there ? No. The caller is required to wire the pages first in some manner. In general bus_dmamap_load_uio() isn't a good idea. I do believe the crypto drivers are careful to wire the buffer first. I think requiring the caller to wire is the only sane way that can be used. Also, doing DMA to a stack variable is absolutely horrible for a related reason since presumably the thread will block while it waits for the DMA to complete, and a sleeping thread can be swapped out (including having it's stack swapped out). -- John Baldwin From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 22:08:17 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E93E4106566C; Mon, 27 Aug 2012 22:08:16 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 6F7F08FC0A; Mon, 27 Aug 2012 22:08:04 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7RM7q0C094077; Mon, 27 Aug 2012 16:07:59 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7RM7Uu6031952; Mon, 27 Aug 2012 16:07:30 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: <9642068B-3C66-42BD-8515-14F734B3FF89@bsdimp.com> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <9642068B-3C66-42BD-8515-14F734B3FF89@bsdimp.com> Content-Type: text/plain; charset="us-ascii" Date: Mon, 27 Aug 2012 16:07:30 -0600 Message-ID: <1346105250.1140.314.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Tim Kientzle , Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 22:08:17 -0000 On Mon, 2012-08-27 at 09:40 -0600, Warner Losh wrote: > On Aug 27, 2012, at 9:20 AM, Adrian Chadd wrote: > > That does remind me, I think the ath(4) driver does the same (since it > > allocates its own descriptor block and then treats it as an array of > > descriptors for the hardware to access) - I should ensure that > > sizeof(ath_desc) is aligned on the relevant architecture. It gets > > slightly scary - AR93xx TX descriptors are "L1 cache == 128 byte > > aligned" which is an enormous waste of memory compared to a 16 or 32 > > byte aligned platform. Alas.. > > The problem is with cache line sharing, not necessarily with alignment. If you are only ever using one of them at a time, or if you have perfect hygiene, you can cope with this situation without undue waste. The perfect hygiene might be hard sometimes. This brings up an interesting tangential issue for this busdma discussion. For some controller hardware you allocate a block of memory which is treated as an array of "descriptors" or some other shared control information, you set a register in the hardware to point to that block of memory, and then there is some degree of concurrent access of that memory by hardware and CPU. The interesting part is that some such hardware cannot operate in phases as anticipated by our busdma model. That is, there's no clear demarkation points between "the CPU has exclusive access to the memory" and "the hardware has exclusive access to the memory." Usually for these schemes to work correctly, the memory has to be mapped as uncached, unbuffered, strongly ordered, or whatever combo of those makes sense for a given platform. We have arm drivers that use bus_dmamem_alloc() with the BUS_DMA_COHERENT flag to obtain such memory, even though that wasn't the intended meaning for that flag. If the armv4 busdma implementation were changed to stop honoring the COHERENT flag (it's supposed to be an optional feature) those drivers would stop working. So we need to track down such mistakes and fix them, but the question is: fix them how? I think it may make sense to let busdma handle it, because you may get some advantage from the allocation being made based upon the constraints encoded in the inherited chain of tags for the driver. On the other hand, drivers doing this sort of thing are usually pretty close to the silicon and have a good idea for themselves what the hardware constraints are. We could just say that drivers with such needs should call kmem_alloc_contig() or kmem_alloc_attr() for themselves. If we say it's a thing that busdma should handle, then I think we need: * A flag that is universal across all platforms that means unambiguously that you need memory that is mapped however device-register memory is mapped on that platform (uncached, unbuffered, strongly ordered; I'm tempted to say "whatever pmap_mapdev() does" but I'm not sure that's rigorously correct). * If the request cannot be honored for some reason it has to return failure, not quietly give you regular cached memory instead (which is what BUS_DMA_COHERENT does). * The busdma sequence of sync operations does not apply to memory allocated with this flag, and indeed you must not call the sync functions on such memory. The x86 busdma code recently grew a BUS_DMA_NOCACHE flag, perhaps that's the name that should be supported across all platforms? -- Ian From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 22:12:56 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C81371065670; Mon, 27 Aug 2012 22:12:56 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id 55C148FC0A; Mon, 27 Aug 2012 22:12:55 +0000 (UTC) Received: from mail.cicely.de ([10.1.1.37]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id q7RMCjUO059568 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 28 Aug 2012 00:12:46 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (cicely7.cicely.de [10.1.1.9]) by mail.cicely.de (8.14.4/8.14.4) with ESMTP id q7RMCWPU082659 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 28 Aug 2012 00:12:32 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.14.2/8.14.2) with ESMTP id q7RMCWaC074506; Tue, 28 Aug 2012 00:12:32 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.14.2/8.14.2/Submit) id q7RMCWMQ074505; Tue, 28 Aug 2012 00:12:32 +0200 (CEST) (envelope-from ticso) Date: Tue, 28 Aug 2012 00:12:32 +0200 From: Bernd Walter To: Warner Losh Message-ID: <20120827221231.GB73992@cicely7.cicely.de> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> <45C204C3-D4CE-4B00-886A-A88A0FE7CAD7@bsdimp.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <45C204C3-D4CE-4B00-886A-A88A0FE7CAD7@bsdimp.com> X-Operating-System: FreeBSD cicely7.cicely.de 7.0-STABLE i386 User-Agent: Mutt/1.5.11 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED=-1, BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01 autolearn=unavailable version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on spamd.cicely.de Cc: Tim Kientzle , freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 22:12:57 -0000 On Mon, Aug 27, 2012 at 09:34:30AM -0600, Warner Losh wrote: > > On Aug 27, 2012, at 9:12 AM, Tim Kientzle wrote: > > > > > On Aug 27, 2012, at 6:38 AM, Warner Losh wrote: > > > >> > >> On Aug 27, 2012, at 12:06 AM, Hans Petter Selasky wrote: > >> > >>> Hi, > >>> Correct. > >>> > >>>> We also need some rules about working with buffers obtained from > >>>> bus_dmamem_alloc() and external buffers passed to bus_dmamap_load(). I > >>>> think the rule should be that a buffer obtained from bus_dmamem_alloc(), > >>>> or more formally any region of memory mapped by a bus_dmamap_load(), is > >>>> a single logical object which can only be accessed by one entity at a > >>>> time. That means that there cannot be two concurrent DMA operations > >>>> happening in different regions of the same buffer, nor can DMA and CPU > >>>> access be happening concurrently even if in different parts of the > >>>> buffer. > >>> > >>> Is this something which we can fix using a simple __align(USB_DMA_ALIGN) on elements in C-structures which are allowed to be DMA loaded. > >> > >> No. I don't think so. the reason is that you can't define USB_DMA_ALGIN to be a constant on MIPS, at least, or I think ARM because that's determined at run time. > > > > But don't mbuf structures do pretty much what Hans is suggesting? > > They kinda do, and kinda don't. > > > Why is mbuf okay? > > mbuf is OK because it is never changed while the DMA is pending to the buffers. That is, m_hdr and m_ext are invariant while the device owns the memory. In addition, mbuf allocations are so large that no two mbufs share the same cache line (although it looks like 256 might be too small to avoid that on some MIPS processors). usb buffers do not obey these same rules, otherwise none of the stuff we're talking about would matter. I've always hated the design of USB controllers with their zillions of tiny DMA buffers. -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. From owner-freebsd-mips@FreeBSD.ORG Tue Aug 28 13:48:53 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2CCE4106566C; Tue, 28 Aug 2012 13:48:53 +0000 (UTC) (envelope-from marktinguely@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id DDDF58FC17; Tue, 28 Aug 2012 13:48:51 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so9660554pbb.13 for ; Tue, 28 Aug 2012 06:48:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=P71mtBUeKJqJJ7uppZZUYzebtvvWMylC3ZFDuLWwN8U=; b=bHZLuzFKqvhefHZ1LPS5tIpUsqC75R1yub1u8o2tFUZoD5ipMaiREpoTrKdMT7Emxe dnj+MRbW+060ycWl/z6nYyRNRPZTFa448jZz9i2j6C9d9ucVZAoRZeVWkxflBRdoLPMg +/SnpzKDAqAgwiHQJwZoTo4N0mHy64YtmDaxMpnKRl0TJxFZovCxGuVOnie5nHCH/rE9 OneNADf1F1c1eEp3mNDK/8FrZeRFWghdsCcbVk747yOV5YDPmORpnyqMNlfCbVgNqcJF gDNyB6UKcpKjaNyb/zQeH/WsKEDkXVME05CdzU/QWMEhNo7GT3GF2v/ITheHdTzvYmRc C6yQ== MIME-Version: 1.0 Received: by 10.68.231.130 with SMTP id tg2mr3863853pbc.70.1346161731445; Tue, 28 Aug 2012 06:48:51 -0700 (PDT) Received: by 10.68.229.227 with HTTP; Tue, 28 Aug 2012 06:48:51 -0700 (PDT) In-Reply-To: <201208271531.42725.jhb@freebsd.org> References: <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> <20120827185346.GE33100@deviant.kiev.zoral.com.ua> <201208271531.42725.jhb@freebsd.org> Date: Tue, 28 Aug 2012 08:48:51 -0500 Message-ID: From: Mark Tinguely To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Cc: Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org, Konstantin Belousov Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2012 13:48:53 -0000 On Mon, Aug 27, 2012 at 2:31 PM, John Baldwin wrote: > On Monday, August 27, 2012 2:53:46 pm Konstantin Belousov wrote: >> On Sun, Aug 26, 2012 at 05:13:31PM -0600, Warner Losh wrote: >> > >> > On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: >> > > In this regard, it's the busdma implementation that's broken, because it >> > > should bounce those IOs through a DMA-safe buffer. There's absolutely >> > > no rule that I've ever heard of in FreeBSD that says IO can only take >> > > place using memory allocated from busdma. >> > >> > That's partially true. Since BUSDMA grew up in the storage area, you >> > must allocate the memory from busdma, or it must be page aligned has >> > been the de-facto rule here. The mbuf and uio variants of load were >> > invented to cope with common cases of mbufs and user I/O to properly >> > flag things. >> >> I once looked at x86 bus_dmamap_load_uio(), and I was unable to >> understand how to use it with usermode uio. I think this is a good >> moment to ask. Most existing users use UIO_SYSSPACE, but several crypto >> drivers might allow the UIO_USERSPACE for them. >> >> For UIO_USERSPACE, if the page is not resident, the pmap_extract() call from >> _bus_dmamap_load_buffer() returns 0. So the i/o happens to the page >> located at 0, which contains real mode IVT and other BIOS sensitive tables. >> >> Worse, if the page is resident, but it is mapped at the region which >> requires COW on write, then DMA will be performed to the wrong page >> which is typically shared with other innocent users. to the COW area >> which was not yet copied, >> >> Am I missing some trick there ? > > No. The caller is required to wire the pages first in some manner. > In general bus_dmamap_load_uio() isn't a good idea. I do believe the > crypto drivers are careful to wire the buffer first. I think requiring > the caller to wire is the only sane way that can be used. > > Also, doing DMA to a stack variable is absolutely horrible for a related > reason since presumably the thread will block while it waits for the DMA > to complete, and a sleeping thread can be swapped out (including having > it's stack swapped out). > > -- > John Baldwin The POST commands for UIO DMA may be a bit hairy because the address space may not be the same as the caller. Someone once told me they got around that (in a Sparc enviroment?) by shadow mapping the address to KVA. --Mark TInguely. From owner-freebsd-mips@FreeBSD.ORG Tue Aug 28 14:31:27 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA758106566B; Tue, 28 Aug 2012 14:31:27 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 571348FC12; Tue, 28 Aug 2012 14:31:23 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q7SEVMEj006002; Tue, 28 Aug 2012 08:31:22 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7SEVJ5w033171; Tue, 28 Aug 2012 08:31:19 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Warner Losh In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> <1346081557.1140.181.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Tue, 28 Aug 2012 08:31:19 -0600 Message-ID: <1346164279.1140.328.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, Hans Petter Selasky , Mark Tinguely , freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2012 14:31:28 -0000 On Mon, 2012-08-27 at 09:57 -0600, Warner Losh wrote: > >> I don't think the cache line problem can be solved with bounce > buffers. Trying to accommodate broken drivers is what lead us to this > spot. We need to fix the broken drivers. If that's impossible, then > the best we can do is have the driver set a 'always bounce' flag in > the tag it creates and use that to always bounce for operations > through that tag. > > > > So you'd be okay with a driver setting a flag that says "always > bounce" > > but not okay with the busdma layer bouncing only when it's actually > > necessary? I'm confused -- do you think the busdma layer will be > unable > > to detect when it's necessary unless directed from the outside? > > Actually, it is good you are confused. I was wrong when I said I'd be > happy with a flag that says always bounce. The reason is that the > driver can do this all the time for those cases like the at91 mci > driver does. After studying the busdma implementation code and pondering this some more, I now understand why the automatic bouncing that I think is the ideal solution to this is so hard to accomplish. I'm not sure "Impossible" is the right description, but I'm afraid that in the long run "impractical" might turn out to be the case. I'm going to think all this over for a few days, maybe do a bit of actual paying work, before revisiting it. -- Ian From owner-freebsd-mips@FreeBSD.ORG Wed Aug 29 02:22:58 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8762F106566B; Wed, 29 Aug 2012 02:22:58 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 575D18FC15; Wed, 29 Aug 2012 02:22:58 +0000 (UTC) Received: by dadr6 with SMTP id r6so45384dad.13 for ; Tue, 28 Aug 2012 19:22:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; bh=jPaxisYnGaNzyL81BFW26rd2oRWNOqjntmaqtrIA9BM=; b=Y6yWxVnQW5rfu4H6GoNtOrjUwqX4T79x0DkWrQ5bM60eF29l7wdRswXtY6kA60kyk8 DwtSR2Nh4SJgrC6iAW+Ro/U5nNw3Hi5qx2+0Heco/CuVUyweypK6J3b43VxnVViCpm/6 pODAkBf+qAhJIt9WFEF3SOaJ+rCNH1FqN+dMkUYODeI8L4ZcMO3B7US8uc8/xIYOy6wq ykHPzF8Dm/GZcA4W30JpFHtjzuXk5S7gM3zUhUfSO6UTcz/CaraOSmKayH0oHP5GVhGg gBTKuXQmI18gqt6Dw+i25szZWV8UAn6yMF8g4HW5+3Rgze1O2hgOy9FsCbxBZwvfVEAd +fyA== MIME-Version: 1.0 Received: by 10.66.74.196 with SMTP id w4mr133242pav.32.1346206977921; Tue, 28 Aug 2012 19:22:57 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.36.106 with HTTP; Tue, 28 Aug 2012 19:22:57 -0700 (PDT) Date: Tue, 28 Aug 2012 19:22:57 -0700 X-Google-Sender-Auth: 1EKXm5nhF29ln4MNaNAWBW3n_r8 Message-ID: From: Adrian Chadd To: scottl@freebsd.org, freebsd-mips@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Subject: MIPS busdma map questions X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 02:22:58 -0000 Hi, I have a couple of busdma map questions. #1 - in mips/mips/busdma_machdep.c, bus_dmamap_destroy() calls _busdma_free_dmamap() and -then- goes and walks the dma map list. Is this a good idea or not? :) #2 - the ath(4) code (in sys/dev/ath/if_ath.c:ath_descdma_alloc_desc() on -HEAD) does the following: * creates a tag * allocates a dmamap (bus_dmamap_create) - storing it in dd->dd_dmamap; * allocates memory via bus_dmamem_alloc - also storing it in dd->dd_dmamap. Now, I saw some NULL pointer derefernce going on when thigns failed to load - and what I found was bus_dmamem_alloc() overwrites the dmat pointer passed in (dd_dmamap above). So the dmamap allocated via bus_dmamap_create() is just plain overwritten. I'm guessing that when allocating memory via bus_dmamem_alloc(), the call to bus_dmamap_create() (and the bus_dmamap_destroy()) isn't needed. Scott/mips people - what do you think? Adrian From owner-freebsd-mips@FreeBSD.ORG Wed Aug 29 03:24:12 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DBF441065673 for ; Wed, 29 Aug 2012 03:24:12 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id A787D8FC0A for ; Wed, 29 Aug 2012 03:24:12 +0000 (UTC) Received: by pbbrp2 with SMTP id rp2so335588pbb.13 for ; Tue, 28 Aug 2012 20:24:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=ZeVQ7YfpiSELJp1WVGxjNwki7zzt5hVAWUzSWav2Rco=; b=UhFQU3Zzx1wX0kzvjiC30k2CHqpr3C+Hz2VTkX29HE+lI5C7tsf3sN+me1gKqjbAoa RfINAaLYpkksQLY2WgFjBWDLUrSBcy06ffUxWZOY2AHNmpiekdO8pLDrQvuD3F9MbbCR MtxV2aF8syCc+2eH2SY41pI2p67TCxtHf6Q0ODog7IIhlx8oOFpISoIeCKImKRnOsY+j kLPgvgEhqOmA2lQHdZ8pgQ66cQeTCUTJfaxCFt8kfYwCQH30GOlgplzn0RYnDOENF+U3 Mb1K5INacncbNTKYIYvb20/pGlH2sc3wLgCp4oTidGy6bK0TdUDZRotFKkP5wclj2dRW M6tQ== Received: by 10.66.82.97 with SMTP id h1mr378975pay.45.1346210651934; Tue, 28 Aug 2012 20:24:11 -0700 (PDT) Received: from [10.0.0.63] (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id qb6sm18364490pbb.18.2012.08.28.20.24.11 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 28 Aug 2012 20:24:11 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Tue, 28 Aug 2012 21:24:08 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <1F780BF4-84C3-48D5-9832-EA32381969A6@bsdimp.com> References: To: Adrian Chadd X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQl9shFHnriu++rFmQrqBmEuByfFGyRB2TwGlEKDGdoqA+1fFRX5yvOPE9mUW9oQzDK70XFs Cc: scottl@freebsd.org, freebsd-mips@freebsd.org Subject: Re: MIPS busdma map questions X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 03:24:13 -0000 On Aug 28, 2012, at 8:22 PM, Adrian Chadd wrote: > Hi, >=20 > I have a couple of busdma map questions. >=20 > #1 - in mips/mips/busdma_machdep.c, bus_dmamap_destroy() calls > _busdma_free_dmamap() and -then- goes and walks the dma map list. Is > this a good idea or not? :) Accessing free memory after freeing it is a free ticket to a free dose = of free pain. It should be freed after the if that checks to see if it = is in use. > #2 - the ath(4) code (in sys/dev/ath/if_ath.c:ath_descdma_alloc_desc() > on -HEAD) does the following: >=20 > * creates a tag > * allocates a dmamap (bus_dmamap_create) - storing it in = dd->dd_dmamap; > * allocates memory via bus_dmamem_alloc - also storing it in = dd->dd_dmamap. >=20 > Now, I saw some NULL pointer derefernce going on when thigns failed to > load - and what I found was bus_dmamem_alloc() overwrites the dmat > pointer passed in (dd_dmamap above). So the dmamap allocated via > bus_dmamap_create() is just plain overwritten. >=20 > I'm guessing that when allocating memory via bus_dmamem_alloc(), the > call to bus_dmamap_create() (and the bus_dmamap_destroy()) isn't > needed. >=20 > Scott/mips people - what do you think? That bug isn't so easy to figure out with my 5 second attention span... Warner >=20 >=20 > Adrian > _______________________________________________ > freebsd-mips@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-mips > To unsubscribe, send any mail to = "freebsd-mips-unsubscribe@freebsd.org" From owner-freebsd-mips@FreeBSD.ORG Wed Aug 29 03:43:01 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 61690106566B; Wed, 29 Aug 2012 03:43:01 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 2DB008FC1A; Wed, 29 Aug 2012 03:43:00 +0000 (UTC) Received: by dadr6 with SMTP id r6so82898dad.13 for ; Tue, 28 Aug 2012 20:43:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=jAkn6SZBu2JgVIZ652zmxYoQqbVJv4IozJRs5iFCGlk=; b=mSMYI5CpEp2gvdQn3kwy3v0Eb6Jn1Y5R9sMr+/e8VpqxpeT01znHkhcxZClTCTC3zJ EJTKhNdZlS8ovVDq1d6t3xxi6S/UTvS8rCm7yFm+UL92xIbkItonPHLk/RY8k+4yG4P0 f1Wr4cuGwmhpieEOLM5Y5Mwdaprzrj7k5PtIPN6SU69P5ZYhRsjSN8LLwYGY5Py3ShV7 xzYHo2uShGfzM6ijEfKxQAkL0VUyq25vtZofVAwgQGt2aZuL8TQzT9AqUAGE6fKhFF7J Hi7+NynwQqiD/D/v92LftQVYwusIUpIQ7CR9fab1gyGHUDD0Z9qRuaZXo4iJX9RkhR99 CaOw== MIME-Version: 1.0 Received: by 10.66.74.196 with SMTP id w4mr514587pav.32.1346211780629; Tue, 28 Aug 2012 20:43:00 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.36.106 with HTTP; Tue, 28 Aug 2012 20:43:00 -0700 (PDT) In-Reply-To: <1F780BF4-84C3-48D5-9832-EA32381969A6@bsdimp.com> References: <1F780BF4-84C3-48D5-9832-EA32381969A6@bsdimp.com> Date: Tue, 28 Aug 2012 20:43:00 -0700 X-Google-Sender-Auth: MsWdTOezouk9q2q6hJT6qOrFyw0 Message-ID: From: Adrian Chadd To: Warner Losh Content-Type: text/plain; charset=ISO-8859-1 Cc: scottl@freebsd.org, freebsd-mips@freebsd.org Subject: Re: MIPS busdma map questions X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 03:43:01 -0000 On 28 August 2012 20:24, Warner Losh wrote: > > On Aug 28, 2012, at 8:22 PM, Adrian Chadd wrote: > >> Hi, >> >> I have a couple of busdma map questions. >> >> #1 - in mips/mips/busdma_machdep.c, bus_dmamap_destroy() calls >> _busdma_free_dmamap() and -then- goes and walks the dma map list. Is >> this a good idea or not? :) > > Accessing free memory after freeing it is a free ticket to a free dose of free pain. It should be freed after the if that checks to see if it is in use. Excellent! Who knows busdma better than I and can wade through what that code path is doing? :) Adrian From owner-freebsd-mips@FreeBSD.ORG Wed Aug 29 05:01:45 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E1D34106566B for ; Wed, 29 Aug 2012 05:01:45 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm16-vm0.bullet.mail.ac4.yahoo.com (nm16-vm0.bullet.mail.ac4.yahoo.com [98.139.52.238]) by mx1.freebsd.org (Postfix) with SMTP id 535108FC18 for ; Wed, 29 Aug 2012 05:01:45 +0000 (UTC) Received: from [98.139.52.195] by nm16.bullet.mail.ac4.yahoo.com with NNFMP; 29 Aug 2012 05:01:39 -0000 Received: from [98.139.52.147] by tm8.bullet.mail.ac4.yahoo.com with NNFMP; 29 Aug 2012 05:01:39 -0000 Received: from [127.0.0.1] by omp1030.mail.ac4.yahoo.com with NNFMP; 29 Aug 2012 05:01:39 -0000 X-Yahoo-Newman-Id: 643645.94976.bm@omp1030.mail.ac4.yahoo.com Received: (qmail 72365 invoked from network); 29 Aug 2012 05:01:09 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=DKIM-Signature:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:References:In-Reply-To:Mime-Version:Content-Transfer-Encoding:Content-Type:Message-Id:Cc:X-Mailer:From:Subject:Date:To; b=Bewsc2VfgInpbgqDSsBq0VSXTX/BfSEnR4UlXwkt79+G3vBvNJ4Q/sKIC2HfGdbFeV6Il2J586P+sqyvH7qcIKaQfAdcImZ3Xf/T4Ggbm8mbRCmgm7UZBDqMm3oA42HNIE8NovD2uXbeIVhF3kxgIH4RFt3D/4rqItT8ka/Yo10= ; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1346216469; bh=kyrYnPmISV3Ti+ptfYHVz56cBptozyTUZOvupOsElf4=; h=X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:References:In-Reply-To:Mime-Version:Content-Transfer-Encoding:Content-Type:Message-Id:Cc:X-Mailer:From:Subject:Date:To; b=6gZizDy4lcHN2uuTfIA5R6rne2c4n0+5oz4ZROBAEk5iFth26HYAN70/x4oUJReUKtFHZ96LqvtYijjx5U2mzxwbjjBnKTW0UJaX9tpWBm0p7MEOsPbOE5I1jJmakfkDhsSePIIHHv+idu0yVtbNyfL/QDf8wsxliCCTjfQM5vI= X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: zyaXTe8VM1n8KQTe16NeaO2tqCeiuKfjvaOCoP4HhZsT43R UbZKEEOPTPU9JdObT6oTV7KBAKatCBgYYoDh_BgbFJ2WuGOhkbkHMrrMvNZ7 GMxbhyjMcdEsoROjOgDFaW6WEQxjnSKBJZw_7idjaJytNuIXnpWFzndM_wVy 9piiwtuDaXtlEtqM2I0UgKmPtNZJGuKL1PLeYKU8144pJzBblU.e_R0P6CI3 07zaxCoFqCNtZRmM81DDdrM_tFLIV5dYp6XIwefnDIJA3D9B9eb8LqNrQxqL WO9bKnY6JLV9gyco1H5a5Tgk3i4QsgIFhatY1kgWyzgmdsZ6k.FLbchWY.8a BfRsfchapX8Cng51HO7IEVR0hcMAnazfK4px0zZ2q7kRuQgK_FWPs9JoJh3c f1z_4Rclj8NLEbR0VYo0nldjcQuDrau94K1wJY0LqCk72pnLnbSfFDC94sUC egt9H X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Received: from [10.37.116.72] (scott4long@198.228.211.25 with xymcookie) by smtp121-mob.biz.mail.ac4.yahoo.com with SMTP; 28 Aug 2012 22:01:09 -0700 PDT References: <1F780BF4-84C3-48D5-9832-EA32381969A6@bsdimp.com> In-Reply-To: Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Message-Id: <08166FD3-8FAD-453D-ABD3-2DC19B52FED1@yahoo.com> X-Mailer: iPhone Mail (9B206) From: Scott Long Date: Tue, 28 Aug 2012 23:01:02 -0600 To: Adrian Chadd Cc: "scottl@freebsd.org" , "freebsd-mips@freebsd.org" Subject: Re: MIPS busdma map questions X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 05:01:46 -0000 On Aug 28, 2012, at 9:43 PM, Adrian Chadd wrote: > On 28 August 2012 20:24, Warner Losh wrote: >>=20 >> On Aug 28, 2012, at 8:22 PM, Adrian Chadd wrote: >>=20 >>> Hi, >>>=20 >>> I have a couple of busdma map questions. >>>=20 >>> #1 - in mips/mips/busdma_machdep.c, bus_dmamap_destroy() calls >>> _busdma_free_dmamap() and -then- goes and walks the dma map list. Is >>> this a good idea or not? :) >>=20 >> Accessing free memory after freeing it is a free ticket to a free dose of= free pain. It should be freed after the if that checks to see if it is in u= se. >=20 > Excellent! Who knows busdma better than I and can wade through what > that code path is doing? :) >=20 >=20 >=20 There is no need to create a separate map for memory allocated via this way.= It's a convenience thing. Scott= From owner-freebsd-mips@FreeBSD.ORG Wed Aug 29 05:25:17 2012 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 17DC31065670 for ; Wed, 29 Aug 2012 05:25:17 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh11.mail.rice.edu (mh11.mail.rice.edu [128.42.199.30]) by mx1.freebsd.org (Postfix) with ESMTP id D84718FC16 for ; Wed, 29 Aug 2012 05:25:16 +0000 (UTC) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 0F66D4C02BE; Wed, 29 Aug 2012 00:25:16 -0500 (CDT) Received: from mh11.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh11.mail.rice.edu (Postfix) with ESMTP id 0DF404C02BB; Wed, 29 Aug 2012 00:25:16 -0500 (CDT) X-Virus-Scanned: by amavis-2.7.0 at mh11.mail.rice.edu, auth channel Received: from mh11.mail.rice.edu ([127.0.0.1]) by mh11.mail.rice.edu (mh11.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id rd-zKxs4CMoF; Wed, 29 Aug 2012 00:25:15 -0500 (CDT) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh11.mail.rice.edu (Postfix) with ESMTPSA id 94A4C4C02B6; Wed, 29 Aug 2012 00:25:15 -0500 (CDT) Message-ID: <503DA7BA.3030102@rice.edu> Date: Wed, 29 Aug 2012 00:25:14 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:8.0) Gecko/20111113 Thunderbird/8.0 MIME-Version: 1.0 To: "Jayachandran C." References: <50228F5C.1000408@rice.edu> <50269AD4.9050804@rice.edu> <5029635A.4050209@rice.edu> <502D2271.6080105@rice.edu> <50325DC3.3090201@rice.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: mips@freebsd.org Subject: Re: mips pmap patch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 05:25:17 -0000 On 08/27/2012 10:24, Jayachandran C. wrote: > On Mon, Aug 20, 2012 at 9:24 PM, Alan Cox wrote: >> On 08/20/2012 05:36, Jayachandran C. wrote: >>> On Thu, Aug 16, 2012 at 10:10 PM, Alan Cox wrote: >>>> On 08/15/2012 17:21, Jayachandran C. wrote: >>>>> On Tue, Aug 14, 2012 at 1:58 AM, Alan Cox wrote: >>>>>> On 08/13/2012 11:37, Jayachandran C. wrote: >>> [...] >>>>>>> I could not test for more than an hour on 32-bit due to another >>>>>>> problem (freelist 1 containing direct-mapped pages runs out of pages >>>>>>> after about an hour of compile test). This issue has been there for a >>>>>>> long time, I am planning to look at it when I get a chance. >>>>>>> >>>>>> What exactly happens? panic? deadlock? >>>>> The build slows down to a crawl and hangs when it runs out of pages in >>>>> the freelist. >>>> >>>> I'd like to see the output of "sysctl vm.phys_segs" and "sysctl >>>> vm.phys_free" from this machine. Even better would be running "sysctl >>>> vm.phys_free" every 60 seconds during the buildworld. Finally, I'd like >>>> to >>>> know whether or not either "ps" or "top" shows any threads blocked on the >>>> "swwrt" wait channel once things slow to a crawl. >>> I spent some time looking at this issue. I use a very large kernel >>> image with built-in root filesystem, and this takes about 120 MB out >>> of the direct mapped area. The remaining pages (~64 MB) are not enough >>> for the build process. If I increase free memory in this area either >>> by reducing the rootfs size of by adding a few more memory segments to >>> this area, the build goes through fine. >> >> I'm still curious to see what "sysctl vm.phys_segs" says. It sounds like >> roughly half of the direct map region is going to DRAM and half to >> memory-mapped I/O devices. Is that correct? > Yes, about half of the direct mapped region in 32-bit is taken by > flash, PCIe and other memory mapped IO. I also made the problem even > worse by not reclaiming some bootloader areas in the direct mapped > region, which reduced the available direct mapped memory. > > Here's the output of sysctls: > > root@testboard:/root # sysctl vm.phys_segs > vm.phys_segs: > SEGMENT 0: > > start: 0x887e000 > end: 0xc000000 > domain: 0 > free list: 0x887a407c > > SEGMENT 1: > > start: 0x1d000000 > end: 0x1fc00000 > domain: 0 > free list: 0x887a407c > > SEGMENT 2: > > start: 0x20000000 > end: 0xbc0b3000 > domain: 0 > free list: 0x887a3f38 > > SEGMENT 3: > > start: 0xe0000000 > end: 0xfffff000 > domain: 0 > free list: 0x887a3f38 > > root@testboard:/root # sysctl vm.phys_free > vm.phys_free: > FREE LIST 0: > > ORDER (SIZE) | NUMBER > | POOL 0 | POOL 1 | POOL 2 > -- -- -- -- -- -- -- -- > 8 ( 1024K) | 2877 | 0 | 0 > 7 ( 512K) | 0 | 1 | 0 > 6 ( 256K) | 1 | 0 | 0 > 5 ( 128K) | 0 | 1 | 0 > 4 ( 64K) | 0 | 1 | 0 > 3 ( 32K) | 0 | 1 | 0 > 2 ( 16K) | 0 | 1 | 0 > 1 ( 8K) | 0 | 0 | 0 > 0 ( 4K) | 0 | 0 | 0 > > FREE LIST 1: > > ORDER (SIZE) | NUMBER > | POOL 0 | POOL 1 | POOL 2 > -- -- -- -- -- -- -- -- > 8 ( 1024K) | 66 | 0 | 0 > 7 ( 512K) | 1 | 1 | 0 > 6 ( 256K) | 0 | 0 | 0 > 5 ( 128K) | 0 | 0 | 0 > 4 ( 64K) | 0 | 1 | 0 > 3 ( 32K) | 0 | 0 | 0 > 2 ( 16K) | 0 | 0 | 0 > 1 ( 8K) | 1 | 1 | 0 > 0 ( 4K) | 0 | 1 | 0 > >>> I also found that when the build slows down, most of the pages taken >>> from freelist 1 are allocated by the UMA subsystem, which seems to >>> keep quite a few pages allocated. >> >> At worst, it may be necessary to disable the use of uma_small_alloc() for >> this machine configuration. At best, uma_small_alloc() could be revised >> opportunistically use pages in the direct map region, but have the ability >> to fall back to pages that have to be mapped. > I think this probably is not a bug, but a configuration problem (we > cannot have such a huge built-in root filesystem when the direct > mapped area is at this low). Anyway, I have checked in code to > recover more areas from the bootloader, and this mostly solves the > issue for me. The above output is taken before the check-in. I'm afraid that exhaustion of freelist 1 is still highly likely to occur under some workloads that require the allocation of a lot of small objects in the kernel's heap. Alan From owner-freebsd-mips@FreeBSD.ORG Wed Aug 29 09:21:25 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B113F1065670; Wed, 29 Aug 2012 09:21:25 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pz0-f54.google.com (mail-pz0-f54.google.com [209.85.210.54]) by mx1.freebsd.org (Postfix) with ESMTP id 788BD8FC08; Wed, 29 Aug 2012 09:21:25 +0000 (UTC) Received: by dadr6 with SMTP id r6so270323dad.13 for ; Wed, 29 Aug 2012 02:21:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=0tzZEYt2JU9E5N9qCchImMx76pvdmx5qcUgxcCquwGo=; b=DWhkbV0q5skdQjUCbZqWUZMlOElbYkqtkksK4E0x1bQZr1mE8d7CqiY7aLixIMPtJO lMOvcuq+vudQQYEv9g7lNhcdAZ2vuC/F1Dr8n6eB1K48sK5MAkyRYp8dER9E246/jgZ0 nk6YlUVDvEyFOSRGTlcrb+Y/dCDgkCF6By9a7k6aWkFkw9/RbFRKmvIrv9qTOCxVremD 8gcCcgGULUpvA5vB6941Xt0456Q1FAfy5SuYQdevWjyNwPhnLcbusSuTt5eLgVmu486q 4SrvCfN7Gz3W1K5Hcjd2tO9lkrCifue3SokKTuoIJRyY7wG5gF7tWAl7u0wxu1UKvhfd iy+g== MIME-Version: 1.0 Received: by 10.68.189.70 with SMTP id gg6mr3255313pbc.125.1346232085097; Wed, 29 Aug 2012 02:21:25 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.36.106 with HTTP; Wed, 29 Aug 2012 02:21:25 -0700 (PDT) In-Reply-To: <08166FD3-8FAD-453D-ABD3-2DC19B52FED1@yahoo.com> References: <1F780BF4-84C3-48D5-9832-EA32381969A6@bsdimp.com> <08166FD3-8FAD-453D-ABD3-2DC19B52FED1@yahoo.com> Date: Wed, 29 Aug 2012 02:21:25 -0700 X-Google-Sender-Auth: PvBmiUBwDqjf0zl4GAM8xG0Tr4w Message-ID: From: Adrian Chadd To: Scott Long Content-Type: text/plain; charset=ISO-8859-1 Cc: "scottl@freebsd.org" , "freebsd-mips@freebsd.org" Subject: Re: MIPS busdma map questions X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 09:21:25 -0000 On 28 August 2012 22:01, Scott Long wrote: > There is no need to create a separate map for memory allocated via this way. It's a convenience thing. Right, but my initial question still stands - the initial map that's being created is completely ignored and overwritten when the bus_dmamem_alloc() call is made. One of the first thing that does is it sets *dmat = NULL; - it's totally trashing the pre-existing map. Adrian From owner-freebsd-mips@FreeBSD.ORG Wed Aug 29 11:07:42 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ABA1E106564A for ; Wed, 29 Aug 2012 11:07:42 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm35-vm6.bullet.mail.bf1.yahoo.com (nm35-vm6.bullet.mail.bf1.yahoo.com [72.30.238.78]) by mx1.freebsd.org (Postfix) with SMTP id 4A5AA8FC15 for ; Wed, 29 Aug 2012 11:07:41 +0000 (UTC) Received: from [98.139.212.147] by nm35.bullet.mail.bf1.yahoo.com with NNFMP; 29 Aug 2012 11:07:35 -0000 Received: from [98.139.215.254] by tm4.bullet.mail.bf1.yahoo.com with NNFMP; 29 Aug 2012 11:07:35 -0000 Received: from [127.0.0.1] by omp1067.mail.bf1.yahoo.com with NNFMP; 29 Aug 2012 11:07:35 -0000 X-Yahoo-Newman-Id: 711512.18216.bm@omp1067.mail.bf1.yahoo.com Received: (qmail 95572 invoked from network); 29 Aug 2012 11:07:35 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=DKIM-Signature:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:References:In-Reply-To:Mime-Version:Content-Transfer-Encoding:Content-Type:Message-Id:Cc:X-Mailer:From:Subject:Date:To; b=vQuA1BEr/gJUScHScMcn0NHez55fhVc9VS+2zJ/lHBjIJG1+o24m2g7dlZSzp8NJEiLX7Gn1RHDERo7ReRannMVGNNE4QpHdKglXlZtZi6vOgf352zN9V/35eO4hIodWKegY2hvI2OZ4QdQqoIbqMA3Rya4rXz1FsP9wQ7Ggtwc= ; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1346238455; bh=sA1kTvSAVq7qdNxjU6amE0cLhQvw1Qbi5pliXf8jEv8=; h=X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:Received:References:In-Reply-To:Mime-Version:Content-Transfer-Encoding:Content-Type:Message-Id:Cc:X-Mailer:From:Subject:Date:To; b=qddTIPL9VZUmCKogZdCvX8xqTBvnKbCtxGdYUoMA0jCDOSunZsiJM94c+8nm7J9PDRJeaCc1YF7t2RPxbjAE2k3s/TuATthdiCaNn8n3lE24UpJ9RSIbybiwC41SDgKuo5apMqpNZbjTT+vrkK125eiByk5NjWglP2qXz93Torg= X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: 9LtRCSYVM1kFWBe8Ld70UeI3U8U8IZRlI6WaRyDQjZKyDTx NHTURQ2oX7fKVlVtK.VSYkn9q1xzuPd8aYvrE3D8YsIi8wDAXOlp0Hv7J0cD r8sxVl4wdqecdcFqOjqFgdfsLr69O63MK86k_AeDq5Ct_31iQ7CcW1e8fuVX jZgs2RweqyhLbsDHgwmNIL9pDlloJmSODuKqs1yfCwEc2pe.qIqnkI20Crkd cu5xoWYPs3S4.GUVKPGobl29oGH5modVbjqgUcyWDvnuvbIxcvZDwWr7EsQQ Cp0G0XjjWeHik5.1rNbBEpK01TBgdORCWyyUIjdaFjRIvDQxQo2b_umQlQwj xdU5Gxq2xZuWAA4.yI4.lDcqXPqddm_7Bw5tIZwAb5SY8EKd4P83ry3d3kRD V9RJFNecbn0qzqNACh5pDh6sQeKmMS8DpwsNT.CpSg9TBKliZoXHGKQA7f_a 4wq0w3hmiT7k1 X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- Received: from [192.168.254.204] (scott4long@168.103.85.57 with xymcookie) by smtp124-mob.biz.mail.bf1.yahoo.com with SMTP; 29 Aug 2012 11:07:35 +0000 UTC References: <1F780BF4-84C3-48D5-9832-EA32381969A6@bsdimp.com> <08166FD3-8FAD-453D-ABD3-2DC19B52FED1@yahoo.com> In-Reply-To: Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Message-Id: <77CDC9C5-A4DA-4CE6-ADEA-57DC6C9A2F0B@yahoo.com> X-Mailer: iPad Mail (9B206) From: Scott Long Date: Wed, 29 Aug 2012 05:07:37 -0600 To: Adrian Chadd Cc: "scottl@freebsd.org" , "freebsd-mips@freebsd.org" Subject: Re: MIPS busdma map questions X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 11:07:42 -0000 On Aug 29, 2012, at 3:21 AM, Adrian Chadd wrote: > On 28 August 2012 22:01, Scott Long wrote: >=20 >> There is no need to create a separate map for memory allocated via this w= ay. It's a convenience thing. >=20 > Right, but my initial question still stands - the initial map that's > being created is completely ignored and overwritten when the > bus_dmamem_alloc() call is made. One of the first thing that does is > it sets *dmat =3D NULL; - it's totally trashing the pre-existing map. >=20 >=20 Right. You don't create a separate map for this kind of allocation, you let= bus_dmamem_alloc do that for you. The inputs to this function are the tag a= nd the flags. The output is the map and the vaddr for the allocation. The m= ap argument is an output, not an input. Does that make more sense? Scott= From owner-freebsd-mips@FreeBSD.ORG Wed Aug 29 11:41:40 2012 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2408C106564A for ; Wed, 29 Aug 2012 11:41:40 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-wi0-f178.google.com (mail-wi0-f178.google.com [209.85.212.178]) by mx1.freebsd.org (Postfix) with ESMTP id A375E8FC16 for ; Wed, 29 Aug 2012 11:41:39 +0000 (UTC) Received: by wibhr14 with SMTP id hr14so389731wib.13 for ; Wed, 29 Aug 2012 04:41:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=dMyd87p1JKf689ZPPrJn8TuTto0/Vc8THNP8setmDCI=; b=fm1AXVtyeHfcqVTSnvbVUuWkAmMyHMEFIxK5k0jf+DhWlnNOrbwsGka3SjPDsvKLLT +P2mxdSqeMGpBNYKQB3ZVvxTUaof+Nvm+HZrqax45bAkOani3ncJivf4aBPr191S7bsi +tGAnqTOX0Psln2yo5wSog9KKNwVsRM4r6KtiW4LizYmy+4tgwpAK6xPdJbYCGPSfQO9 nxvwl56VINO6TxjSt/x0CxY6I02nUYpRQHfQ/s+Rmv3xOv+xUG9dY1AlipQ4DzdJn2aT tFFjhR/PM1UZrLsUvxKMMw7n5Ye25q8T+3CzBNQJiaT0eCEqz5RV2zZvGcuu3i+r8E3H ZQqA== MIME-Version: 1.0 Received: by 10.216.123.130 with SMTP id v2mr805547weh.117.1346240498441; Wed, 29 Aug 2012 04:41:38 -0700 (PDT) Received: by 10.216.115.3 with HTTP; Wed, 29 Aug 2012 04:41:38 -0700 (PDT) In-Reply-To: <503AF695.9060405@rice.edu> References: <50228F5C.1000408@rice.edu> <50269AD4.9050804@rice.edu> <5029635A.4050209@rice.edu> <502D2271.6080105@rice.edu> <503AF695.9060405@rice.edu> Date: Wed, 29 Aug 2012 17:11:38 +0530 Message-ID: From: "Jayachandran C." To: Alan Cox Content-Type: text/plain; charset=ISO-8859-1 Cc: mips@freebsd.org Subject: Re: mips pmap patch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2012 11:41:40 -0000 On Mon, Aug 27, 2012 at 9:54 AM, Alan Cox wrote: > Could you please test the attached patch? For a change, this patch is an > optimization (as opposed to a bug fix). It carves off a third high-order > bit from the PTE in order to implement another software flag. For 64-bit > systems, this bit was previously unused. For 32-bit systems, this patch > effectively reduces the size of the PTE's page frame number (PFN) field by 1 > bit. However, the PFN still has more than enough bits to support the 32-bit > vm_paddr_t. > > This patch needs both 32-bit and 64-bit testing. I'm fairly confident of > the correctness of the 64-bit case, but less so of the 32-bit case. Tested this on both 32 and 64 bit configuration on XLP and no problems were seen. > Going forward, I expect to claim a fourth bit from the PTE in order to > implement a proper access bit emulation. Regards, JC From owner-freebsd-mips@FreeBSD.ORG Fri Aug 31 04:47:52 2012 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB1C4106564A; Fri, 31 Aug 2012 04:47:52 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-stable.sentex.ca (freebsd-stable.sentex.ca [IPv6:2607:f3e0:0:3::6502:9b]) by mx1.freebsd.org (Postfix) with ESMTP id 935BD8FC14; Fri, 31 Aug 2012 04:47:52 +0000 (UTC) Received: from freebsd-stable.sentex.ca (localhost [127.0.0.1]) by freebsd-stable.sentex.ca (8.14.5/8.14.5) with ESMTP id q7V4lq36044863; Fri, 31 Aug 2012 04:47:52 GMT (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-stable.sentex.ca (8.14.5/8.14.5/Submit) id q7V4lprS044832; Fri, 31 Aug 2012 04:47:51 GMT (envelope-from tinderbox@freebsd.org) Date: Fri, 31 Aug 2012 04:47:51 GMT Message-Id: <201208310447.q7V4lprS044832@freebsd-stable.sentex.ca> X-Authentication-Warning: freebsd-stable.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Precedence: bulk Cc: Subject: [releng_9 tinderbox] failure on mips/mips X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2012 04:47:53 -0000 TB --- 2012-08-31 03:44:17 - tinderbox 2.9 running on freebsd-stable.sentex.ca TB --- 2012-08-31 03:44:17 - FreeBSD freebsd-stable.sentex.ca 8.2-STABLE FreeBSD 8.2-STABLE #4: Wed Sep 28 13:48:49 UTC 2011 mdtancsa@freebsd-stable.sentex.ca:/usr/obj/usr/src/sys/server amd64 TB --- 2012-08-31 03:44:17 - starting RELENG_9 tinderbox run for mips/mips TB --- 2012-08-31 03:44:17 - cleaning the object tree TB --- 2012-08-31 03:44:17 - cvsupping the source tree TB --- 2012-08-31 03:44:17 - /usr/bin/csup -z -r 3 -g -L 1 -h cvsup.sentex.ca /tinderbox/RELENG_9/mips/mips/supfile TB --- 2012-08-31 03:45:20 - building world TB --- 2012-08-31 03:45:20 - CROSS_BUILD_TESTING=YES TB --- 2012-08-31 03:45:20 - MAKEOBJDIRPREFIX=/obj TB --- 2012-08-31 03:45:20 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2012-08-31 03:45:20 - SRCCONF=/dev/null TB --- 2012-08-31 03:45:20 - TARGET=mips TB --- 2012-08-31 03:45:20 - TARGET_ARCH=mips TB --- 2012-08-31 03:45:20 - TZ=UTC TB --- 2012-08-31 03:45:20 - __MAKE_CONF=/dev/null TB --- 2012-08-31 03:45:20 - cd /src TB --- 2012-08-31 03:45:20 - /usr/bin/make -B buildworld >>> World build started on Fri Aug 31 03:45:23 UTC 2012 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything [...] /obj/mips.mipsel/src/tmp/usr/lib/libcrypto.a(sha512.o): In function `SHA512_Update': sha512.c:(.text+0x5178): multiple definition of `SHA512_Update' /obj/mips.mipsel/src/tmp/usr/lib/libmd.a(sha512c.o):sha512c.c:(.text+0xab9c): first defined here /obj/mips.mipsel/src/tmp/usr/lib/libcrypto.a(sha512.o): In function `SHA512_Final': sha512.c:(.text+0x5408): multiple definition of `SHA512_Final' /obj/mips.mipsel/src/tmp/usr/lib/libmd.a(sha512c.o):sha512c.c:(.text+0xadb0): first defined here nc.lo: In function `_$$hide$$ nc.lo main': (.text+0x2290): warning: warning: mktemp() possibly used unsafely; consider using mkstemp() *** Error code 1 Stop in /obj/mips.mipsel/src/rescue/rescue. *** Error code 1 Stop in /src/rescue/rescue. *** Error code 1 Stop in /src/rescue. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. *** Error code 1 Stop in /src. TB --- 2012-08-31 04:47:51 - WARNING: /usr/bin/make returned exit code 1 TB --- 2012-08-31 04:47:51 - ERROR: failed to build world TB --- 2012-08-31 04:47:51 - 2214.20 user 499.62 system 3814.45 real http://tinderbox.freebsd.org/tinderbox-releng_9-RELENG_9-mips-mips.full From owner-freebsd-mips@FreeBSD.ORG Fri Aug 31 17:25:18 2012 Return-Path: Delivered-To: freebsd-mips@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 85592106567B; Fri, 31 Aug 2012 17:25:18 +0000 (UTC) (envelope-from ray@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 598478FC17; Fri, 31 Aug 2012 17:25:18 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q7VHPIJY048442; Fri, 31 Aug 2012 17:25:18 GMT (envelope-from ray@freefall.freebsd.org) Received: (from ray@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q7VHPICs048438; Fri, 31 Aug 2012 17:25:18 GMT (envelope-from ray) Date: Fri, 31 Aug 2012 17:25:18 GMT Message-Id: <201208311725.q7VHPICs048438@freefall.freebsd.org> To: loos.br@gmail.com, ray@FreeBSD.org, freebsd-mips@FreeBSD.org From: ray@FreeBSD.org Cc: Subject: Re: kern/170864: [mips] [patch] Move SPI flash from AR71XX_BASE (kernel) to board specific files X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2012 17:25:18 -0000 Synopsis: [mips] [patch] Move SPI flash from AR71XX_BASE (kernel) to board specific files State-Changed-From-To: open->closed State-Changed-By: ray State-Changed-When: Fri Aug 31 17:20:36 UTC 2012 State-Changed-Why: After conversation on IRC, decided to leave spibus definitions in AR71XX_BASE because most of AR71XX based devices use SPI, but others able to disable it with option. http://www.freebsd.org/cgi/query-pr.cgi?pr=170864