From owner-svn-src-all@FreeBSD.ORG Wed Jan 8 15:20:04 2014 Return-Path: Delivered-To: svn-src-all@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8449CDBD; Wed, 8 Jan 2014 15:20:04 +0000 (UTC) Received: from mho-02-ewr.mailhop.org (mho-02-ewr.mailhop.org [204.13.248.72]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 55A2E135A; Wed, 8 Jan 2014 15:20:03 +0000 (UTC) Received: from c-24-8-230-52.hsd1.co.comcast.net ([24.8.230.52] helo=damnhippie.dyndns.org) by mho-02-ewr.mailhop.org with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1W0uvF-00039U-7Z; Wed, 08 Jan 2014 15:19:57 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id s08FJsLI030630; Wed, 8 Jan 2014 08:19:54 -0700 (MST) (envelope-from ian@FreeBSD.org) X-Mail-Handler: Dyn Standard SMTP by Dyn X-Originating-IP: 24.8.230.52 X-Report-Abuse-To: abuse@dyndns.com (see http://www.dyndns.com/services/sendlabs/outbound_abuse.html for abuse reporting information) X-MHO-User: U2FsdGVkX1/BFoLYJN/ST1o39NIrjhJN Subject: Re: svn commit: r260440 - head/sys/arm/conf From: Ian Lepore To: Nathan Whitehorn In-Reply-To: <52CCD1DA.7010008@freebsd.org> References: <201401080340.s083eIDG054652@svn.freebsd.org> <52CCD1DA.7010008@freebsd.org> Content-Type: text/plain; charset="us-ascii" Date: Wed, 08 Jan 2014 08:19:54 -0700 Message-ID: <1389194394.1158.362.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: svn-src-head@FreeBSD.org, svn-src-all@FreeBSD.org, src-committers@FreeBSD.org X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 15:20:04 -0000 On Tue, 2014-01-07 at 23:19 -0500, Nathan Whitehorn wrote: > On 01/07/14 22:40, Ian Lepore wrote: > > Author: ian > > Date: Wed Jan 8 03:40:18 2014 > > New Revision: 260440 > > URL: http://svnweb.freebsd.org/changeset/base/260440 > > > > Log: > > Add option USB_HOST_ALIGN to configs that contain 'device usb'. Setting > > this to the cache line size is required to avoid data corruption on armv4 > > and armv5, and improves performance on armv6, in both cases by avoiding > > partial cacheline flushes for USB IO. > > > > All these configs already exist in 10-stable. A few that don't (and > > thus can't be MFC'd yet) will be committed separately. > > > > There has to be -- and I do not mean this as a criticism of your patch > -- a better solution to this problem than USB_HOST_ALIGN. Isn't busdma > supposed to handle this kind of thing? Why is USB different? > -Nathan > USB is different because it doesn't follow the busdma rules. It allocates one large buffer, then sub-divides it internally into bits that are used for DMA IO and adjacent bits that are accessed by the cpu concurrently with the DMA. If it doesn't do that subdividing with an awareness of the cache line boundaries, it ends up with concurrent CPU and DMA access to data in the same cache line, and there's no way a software-assisted cache coherency scheme can reliably do busdma sync ops that don't corrupt either the CPU data or the DMA data. On armv6 we now automatically bounce IO that's not sized and aligned on cache line boundaries. The overhead for doing so is non-trivial, doubly so in the case of USB, because it's the only consumer of busdma in the system that requires that the offset-within-page for a bounced IO be the same as the offset in the original page (so a pool of small bounce buffers for small unligned IOs is not an option, it must allocate full bounce pages for every IO). It used to be (on armv4) that when you used the busdma alloc functions to allocate small DMA buffers (a few bytes) the implementation allocated entire pages, which is pretty inefficient and can add up to a lot of allocation overhead. That was cited as a reason not to change USB's "allocate big then subdivide" scheme. I wrote new busdma allocators that use UMA pools to efficiently handle small aligned buffers of both normal and uncachable (BUSDMA_COHERENT) memory, so that's not a roadblock anymore. (Arm uses the new allocator, mips never got converted.) So, since we keep getting reports on arm@ of data corruption that shows up as 32-byte chunks of bad data, and it costs real time and resources to try to debug each case, I figured we should just go with the fix that nobody likes but it actually works. -- Ian