From owner-svn-src-head@FreeBSD.ORG Wed Jan 8 15:50:12 2014 Return-Path: Delivered-To: svn-src-head@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1E5529EA; Wed, 8 Jan 2014 15:50:12 +0000 (UTC) Received: from smtpauth3.wiscmail.wisc.edu (wmauth3.doit.wisc.edu [144.92.197.226]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DC941164F; Wed, 8 Jan 2014 15:50:11 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Received: from avs-daemon.smtpauth3.wiscmail.wisc.edu by smtpauth3.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MZ300300ADYQ100@smtpauth3.wiscmail.wisc.edu>; Wed, 08 Jan 2014 09:50:10 -0600 (CST) X-Spam-PmxInfo: Server=avs-3, Version=6.0.3.2322014, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2014.1.8.153914, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from wanderer.tachypleus.net (pool-72-66-107-173.washdc.fios.verizon.net [72.66.107.173]) by smtpauth3.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MZ300GMWANJU820@smtpauth3.wiscmail.wisc.edu>; Wed, 08 Jan 2014 09:50:09 -0600 (CST) Message-id: <52CD73AF.1030205@freebsd.org> Date: Wed, 08 Jan 2014 10:50:07 -0500 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 To: Ian Lepore Subject: Re: svn commit: r260440 - head/sys/arm/conf References: <201401080340.s083eIDG054652@svn.freebsd.org> <52CCD1DA.7010008@freebsd.org> <1389194394.1158.362.camel@revolution.hippie.lan> In-reply-to: <1389194394.1158.362.camel@revolution.hippie.lan> X-Enigmail-Version: 1.6 Cc: svn-src-head@FreeBSD.org, svn-src-all@FreeBSD.org, src-committers@FreeBSD.org X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 15:50:12 -0000 On 01/08/14 10:19, Ian Lepore wrote: > On Tue, 2014-01-07 at 23:19 -0500, Nathan Whitehorn wrote: >> On 01/07/14 22:40, Ian Lepore wrote: >>> Author: ian >>> Date: Wed Jan 8 03:40:18 2014 >>> New Revision: 260440 >>> URL: http://svnweb.freebsd.org/changeset/base/260440 >>> >>> Log: >>> Add option USB_HOST_ALIGN to configs that contain 'device usb'. Setting >>> this to the cache line size is required to avoid data corruption on armv4 >>> and armv5, and improves performance on armv6, in both cases by avoiding >>> partial cacheline flushes for USB IO. >>> >>> All these configs already exist in 10-stable. A few that don't (and >>> thus can't be MFC'd yet) will be committed separately. >>> >> There has to be -- and I do not mean this as a criticism of your patch >> -- a better solution to this problem than USB_HOST_ALIGN. Isn't busdma >> supposed to handle this kind of thing? Why is USB different? >> -Nathan >> > USB is different because it doesn't follow the busdma rules. It > allocates one large buffer, then sub-divides it internally into bits > that are used for DMA IO and adjacent bits that are accessed by the cpu > concurrently with the DMA. If it doesn't do that subdividing with an > awareness of the cache line boundaries, it ends up with concurrent CPU > and DMA access to data in the same cache line, and there's no way a > software-assisted cache coherency scheme can reliably do busdma sync ops > that don't corrupt either the CPU data or the DMA data. > > On armv6 we now automatically bounce IO that's not sized and aligned on > cache line boundaries. The overhead for doing so is non-trivial, doubly > so in the case of USB, because it's the only consumer of busdma in the > system that requires that the offset-within-page for a bounced IO be the > same as the offset in the original page (so a pool of small bounce > buffers for small unligned IOs is not an option, it must allocate full > bounce pages for every IO). > > It used to be (on armv4) that when you used the busdma alloc functions > to allocate small DMA buffers (a few bytes) the implementation allocated > entire pages, which is pretty inefficient and can add up to a lot of > allocation overhead. That was cited as a reason not to change USB's > "allocate big then subdivide" scheme. I wrote new busdma allocators > that use UMA pools to efficiently handle small aligned buffers of both > normal and uncachable (BUSDMA_COHERENT) memory, so that's not a > roadblock anymore. (Arm uses the new allocator, mips never got > converted.) > > So, since we keep getting reports on arm@ of data corruption that shows > up as 32-byte chunks of bad data, and it costs real time and resources > to try to debug each case, I figured we should just go with the fix that > nobody likes but it actually works. > > -- Ian > > Thanks for the explanation, the debugging, and the fix. This seems like a straightforward bug in the USB stack. Can it be fixed, or are there architectural reasons why it is the way it is? -Nathan