From owner-freebsd-arch@FreeBSD.ORG Thu Aug 23 23:27:12 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1BBE71065670 for ; Thu, 23 Aug 2012 23:27:12 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [76.96.30.16]) by mx1.freebsd.org (Postfix) with ESMTP id EE4128FC19 for ; Thu, 23 Aug 2012 23:27:11 +0000 (UTC) Received: from omta17.emeryville.ca.mail.comcast.net ([76.96.30.73]) by qmta01.emeryville.ca.mail.comcast.net with comcast id qEbq1j04X1afHeLA1PTBbX; Thu, 23 Aug 2012 23:27:11 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta17.emeryville.ca.mail.comcast.net with comcast id qPTA1j00D4NgCEG8dPTAp2; Thu, 23 Aug 2012 23:27:11 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q7NNR8Lw025556; Thu, 23 Aug 2012 17:27:08 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Adrian Chadd In-Reply-To: References: <1345757300.27688.535.camel@revolution.hippie.lan> Content-Type: text/plain; charset="us-ascii" Date: Thu, 23 Aug 2012 17:27:08 -0600 Message-ID: <1345764428.27688.591.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Aug 2012 23:27:12 -0000 On Thu, 2012-08-23 at 16:09 -0700, Adrian Chadd wrote: > [snip] > > Whoa, there's USB code that has these small buffers straddle cache lines? > > Why aren't they just allocated to always be in their own separate > buffers, so they don't ever have to worry about overlapping cache > lines? Yes, but note that with our current code, a small buffer doesn't have to straddle a cacheline boundary to invoke the partial cacheline flush logic. Its very smallness does so -- if a buffer doesn't start on a cacheline boundary, and/or if its size doesn't exactly equal a multiple of the cacheline size, then it's a partial cacheline flush situation. So even a driver striving to work around this trouble can get into trouble -- it uses bus_dmamem_alloc() to grab a buffer for a 13 byte status message. It gets a 16-byte chunk from the kernel allocator, and the other 16 bytes of that cacheline are given to something else in the kernel, and now you have DMA and CPU access going on in the same cacheline despite your efforts to Do The Right Thing with buffer allocation. (At work we avoid this by setting the kernel minimum allocation size to 32 bytes instead of 16. We did so after being burned by exactly this situation -- the "other 16 bytes" in our case belonged to a struct ithread, and trying to track down a lockup that was caused by a partial cacheline flush corrupting the struct ithread kept me entertained for a couple weeks.) -- Ian