From owner-freebsd-arm@FreeBSD.ORG Mon Aug 27 15:57:54 2012 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 67F9F1065673 for ; Mon, 27 Aug 2012 15:57:54 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0C5738FC14 for ; Mon, 27 Aug 2012 15:57:53 +0000 (UTC) Received: by obbun3 with SMTP id un3so11111201obb.13 for ; Mon, 27 Aug 2012 08:57:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=+6b5VLtRymOQqJ25JKslOXKH/1wAxmLCqLr7LE+kYFs=; b=Pxzv2SX3IBwvPqtEHWj5G/kPuCtQMvc3OoLE36JGWitV9nBYGHJfR73iUeFvhfa/hP R1hb/UCLyeP+t0j7ZRfwrFGGNsjQzcN7AiKH8YDoEwRGDv6OyVipeyvDDdpLKv9M2jwk ssppQOsdgLXZeT/9K0R6Odh/fFfkDP5nSDsl7+EPoYRWzZlNQzuzbpVVPB+Kl6qMYzjA NmZ632j0Xq72LN6NgPouRoTPs/T7rJAmFNC84sy8StNBvU7mrwDBGhAfM7cri1GPZ2XZ 7p3wvmD9uAzdAXRvDaajBNBVHU7ikBeG/NdMT8yq5XWsXUTWarxiiAurl0cCfr2VGKFe MRwQ== Received: by 10.60.19.34 with SMTP id b2mr10134983oee.41.1346083073414; Mon, 27 Aug 2012 08:57:53 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id ov5sm9250003obc.2.2012.08.27.08.57.51 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 08:57:52 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1346081557.1140.181.camel@revolution.hippie.lan> Date: Mon, 27 Aug 2012 09:57:48 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <1345757300.27688.535.camel@revolution.hippie.lan> <3A08EB08-2BBF-4B0F-97F2-A3264754C4B7@bsdimp.com> <1345763393.27688.578.camel@revolution.hippie.lan> <1345765503.27688.602.camel@revolution.hippie.lan> <1345766109.27688.606.camel@revolution.hippie.lan> <1346002922.1140.56.camel@revolution.hippie.lan> <1346005507.1140.69.camel@revolution.hippie.lan> <10307B47-13F3-45C0-87F7-66FD3ACA3F86@bsdimp.com> <1346081557.1140.181.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQn7xsdR/TiJaMnB8KzFil08ZVW1k19ZkMYkdepb/k438KhTO+NXhGG5mLgi7j3C4dhxTTog Cc: freebsd-arm@freebsd.org, freebsd-arch@freebsd.org, freebsd-mips@freebsd.org, Hans Petter Selasky Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:57:54 -0000 On Aug 27, 2012, at 9:32 AM, Ian Lepore wrote: > On Sun, 2012-08-26 at 17:13 -0600, Warner Losh wrote: >> On Aug 26, 2012, at 12:25 PM, Ian Lepore wrote: >>> In this regard, it's the busdma implementation that's broken, = because it >>> should bounce those IOs through a DMA-safe buffer. There's = absolutely >>> no rule that I've ever heard of in FreeBSD that says IO can only = take >>> place using memory allocated from busdma. >>=20 >> That's partially true. Since BUSDMA grew up in the storage area, you = must allocate the memory from busdma, or it must be page aligned has = been the de-facto rule here. =20 >=20 > Where does anything say that you must allocate the memory from busdma = if > it's not mbuf/page-aligned? I've never seen it in any docs. I > certainly find contrary evidence in existing driver code (and I'm not > talking about USB here). Where does it say that you are allowed to not use the routines? >> The mbuf and uio variants of load were invented to cope with common = cases of mbufs and user I/O to properly flag things. >>=20 >=20 > What does that mean, "to properly flag things"? >=20 > I think with uio we come to the crux of the issue. Userland buffers = are > not allocated with knowledge of the busdma constraints. That leads to > some choices: >=20 > * We don't support IO with a userland buffer at all. You may have to bounce here. > * We support userland IO if the userland buffers are accidentally > aligned properly for the platform, otherwise the call fails. You may have to bounce here. > * We support userland buffers by making any driver which wants to > do so responsible for always copying the data to aligned = buffers > (each individual driver handles bounces). This is what most drivers that want to do zero copying do. > * We support userland buffers by having the busdma layer handle > the bouncing when required. I thought that's what the uio variants of the map loading routines were = supposed to do. > The first two seem untenable to me. The third one comes down to = "every > driver must always bounce all userland data" because the driver = doesn't > actually have access to the info to decide whether a userland buffer = is > aligned properly or not for a given platform. >=20 > That leaves the option where the busdma layer handles bouncing for > unaligned userland buffers. If that's possible, then the busdma layer > could automatically bounce any unaligned request, whether it came from > userland or a driver reading a status response into an unaligned local > buffer in kernel memory. Right, the uio load option is supposed to do this. >> How does busdma know that it is using memory that's not from its = allocator? >=20 > The busdma code allocates the map along with the buffer, and can = record > information in the map that it can use during mapping and sync > operations to know whether it allocated the buffer. >=20 >>> The rule is only that the >>> proper sequence of busdma operation must be called, and beyond that = it's >>> up to the busdma implementation to make it work. =20 >>=20 >> No. Bouncing is needed due to poor alignment of the underlying = device. Not due to cache effects. >=20 > Says who? This statement seems to me to be more about opinion and = dogma > than about a documented API or technical concerns about how to = implement > it. IMO, bouncing is needed when that's the only way to make a > particular busdma sequence work correctly given the parameters of the > hardware and the transfer. Says the people that wrote and use busdma? There's no requirement for = cache effects for the transfer, so there's no constraints for the = transfer. The requirement is about having a buffer that's suitable for = DMA. This means either the buffer (not the start of the transfer) be = aligned to a cache line and be an integral number of cache lines in = size, or it means that you have 100% perfect hygiene when it comes to = ensuring the CPU touches the buffer or the device touches it and ever = have cross threading. I'd agree that better docs here would help. > To put it another way, the busdma subsystem is responsible for helping > make DMA transfers work on a platform without every driver needing to > contain platform-specific code, and bouncing is one of the techniques > available to it. Buffers allocated through the busdma system do this. >> There's a limited number of things that we support with busdma. = Arbitrary data from malloc that might be shared with the CPU isn't on = that list. >>=20 >=20 > Where are those limitations documented? =20 Where is it documented that memory returned from malloc may be used for = DMA? I agree that docs could be better here. > This isn't some small oversight in the documention, if the rule is = "IO, > any IO at all, can only be performed using buffers allocated from > busdma" that's a pretty major thing that ought to appear in multiple > manpages relating to driver development. >=20 > If you don't think "any IO at all" is the right characterization, read > all the way to the end before responding. Yes. I'd state it another way. "Buffers returned from the busdma = allocation routines is always safe." Everything else may or may not be = safe. mbufs, for example, happen to be safe because there is a rigid = protocol for sharing between the driver and the device. uio buffers can = be made safe. busdma has routines to ensure that these types of data = are handled properly. >>> Our biggest problem, I think, is that we don't have a sufficient >>> definition of "the proper sequence of busdma operations." >>=20 >> I disagree. The sequence has been known for a long time. >>=20 >>> I don't think it will be very hard to make the arm and mips busdma >>> implementations work correctly. It won't even be too hard to make = them >>> fairly efficient at bouncing small IOs (my thinking is that we can = make >>> small bounces no more expensive than the current partial cacheline = flush >>> implementation which copies the data multiple times). Bouncing = large IO >>> will never be efficient, but the inefficiency will be a powerful >>> motivator to update drivers that do large IO to work better, such as >>> using buffers allocated from busdma. >>=20 >> I don't think the cache line problem can be solved with bounce = buffers. Trying to accommodate broken drivers is what lead us to this = spot. We need to fix the broken drivers. If that's impossible, then = the best we can do is have the driver set a 'always bounce' flag in the = tag it creates and use that to always bounce for operations through that = tag. >=20 > So you'd be okay with a driver setting a flag that says "always = bounce" > but not okay with the busdma layer bouncing only when it's actually > necessary? I'm confused -- do you think the busdma layer will be = unable > to detect when it's necessary unless directed from the outside? Actually, it is good you are confused. I was wrong when I said I'd be = happy with a flag that says always bounce. The reason is that the driver = can do this all the time for those cases like the at91 mci driver does. > Let me pose a question back to you... if it is up to a driver to > allocate DMA buffers using the busdma allocator, how is any given = driver > to know whether DMA is going to be involved in the transfer? =20 because it does the dma? > Don't think USB or ATA here... think iicbus/foo_temperature_sensor, or > the mmc/sd driver. Those drivers know nothing about the hardware = bridge > layers beneath them and whether DMA is involved or not. They just = know > "I need to read a 2-byte value from a register on this IIC chip" or "I > need to retrieve the 32-bit CSD data from the SD card" and they > accomplish that by calling a read method of some lower-level bridge > driver. In each of those cases we have both PIO and DMA = implementations > of the lower-level drivers that move bits over a wire. The bridge here would know and have to cope. However, the real issue in = this case wouldn't be the transfer alignment, but what traffic might be = happening to other parts of the cache line. For small things like this, typically data copies are involved. There's = a few places that break these rules (I think mci might be breaking the = rules for small transfers, for example). > If the concensus is that such drivers should always allocate all IO > buffers using busdma, even though they have no idea whether DMA is = going > to be involved, then we certainly have a lot of work ahead of us in > terms of "fixing broken drivers." That also implies that > bus_dmamem_alloc() is misnamed and implemented in the wrong place, > because it's not really about dma at all. You can't get around the hardware requirement that any I/O going to the = device cannot share cache lines with other data on the system. > If you push the responsibility down to the layer where it is known > whether DMA is going to be involved, then we're back to the need for > many individual drivers to bounce every request, because they don't = have > access to the info needed to know whether bouncing is required or not > for a given buffer they were handed from higher layers. (Remember = when > I proposed exporting that info to drivers so they could decide whether = a > buffer is dma-aligned or not, the response was "No, keep it all hidden > in the busdma layer," which I think is the right response.) Right now we support only a few things doing DMA on the system. We = support pages, mbufs and to a limited extent uio buffers (but to be = honest, we may have exposure on smaller transfers here). > Or if you push the responsibility down to the busdma layer where all > that info resides, all drivers which use DMA and adhere to the busdma > rules just work, without needing to know anything about the buffers = that > were handed to them and without needing to know anything about the > platform requirements. I'm not sure I follow this last bit... Warner