From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 15:40:24 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 8E9A71065679 for ; Mon, 27 Aug 2012 15:40:24 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1C1648FC26 for ; Mon, 27 Aug 2012 15:40:23 +0000 (UTC) Received: by obbun3 with SMTP id un3so11051468obb.13 for ; Mon, 27 Aug 2012 08:40:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=t1YrUXxwc6T+C4ku+1nQRySvTX9kp0FRv2Gmd0Bg2co=; b=a1mCsdFrHrOgO6CVnsUzJ/bQ5l2PdH4hQzVpxCAMFP71oRYZJfQSJZEcrHdz1miCb1 /1HIuPzO9btcE8ygCgoiXEly76S+9A2/ZzwumpBOCvXLZlpaNBakJyJfy1AwUD4trruj LqjTjjjKQFh8132Tqf2rg0L9ktRLNdyg7qJ5Q9uLovUryfenkjpRbxXsgqfT5oxpNKgX WoFzxY/W97EF7Q5Ji/ARRT3haPaA9JxK0pC8EatyI8UpGP38viz3t0T2tJyJlnSeQGqU LvB8nKRxWYk4/hZDrXzJQrOSWA3AKPXF74+bSXtkpFhDr69R7t+BgJD7qiVdCEZ1Zv9F BywQ== Received: by 10.60.3.35 with SMTP id 3mr10500585oez.139.1346082023466; Mon, 27 Aug 2012 08:40:23 -0700 (PDT) Received: from [10.30.101.53] ([209.117.142.2]) by mx.google.com with ESMTPS id hz6sm17394331obb.1.2012.08.27.08.40.21 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 27 Aug 2012 08:40:21 -0700 (PDT) Sender: Warner Losh Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Mon, 27 Aug 2012 09:40:19 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <9642068B-3C66-42BD-8515-14F734B3FF89@bsdimp.com> References: <6D83AF9D-577B-4C83-84B7-C4E3B32695FC@bsdimp.com> To: Adrian Chadd X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQnV7kuwDh98NncF+eDHzMT5XGC25yNMWbW20dT4+ep96VudHChGlVhuHxD8mwOx8oIcEBA9 Cc: Tim Kientzle , Hans Petter Selasky , freebsd-arm@freebsd.org, freebsd-mips@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Partial cacheline flush problems on ARM and MIPS X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:40:24 -0000 On Aug 27, 2012, at 9:20 AM, Adrian Chadd wrote: > On 27 August 2012 08:12, Tim Kientzle wrote: >> But don't mbuf structures do pretty much what Hans is suggesting? >>=20 >> Why is mbuf okay? >=20 > Which part of mbufs? >=20 > I think the difference here is that there's no concurrent access. Ie, > although you may have an mbuf header + data inside the same cache > line, you wouldn't go fondling the mbuf once it's handed to the > hardware. >=20 > But I was under the impression that mbuf + mbuf buffers are already > correctly aligned. Not quite. mbufs are large enough and aligned enough so that no two = mbufs are in the same cache line. The data inside the mbuf on 32-bit = platforms is 24 bytes from the start of the mbuf, so the mbuf data and = mbuf header do share the same cache line, but as you say, nobody touches = any part of the mbuf will the hardware has it. Recall that it isn't = alignment of where the buffer starts that matters here, but rather = alignment to ensure that there's no sharing of cache lines between = structures. > This all honestly looks like a very i386-centric interpretation of the > busdma "intention", which I think illustrates a need to better > document what assumptions busdma actually makes. Yes. x86 lets you play fast and loose with many of these things. We've = tried to accommodate this for a long time, but that is lame. > That does remind me, I think the ath(4) driver does the same (since it > allocates its own descriptor block and then treats it as an array of > descriptors for the hardware to access) - I should ensure that > sizeof(ath_desc) is aligned on the relevant architecture. It gets > slightly scary - AR93xx TX descriptors are "L1 cache =3D=3D 128 byte > aligned" which is an enormous waste of memory compared to a 16 or 32 > byte aligned platform. Alas.. The problem is with cache line sharing, not necessarily with alignment. = If you are only ever using one of them at a time, or if you have perfect = hygiene, you can cope with this situation without undue waste. The = perfect hygiene might be hard sometimes. Warner