From owner-svn-src-all@FreeBSD.ORG Tue Feb 24 19:26:57 2015 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CFCD03B4; Tue, 24 Feb 2015 19:26:57 +0000 (UTC) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A7717AA8; Tue, 24 Feb 2015 19:26:57 +0000 (UTC) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t1OJQuU5005046 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 24 Feb 2015 11:26:56 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t1OJQu41005045; Tue, 24 Feb 2015 11:26:56 -0800 (PST) (envelope-from jmg) Date: Tue, 24 Feb 2015 11:26:56 -0800 From: John-Mark Gurney To: Ian Lepore Subject: Re: svn commit: r279236 - head/sys/netinet Message-ID: <20150224192656.GK46794@funkthat.com> References: <201502241257.t1OCv40V097418@svn.freebsd.org> <20150224173413.GF46794@funkthat.com> <1424804342.3293.9.camel@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1424804342.3293.9.camel@freebsd.org> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Tue, 24 Feb 2015 11:26:56 -0800 (PST) Cc: Zbigniew Bodek , svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Feb 2015 19:26:57 -0000 Ian Lepore wrote this message on Tue, Feb 24, 2015 at 11:59 -0700: > On Tue, 2015-02-24 at 09:34 -0800, John-Mark Gurney wrote: > > Zbigniew Bodek wrote this message on Tue, Feb 24, 2015 at 12:57 +0000: > > > Author: zbb > > > Date: Tue Feb 24 12:57:03 2015 > > > New Revision: 279236 > > > URL: https://svnweb.freebsd.org/changeset/base/279236 > > > > > > Log: > > > Change struct attribute to avoid aligned operations mismatch > > > > > > Previous __alignment(4) allowed compiler to assume that operations are > > > performed on aligned region. On ARM processor, this led to alignment fault > > > as shown below: > > > trapframe: 0xda9e5b10 > > > FSR=00000001, FAR=a67b680e, spsr=60000113 > > > r0 =00000000, r1 =00000068, r2 =0000007c, r3 =00000000 > > > r4 =a67b6826, r5 =a67b680e, r6 =00000014, r7 =00000068 > > > r8 =00000068, r9 =da9e5bd0, r10=00000011, r11=da9e5c10 > > > r12=da9e5be0, ssp=da9e5b60, slr=a054f164, pc =a054f2cc > > > <...> > > > udp_input+0x264: ldmia r5, {r0-r3, r6} > > > udp_input+0x268: stmia r12, {r0-r3, r6} > > > > > > This was due to instructions which do not support unaligned access, > > > whereas for __alignment(2) compiler replaced ldmia/stmia with some > > > logically equivalent memcpy operations. > > > In fact, the assumption that 'struct ip' is always 4-byte aligned > > > is definitely false, as we have no impact on data alignment of packet > > > stream received. > > > > So, the whole point of ETHER_ALIGN is to make struct ip aligned on > > 4 byte offsets... This will probably impact performance on arm for > > properly aligned struct ip... > > > > ETHER_ALIGN is wonderful... if you're on a platform that can DMA to an > arbitrary boundary. Of course, if you're on such a platform it can > probably just access the word-sized values on halfword boundaries > anyway. For arm, the only solution at the driver level is to memcpy() > every incoming packet to another buffer to realign it. If you think > that makes receive performance really bad, you'd be right. Having working on the TS-7200 which had this restriction, I know very well the issues... > Many arm platforms can only DMA on a cacheline boundary. The size of an Are you sure? Last time I looked at the various drivers, about a year or two ago, I think I found only one ARM driver that didn't do at least 2 byte DMA alignment.. Either that, or they haven't been declaring their restrictions properly in bus_dma_tag_create... > mbuf header is like 24 or 28 or something, definitely not cache aligned. > So in addition to the extra copying the drivers do for ETHER_ALIGN, > there could also be bounce-buffer copying involved due to the alignment > in the busdma tag. Can you list a few of these drivers for me? I'd be curious to look at them again... > The latter issue could be fixed with an MD padding field at the end of > the mbuf header to make the data portion start on a cache line boundary. > When I experimented with that concept on imx6 I gained 10 MB/sec > performance from the reduced copying. ffec allows byte aligned rx dma, but it looks like 16 byte TX align... If the ffec allows segments that are not a multiple of 16, there is/was talk about supporting this, and then you'd only have to bounce a few bytes instead of the entire packet, but I'm not sure if we can do that w/ the existing bus_dma framework.. I can't find the define/var right now, but I remeber there being an offset that is processed to try to put the packet at the correct byte offset... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."