From owner-freebsd-hackers@FreeBSD.ORG Sat Nov 4 16:24:59 2006 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A0ECB16A6D1 for ; Sat, 4 Nov 2006 16:24:59 +0000 (UTC) (envelope-from sam@errno.com) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4B32A43D53 for ; Sat, 4 Nov 2006 16:24:57 +0000 (GMT) (envelope-from sam@errno.com) Received: from [10.0.0.248] (trouble.errno.com [10.0.0.248]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id kA4GOtS3069505 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 4 Nov 2006 08:24:57 -0800 (PST) (envelope-from sam@errno.com) Message-ID: <454CBED7.2000103@errno.com> Date: Sat, 04 Nov 2006 08:24:55 -0800 From: Sam Leffler User-Agent: Thunderbird 1.5.0.7 (X11/20060920) MIME-Version: 1.0 To: pyunyh@gmail.com References: <9ab217670611021511l3120d58bhd0b61bf44f8ecc87@mail.gmail.com> <454A7EF2.5090201@errno.com> <20061103003311.GD69214@cdnetworks.co.kr> In-Reply-To: <20061103003311.GD69214@cdnetworks.co.kr> X-Enigmail-Version: 0.94.0.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: hackers@freebsd.org, "Devon H. O'Dell" Subject: Re: vr(4) performance X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Nov 2006 16:24:59 -0000 Pyun YongHyeon wrote: > On Thu, Nov 02, 2006 at 03:27:46PM -0800, Sam Leffler wrote: > > Devon H. O'Dell wrote: > > > Hey all, > > > > > > So, vr(4) kind of sucks, and it seems like this is mostly due to the > > > fact that we call m_defrag() on every mbuf that we send through it. > > > This seems to really screw performance on outgoing packets (something > > > like 33% the output efficiency of fxp(4), if I'm understanding this > > > all correctly). > > > > > > I'm sort of wondering if anybody has attempted to address this before > > > and if there's a way to possibly mitigate this behavior. I know Bill > > > Paul's comments say ``Unfortunately, FreeBSD FreeBSD doesn't guarantee > > > that mbufs will be filled in starting at longword boundaries, so we > > > have to do a buffer copy before transmission.'' -- since it's been a > > > long day, and I'm about to go home to grab a pizza and stop thinking > > > about code, would anybody mind offering suggestions as to either: > > > > > > a) Pros and cons of guaranteeing that they're filled in aligned (and > > > possibly hints on doing it), or > > > b) Possible workarounds / hacks to do this faster for vr(4) > > > > > > Any input is appreciated! (Except ``vr(4) is lol'') > > > > m_defrag is ~10x slower than it needs to be. I proposed changes to > > address this a while back but eventually gave up and put driver-specific > > code in ath. You can look there or I can send you some patches to > > m_defrag to try out in vr. > > > > Because the purpose of m_defrag(9) in vr(4) is to guarantee longword > aligned mbufs I'm not sure ath_defrag can be used here. If memory > serve me right ath_defrag would not change the first mbuf address > in a chain. If the first mbuf is not aligned on longword boundary > it wouldn't work I guess. Of course we can check the first mbuf in > the chain before calling super-fast ath_defrag, I guess. > m_defrag is used for two purposes (mainly) in the system: reducing the mbuf count in a chain so that an outbound packet fits in a limited number of h/w tx descriptors and aligning packet data for cards with constrained dma engines. Both these operations belong in bus_dma. Combining both these operations in a single routine results in overly pessimistic code for the common case. Separately the algorithm in m_defrag is suboptimal (e.g. it makes a complete copy even when a packet needs no changes). ath_defrag is example code tailored to the ath driver that handles only the mbuf chain too long issue. I have other code that can do packet alignment and/or both alignment+mbuf coalescing far better than the current logic in m_defrag. The right solution to this problem--as suggested by John Baldwin and Scott Long is to improve the bus_dma code so these things happen automatically for the driver according to the dma tag config. This would eliminate the need for m_defrag in all cases I'm aware of. Since bus_dma has info like the max # segments a device can accept and any alignment constraints it can do a much more efficient job. Sam