From owner-freebsd-current@FreeBSD.ORG Mon Jan 26 17:29:03 2009 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7B14F106564A for ; Mon, 26 Jan 2009 17:29:03 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.freebsd.org (Postfix) with ESMTP id DFB838FC17 for ; Mon, 26 Jan 2009 17:29:02 +0000 (UTC) (envelope-from scottl@samsco.org) Received: from phobos.local ([192.168.254.200]) (authenticated bits=0) by pooker.samsco.org (8.14.2/8.14.2) with ESMTP id n0QGqYSo029635; Mon, 26 Jan 2009 09:52:35 -0700 (MST) (envelope-from scottl@samsco.org) Message-ID: <497DEA52.3040004@samsco.org> Date: Mon, 26 Jan 2009 09:52:34 -0700 From: Scott Long User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.13) Gecko/20080313 SeaMonkey/1.1.9 MIME-Version: 1.0 To: Sepherosa Ziehau References: <497758.90319.qm@web63903.mail.re1.yahoo.com> In-Reply-To: X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-4.4 required=3.8 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on pooker.samsco.org Cc: barney_cordoba@yahoo.com, current@freebsd.org Subject: Re: DMA bounce buffers X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Jan 2009 17:29:04 -0000 Sepherosa Ziehau wrote: > On Mon, Jan 26, 2009 at 9:05 PM, Barney Cordoba > wrote: >> I have a problem with the ixgbe driver, where bus_dmamap_load_mbuf_sg() is >> failing. The underlying failure is reserve_bounce_pages(). It seems that >> there aren't any bounce pages allocated. What could be the cause of this? Is there some tunable? > > I don't think the hardware supported ixgbe(4) will need bounce pages > on RX/TX buffers (according to data sheet, the hardware supports full > 64bits address space accessing and no alignment constraints on RX/TX > buffer start address). However, the driver's buffer busdma is kinda > misconfigured: in ixgbe_allocate_{transmit,receive}_buffers(), the > buffers' alignment is set to PAGE_SIZE (?!), this means that almost > all TX/RX operation will require bouncing. The best case is poor > performance due to additional and unnecessary memory copy in > bus_dmamap_sync(); the worst case is completely unfunction NIC due to > RX ring could not be filled during ifnet.if_init(). > Yes, no high-speed NIC should be using bounce buffers. I haven't looked at the ixgbe data sheets, but I'd be highly surprised if this hardware actually needed buffer alignment like that. If it does, then the better thing to do is to either use 4k mbuf clusters (and make sure that the system doesn't try to treat them as jumbo frames), or have the rx ring allocator throw away and reallocate any clusters that aren't already on 4k boundaries (normal clusters are 2k, so it'll have a nominal 50% chance of getting what it needs). Neither of these solutions is ideal, though. Scott