From owner-freebsd-net@FreeBSD.ORG Thu Jan 30 02:05:27 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3DF5EDB1; Thu, 30 Jan 2014 02:05:27 +0000 (UTC) Received: from mail-pb0-x230.google.com (mail-pb0-x230.google.com [IPv6:2607:f8b0:400e:c01::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 09F92149D; Thu, 30 Jan 2014 02:05:27 +0000 (UTC) Received: by mail-pb0-f48.google.com with SMTP id rr13so2504520pbb.21 for ; Wed, 29 Jan 2014 18:05:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=BYVlWTcCBZVAZbvxJEeExuf3OB4kxF3bOLO3wrj0Qx8=; b=C7rLbW3saSuaYMczwhkmx2ff4KJh/4VWLQhfmKDZurfSSwZ+GoB/iDJbwaFz0w0QYI AqfymKhS2JfGGYF4YLXT12XrjW+VI4Afxna60MQv7QuuXHxSmihnEe1bwezVkI3OxT6u 8N5BY8kqFO/V4BE78nYOhmQ2VMahspB2DxwhhaXQ/40HWR/QCvse39qaz93qvQxqDZ9M cWs0k9O6iH6dZ+m13zTDPRy6+8gFaQwGr36Nry1qjCAlzav3qPa0G5b9ulMUMb/OYlTL /eRxclfsVVX/1VGCRNIFuMjMUYOqPE1RISCDcYw3MPjuMWb0x7NSU75NztfpyKCcT3Wa TIhQ== X-Received: by 10.66.163.164 with SMTP id yj4mr11461589pab.91.1391047526709; Wed, 29 Jan 2014 18:05:26 -0800 (PST) Received: from ox (c-24-6-44-228.hsd1.ca.comcast.net. [24.6.44.228]) by mx.google.com with ESMTPSA id ns7sm11640705pbc.32.2014.01.29.18.05.25 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Wed, 29 Jan 2014 18:05:25 -0800 (PST) Date: Wed, 29 Jan 2014 18:05:23 -0800 From: Navdeep Parhar To: Adrian Chadd , Garrett Wollman , FreeBSD Net Subject: Re: Big physically contiguous mbuf clusters Message-ID: <20140130020523.GB18434@ox> Mail-Followup-To: Adrian Chadd , Garrett Wollman , FreeBSD Net References: <21225.20047.947384.390241@khavrinen.csail.mit.edu> <20140129231121.GA18434@ox> <20140130013434.GP93141@funkthat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140130013434.GP93141@funkthat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jan 2014 02:05:27 -0000 On Wed, Jan 29, 2014 at 05:34:34PM -0800, John-Mark Gurney wrote: > Navdeep Parhar wrote this message on Wed, Jan 29, 2014 at 15:11 -0800: > > > (scatter list has to be fed to the chip and now it's 3x what it has to > > be), 3x the number of "wrapper" mbuf allocations (one for each 4K > > cluster) which will then be stitched together to form a frame, etc. etc. > > And what is that in percentage of overall traffic? .4% (assuming 16 bytes > per 4k page)... If your PCIe bus is saturating and you need that extra > .4% traffic, then you have a serious issue w/ your bus layout... The 16B and 4KB are in different directions, the former is from host to chip and the latter from chip to host memory. So the 16B eats into the transmit bandwidth. FWIW, I do deal with cards where PCIe is the limiting factor (a 4x10G card with a pcie gen2 x8 block, a 2x40G card with pcie gen3 x8 block) and the effects of 4K vs. 9K rx on the transmit bandwidth are measurable. These days chips can even place multiple frames into a single buffer (if they'd fit) and that's another reason I tend to advocate for larger contiguous buffer sizes. Regards, Navdeep