From owner-freebsd-net@FreeBSD.ORG Wed Jan 29 23:11:27 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4D55F561; Wed, 29 Jan 2014 23:11:27 +0000 (UTC) Received: from mail-pb0-x22e.google.com (mail-pb0-x22e.google.com [IPv6:2607:f8b0:400e:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 191CA118C; Wed, 29 Jan 2014 23:11:27 +0000 (UTC) Received: by mail-pb0-f46.google.com with SMTP id um1so2364408pbc.19 for ; Wed, 29 Jan 2014 15:11:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=DY5Bn7ksljXPjIRVBlEkpA6d9joXOdvyunVoYwCRcQ0=; b=GCPISSPhA2hQ50K2ncGUe2/qjFs3fCX8ObF/Ag8uNzyYiEPPnYiNrn/2DUPTu49+fY SkHSmy6ch/L9zgekJUNHGu1mkeBzyEC5WWYjoBs7DBXU3ksGuUj5l2T6I73cLB7P/LMw kWIZubTW1GBkDWROPLHqLuLl2mWu/yY4GJKvTo1GRjpl4Hn1WEUSjPtj5MlzUtYw8AOE VgRh4UWiG37CfwIcTsUoZU4oL+TLQtywwBoidRZFcHGS/3pJxy6dwPD7fKQvsdtSbgYs 0SFbIt28QTmXozQ5MaNy6cr45kql59aLcWJYbtVBCxvTe4Ygcc9OiZo3igVeYDm7ewXc EqIA== X-Received: by 10.68.130.202 with SMTP id og10mr10744421pbb.133.1391037086785; Wed, 29 Jan 2014 15:11:26 -0800 (PST) Received: from ox (c-24-6-44-228.hsd1.ca.comcast.net. [24.6.44.228]) by mx.google.com with ESMTPSA id sy10sm26834530pac.15.2014.01.29.15.11.25 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Wed, 29 Jan 2014 15:11:26 -0800 (PST) Date: Wed, 29 Jan 2014 15:11:21 -0800 From: Navdeep Parhar To: Adrian Chadd Subject: Re: Big physically contiguous mbuf clusters Message-ID: <20140129231121.GA18434@ox> Mail-Followup-To: Adrian Chadd , Garrett Wollman , FreeBSD Net References: <21225.20047.947384.390241@khavrinen.csail.mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Garrett Wollman , FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jan 2014 23:11:27 -0000 On Wed, Jan 29, 2014 at 02:21:21PM -0800, Adrian Chadd wrote: > Hi, > > On 29 January 2014 10:54, Garrett Wollman wrote: > > Resolved: that mbuf clusters longer than one page ought not be > > supported. There is too much physical-memory fragmentation for them > > to be of use on a moderately active server. 9k mbufs are especially > > bad, since in the fragmented case they waste 3k per allocation. > > I've been wondering whether it'd be feasible to teach the physical > memory allocator about >page sized allocations and to create zones of > slightly more physically contiguous memory. I think this would be very useful. For example, a zone_jumbo32 would hit a sweet spot -- enough to fit 3 jumbo frames and some loose change for metadata. I'd like to see us improve our allocators and VM system to work better with larger contiguous allocations, rather than deprecating the larger zones. It seems backwards to push towards smaller allocation units when installed physical memory in a typical system continues to rise. Allocating 3 x 4K instead of 1 x 9K for a jumbo means 3x the number of vtophys translations, 3x the phys_addr/len traffic on the PCIe bus (scatter list has to be fed to the chip and now it's 3x what it has to be), 3x the number of "wrapper" mbuf allocations (one for each 4K cluster) which will then be stitched together to form a frame, etc. etc. Regards, Navdeep > > For servers with lots of memory we could then keep these around and > only dip into them for temporary allocations (eg not VM pages that may > be held for some unknown amount of time.) > > Question is - can we enforce that kind of behaviour? > > > > -a > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"