From owner-svn-src-all@FreeBSD.ORG Thu Aug 14 22:47:25 2014 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 558DE3B9; Thu, 14 Aug 2014 22:47:25 +0000 (UTC) Received: from pp2.rice.edu (proofpoint2.mail.rice.edu [128.42.201.101]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 185B34C8C; Thu, 14 Aug 2014 22:47:24 +0000 (UTC) Received: from pps.filterd (pp2.rice.edu [127.0.0.1]) by pp2.rice.edu (8.14.5/8.14.5) with SMTP id s7EMlIsP008197; Thu, 14 Aug 2014 17:47:18 -0500 Received: from mh1.mail.rice.edu (mh1.mail.rice.edu [128.42.201.20]) by pp2.rice.edu with ESMTP id 1nrys908br-1; Thu, 14 Aug 2014 17:47:17 -0500 X-Virus-Scanned: by amavis-2.7.0 at mh1.mail.rice.edu, auth channel Received: from 108-254-203-201.lightspeed.hstntx.sbcglobal.net (108-254-203-201.lightspeed.hstntx.sbcglobal.net [108.254.203.201]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh1.mail.rice.edu (Postfix) with ESMTPSA id 20A2D460110; Thu, 14 Aug 2014 17:47:11 -0500 (CDT) Message-ID: <53ED3C6E.10204@rice.edu> Date: Thu, 14 Aug 2014 17:47:10 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: d@delphij.net, John-Mark Gurney , Xin LI Subject: Re: svn commit: r269963 - head/sys/kern References: <201408140513.s7E5DPRb069698@svn.freebsd.org> <20140814053518.GO83475@funkthat.com> <53EC560B.5000104@delphij.net> In-Reply-To: <53EC560B.5000104@delphij.net> X-Enigmail-Version: 1.6 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=0 kscore.compositescore=0 circleOfTrustscore=0 compositescore=0.248919945447816 urlsuspect_oldscore=0.248919945447816 suspectscore=13 recipient_domain_to_sender_totalscore=0 phishscore=0 bulkscore=0 kscore.is_spamscore=0 recipient_to_sender_totalscore=0 recipient_domain_to_sender_domain_totalscore=0 rbsscore=0.248919945447816 spamscore=0 recipient_to_sender_domain_totalscore=0 urlsuspectscore=0.9 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1408140260 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 22:47:25 -0000 On 08/14/2014 01:24, Xin Li wrote: > On 8/13/14 10:35 PM, John-Mark Gurney wrote: > > Xin LI wrote this message on Thu, Aug 14, 2014 at 05:13 +0000: > >> Author: delphij Date: Thu Aug 14 05:13:24 2014 New Revision: > >> 269963 URL: http://svnweb.freebsd.org/changeset/base/269963 > >> > >> Log: Re-instate UMA cached backend for 4K - 64K allocations. New > >> consumers like geli(4) uses malloc(9) to allocate temporary > >> buffers that gets free'ed shortly, causing frequent TLB shootdown > >> as observed in hwpmc supported flame graph. > > > Can we do even larger, like 128k for phys io sized blocks? > > Sure (Actually I'm running with 128k and 256k buckets enabled on my > own storage box; with r269964 we can easily add new buckets without > actually activating them by default). > > However, I'm relented to add them right now because the current > malloc(9) implementation would use the next bucket size, which is 2x > of the previous one, when the requested size is only a little bit > larger than the smaller chunk's size. In real world the larger bucket > could eat more memory than all smaller but greater than page-sized > bucket combined (the actual consumption is still small, though). > The current code already supports sizes that are not powers of 2. For example, with Index: kern/kern_malloc.c =================================================================== --- kern/kern_malloc.c (revision 269997) +++ kern/kern_malloc.c (working copy) @@ -152,8 +152,11 @@ struct { {2048, "2048", }, {4096, "4096", }, {8192, "8192", }, + {12228, "12228", }, {16384, "16384", }, + {24576, "24576", }, {32768, "32768", }, + {49152, "49152", }, {65536, "65536", }, {0, NULL}, }; I see ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP UMA Kegs: 384, 0, 94, 6, 94, 0, 0 ... 16: 16, 0, 2501, 260, 36924, 0, 0 32: 32, 0, 2405, 470, 94881, 0, 0 64: 64, 0, 12480, 8042, 1365658, 0, 0 128: 128, 0, 12886, 26019, 211536, 0, 0 256: 256, 0, 5352, 2223, 463546, 0, 0 512: 512, 0, 2797, 7819, 46986, 0, 0 1024: 1024, 0, 70, 126, 89345, 0, 0 2048: 2048, 0, 2037, 1353, 168857, 0, 0 4096: 4096, 0, 289, 17, 108610, 0, 0 8192: 8192, 0, 26, 1, 323, 0, 0 12228: 12228, 0, 9, 0, 159, 0, 0 16384: 16384, 0, 4, 2, 97, 0, 0 24576: 24576, 0, 7, 2, 55, 0, 0 32768: 32768, 0, 1, 1, 34, 0, 0 49152: 49152, 0, 6, 1, 56, 0, 0 65536: 65536, 0, 8, 2, 784, 0, 0 after a few minutes of activity. > I think eventually the right way to go is to adopt more sophisticated > allocation strategy like the one used in jemalloc(3) and this > changeset is more-or-less temporary for now: I committed it mainly > because it eliminated a large portion of unwanted TLB shootdowns I > have observed with very reasonable overhead (a few megabytes of RAM). > > > geli can do allocations >128k, which could be broken into two > > parts, one in the <8k sized range and the other in 128k... > > Yes, this is another issue that I'd like to solve. > >