From owner-freebsd-hackers@FreeBSD.ORG Fri Sep 17 12:56:53 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4224E106566C for ; Fri, 17 Sep 2010 12:56:53 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id A757D8FC1D for ; Fri, 17 Sep 2010 12:56:52 +0000 (UTC) Received: (qmail 14323 invoked from network); 17 Sep 2010 12:24:36 -0000 Received: from unknown (HELO [62.48.0.92]) ([62.48.0.92]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 17 Sep 2010 12:24:36 -0000 Message-ID: <4C935F56.4030903@freebsd.org> Date: Fri, 17 Sep 2010 14:30:14 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.9) Gecko/20100825 Thunderbird/3.1.3 MIME-Version: 1.0 To: Andriy Gapon References: <4C93236B.4050906@freebsd.org> In-Reply-To: <4C93236B.4050906@freebsd.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, Jeff Roberson Subject: Re: zfs + uma X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Sep 2010 12:56:53 -0000 On 17.09.2010 10:14, Andriy Gapon wrote: > > I've been investigating interaction between zfs and uma for a while. > You might remember that there is a noticeable fragmentation in zfs uma zones > when uma use is not enabled for actual data/metadata buffers. > > I also noticed that when uma use is enabled for data/metadata buffers > (zio.use_uma=1) amount of memory reserved in free items of zfs uma zones becomes > really huge. And this is despite the fact that the vast majority of the > data/metadata zone have items with sizes that are multiples of page size. > This couldn't really be because of fragmentation. > > Further checks show that the free items are accumulated in per-cpu cache > buckets. uz_count for those buckets starts with 1, but over time, during bursts > of activity, it grows up to maximum of 128. > Problem with those buckets is that they are not drained on low memory conditions > and uz_count never goes down. > > So, after a while, I observe about 300 free items (on a mere two core system) > cached in 4 per-cpu buckets for a single zone with 128KB item size. > That's 30MB right there. > For all data and metadata zones the number goes as high as 500MB on my machine > with 4GB physical RAM. > This seems like a bit too much to me. > > Although keeping free items around improves performance, it does consume memory > too. And the fact that that memory is not freed on lowmem condition makes the > situation worse. Interesting. We may run into related issues with excessive mbuf (cluster) caching in the per-cpu buckets as well. Having a general solutions for that is appreciated. Maybe the size of the free per-cpu buckets should be specified when setting up the UMA zone. Of certain frequently re-used elements we may want to cache more, other less. -- Andre