From owner-freebsd-hackers@FreeBSD.ORG  Sat Sep 18 11:23:48 2010
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id EAC9C1065697;
	Sat, 18 Sep 2010 11:23:48 +0000 (UTC)
	(envelope-from rwatson@FreeBSD.org)
Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42])
	by mx1.freebsd.org (Postfix) with ESMTP id C44A18FC1F;
	Sat, 18 Sep 2010 11:23:48 +0000 (UTC)
Received: from fledge.watson.org (fledge.watson.org [65.122.17.41])
	by cyrus.watson.org (Postfix) with ESMTPS id 75C8546B2D;
	Sat, 18 Sep 2010 07:23:48 -0400 (EDT)
Date: Sat, 18 Sep 2010 12:23:48 +0100 (BST)
From: Robert Watson <rwatson@FreeBSD.org>
X-X-Sender: robert@fledge.watson.org
To: Andre Oppermann <andre@freebsd.org>
In-Reply-To: <4C935F56.4030903@freebsd.org>
Message-ID: <alpine.BSF.2.00.1009181221560.86826@fledge.watson.org>
References: <4C93236B.4050906@freebsd.org> <4C935F56.4030903@freebsd.org>
User-Agent: Alpine 2.00 (BSF 1167 2008-08-23)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: freebsd-hackers@freebsd.org, Jeff Roberson <jeff@freebsd.org>,
	Andriy Gapon <avg@freebsd.org>
Subject: Re: zfs + uma
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
	<freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>, 
	<mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
	<mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 18 Sep 2010 11:23:49 -0000


On Fri, 17 Sep 2010, Andre Oppermann wrote:

>> Although keeping free items around improves performance, it does consume 
>> memory too.  And the fact that that memory is not freed on lowmem condition 
>> makes the situation worse.
>
> Interesting.  We may run into related issues with excessive mbuf (cluster) 
> caching in the per-cpu buckets as well.
>
> Having a general solutions for that is appreciated.  Maybe the size of the 
> free per-cpu buckets should be specified when setting up the UMA zone.  Of 
> certain frequently re-used elements we may want to cache more, other less.

I've been keeping a vague eye out for this over the last few years, and 
haven't spotted many problems in production machines I've inspected.  You can 
use the umastat tool in the tools tree to look at the distribution of memory 
over buckets (etc) in UMA manually.  It would be nice if it had some automated 
statistics on fragmentation however.  Short-lived fragmentation is likely, and 
isn't an issue, so what you want is a tool that monitors over time and reports 
on longer-lived fragmentation.

The main fragmentation issue we've had in the past has been due to 
mbuf+cluster caching, which prevented mbufs from being freed usefully in some 
cases.  Jeff's ongoing work on variable-sized mbufs would entirely eliminate 
that problem...

Robert