From owner-freebsd-hackers@FreeBSD.ORG Wed Sep 22 07:25:28 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DFF43106566C; Wed, 22 Sep 2010 07:25:28 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 5B8868FC1E; Wed, 22 Sep 2010 07:25:26 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA24079; Wed, 22 Sep 2010 10:25:24 +0300 (EEST) (envelope-from avg@freebsd.org) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1OyJhk-0001A1-Fu; Wed, 22 Sep 2010 10:25:24 +0300 Message-ID: <4C99AF63.3000900@freebsd.org> Date: Wed, 22 Sep 2010 10:25:23 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.9) Gecko/20100918 Lightning/1.0b2 Thunderbird/3.1.4 MIME-Version: 1.0 To: alc@freebsd.org, Jeff Roberson References: <4C93236B.4050906@freebsd.org> <4C935F56.4030903@freebsd.org> <4C95C804.1010701@freebsd.org> <4C95CCDA.7010007@freebsd.org> <4C984E90.60507@freebsd.org> In-Reply-To: X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Robert Watson , Jeff Roberson , Alan Cox , Andre Oppermann , freebsd-hackers@freebsd.org Subject: Re: zfs + uma X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Sep 2010 07:25:29 -0000 on 21/09/2010 19:16 Alan Cox said the following: > Actually, I think that there is a middle ground between "per-cpu caches" and > "directly from the VM" that we are missing. When I've looked at the default > configuration of ZFS (without the extra UMA zones enabled), there is an > incredible amount of churn on the kmem map caused by the implementation of > uma_large_malloc() and uma_large_free() going directly to the kmem map. Not > only are the obvious things happening, like allocating and freeing kernel > virtual addresses and underlying physical pages on every call, but also > system-wide TLB shootdowns and sometimes superpage demotions are occurring. > > I have some trouble believing that the large allocations being performed by ZFS > really need per-CPU caching, but I can certainly believe that they could benefit > from not going directly to the kmem map on every uma_large_malloc() and > uma_large_free(). In other words, I think it would make a lot of sense to have > a thin layer between UMA and the kmem map that caches allocated but unused > ranges of pages. Alan, thank you very much for the testing and analysis. These are very good points. So, for the reference, here are two patches that I came up with: 1. original patch that attempts to implement Solaris-like behavior but doesn't go all the way to disabling per-CPU caches: http://people.freebsd.org/~avg/uma-1.diff 2. patch that attempts to implement Jeff's three suggestions; I've tested per-CPU cache size adaptive behavior, works well, but haven't tested per-CPU cache draining yet: http://people.freebsd.org/~avg/uma-2.diff -- Andriy Gapon