Date: Wed, 22 Sep 2010 10:25:23 +0300 From: Andriy Gapon <avg@freebsd.org> To: alc@freebsd.org, Jeff Roberson <jroberson@jroberson.net> Cc: Robert Watson <rwatson@freebsd.org>, Jeff Roberson <jeff@freebsd.org>, Alan Cox <alan.l.cox@gmail.com>, Andre Oppermann <andre@freebsd.org>, freebsd-hackers@freebsd.org Subject: Re: zfs + uma Message-ID: <4C99AF63.3000900@freebsd.org> In-Reply-To: <AANLkTimy=2WUcH59R5spajrKkUYQnii9SD1ZDdMymNC%2B@mail.gmail.com> References: <4C93236B.4050906@freebsd.org> <4C935F56.4030903@freebsd.org> <alpine.BSF.2.00.1009181221560.86826@fledge.watson.org> <alpine.BSF.2.00.1009181135430.23448@desktop> <4C95C804.1010701@freebsd.org> <alpine.BSF.2.00.1009182225050.23448@desktop> <4C95CCDA.7010007@freebsd.org> <4C984E90.60507@freebsd.org> <alpine.BSF.2.00.1009202037260.23448@desktop> <AANLkTimy=2WUcH59R5spajrKkUYQnii9SD1ZDdMymNC%2B@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
on 21/09/2010 19:16 Alan Cox said the following: > Actually, I think that there is a middle ground between "per-cpu caches" and > "directly from the VM" that we are missing. When I've looked at the default > configuration of ZFS (without the extra UMA zones enabled), there is an > incredible amount of churn on the kmem map caused by the implementation of > uma_large_malloc() and uma_large_free() going directly to the kmem map. Not > only are the obvious things happening, like allocating and freeing kernel > virtual addresses and underlying physical pages on every call, but also > system-wide TLB shootdowns and sometimes superpage demotions are occurring. > > I have some trouble believing that the large allocations being performed by ZFS > really need per-CPU caching, but I can certainly believe that they could benefit > from not going directly to the kmem map on every uma_large_malloc() and > uma_large_free(). In other words, I think it would make a lot of sense to have > a thin layer between UMA and the kmem map that caches allocated but unused > ranges of pages. Alan, thank you very much for the testing and analysis. These are very good points. So, for the reference, here are two patches that I came up with: 1. original patch that attempts to implement Solaris-like behavior but doesn't go all the way to disabling per-CPU caches: http://people.freebsd.org/~avg/uma-1.diff 2. patch that attempts to implement Jeff's three suggestions; I've tested per-CPU cache size adaptive behavior, works well, but haven't tested per-CPU cache draining yet: http://people.freebsd.org/~avg/uma-2.diff -- Andriy Gapon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C99AF63.3000900>