From owner-freebsd-fs@FreeBSD.ORG Sat Oct 22 03:04:06 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 52F431065673 for ; Sat, 22 Oct 2011 03:04:06 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta08.westchester.pa.mail.comcast.net (qmta08.westchester.pa.mail.comcast.net [76.96.62.80]) by mx1.freebsd.org (Postfix) with ESMTP id 139068FC17 for ; Sat, 22 Oct 2011 03:04:05 +0000 (UTC) Received: from omta16.westchester.pa.mail.comcast.net ([76.96.62.88]) by qmta08.westchester.pa.mail.comcast.net with comcast id nepN1h0011uE5Es58f462m; Sat, 22 Oct 2011 03:04:06 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta16.westchester.pa.mail.comcast.net with comcast id nf451h00J1t3BNj3cf45rQ; Sat, 22 Oct 2011 03:04:06 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id BFFAA102C1C; Fri, 21 Oct 2011 20:04:03 -0700 (PDT) Date: Fri, 21 Oct 2011 20:04:03 -0700 From: Jeremy Chadwick To: Doug Barton Message-ID: <20111022030403.GA176@icarus.home.lan> References: <4E97FEDD.7060205@quip.cz> <4EA19203.5050503@quip.cz> <4EA2277B.5080306@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EA2277B.5080306@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, Ivan Voras Subject: Re: dirhash and dynamic memory allocation X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 22 Oct 2011 03:04:06 -0000 On Fri, Oct 21, 2011 at 07:16:27PM -0700, Doug Barton wrote: > On 10/21/2011 08:38, Miroslav Lachman wrote: > > Hi, I am back on this topic... > > > > Ivan Voras wrote: > >> On 14/10/2011 11:20, Miroslav Lachman wrote: > >>> Hi all, > >>> > >>> I tried some tuning of dirhash on our servers and after googlig a bit, I > >>> found an old GSoC project wiki page about Dynamic Memory Allocation for > >>> Dirhash: http://wiki.freebsd.org/DirhashDynamicMemory > >>> Is there any reason not to use it / not commit it to HEAD? > >> > >> AFAIK it's sort-of already present. In 8-stable and recent kernels you > >> can give huge amounts of memory to dirhash via vfs.ufs.dirhash_maxmem > >> (but except in really large edge cases I don't think you *need* more > >> than 32 MB), and the kernel will scale-down or free the memory if not > >> needed. > >> > >> In effect, vfs.ufs.dirhash_maxmem is the upper limit - the kernel will > >> use less and will free the allocated memory in low memory situations > >> (which I've tried and it works). > > > > So the current behavior is that on 7.3+ and 8.x we have smaller average > > dirhash buffer (by default) than it was initialy 10 years ago. Because > > it starts as 2MB fixed size and now we have 2MB max, which is lowered in > > low mem situations... and sometimes it is set to 0MB! > > > > I caught this 2 days ago: > > > > root@rip ~/# sysctl vfs.ufs > > vfs.ufs.dirhash_reclaimage: 5 > > vfs.ufs.dirhash_lowmemcount: 36953 > > vfs.ufs.dirhash_docheck: 0 > > vfs.ufs.dirhash_mem: 0 > > vfs.ufs.dirhash_maxmem: 8388608 > > vfs.ufs.dirhash_minsize: 2560 > > > > I set maxmem to 8MB in sysctl.conf to increase performance and > > dirhash_mem 0 is really bad surprise! > > > > I am worrying about bad performance in situation where dirhash is > > emptied in situations, where server is already running at maximum > > performance (there is some memory hungry process and system can start > > swapping to disk + dirhash is efectively disabled) > > > > I found a PR kern/145246 > > http://www.freebsd.org/cgi/query-pr.cgi?pr=145246 > > > > Is it possible to add some dirhash_minmem limit to not clear all the > > dirhash memory? > > So I can set dirhash_minmem=2MB dirhash_maxmem=16MB and then dirhash_mem > > will be allways between these two limits? > > Isn't that what vfs.ufs.dirhash_minsize is for? I think given that there > is a lot more memory in modern systems setting that higher by default is > probably a good idea. Or maybe I'm misunderstanding what that knob does? I believe the function of that sysctl is different. It's not the "minimum amount of dirhash memory to retain", it's: $ sysctl -d vfs.ufs.dirhash_minsize vfs.ufs.dirhash_minsize: minimum directory size in bytes for which to use hashed lookup The sysctl should really be named "dirhash_mindirsize". -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |