From owner-freebsd-fs@FreeBSD.ORG Wed Aug 11 06:17:13 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC4C41065689 for ; Wed, 11 Aug 2010 06:17:13 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta10.emeryville.ca.mail.comcast.net (qmta10.emeryville.ca.mail.comcast.net [76.96.30.17]) by mx1.freebsd.org (Postfix) with ESMTP id 97C5E8FC15 for ; Wed, 11 Aug 2010 06:17:13 +0000 (UTC) Received: from omta11.emeryville.ca.mail.comcast.net ([76.96.30.36]) by qmta10.emeryville.ca.mail.comcast.net with comcast id suDY1e0030mlR8UAAuHDYi; Wed, 11 Aug 2010 06:17:13 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta11.emeryville.ca.mail.comcast.net with comcast id suHC1e0013LrwQ28XuHCUY; Wed, 11 Aug 2010 06:17:12 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 1B9D59B42E; Tue, 10 Aug 2010 23:17:12 -0700 (PDT) Date: Tue, 10 Aug 2010 23:17:12 -0700 From: Jeremy Chadwick To: freebsd-fs@freebsd.org Message-ID: <20100811061712.GA60244@icarus.home.lan> References: <20100810214418.GA28288@tolstoy.tols.org> <4C61F9CB.1010506@ukr.net> <20100811060910.GA34883@tolstoy.tols.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20100811060910.GA34883@tolstoy.tols.org> User-Agent: Mutt/1.5.20 (2009-06-14) Subject: Re: zfs arc - just take it all and be good to me X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Aug 2010 06:17:13 -0000 On Wed, Aug 11, 2010 at 06:09:10AM +0000, Marco van Tol wrote: > On Wed, Aug 11, 2010 at 07:43:24AM +0200, Matthias Gamsjager wrote: > > > Please, insert into /boot/loader.conf only this options: > > > vm.kmem_size="999M" > > > vm.kmem_size_max="999M" > > > vfs.zfs.arc_max="160M" > > > vfs.zfs.prefetch_disable=1 > > > > > > And after reboot, please run dd... > > > > vfs.zfs.arc_max="160M" should be enough to limit the size of the arc. > > But thats the point. I don't want to limit its size, I want it to take > every available piece of memory it can find, while allowing applications > to temporarely use large slabs of that same memory during their runtime. As I described in my previous post (not sure if you've seen it yet), ZFS on FreeBSD / the VM, at this moment in time, doesn't support this ability. Solaris offers this. For the time being, you're going to have to adjust /boot/loader.conf as described below. That's the bottom line. > > and I'm not sure if that ( vm.kmem_size="999M" > > ,vm.kmem_size_max="999M") is such a good idea. > > Me neither. I'm pretty sure I will hit the "kmem map too small" > scenario real quick. What is it you are testing with those settings > Vladislav? You need to adjust/increase vm.kmem_size to ensure you don't experience the "kmem map too small" condition, and make sure that vfs.zfs.arc_max is set to something **lower** than that amount. E.g.: vm.kmem_size="2048M" vfs.zfs.arc_max="1536M" Folks need to remember that vm.kmem_size affects more than just ZFS ARC. It's best to give yourself breathing room. I've been harping about this problem for way too long, and complaining in equal amount that something official needs to be written/documented about tuning. The FreeBSD Wiki for ZFS tuning isn't sufficient as we all know, and as I've stated before I'm willing to write something up but I need open communication with folks familiar with both ZFS and the VM to help in explaining the interaction on a technical level. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |