From owner-freebsd-fs@FreeBSD.ORG Tue Jun 1 19:23:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 20A28106567B for ; Tue, 1 Jun 2010 19:23:47 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta14.westchester.pa.mail.comcast.net (qmta14.westchester.pa.mail.comcast.net [76.96.59.212]) by mx1.freebsd.org (Postfix) with ESMTP id C08C68FC15 for ; Tue, 1 Jun 2010 19:23:45 +0000 (UTC) Received: from omta03.westchester.pa.mail.comcast.net ([76.96.62.27]) by qmta14.westchester.pa.mail.comcast.net with comcast id QafB1e0040bG4ec5EjPlJs; Tue, 01 Jun 2010 19:23:46 +0000 Received: from koitsu.dyndns.org ([98.248.46.159]) by omta03.westchester.pa.mail.comcast.net with comcast id QjPk1e00U3S48mS3PjPlqc; Tue, 01 Jun 2010 19:23:45 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 461E69B418; Tue, 1 Jun 2010 12:23:43 -0700 (PDT) Date: Tue, 1 Jun 2010 12:23:43 -0700 From: Jeremy Chadwick To: Darren Pilgrim Message-ID: <20100601192343.GA44238@icarus.home.lan> References: <060401cafe37$a411b240$ec3516c0$@net> <4BFF894F.4010008@icyb.net.ua> <20100528134549.GA75411@icarus.home.lan> <4C05577F.7040001@bitfreak.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C05577F.7040001@bitfreak.org> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: amd64@freebsd.org, freebsd-fs@freebsd.org Subject: Re: FreeBSD 8.1-Prerelease Panic amd64 w/ZFS.. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jun 2010 19:23:47 -0000 On Tue, Jun 01, 2010 at 11:54:55AM -0700, Darren Pilgrim wrote: > Jeremy Chadwick wrote: > >To the OP: you will need to increase vm.kmem_size in /boot/loader.conf > >and reboot the system. "What value do I pick?" With 4GB, I would > >recommend you use these two variables: > > > >vm.kmem_size="2048M" > >vfs.zfs.arc_max="1536M" > > > >This will increase the available kmem, and also limit the ARC size > >explicitly to nothing larger than 1.5GB. This should stabilise your > >system. > > What are the general guidelines for sizing these given the amount of > physical RAM? For example, I have a machine with 6 GB, so I have: > > vm.kmem_size="3072M" > vfs.zfs.arc_max="2048M" There really aren't any, as far as I know, since it depends on each system, it's role, and what architecture type (i386 vs. amd64). I'm focusing solely on amd64. Method I tend to follow: vm.kmem_size = 1/2 of available RAM (think physical/installed) vfs.zfs.arc_max = vm.kmem_size minus 512MB I believe there were some reported cases of panics on systems with vm.kmem_size == vfs.zfs.arc_max, but I could be remembering incorrectly. There's too many posts/people to remember who said what and when. I think there's a kstat.zfs.misc.arcstats sysctl counter which tracks the maximum ARC ever used, but could be mistaken here too. One of our 8GB systems that uses ZFS also runs MySQL, which requires me to use these values in addition to the above. Note that, AFAIK, the below values *do not* affect ZFS directly; I'm pointing them out as necessities, else a "heavy" mysqld crashes and complains about there not being enough memory (directly correlating to maxdsiz). kern.maxdsiz="1536M" kern.dfldsiz="1536M" kern.maxssiz="256M" Rather than try to pick values that "all add up to 8GB" (don't do this), I leave a lot of extra space for other things (kernel bits, network buffers, blah blah) that I don't want to think about. I try to avoid being OCD with tuning unless I have reason to be. All I really care about is providing enough memory for userland daemons and ZFS to run reliably/not panic the box. I'd welcome the community and developers to chime in here, especially folks familiar with the VM and how all of these tunables interact with one another. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |