From owner-freebsd-fs@FreeBSD.ORG Tue Mar 8 00:28:23 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3CC51065672 for ; Tue, 8 Mar 2011 00:28:23 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta10.emeryville.ca.mail.comcast.net (qmta10.emeryville.ca.mail.comcast.net [76.96.30.17]) by mx1.freebsd.org (Postfix) with ESMTP id 885368FC15 for ; Tue, 8 Mar 2011 00:28:23 +0000 (UTC) Received: from omta04.emeryville.ca.mail.comcast.net ([76.96.30.35]) by qmta10.emeryville.ca.mail.comcast.net with comcast id GQTN1g00A0lTkoCAAQUN9c; Tue, 08 Mar 2011 00:28:23 +0000 Received: from koitsu.dyndns.org ([98.248.33.18]) by omta04.emeryville.ca.mail.comcast.net with comcast id GQUM1g0020PUQVN8QQUMty; Tue, 08 Mar 2011 00:28:22 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 92F089B422; Mon, 7 Mar 2011 16:28:20 -0800 (PST) Date: Mon, 7 Mar 2011 16:28:20 -0800 From: Jeremy Chadwick To: Martin Matuska Message-ID: <20110308002820.GA26744@icarus.home.lan> References: <1299232133.18671.3.camel@pc286.embl.fr> <20110304100517.GA23249@icarus.home.lan> <4D74AA26.10606@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D74AA26.10606@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org, Micka??l Can??vet Subject: Re: kmem_map too small with ZFS and 8.2-RELEASE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Mar 2011 00:28:23 -0000 On Mon, Mar 07, 2011 at 10:49:26AM +0100, Martin Matuska wrote: > In 8-STABLE (amd64), starting with SVN revision 214620 > vm.kmem_size_scale defaults to 1. > > This means in 8.2-RELEASE, vm.kmem_size is automatically set to the > amout of your system RAM, > so vm.kmem_size="16G" is automatically set on a system with 16GB RAM. > > D??a 04.03.2011 11:05, Jeremy Chadwick wrote / nap??sal(a): > > On Fri, Mar 04, 2011 at 10:48:53AM +0100, Micka??l Can??vet wrote: > >>> I'd use vm.kmem_size="32G" (i.e. twice your RAM) and that's it. > >> Should I also increase vfs.zfs.arc_max ? > > You should adjust vm.kmem_size, but not vm.kmem_size_max. > > > > You can adjust vfs.zfs.arc_max to basically ensure system stability. > > This thread is acting as evidence that there are probably edge cases > > where the kmem too small panic can still happen despite the limited ARC > > maximum defaults. > > > > For a 16GB system, I'd probably use these settings: > > > > vm.kmem_size="16384M" > > vfs.zfs.arc_max="13312M" > > > > I would also use these two settings: > > > > # Disable ZFS prefetching > > # http://southbrain.com/south/2008/04/the-nightmare-comes-slowly-zfs.html > > # Increases overall speed of ZFS, but when disk flushing/writes occur, > > # system is less responsive (due to extreme disk I/O). > > # NOTE: Systems with 8GB of RAM or more have prefetch enabled by > > # default. > > vfs.zfs.prefetch_disable="1" > > > > # Decrease ZFS txg timeout value from 30 (default) to 5 seconds. This > > # should increase throughput and decrease the "bursty" stalls that > > # happen during immense I/O with ZFS. > > # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html > > # http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html > > vfs.zfs.txg.timeout="5" > > > > The advice in the Wiki is outdated, especially for 8.2-RELEASE. Best > > not to follow it as of this writing. > > > >> Do you have any idea why the kernel panicked at only 8GB allocated ? > > I do not. A kernel developer will have to comment on that. > > > > Please attempt to reproduce the problem. If you can reproduce it > > reliably, this will greatly help kernel developers tracking down the > > source of the problem. Thanks -- you're absolutely right. I had forgotten all about vm.kmem_size_scale. :-) So yeah, with 8.2-RELEASE onward (rather than get into individual SVN revisions I'm using 8.2-RELEASE as "a point in time"), vm.kmem_size will default to the amount of usable memory (usually slightly less than hw.physmem). Validation: $ sysctl hw.realmem hw.physmem hw.usermem hw.realmem: 9395240960 hw.physmem: 8579981312 hw.usermem: 1086521344 $ sysctl vm.kmem_size vm.kmem_size: 8303894528 As such, one only needs to tune vfs.zfs.arc_max if desired. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |