From owner-freebsd-fs@FreeBSD.ORG Fri Mar 4 14:38:13 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 104501065692 for ; Fri, 4 Mar 2011 14:38:13 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta10.emeryville.ca.mail.comcast.net (qmta10.emeryville.ca.mail.comcast.net [76.96.30.17]) by mx1.freebsd.org (Postfix) with ESMTP id EB8688FC15 for ; Fri, 4 Mar 2011 14:38:12 +0000 (UTC) Received: from omta21.emeryville.ca.mail.comcast.net ([76.96.30.88]) by qmta10.emeryville.ca.mail.comcast.net with comcast id F2K11g00A1u4NiLAA2eCMP; Fri, 04 Mar 2011 14:38:12 +0000 Received: from koitsu.dyndns.org ([98.248.33.18]) by omta21.emeryville.ca.mail.comcast.net with comcast id F2e21g00G0PUQVN8h2e7M0; Fri, 04 Mar 2011 14:38:09 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 21FE49B422; Fri, 4 Mar 2011 06:38:01 -0800 (PST) Date: Fri, 4 Mar 2011 06:38:01 -0800 From: Jeremy Chadwick To: Adam McDougall Message-ID: <20110304143801.GA1734@icarus.home.lan> References: <1299232133.18671.3.camel@pc286.embl.fr> <4D70E7D1.5030205@egr.msu.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <4D70E7D1.5030205@egr.msu.edu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: kmem_map too small with ZFS and 8.2-RELEASE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Mar 2011 14:38:13 -0000 On Fri, Mar 04, 2011 at 08:23:29AM -0500, Adam McDougall wrote: > On 03/04/11 04:48, Mickaël Canévet wrote: > >>I'd use vm.kmem_size="32G" (i.e. twice your RAM) and that's it. > > > >Should I also increase vfs.zfs.arc_max ? > > > >Do you have any idea why the kernel panicked at only 8GB allocated ? > > > >Thank you > > I believe ARC allocations in kmem can become fragmented, so when it > is searching for a place to store a new contiguous segment of > memory, the remaining fragmented free spaces may all be too small. > I also set > vm.kmem_size to about twice the amount of ram to help it avoid this > issue. I suspect if kmem is badly fragmented then performance of > ZFS can downgrade, so that is another reason to keep kmem bigger. My findings on 8.2-RELEASE indicate that doing this results in very unexpected behaviour regarding the ARC maximum. As such, I cannot recommend this model. For example, on an amd64 system with 8GB physical RAM and these two settings in /boot/loader.conf: vm.kmem_size="8192M" vfs.zfs.arc_max="6144M" kstat.zfs.misc.arcstats.size tops out at around 6240986896, with Wired in top(1) showing ~6.3GB. This is expected behaviour and fits (I think) what people expect. However, on the exact same system with these two settings: vm.kmem_size="16384M" vfs.zfs.arc_max="6144M" The above ARC numbers are exactly *half* that amount. This is easily reproducible. Can someone 1) justify the "2x the amount of RAM for vm.kmem_size" setting, and 2) explain in detail the above behaviour? -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |