From owner-freebsd-stable@FreeBSD.ORG Wed Dec 15 23:12:54 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8BEF61065679 for ; Wed, 15 Dec 2010 23:12:54 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta02.westchester.pa.mail.comcast.net (qmta02.westchester.pa.mail.comcast.net [76.96.62.24]) by mx1.freebsd.org (Postfix) with ESMTP id 35C958FC1A for ; Wed, 15 Dec 2010 23:12:53 +0000 (UTC) Received: from omta16.westchester.pa.mail.comcast.net ([76.96.62.88]) by qmta02.westchester.pa.mail.comcast.net with comcast id jQEp1f0021uE5Es52bCuxx; Wed, 15 Dec 2010 23:12:54 +0000 Received: from koitsu.dyndns.org ([98.248.34.134]) by omta16.westchester.pa.mail.comcast.net with comcast id jbCs1f00J2tehsa3cbCtmQ; Wed, 15 Dec 2010 23:12:54 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 1DF749B422; Wed, 15 Dec 2010 15:12:51 -0800 (PST) Date: Wed, 15 Dec 2010 15:12:51 -0800 From: Jeremy Chadwick To: Jeremie Le Hen Message-ID: <20101215231251.GA2443@icarus.home.lan> References: <20101215082837.GA8734@felucia.tataz.chchile.org> <4D08FAD9.4010300@freebsd.org> <20101215225540.GD8734@felucia.tataz.chchile.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20101215225540.GD8734@felucia.tataz.chchile.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-stable@freebsd.org, Andriy Gapon Subject: Re: Panic in ZFS layer on 8.1-STABLE X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Dec 2010 23:12:54 -0000 On Wed, Dec 15, 2010 at 11:55:40PM +0100, Jeremie Le Hen wrote: > On Wed, Dec 15, 2010 at 07:28:57PM +0200, Andriy Gapon wrote: > > on 15/12/2010 10:28 Jeremie Le Hen said the following: > > > Hi, > > > > > > [ Please Cc: me when replying, as I'm not subscribed to -stable@. ] > > > > > > My filer at home runs FreeBSD. A single data RAID-1 zpool with 10~15 > > > datasets, two of them using compression. Over the night, I got the > > > following panic: > > > > Thanks for the stack trace! > > But where is the promised panic message? :) > > Eheh, sorry for this :-). > > > I suspect that you ran out of kernel address space. > > You'd probably have to tune your system and/or add more memory. > > Please research this topic via mailing lists archives. > > You guessed right! > > panic: kmem_malloc(114688): kmem_map too small: 775847936 total allocated Adjust the following two tunables (loader.conf, not sysctl) in combination: vm.kmem_size vfs.zfs.arc_max I strongly recommend setting vfs.zfs.arc_max to about half of your system memory in your case (so, vfs.zfs.arc_max="512M"). For example, on a 4GB system, I use: vm.kmem_size="4096M" vfs.zfs.arc_max="3072M" Make sure vm.kmem_size is larger than arc_max. Other people will advocate increasing your physical RAM, which I agree with, but the problem isn't technically the amount of RAM you have, it's that (paraphrasing to keep it simple) the kernel ran out of usable memory to use, even though the permissible size (vm.kmem_size_max -- note the difference in the tunable) is much larger. So, what I'm saying: you can use the above two tunables I mentioned to effectively ensure you don't encounter this problem. If after tuning you DO encounter this *exact same problem* (kmem_map too small), please report back, as there may be another bug with ARC limiting that needs to be looked at. (Every situation is unique/different) -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |