From owner-freebsd-fs@FreeBSD.ORG Tue Sep 28 12:04:05 2010 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 124591065673 for ; Tue, 28 Sep 2010 12:04:05 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta01.westchester.pa.mail.comcast.net (qmta01.westchester.pa.mail.comcast.net [76.96.62.16]) by mx1.freebsd.org (Postfix) with ESMTP id B4A3C8FC0C for ; Tue, 28 Sep 2010 12:04:04 +0000 (UTC) Received: from omta10.westchester.pa.mail.comcast.net ([76.96.62.28]) by qmta01.westchester.pa.mail.comcast.net with comcast id CAax1f0060cZkys51BqqWX; Tue, 28 Sep 2010 11:50:50 +0000 Received: from koitsu.dyndns.org ([98.248.41.155]) by omta10.westchester.pa.mail.comcast.net with comcast id CBqo1f00P3LrwQ23WBqp1Z; Tue, 28 Sep 2010 11:50:50 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 5DA489B418; Tue, 28 Sep 2010 04:50:47 -0700 (PDT) Date: Tue, 28 Sep 2010 04:50:47 -0700 From: Jeremy Chadwick To: Willem Jan Withagen Message-ID: <20100928115047.GA62142@icarus.home.lan> References: <4CA1D06C.9050305@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4CA1D06C.9050305@digiware.nl> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: stable@freebsd.org, fs@freebsd.org Subject: Re: Still getting kmem exhausted panic X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Sep 2010 12:04:05 -0000 On Tue, Sep 28, 2010 at 01:24:28PM +0200, Willem Jan Withagen wrote: > This is with stable as of yesterday,but with an un-tunned ZFS box I > was still able to generate a kmem exhausted panic. > Hard panic, just 3 lines. > > The box contains 12Gb memory, runs on a 6 core (with HT) xeon. > 6* 2T WD black caviar in raidz2 with 2*512Mb mirrored log. > > The box died while rsyncing 5.8T from its partnering system. > (that was the only activity on the box) It would help if you could provide output from the following commands (even after the box has rebooted): $ sysctl -a | egrep ^vm.kmem $ sysctl -a | egrep ^vfs.zfs.arc $ sysctl kstat.zfs.misc.arcstats > So the obvious would to conclude that auto-tuning voor ZFS on > 8.1-Stable is not yet quite there. > > So I guess that we still need tuning advice even for 8.1. > And thus prevent a hard panic. Andriy Gapon provides this general recommendation: http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/059114.html The advice I've given for RELENG_8 (as of the time of this writing), 8.1-STABLE, and 8.1-RELEASE, is that for amd64 you'll need to tune: vm.kmem_size vfs.zfs.arc_max An example machine: amd64, with 4GB physical RAM installed (3916MB available for use (verified via dmesg)) uses values: vm.kmem_size="4096M" vfs.zfs.arc_max="3584M" Another example machine: amd64, with 8GB physical RAM installed (7875MB available for use) uses values: vm.kmem_size="8192M" vfs.zfs.arc_max="6144M" I believe the trick -- Andriy, please correct me if I'm wrong -- is the tuning of vfs.zfs.arc_max, which is now a hard limit rather than a "high watermark". However, I believe there have been occasional reports of exhaustion panics despite both of these being set[1]. Those reports are being investigated on an individual basis. I set some other ZFS-related parameters as well (disabling prefetch, adjusting txg.timeout, etc.), but those shouldn't be necessary to gain stability at this point in time. I can't provide tuning advice for i386. [1]: http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/059109.html -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |