From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 5 06:53:30 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87669106566B; Thu, 5 Jun 2008 06:53:30 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 7361A8FC13; Thu, 5 Jun 2008 06:53:30 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 4C5641CC05B; Wed, 4 Jun 2008 23:53:30 -0700 (PDT) Date: Wed, 4 Jun 2008 23:53:30 -0700 From: Jeremy Chadwick To: Pawel Jakub Dawidek Message-ID: <20080605065330.GA62591@eos.sc1.parodius.com> References: <6a7033710805302252v43a7b240x66ca3f5e3dd5fda4@mail.gmail.com> <20080603135308.GC3434@garage.freebsd.pl> <6a7033710806032317g4dbe8845h26a1196016b9c440@mail.gmail.com> <86zlq140x0.fsf@ds4.des.no> <6a7033710806041053g4a5c2fdftd7202b708bff363c@mail.gmail.com> <20080605062728.GA4278@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080605062728.GA4278@garage.freebsd.pl> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: Dag-Erling Sm??rgrav , Tz-Huan Huang , freebsd-hackers@freebsd.org Subject: Re: Is there any way to increase the KVM? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Jun 2008 06:53:30 -0000 On Thu, Jun 05, 2008 at 08:27:28AM +0200, Pawel Jakub Dawidek wrote: > On Thu, Jun 05, 2008 at 01:53:37AM +0800, Tz-Huan Huang wrote: > > On Thu, Jun 5, 2008 at 12:31 AM, Dag-Erling Sm??rgrav wrote: > > > "Tz-Huan Huang" writes: > > >> The vfs.zfs.arc_max was set to 512M originally, the machine survived for > > >> 4 days and panicked this morning. Now the vfs.zfs.arc_max is set to 64M > > >> by Oliver's suggestion, let's see how long it will survive. :-) > > > > > > des@ds4 ~% uname -a > > > FreeBSD ds4.des.no 8.0-CURRENT FreeBSD 8.0-CURRENT #27: Sat Feb 23 01:24:32 CET 2008 des@ds4.des.no:/usr/obj/usr/src/sys/ds4 amd64 > > > des@ds4 ~% sysctl -h vm.kmem_size_min vm.kmem_size_max vm.kmem_size vfs.zfs.arc_min vfs.zfs.arc_max > > > vm.kmem_size_min: 1,073,741,824 > > > vm.kmem_size_max: 1,073,741,824 > > > vm.kmem_size: 1,073,741,824 > > > vfs.zfs.arc_min: 67,108,864 > > > vfs.zfs.arc_max: 536,870,912 > > > des@ds4 ~% zpool list > > > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > > > raid 1.45T 435G 1.03T 29% ONLINE - > > > des@ds4 ~% zfs list | wc -l > > > 210 > > > > > > Haven't had a single panic in over six months. > > > > Thanks for your information, the major difference is that we > > runs on 7-stable and the size of our zfs pool is much bigger. > > I'm don't think the panics are related to pool size. More to the load > and characteristics of your workload. Not to add superfluous comments, but I agree. It has little to do with actual pool size, but more with I/O activity. However, multiple zpools will very likely result in the panic happening sooner, just based on the "nature of the beast" -- more zpools likely means more "overall" I/O because there's more things people will be utilising via ZFS, thus you'll exhaust kmem quicker. > beast:root:~# zpool list > NAME SIZE USED AVAIL CAP HEALTH ALTROOT > tank 732G 604G 128G 82% ONLINE - > > but: > > beast:root:~# zfs list | wc -l > 1932 > > No panics. > > PS. I'm quite sure the ZFS version I've in perforce will fix most if not > all 'kmem_map too small' panics. It's not yet committed, but I do want > to MFC it into RELENG_7. That's great to hear, but the point I've made regarding kmem_size not being able to extend past 2GB (on i386 and amd64) still stands. I've looked at the code myself, in attempt to figure out where the actual limitation is, and the code is beyond my understanding. (It's somewhat abstracted, but only to those who are completely unfamiliar to the VM piece -- like me :-) ). -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |