From owner-freebsd-stable@FreeBSD.ORG Fri May 30 07:05:34 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B75361065679 for ; Fri, 30 May 2008 07:05:34 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 9BEF88FC19 for ; Fri, 30 May 2008 07:05:34 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 5E0801CC031; Fri, 30 May 2008 00:05:34 -0700 (PDT) Date: Fri, 30 May 2008 00:05:34 -0700 From: Jeremy Chadwick To: freebsd-stable@freebsd.org Message-ID: <20080530070534.GA25479@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.17 (2007-11-01) Subject: RELENG_7 amd64; memory and vm.kmem_size X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 May 2008 07:05:34 -0000 As has been pointed out on wikis, mailing lists, and even on IRC, ZFS requires a bit of tuning -- specifically in regards to vm.kmem_size and vm.kmem_size_max. The opinion is: ZFS is memory-hungry. On my home RELENG_7 amd64 box (2GB RAM), I could panic the system with heavy I/O due to kmem_size being too small, until I used the following values: vm.kmem_size="1536M" vm.kmem_size_max="1536M" I decided to upgrade the box to 4GB of RAM, since I was worried about memory exhaustion under even higher loads (during heavy I/O with ZFS, I'd often see the "Wired" value in top reach 1.3-1.4GB). I received the RAM today, installed it, works fine. I then chose to adjust the vm.kmem_size and kmem_size_max settings to something larger, which seemed like the logical choice. I went with: vm.kmem_size="3584M" vm.kmem_size_max="3584M" Upon reboot, the kernel immediately panic'd with the following message: kmem_suballoc(): bad status return of 3. I then chose smaller values (going with 2048M); same panic. Can someone shed some light on this? I'm guessing it's intentional; from what I've found online, it seems to indicate that when the kmem_size value is set too large, there isn't enough memory available for allocation in other pieces of the kernel, hence the panic. I'm worried that there's a limit of some sort being hit, and that inadvertantly systems with lots of ZFS usage (multiple zpools comes to mind), one will not be able to increase kmem_size past ~1.5GB, despite how much memory is physically installed. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |