From owner-freebsd-fs@FreeBSD.ORG Thu Jan 21 22:41:05 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4ECC51065670 for ; Thu, 21 Jan 2010 22:41:05 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mx.egr.msu.edu (surfnturf.egr.msu.edu [35.9.37.164]) by mx1.freebsd.org (Postfix) with ESMTP id 1F6D48FC15 for ; Thu, 21 Jan 2010 22:41:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mx.egr.msu.edu (Postfix) with ESMTP id EF6B3EE879; Thu, 21 Jan 2010 17:27:31 -0500 (EST) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mx.egr.msu.edu ([127.0.0.1]) by localhost (surfnturf.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cAaGVpmm-sbq; Thu, 21 Jan 2010 17:27:31 -0500 (EST) Received: from [35.9.44.65] (daemon.egr.msu.edu [35.9.44.65]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: mcdouga9) by mx.egr.msu.edu (Postfix) with ESMTPSA id ACB18EE873; Thu, 21 Jan 2010 17:27:31 -0500 (EST) Message-ID: <4B58D4D3.80009@egr.msu.edu> Date: Thu, 21 Jan 2010 17:27:31 -0500 From: Adam McDougall User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.5) Gecko/20100103 Thunderbird/3.0 MIME-Version: 1.0 To: Artem Belevich References: <4B58976E.1020402@polands.org> <4B58A069.8000802@egr.msu.edu> <4B58BD2D.30803@rcn.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Repeatable ZFS "kmem map too small" panic on 8.0-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2010 22:41:05 -0000 On 01/21/10 17:21, Artem Belevich wrote: > On Thu, Jan 21, 2010 at 12:46 PM, Gary Corcoran wrote: >> Adam McDougall wrote: >>> >>> Put this in /boot/loader.conf: >>> vm.kmem_size="20G" >>> >>> It is intentionally higher than your amount of ram. >> >> Would you mind explaining... >> 1) why this fixes the kmem_map too small problem ? > > Because it explicitly makes kmem_map larger. > >> 2) why it should be larger than the amount of RAM, and by how much ? > > ZFS needs access to a lot of memory for ARC and it allocates/frees > memory fairly randomly. That raises two issues. > > First issue is that kernel is by default fairly conservative about its > memory needs. vm.kmem_size which limits address space for in-kernel > memory allocations is by default set to a fairly low value which works > reasonably well in most of the cases. However, for ZFS it needs to be > bumped up allow large amounts of memory to be allocated by ZFS. > > Second problem is memory fragmentation. If you set vm.kmem_size == > physical memory size, over time you may end up with a situation when > there is plenty of physical memory available, but there is no single > contiguous block of address space to map that memory into. FreeBSD > allocator is pretty good about avoiding fragmentation but you still do > need more address space than the amount of memory that could > potentially be allocated. I'd say that vm.kmem_size should be few > multiples of amounts of memory that you're planning to allocate. > > Just my $0.02 > > --Artem > Exactly what I would have said, thanks :) I'd imagine the kmem_size could be much larger still, closer to kmem_size_max, but I just picked 20G as a default for my servers that have 8G or less and I haven't seen an out of kmem panic for as long as I could raise kmem_size sufficiently high (a change was made around 6 months ago). kmem_size doesn't seem to "grow" (much?) towards kmem_size_max, it is what it is, and you need to make sure it is big enough for your needs. I have systems with just one gig and they run fine.