From owner-freebsd-fs@FreeBSD.ORG Fri Jan 22 04:28:47 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2A4081065692 for ; Fri, 22 Jan 2010 04:28:47 +0000 (UTC) (envelope-from djp@polands.org) Received: from hrndva-omtalb.mail.rr.com (hrndva-omtalb.mail.rr.com [71.74.56.122]) by mx1.freebsd.org (Postfix) with ESMTP id D54D38FC16 for ; Fri, 22 Jan 2010 04:28:46 +0000 (UTC) X-Authority-Analysis: v=1.0 c=1 a=GSN_Y9T6cv4A:10 a=OA2lqS22AAAA:8 a=VDDltm6BRT3w7GqL0QUA:9 a=rtW85IiFBK7f9twzNHwA:7 a=x49OsMu7hDFgt7HixzlrjwBuJWQA:4 a=ZZAfTtC2Ym4A:10 a=azDIrACOat2JRc_j:21 a=TT84m8gFpZP1GIKG:21 X-Cloudmark-Score: 0 X-Originating-IP: 75.87.219.217 Received: from [75.87.219.217] ([75.87.219.217:52983] helo=haran.polands.org) by hrndva-oedge03.mail.rr.com (envelope-from ) (ecelerity 2.2.2.39 r()) with ESMTP id EA/9C-05903-D79295B4; Fri, 22 Jan 2010 04:28:45 +0000 Received: from moab.polands.org (moab.polands.org [172.16.1.8]) by haran.polands.org (8.14.3/8.14.3) with ESMTP id o0M4Si9U028661; Thu, 21 Jan 2010 22:28:44 -0600 (CST) (envelope-from djp@polands.org) Received: from moab.polands.org (localhost [127.0.0.1]) by moab.polands.org (8.14.3/8.14.3) with ESMTP id o0M4SiH8008964; Thu, 21 Jan 2010 22:28:44 -0600 (CST) (envelope-from djp@moab.polands.org) Received: (from djp@localhost) by moab.polands.org (8.14.3/8.14.3/Submit) id o0M4ShMH008963; Thu, 21 Jan 2010 22:28:43 -0600 (CST) (envelope-from djp) Date: Thu, 21 Jan 2010 22:28:43 -0600 From: Doug Poland To: Adam McDougall Message-ID: <20100122042843.GA8858@polands.org> References: <4B58976E.1020402@polands.org> <4B58A069.8000802@egr.msu.edu> <4B58BD2D.30803@rcn.com> <4B58D4D3.80009@egr.msu.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4B58D4D3.80009@egr.msu.edu> User-Agent: Mutt/1.5.20 (2009-06-14) Cc: freebsd-fs@freebsd.org Subject: Re: Repeatable ZFS "kmem map too small" panic on 8.0-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2010 04:28:47 -0000 On Thu, Jan 21, 2010 at 05:27:31PM -0500, Adam McDougall wrote: > On 01/21/10 17:21, Artem Belevich wrote: > >On Thu, Jan 21, 2010 at 12:46 PM, Gary Corcoran wrote: > >>Adam McDougall wrote: > >>> > >>>Put this in /boot/loader.conf: > >>>vm.kmem_size="20G" > >>> > >>>It is intentionally higher than your amount of ram. > >> > >>Would you mind explaining... > >>1) why this fixes the kmem_map too small problem ? > > > >Because it explicitly makes kmem_map larger. > > > >>2) why it should be larger than the amount of RAM, and by how much ? > > > >ZFS needs access to a lot of memory for ARC and it allocates/frees > >memory fairly randomly. That raises two issues. > > > >First issue is that kernel is by default fairly conservative about > >its memory needs. vm.kmem_size which limits address space for > >in-kernel memory allocations is by default set to a fairly low value > >which works reasonably well in most of the cases. However, for ZFS it > >needs to be bumped up allow large amounts of memory to be allocated > >by ZFS. > > > >Second problem is memory fragmentation. If you set vm.kmem_size == > >physical memory size, over time you may end up with a situation when > >there is plenty of physical memory available, but there is no single > >contiguous block of address space to map that memory into. FreeBSD > >allocator is pretty good about avoiding fragmentation but you still > >do need more address space than the amount of memory that could > >potentially be allocated. I'd say that vm.kmem_size should be few > >multiples of amounts of memory that you're planning to allocate. > > > >Just my $0.02 > > > > Exactly what I would have said, thanks :) I'd imagine the kmem_size > could be much larger still, closer to kmem_size_max, but I just picked > 20G as a default for my servers that have 8G or less and I haven't > seen an out of kmem panic for as long as I could raise kmem_size > sufficiently high (a change was made around 6 months ago). kmem_size > doesn't seem to "grow" (much?) towards kmem_size_max, it is what it > is, and you need to make sure it is big enough for your needs. I have > systems with just one gig and they run fine. > Interesting discussion :) I added vm.kmem_size="20G" to /boot/loader.conf per your instructions. This time, it didn't panic at the same point in the test, however, it appears the filesystem is "hanging". On the fdisk test, I hit T and get: cmd: fsdisk 37066 [zio->io_cv)] 245.62r 0.12u 25.10s 0 My various metrics are still running, but anything that needs the filesystem appears "stuck". The memory usage of the item "solaris" (vmstat -m | grep solaris) spiked at 3334781952 (3180.30 MiB). # zpool iostat 2, a T shows: load: 0.00 cmd: zpool 934 [tx->tx_quiesce_done_cv)] 2052.45r 0.06 u 0.39s 0% 0k # vmstat -v | grep solaris to disk every second and it's hung at: load: 0.00 cmd: sh 38551 [zfs] 909.85r 0.00u 0.00s 0% 16k Any suggestions! -- Regards, Doug