Date: Thu, 21 Jan 2010 22:28:43 -0600 From: Doug Poland <doug@polands.org> To: Adam McDougall <mcdouga9@egr.msu.edu> Cc: freebsd-fs@freebsd.org Subject: Re: Repeatable ZFS "kmem map too small" panic on 8.0-STABLE Message-ID: <20100122042843.GA8858@polands.org> In-Reply-To: <4B58D4D3.80009@egr.msu.edu> References: <4B58976E.1020402@polands.org> <4B58A069.8000802@egr.msu.edu> <4B58BD2D.30803@rcn.com> <ed91d4a81001211421r4f7ba7a8n1c92bfc413e5feed@mail.gmail.com> <4B58D4D3.80009@egr.msu.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jan 21, 2010 at 05:27:31PM -0500, Adam McDougall wrote: > On 01/21/10 17:21, Artem Belevich wrote: > >On Thu, Jan 21, 2010 at 12:46 PM, Gary Corcoran<gcorcoran@rcn.com> wrote: > >>Adam McDougall wrote: > >>> > >>>Put this in /boot/loader.conf: > >>>vm.kmem_size="20G" > >>> > >>>It is intentionally higher than your amount of ram. > >> > >>Would you mind explaining... > >>1) why this fixes the kmem_map too small problem ? > > > >Because it explicitly makes kmem_map larger. > > > >>2) why it should be larger than the amount of RAM, and by how much ? > > > >ZFS needs access to a lot of memory for ARC and it allocates/frees > >memory fairly randomly. That raises two issues. > > > >First issue is that kernel is by default fairly conservative about > >its memory needs. vm.kmem_size which limits address space for > >in-kernel memory allocations is by default set to a fairly low value > >which works reasonably well in most of the cases. However, for ZFS it > >needs to be bumped up allow large amounts of memory to be allocated > >by ZFS. > > > >Second problem is memory fragmentation. If you set vm.kmem_size == > >physical memory size, over time you may end up with a situation when > >there is plenty of physical memory available, but there is no single > >contiguous block of address space to map that memory into. FreeBSD > >allocator is pretty good about avoiding fragmentation but you still > >do need more address space than the amount of memory that could > >potentially be allocated. I'd say that vm.kmem_size should be few > >multiples of amounts of memory that you're planning to allocate. > > > >Just my $0.02 > > > > Exactly what I would have said, thanks :) I'd imagine the kmem_size > could be much larger still, closer to kmem_size_max, but I just picked > 20G as a default for my servers that have 8G or less and I haven't > seen an out of kmem panic for as long as I could raise kmem_size > sufficiently high (a change was made around 6 months ago). kmem_size > doesn't seem to "grow" (much?) towards kmem_size_max, it is what it > is, and you need to make sure it is big enough for your needs. I have > systems with just one gig and they run fine. > Interesting discussion :) I added vm.kmem_size="20G" to /boot/loader.conf per your instructions. This time, it didn't panic at the same point in the test, however, it appears the filesystem is "hanging". On the fdisk test, I hit <CTRL> T and get: cmd: fsdisk 37066 [zio->io_cv)] 245.62r 0.12u 25.10s 0 My various metrics are still running, but anything that needs the filesystem appears "stuck". The memory usage of the item "solaris" (vmstat -m | grep solaris) spiked at 3334781952 (3180.30 MiB). # zpool iostat 2, a <CTRL> T shows: load: 0.00 cmd: zpool 934 [tx->tx_quiesce_done_cv)] 2052.45r 0.06 u 0.39s 0% 0k # vmstat -v | grep solaris to disk every second and it's hung at: load: 0.00 cmd: sh 38551 [zfs] 909.85r 0.00u 0.00s 0% 16k Any suggestions! -- Regards, Doug
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100122042843.GA8858>