From owner-freebsd-fs@FreeBSD.ORG Fri Jan 22 18:09:45 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E839106566C for ; Fri, 22 Jan 2010 18:09:45 +0000 (UTC) (envelope-from doug@polands.org) Received: from hrndva-omtalb.mail.rr.com (hrndva-omtalb.mail.rr.com [71.74.56.125]) by mx1.freebsd.org (Postfix) with ESMTP id 096858FC0A for ; Fri, 22 Jan 2010 18:09:44 +0000 (UTC) X-Authority-Analysis: v=1.0 c=1 a=GSN_Y9T6cv4A:10 a=5oGaD+IacabtnYYqBVNCkQ==:17 a=bqq2Vc5EAAAA:8 a=flJLQrt4EQDhqcY5gHkA:9 a=aOOHSjQ4rLumRQA88O0A:7 a=ZuPtNL6XV-s6dh0S0N5OpwVYNu0A:4 a=5ERLOmoKdHQA:10 X-Cloudmark-Score: 0 X-Originating-IP: 75.87.219.217 Received: from [75.87.219.217] ([75.87.219.217:50571] helo=haran.polands.org) by hrndva-oedge02.mail.rr.com (envelope-from ) (ecelerity 2.2.2.39 r()) with ESMTP id 47/4F-11553-7E9E95B4; Fri, 22 Jan 2010 18:09:44 +0000 Received: from email.polands.org (ammon.polands.org [172.16.1.7]) by haran.polands.org (8.14.3/8.14.3) with ESMTP id o0MI9hEF031285; Fri, 22 Jan 2010 12:09:43 -0600 (CST) (envelope-from doug@polands.org) Received: from 209.103.214.34 (SquirrelMail authenticated user djp) by email.polands.org with HTTP; Fri, 22 Jan 2010 12:09:43 -0600 Message-ID: In-Reply-To: References: <4B58976E.1020402@polands.org> <4B58A069.8000802@egr.msu.edu> <4B58BD2D.30803@rcn.com> <4B58D4D3.80009@egr.msu.edu> <20100122042843.GA8858@polands.org> Date: Fri, 22 Jan 2010 12:09:43 -0600 From: "Doug Poland" To: "Artem Belevich" User-Agent: SquirrelMail/1.4.20-RC2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: freebsd-fs@freebsd.org Subject: Re: Repeatable ZFS "kmem map too small" panic on 8.0-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2010 18:09:45 -0000 On Fri, January 22, 2010 07:22, Doug Poland wrote: > On Fri, January 22, 2010 00:09, Artem Belevich wrote: >> On Thu, Jan 21, 2010 at 8:28 PM, Doug Poland >> wrote: >>> Interesting discussion :)  I added vm.kmem_size="20G" to >>> /boot/loader.conf per your instructions.  This time, it didn't >>> panic at the same point in the test, however, it appears the >>> filesystem is "hanging". >>> >>> On the fdisk test, I hit T and get: >>> cmd: fsdisk 37066 [zio->io_cv)] 245.62r 0.12u 25.10s 0 >>> >>> My various metrics are still running, but anything that needs the >>> filesystem appears "stuck". >>> >>> The memory usage of the item "solaris" (vmstat -m | grep solaris) >>> spiked at 3334781952 (3180.30 MiB). >>> >>> # zpool iostat 2, a T shows: >>> load: 0.00  cmd: zpool 934 [tx->tx_quiesce_done_cv)] 2052.45r 0.06 > u 0.39s 0% 0k >>> >>> # vmstat -v | grep solaris to disk every second and it's hung at: >>> load: 0.00  cmd: sh 38551 [zfs] 909.85r 0.00u 0.00s 0% 16k >>> >>> Any suggestions! >>> >> Next step would be to set vfs.zfs.arc_max to a value that's somewhat >> below your physical memory size. Let's say - 1G or so. On my box >> with >> 8GB of RAM I set these values: >> >> vfs.zfs.arc_max="6500M" >> vfs.zfs.arc_min="4G" >> > OK, I'll give that a shot... > panic: kmem_malloc(131072): kmem_map too small: 3593236480 total allocate d # cat /boot/loader.conf vfs.root.mountfrom="zfs:bethesda" vfs.zfs.arc_max="1G" vm.kmem_size="20G" zfs_load="YES" OK, what do we do next? :) -- Regards, Doug