From owner-freebsd-fs@FreeBSD.ORG Fri Jan 22 20:11:48 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C57A3106566B for ; Fri, 22 Jan 2010 20:11:48 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-iw0-f198.google.com (mail-iw0-f198.google.com [209.85.223.198]) by mx1.freebsd.org (Postfix) with ESMTP id 8548D8FC14 for ; Fri, 22 Jan 2010 20:11:48 +0000 (UTC) Received: by iwn36 with SMTP id 36so1306785iwn.3 for ; Fri, 22 Jan 2010 12:11:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:date:x-google-sender-auth:message-id:subject:from:to:cc :content-type:content-transfer-encoding; bh=j3BI5kzaWarBNgg3UJNNDlyot/unjm7SYz2wMaVAmdI=; b=ltqwNZat3WPc2SwdutzReHgXqAAIaKkXZl2wBypvkYyAU+Wxee9/Fo2EgWYt4vMs3D sUGHjH8+Uenh+kq2b3fgb5OWrt2Y31yE/rKLwLyqiwya2h4rRsa+8jwBNbSjwBRCr20X 5VtI1J+QlXgR8fdcYNXPDJPKiIKn3XfdyXQss= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=fV9VO38wNofaoYJCcM2DgiF+t3cwK6TJX6DWQdstIIGeiyxuN2mdyYb2s2uu3E6ZZ/ wy2adiLPkH6ketM74eNPrdYjgZ3qR+QL/4P9KHhD89uWF2alS6sc/418qPSu/FfJlcKE M7kGwIdLduCg5WK/9Jt9J7+aaytBJnuLDc31w= MIME-Version: 1.0 Sender: artemb@gmail.com Received: by 10.231.59.7 with SMTP id j7mr1677464ibh.12.1264191107683; Fri, 22 Jan 2010 12:11:47 -0800 (PST) In-Reply-To: References: <4B58976E.1020402@polands.org> <4B58A069.8000802@egr.msu.edu> <4B58BD2D.30803@rcn.com> <4B58D4D3.80009@egr.msu.edu> <20100122042843.GA8858@polands.org> Date: Fri, 22 Jan 2010 12:11:47 -0800 X-Google-Sender-Auth: ca6a295921f642fc Message-ID: From: Artem Belevich To: Doug Poland Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: Repeatable ZFS "kmem map too small" panic on 8.0-STABLE X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2010 20:11:48 -0000 What do following sysctls show on your box? hw.physmem: vm.kmem_size vm.kmem_size_max vfs.zfs.arc_max --Artem On Fri, Jan 22, 2010 at 10:09 AM, Doug Poland wrote: > > On Fri, January 22, 2010 07:22, Doug Poland wrote: >> On Fri, January 22, 2010 00:09, Artem Belevich wrote: >>> On Thu, Jan 21, 2010 at 8:28 PM, Doug Poland >>> wrote: >>>> Interesting discussion :) =A0I added vm.kmem_size=3D"20G" to >>>> /boot/loader.conf per your instructions. =A0This time, it didn't >>>> panic at the same point in the test, however, it appears the >>>> filesystem is "hanging". >>>> >>>> On the fdisk test, I hit T and get: >>>> cmd: fsdisk 37066 [zio->io_cv)] 245.62r 0.12u 25.10s 0 >>>> >>>> My various metrics are still running, but anything that needs the >>>> filesystem appears "stuck". >>>> >>>> The memory usage of the item "solaris" (vmstat -m | grep solaris) >>>> spiked at 3334781952 (3180.30 MiB). >>>> >>>> # zpool iostat 2, a T shows: >>>> load: 0.00 =A0cmd: zpool 934 [tx->tx_quiesce_done_cv)] 2052.45r 0.06 >> u 0.39s 0% 0k >>>> >>>> # vmstat -v | grep solaris to disk every second and it's hung at: >>>> load: 0.00 =A0cmd: sh 38551 [zfs] 909.85r 0.00u 0.00s 0% 16k >>>> >>>> Any suggestions! >>>> >>> Next step would be to set vfs.zfs.arc_max to a value that's somewhat >>> below your physical memory size. Let's say - 1G or so. On my box >>> with >>> 8GB of RAM I set these values: >>> >>> vfs.zfs.arc_max=3D"6500M" >>> vfs.zfs.arc_min=3D"4G" >>> >> OK, I'll give that a shot... >> > > panic: kmem_malloc(131072): kmem_map too small: 3593236480 total > allocate d > > # cat /boot/loader.conf > vfs.root.mountfrom=3D"zfs:bethesda" > vfs.zfs.arc_max=3D"1G" > vm.kmem_size=3D"20G" > zfs_load=3D"YES" > > OK, what do we do next? :) > > > -- > Regards, > Doug > >