From owner-freebsd-current@FreeBSD.ORG Fri Jun 5 01:53:26 2009 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A7168106566C; Fri, 5 Jun 2009 01:53:26 +0000 (UTC) (envelope-from amdmi3@amdmi3.ru) Received: from smtp.timeweb.ru (smtp.timeweb.ru [217.170.79.85]) by mx1.freebsd.org (Postfix) with ESMTP id 487988FC14; Fri, 5 Jun 2009 01:53:25 +0000 (UTC) (envelope-from amdmi3@amdmi3.ru) Received: from [213.148.20.85] (helo=hive.panopticon) by smtp.timeweb.ru with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.69) (envelope-from ) id 1MCOcW-0002BB-V8; Fri, 05 Jun 2009 05:53:25 +0400 Received: from hades.panopticon (hades.panopticon [192.168.0.32]) by hive.panopticon (Postfix) with ESMTP id A4D97B860; Fri, 5 Jun 2009 05:53:16 +0400 (MSD) Received: by hades.panopticon (Postfix, from userid 1000) id D5E81108839; Fri, 5 Jun 2009 05:53:17 +0400 (MSD) Date: Fri, 5 Jun 2009 05:53:17 +0400 From: Dmitry Marakasov To: freebsd-current@FreeBSD.org Message-ID: <20090605015317.GA23952@hades.panopticon> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline User-Agent: Mutt/1.5.19 (2009-01-05) Cc: freebsd-fs@FreeBSD.org Subject: [ZFS] still kmem_too_small on recent current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jun 2009 01:53:27 -0000 Hi! Just got a kmem_too_small panic on yesterday's current after writing 30GB disk image via NFS. This is amd64 system, with 8GB mem and those tunables: vm.kmem_size_max="2G" vm.kmem_size="2G" vfs.zfs.arc_max="1G" ZFS is 6x1TB raidz2. I have a coredump of it, so just ask if you need additional details. --- #0 doadump () at pcpu.h:223 #1 0xffffffff8054bec6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:420 #2 0xffffffff8054c356 in panic (fmt=Variable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:576 #3 0xffffffff806ed86e in kmem_malloc (map=0xffffff00010000e0, size=131072, flags=2) at /usr/src/sys/vm/vm_kern.c:304 #4 0xffffffff806e5e75 in uma_large_malloc (size=131072, wait=2) at /usr/src/sys/vm/uma_core.c:3001 #5 0xffffffff8053b271 in malloc (size=131072, mtp=0xffffffff80e1a1e0, flags=2) at /usr/src/sys/kern/kern_malloc.c:391 #6 0xffffffff80d90ca6 in vdev_queue_io_to_issue (vq=0xffffff00065bf420, pending_limit=Variable "pending_limit" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:227 #7 0xffffffff80d90e1c in vdev_queue_io_done (zio=0xffffff018e1995a0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:313 #8 0xffffffff80da2370 in zio_vdev_io_done (zio=0xffffff013bc26870) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1845 #9 0xffffffff80da0a10 in zio_execute (zio=0xffffff013bc26870) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:996 #10 0xffffffff80d49f1c in taskq_thread (arg=Variable "arg" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/taskq.c:854 #11 0xffffffff80525fad in fork_exit (callout=0xffffffff80d49d48 , arg=0xffffff000188fd80, frame=0xffffff80c516cc90) at /usr/src/sys/kern/kern_fork.c:829 #12 0xffffffff8076d56e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:552 --- I've monitored kstat.zfs.misc.arcstats.size, top and vmstat during it, and here are the observations: - normal copying process uses ~1GB arc (sysctl) and ~1GB wired (top), which I assume includes arc. - actual disk writes are not continuous, ZFS writes data in >512MB packs. - usually those writes are pretty uniform (a write in ~10 seconds, while thoughput is 50MB/s, array can handle 400MB/s writes), but sometimes the write is either larger, or the disks can't keep up, or both - those moments are visible as much higher CPU load for 3-5 seconds, wired kicks up to 1700 mb and more, and arc decreases to ~800MB. I assume the latter is recently committed VM backpressure, and the former means panic if it rolls over kmem_max. Seems so, as I've just got another panic. Just curious, what uses that much memory in those moments? Here are last contents of ssh consoles: --- kstat.zfs.misc.arcstats.size: 804244704 kstat.zfs.misc.arcstats.size: 804244704 kstat.zfs.misc.arcstats.size: 804244704 --- procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad8 ad10 in sy cs us sy id 0 0 0 1017M 5348M 308 0 0 0 253 0 0 0 12 411 655 0 0 100 0 0 0 1017M 5348M 241 0 0 0 257 0 0 0 18 381 748 0 0 100 0 0 0 1017M 5348M 308 0 0 0 253 0 0 0 16 411 645 0 0 100 --- last pid: 7620; load averages: 1.91, 1.80, 1.42 up 0+00:48:56 05:44:11 83 processes: 1 running, 82 sleeping CPU: 0.2% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.8% idle Mem: 149M Active, 60M Inact, 2317M Wired, 1292K Cache, 821M Buf, 5347M Free Swap: 10G Total, 10G Free --- Will give it another go with 512MB arc and 4GB kmem. -- Dmitry Marakasov . 55B5 0596 FF1E 8D84 5F56 9510 D35A 80DD F9D2 F77D amdmi3@amdmi3.ru ..: jabber: amdmi3@jabber.ru http://www.amdmi3.ru