From owner-freebsd-stable@FreeBSD.ORG Thu Jun 4 15:05:39 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4231B1065674 for ; Thu, 4 Jun 2009 15:05:39 +0000 (UTC) (envelope-from tim@chase2k.com) Received: from smtp.onlight.com (mx1.onlight.com [96.61.34.6]) by mx1.freebsd.org (Postfix) with ESMTP id F3F4D8FC18 for ; Thu, 4 Jun 2009 15:05:38 +0000 (UTC) (envelope-from tim@chase2k.com) Received: from localhost (livingston [127.0.0.1]) by clean.onlight.com (Postfix) with ESMTP id E670F82012AB for ; Thu, 4 Jun 2009 09:47:35 -0500 (CDT) X-Virus-Scanned: amavisd-new at onlight.com Received: from smtp.onlight.com ([127.0.0.1]) by localhost (filter.onlight.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 9Rv+VqVEugu7 for ; Thu, 4 Jun 2009 09:47:34 -0500 (CDT) Received: from oink.onlight.com (oink.onlight.com [10.10.1.29]) by smtp.onlight.com (Postfix) with ESMTPA id ACA6082012A7 for ; Thu, 4 Jun 2009 09:47:34 -0500 (CDT) Date: Thu, 4 Jun 2009 09:48:15 -0500 (CDT) From: Tim Chase Sender: tim@localhost.localdomain To: freebsd-stable@freebsd.org Message-ID: User-Agent: Alpine 2.00 (LFD 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Subject: kmem map too small panic after updating to STABLE-7 r192996 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jun 2009 15:05:39 -0000 Hello, I decided to give the new zfs code a try and upgraded my stable-7 system and discovered a panic that I can reproduce at will. This is an i386 system with 1.5GB of RAM and a single zfs pool: appbuild# zpool status pool: backup state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: none requested config: NAME STATE READ WRITE CKSUM backup ONLINE 0 0 0 mirror ONLINE 0 0 0 ad4 ONLINE 0 0 0 ad6 ONLINE 0 0 0 errors: No known data errors I had previously used the following zfs tuning: vm.kmem_size="512M" vm.kmem_size_max="512M" vfs.zfs.arc_max="100M" After getting this panic, I removed these tunables. The panic happens in either case. Reproducing the panic is easy: I keep a copy of the FreeBSD svn tree in a compressed zfs file system (compression ratio runs around 1.35x). An "svn update" (or the requisite "svn cleanup" following a system crash) in my checkout area will _always_ result in a "kmem map too small" panic. Here's a stack trace from my most recent panic: (kgdb) bt #0 doadump () at pcpu.h:196 #1 0xc052c963 in boot (howto=260) at /root/stable-7/sys/kern/kern_shutdown.c:418 #2 0xc052cb9d in panic (fmt=Variable "fmt" is not available. ) at /root/stable-7/sys/kern/kern_shutdown.c:574 #3 0xc069fa2a in kmem_malloc (map=0xc105408c, size=131072, flags=2) at /root/stable-7/sys/vm/vm_kern.c:305 #4 0xc0696587 in page_alloc (zone=0x0, bytes=131072, pflag=0xe6dbcb47 "\002", wait=2) at /root/stable-7/sys/vm/uma_core.c:952 #5 0xc0699000 in uma_large_malloc (size=131072, wait=2) at /root/stable-7/sys/vm/uma_core.c:2706 #6 0xc051af38 in malloc (size=131072, mtp=0xc094f060, flags=2) at /root/stable-7/sys/kern/kern_malloc.c:393 #7 0xc085db00 in zfs_kmem_alloc (size=131072, kmflags=2) at /root/stable-7/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_kmem.c:74 #8 0xc08d1c79 in zio_buf_alloc (size=131072) at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:207 #9 0xc08bcc40 in vdev_queue_io_to_issue (vq=0xc492b334, pending_limit=Variable "pending_limit" is not available. ) at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:227 #10 0xc08bce42 in vdev_queue_io_done (zio=0xd420a000) at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:313 #11 0xc08d4162 in zio_vdev_io_done (zio=0xd420a000) at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1847 #12 0xc08d2532 in zio_execute (zio=0xd420a000) at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998 #13 0xc0862aa0 in taskq_thread (arg=0xc44a10cc) at /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/taskq.c:854 #14 0xc0506816 in fork_exit (callout=0xc0862960 , arg=0xc44a10cc, frame=0xe6dbcd38) at /root/stable-7/sys/kern/kern_fork.c:811 #15 0xc06d4670 in fork_trampoline () at /root/stable-7/sys/i386/i386/exception.s:264 (kgdb) print panicstr $1 = 0xc0792320 "kmem_malloc(131072): kmem_map too small: 325656576 total allocated" The arc size is now auto-tuned as: kstat.zfs.misc.arcstats.c_min: 26214400 kstat.zfs.misc.arcstats.c_max: 209715200 and while monitoring kstat.zfs.misc.arcstats.size I noticed it fluctuated between around 140 and 150 million. Interestingly enough, immediately before the last panic, It (arcstats.size) had just been reduced from 150038268 to 128790020. I'm going to continue to poke around in my core dumps and see if I can figure out what's going on. Any hints as to what to look for or monitor while the system is running would be appreciated. Thanks, Tim