Date: Thu, 4 Jun 2009 14:13:44 -0700 From: Kip Macy <kmacy@freebsd.org> To: Tim Chase <tim@chase2k.com> Cc: freebsd-stable@freebsd.org Subject: Re: kmem map too small panic after updating to STABLE-7 r192996 Message-ID: <3c1674c90906041413n15e2ce88q9f279818641515e7@mail.gmail.com> In-Reply-To: <alpine.LFD.2.00.0906040932280.27799@localhost.localdomain> References: <alpine.LFD.2.00.0906040932280.27799@localhost.localdomain>
next in thread | previous in thread | raw e-mail | index | archive | help
As I mentioned in the initial e-mail, auto-tuning is only safe to rely on on amd64. -Kip On Thu, Jun 4, 2009 at 7:48 AM, Tim Chase <tim@chase2k.com> wrote: > Hello, > > I decided to give the new zfs code a try and upgraded my stable-7 system > and discovered a panic that I can reproduce at will. > > This is an i386 system with 1.5GB of RAM and a single zfs pool: > > appbuild# zpool status > pool: backup > state: ONLINE > status: The pool is formatted using an older on-disk format. The > pool can > still be used, but some features are unavailable. > action: Upgrade the pool using 'zpool upgrade'. Once this is done, > the > pool will no longer be accessible on older software versions. > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > backup ONLINE 0 0 0 > mirror ONLINE 0 0 0 > ad4 ONLINE 0 0 0 > ad6 ONLINE 0 0 0 > > errors: No known data errors > > I had previously used the following zfs tuning: > > vm.kmem_size="512M" > vm.kmem_size_max="512M" > vfs.zfs.arc_max="100M" > > After getting this panic, I removed these tunables. The panic happens > in either case. > > Reproducing the panic is easy: I keep a copy of the FreeBSD svn tree > in a compressed zfs file system (compression ratio runs around 1.35x). > An "svn update" (or the requisite "svn cleanup" following a system crash) > in my checkout area will _always_ result in a "kmem map too small" panic. > Here's a stack trace from my most recent panic: > > (kgdb) bt > #0 doadump () at pcpu.h:196 > #1 0xc052c963 in boot (howto=260) at > /root/stable-7/sys/kern/kern_shutdown.c:418 > #2 0xc052cb9d in panic (fmt=Variable "fmt" is not available. > ) at /root/stable-7/sys/kern/kern_shutdown.c:574 > #3 0xc069fa2a in kmem_malloc (map=0xc105408c, size=131072, flags=2) at > /root/stable-7/sys/vm/vm_kern.c:305 > #4 0xc0696587 in page_alloc (zone=0x0, bytes=131072, pflag=0xe6dbcb47 > "\002", wait=2) > at /root/stable-7/sys/vm/uma_core.c:952 > #5 0xc0699000 in uma_large_malloc (size=131072, wait=2) at > /root/stable-7/sys/vm/uma_core.c:2706 > #6 0xc051af38 in malloc (size=131072, mtp=0xc094f060, flags=2) at > /root/stable-7/sys/kern/kern_malloc.c:393 > #7 0xc085db00 in zfs_kmem_alloc (size=131072, kmflags=2) > at > /root/stable-7/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_kmem.c:74 > #8 0xc08d1c79 in zio_buf_alloc (size=131072) > at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:207 > #9 0xc08bcc40 in vdev_queue_io_to_issue (vq=0xc492b334, > pending_limit=Variable "pending_limit" is not available. > ) > at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:227 > #10 0xc08bce42 in vdev_queue_io_done (zio=0xd420a000) > at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:313 > #11 0xc08d4162 in zio_vdev_io_done (zio=0xd420a000) > at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1847 > #12 0xc08d2532 in zio_execute (zio=0xd420a000) > at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998 > #13 0xc0862aa0 in taskq_thread (arg=0xc44a10cc) > at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/taskq.c:854 > #14 0xc0506816 in fork_exit (callout=0xc0862960 <taskq_thread>, > arg=0xc44a10cc, frame=0xe6dbcd38) > at /root/stable-7/sys/kern/kern_fork.c:811 > #15 0xc06d4670 in fork_trampoline () at > /root/stable-7/sys/i386/i386/exception.s:264 > (kgdb) print panicstr > $1 = 0xc0792320 "kmem_malloc(131072): kmem_map too small: 325656576 total > allocated" > > > The arc size is now auto-tuned as: > > kstat.zfs.misc.arcstats.c_min: 26214400 > kstat.zfs.misc.arcstats.c_max: 209715200 > > and while monitoring kstat.zfs.misc.arcstats.size I noticed it fluctuated > between around 140 and 150 million. Interestingly enough, immediately > before the last panic, It (arcstats.size) had just been reduced from > 150038268 to 128790020. > > I'm going to continue to poke around in my core dumps and see if I can > figure out what's going on. Any hints as to what to look for or monitor > while the system is running would be appreciated. > > Thanks, > Tim > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > -- When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle. Edmund Burke
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3c1674c90906041413n15e2ce88q9f279818641515e7>