From owner-freebsd-stable@FreeBSD.ORG Fri Dec 4 19:56:08 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C16FA106566B for ; Fri, 4 Dec 2009 19:56:08 +0000 (UTC) (envelope-from stb@lassitu.de) Received: from koef.zs64.net (koef.zs64.net [212.12.50.230]) by mx1.freebsd.org (Postfix) with ESMTP id 310C58FC08 for ; Fri, 4 Dec 2009 19:56:07 +0000 (UTC) Received: from localhost by koef.zs64.net (8.14.3/8.14.3) with ESMTP id nB4Ju64q055358 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO) for ; Fri, 4 Dec 2009 20:56:06 +0100 (CET) (envelope-from stb@lassitu.de) (authenticated as stb) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1077) From: Stefan Bethke In-Reply-To: <831421F9-6344-4E68-BD64-9C013EB86523@lassitu.de> Date: Fri, 4 Dec 2009 20:56:05 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <06D8F596-649B-4478-8A2F-F9EA133B8DDC@lassitu.de> References: <831421F9-6344-4E68-BD64-9C013EB86523@lassitu.de> To: FreeBSD Stable X-Mailer: Apple Mail (2.1077) Subject: Re: Fatal trap 9 triggered by zfs? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Dec 2009 19:56:08 -0000 Am 04.12.2009 um 17:52 schrieb Stefan Bethke: > I'm getting panics like this every so often (couple weeks, sometimes = just a few days.) A second machine that has identical hardware and is = running the same source has no such problems. >=20 > FreeBSD XXX.hanse.de 8.0-STABLE FreeBSD 8.0-STABLE #16: Tue Dec 1 = 14:30:54 UTC 2009 root@XXX.hanse.de:/usr/obj/usr/src/sys/EISENBOOT = amd64 >=20 > # zpool status > pool: tank > state: ONLINE > scrub: none requested > config: >=20 > NAME STATE READ WRITE CKSUM > tank ONLINE 0 0 0 > ad4s1d ONLINE 0 0 0 > # cat /boot/loader.conf > vfs.zfs.arc_max=3D"512M" > vfs.zfs.prefetch_disable=3D"1" > vfs.zfs.zil_disable=3D"1" Got another, different one. Any tuning suggestions or similar? #0 doadump () at pcpu.h:223 223 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:223 #1 0xffffffff80337bd9 in boot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0xffffffff8033802c in panic (fmt=3DVariable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:579 #3 0xffffffff805cc2ad in trap_fatal (frame=3D0x9, eva=3DVariable "eva" = is not available. ) at /usr/src/sys/amd64/amd64/trap.c:857 #4 0xffffffff805cce12 in trap (frame=3D0xffffff80625db030) at /usr/src/sys/amd64/amd64/trap.c:644 #5 0xffffffff805b2943 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #6 0xffffffff80586c7a in vm_map_entry_splay (addr=3DVariable "addr" is = not available. ) at /usr/src/sys/vm/vm_map.c:771 #7 0xffffffff80587f37 in vm_map_lookup_entry (map=3D0xffffff00010000e8,=20= address=3D18446743523979624448, entry=3D0xffffff80625db170) at /usr/src/sys/vm/vm_map.c:1021 #8 0xffffffff80588aa3 in vm_map_delete (map=3D0xffffff00010000e8,=20 start=3D18446743523979624448, end=3D18446743523979689984) at /usr/src/sys/vm/vm_map.c:2685 #9 0xffffffff80588e61 in vm_map_remove (map=3D0xffffff00010000e8,=20 start=3D18446743523979624448, end=3D18446743523979689984) at /usr/src/sys/vm/vm_map.c:2774 #10 0xffffffff8057db85 in uma_large_free (slab=3D0xffffff005fcc7000) at /usr/src/sys/vm/uma_core.c:3021 #11 0xffffffff80325987 in free (addr=3D0xffffff80018b0000,=20 mtp=3D0xffffffff80ac61e0) at /usr/src/sys/kern/kern_malloc.c:471 #12 0xffffffff80a36d03 in vdev_cache_evict (vc=3D0xffffff0001723ce0,=20 ve=3D0xffffff003dd52200) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= vdev_cache.c:151 #13 0xffffffff80a372ad in vdev_cache_read (zio=3D0xffffff005f5ca2d0) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= vdev_cache.c:182 #14 0xffffffff80a4a954 in zio_vdev_io_start (zio=3D0xffffff005f5ca2d0) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= zio.c:1814 #15 0xffffffff80a4ae87 in zio_execute (zio=3D0xffffff005f5ca2d0) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= zio.c:996 #16 0xffffffff80a3a080 in vdev_mirror_io_start (zio=3D0xffffff005f811b40) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= vdev_mirror.c:303 #17 0xffffffff80a4ae87 in zio_execute (zio=3D0xffffff005f811b40) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= zio.c:996 #18 0xffffffff809ff45a in arc_read_nolock (pio=3D0xffffff005f66d5a0,=20 spa=3D0xffffff000150a000, bp=3D0xffffff800a91c440,=20 done=3D0xffffffff80a02630 , private=3DVariable = "private" is not available. ) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= arc.c:2763 #19 0xffffffff809ff8ec in arc_read (pio=3D0xffffff005f66d5a0,=20 spa=3D0xffffff000150a000, bp=3D0xffffff800a91c440, = pbuf=3D0xffffff0042a3ca20,=20 done=3D0xffffffff80a02630 , = private=3D0xffffff005fbfc620,=20 priority=3D0, zio_flags=3D1, arc_flags=3D0xffffff80625db5ec,=20 zb=3D0xffffff80625db5c0) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= arc.c:2508 #20 0xffffffff80a02aba in dbuf_read (db=3D0xffffff005fbfc620,=20 zio=3D0xffffff005f66d5a0, flags=3D2) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= dbuf.c:521 #21 0xffffffff80a0602c in dmu_buf_hold (os=3DVariable "os" is not = available. ) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= dmu.c:106 #22 0xffffffff80a40db5 in zap_lockdir (os=3D0xffffff005f937610, = obj=3D247890,=20 tx=3D0x0, lti=3DRW_READER, fatreader=3D1, adding=3D0, = zapp=3D0xffffff80625db888) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= zap_micro.c:388 #23 0xffffffff80a41724 in zap_cursor_retrieve (zc=3D0xffffff80625db880,=20= za=3D0xffffff80625db8c0) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= zap_micro.c:1004 #24 0xffffffff80a61b66 in zfs_freebsd_readdir (ap=3DVariable "ap" is not = available. ) at = /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/= zfs_vnops.c:2157 #25 0xffffffff803cfde9 in kern_getdirentries (td=3D0xffffff0057bfe000, = fd=3DVariable "fd" is not available. ) at vnode_if.h:758 #26 0xffffffff803d0093 in getdirentries (td=3DVariable "td" is not = available. ) at /usr/src/sys/kern/vfs_syscalls.c:4051 #27 0xffffffff805cc906 in syscall (frame=3D0xffffff80625dbc80) at /usr/src/sys/amd64/amd64/trap.c:989 #28 0xffffffff805b2c21 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:373 #29 0x0000000800724cdc in ?? () Previous frame inner to this frame (corrupt stack?) --=20 Stefan Bethke Fon +49 151 14070811