From owner-freebsd-fs@FreeBSD.ORG Tue Jun 10 17:31:19 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C327989A for ; Tue, 10 Jun 2014 17:31:19 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7BE9F2E25 for ; Tue, 10 Jun 2014 17:31:19 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 95CA0B94A; Tue, 10 Jun 2014 13:31:17 -0400 (EDT) From: John Baldwin To: freebsd-fs@freebsd.org Subject: Re: ZFS import panic (kgdb backtrace attached) Date: Tue, 10 Jun 2014 11:58:08 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201406101158.08599.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 10 Jun 2014 13:31:17 -0400 (EDT) Cc: grarpamp X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jun 2014 17:31:19 -0000 On Monday, June 09, 2014 2:37:36 pm grarpamp wrote: > ZFS pool was 96% full and under heavy sequential write, and panicked. > Dumps were not enabled so this first panic was lost. Fixed that, then... > > # zpool import -o readonly=on -f pool > (ok, zpool export pool) > # zpool import -f pool > (repeatably panics, coredump, reboot) > > FreeBSD 8.4-STABLE #0 r265935 i386 > > [/usr/include]% kgdb /boot/kernel/kernel /.../vmcore.1 > GNU gdb 6.1.1 [FreeBSD] > This GDB was configured as "i386-marcel-freebsd"... > > Unread portion of the kernel message buffer: > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 00 > fault virtual address = 0x11 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc13cb9f4 > stack pointer = 0x28:0xfcfb5ac0 > frame pointer = 0x28:0xfcfb5ae4 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 8 (txg_thread_enter) > trap number = 12 > panic: page fault > cpuid = 0 > KDB: stack backtrace: > #0 0xc094cd8f at kdb_backtrace+0x4f > #1 0xc091c5bc at panic+0x15c > #2 0xc0d75193 at trap_fatal+0x323 > #3 0xc0d7529c at trap_pfault+0xfc > #4 0xc0d7600a at trap+0x44a > #5 0xc0d5c2dc at calltrap+0x6 > #6 0xc13c91d9 at metaslab_sync+0x509 > #7 0xc13eb280 at vdev_sync+0x90 > #8 0xc13dded6 at spa_sync+0x496 > #9 0xc13e8835 at txg_sync_thread+0x145 > #10 0xc08ef767 at fork_exit+0x97 > #11 0xc0d5c354 at fork_trampoline+0x8 > Uptime: 17m21s > Physical memory: 2026 MB > Dumping 162 MB: 147 131 115 99 83 67 51 35 19 3 > > Loaded symbols for /boot/kernel/zfs.ko > Loaded symbols for /boot/kernel/opensolaris.ko > Loaded symbols for /boot/kernel/geom_eli.ko > Loaded symbols for /boot/kernel/crypto.ko > Loaded symbols for /boot/kernel/zlib.ko > Loaded symbols for /boot/kernel/snd_ich.ko > Loaded symbols for /boot/kernel/sound.ko > Loaded symbols for /boot/kernel/drm.ko > Loaded symbols for /boot/kernel/i915.ko > Loaded symbols for /boot/kernel/atapicam.ko > Loaded symbols for /boot/kernel/cpuctl.ko > > #0 doadump () at pcpu.h:244 > 244 __asm("movl %%fs:0,%0" : "=r" (td)); > > (kgdb) bt > #0 doadump () at pcpu.h:244 > #1 0xc091c313 in boot (howto=260) at /.../src/sys/kern/kern_shutdown.c:443 > #2 0xc091c5fe in panic (fmt=) at > /.../src/sys/kern/kern_shutdown.c:634 > #3 0xc0d75193 in trap_fatal (frame=0xfcfb5a80, eva=17) at > /.../src/sys/i386/i386/trap.c:1010 > #4 0xc0d7529c in trap_pfault (frame=0xfcfb5a80, usermode=0, eva=17) > at /.../src/sys/i386/i386/trap.c:872 > #5 0xc0d7600a in trap (frame=0xfcfb5a80) at /.../src/sys/i386/i386/trap.c:546 > #6 0xc0d5c2dc in calltrap () at /.../src/sys/i386/i386/exception.s:168 > #7 0xc13cb9f4 in range_tree_vacate (rt=0xc83dc000, func=0, arg=0x0) > at /.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c:364 > #8 0xc13c91d9 in metaslab_sync (msp=0xc8309000, txg=21088308) > at /.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1486 > #9 0xc13eb280 in vdev_sync (vd=0xc7f69800, txg=Unhandled dwarf > expression opcode 0x93 > ) at /.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:2274 > #10 0xc13dded6 in spa_sync (spa=0xdf3bf000, txg=21088308) > at /.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:6506 > #11 0xc13e8835 in txg_sync_thread (arg=0xc7907400) at > /.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:518 > #12 0xc08ef767 in fork_exit (callout=0xc13e86f0 , > arg=0xc7907400, frame=0xfcfb5d28) at /.../src/sys/kern/kern_fork.c:872 > #13 0xc0d5c354 in fork_trampoline () at /.../src/sys/i386/i386/exception.s:275 > > (kgdb) list *0xc13cb9f4 > 0xc13cb9f4 is in range_tree_vacate > (/.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c:364). > 359 void *cookie = NULL; > 360 > 361 ASSERT(MUTEX_HELD(rt->rt_lock)); > 362 > 363 if (rt->rt_ops != NULL) > 364 rt->rt_ops->rtop_vacate(rt, rt->rt_arg); > 365 > 366 while ((rs = avl_destroy_nodes(&rt->rt_root, &cookie)) > != NULL) { > 367 if (func != NULL) > 368 func(arg, rs->rs_start, rs->rs_end - > rs->rs_start); Can you do 'frame 7' and 'p *rt' and 'p *rt->rt_ops'? -- John Baldwin