From owner-freebsd-fs@FreeBSD.ORG Mon Jun 9 18:37:37 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 550E763E for ; Mon, 9 Jun 2014 18:37:37 +0000 (UTC) Received: from mail-vc0-x230.google.com (mail-vc0-x230.google.com [IPv6:2607:f8b0:400c:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 18CE42480 for ; Mon, 9 Jun 2014 18:37:37 +0000 (UTC) Received: by mail-vc0-f176.google.com with SMTP id im17so6630199vcb.7 for ; Mon, 09 Jun 2014 11:37:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=xulQIJMiDQa4tRjvx25Mo9MQTGsU0YZAwQCldU/C2IY=; b=jbVq3dborqF0MyjmRBWg0M31V6kPD5r0mrrX6L6ukRqqBZ6aBvum/ge8xl8tbbeqsH RS6hWVAZw+l1nhljAw8nnvPrNzEhwmumkLTji30JTfQeUksS60M1wDiIq87oBzQBJR5V nyoAcrBRL7CzVd5d37MMVd69qjxHG6DrZhtHMcBgefQQhfo6GoKKNj1WjNwCBax9+D4m 4+CBYOyxsdb9Lh4vnqWLWgzusvf+J1LolCt46bdIxH/wQcsILThegs+qMOBDOHV9LW9K VLm/PutXdpx9veYx6rsQKQ7QQwXviDcCrhAm5APA7aFLuTn6KC9/6TWVfgoyYU0tnvTT 7fig== MIME-Version: 1.0 X-Received: by 10.58.106.104 with SMTP id gt8mr11362048veb.46.1402339056138; Mon, 09 Jun 2014 11:37:36 -0700 (PDT) Received: by 10.221.65.198 with HTTP; Mon, 9 Jun 2014 11:37:36 -0700 (PDT) Date: Mon, 9 Jun 2014 14:37:36 -0400 Message-ID: Subject: ZFS import panic (kgdb backtrace attached) From: grarpamp To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2014 18:37:37 -0000 ZFS pool was 96% full and under heavy sequential write, and panicked. Dumps were not enabled so this first panic was lost. Fixed that, then... # zpool import -o readonly=on -f pool (ok, zpool export pool) # zpool import -f pool (repeatably panics, coredump, reboot) FreeBSD 8.4-STABLE #0 r265935 i386 [/usr/include]% kgdb /boot/kernel/kernel /.../vmcore.1 GNU gdb 6.1.1 [FreeBSD] This GDB was configured as "i386-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x11 fault code = supervisor read, page not present instruction pointer = 0x20:0xc13cb9f4 stack pointer = 0x28:0xfcfb5ac0 frame pointer = 0x28:0xfcfb5ae4 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 8 (txg_thread_enter) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: #0 0xc094cd8f at kdb_backtrace+0x4f #1 0xc091c5bc at panic+0x15c #2 0xc0d75193 at trap_fatal+0x323 #3 0xc0d7529c at trap_pfault+0xfc #4 0xc0d7600a at trap+0x44a #5 0xc0d5c2dc at calltrap+0x6 #6 0xc13c91d9 at metaslab_sync+0x509 #7 0xc13eb280 at vdev_sync+0x90 #8 0xc13dded6 at spa_sync+0x496 #9 0xc13e8835 at txg_sync_thread+0x145 #10 0xc08ef767 at fork_exit+0x97 #11 0xc0d5c354 at fork_trampoline+0x8 Uptime: 17m21s Physical memory: 2026 MB Dumping 162 MB: 147 131 115 99 83 67 51 35 19 3 Loaded symbols for /boot/kernel/zfs.ko Loaded symbols for /boot/kernel/opensolaris.ko Loaded symbols for /boot/kernel/geom_eli.ko Loaded symbols for /boot/kernel/crypto.ko Loaded symbols for /boot/kernel/zlib.ko Loaded symbols for /boot/kernel/snd_ich.ko Loaded symbols for /boot/kernel/sound.ko Loaded symbols for /boot/kernel/drm.ko Loaded symbols for /boot/kernel/i915.ko Loaded symbols for /boot/kernel/atapicam.ko Loaded symbols for /boot/kernel/cpuctl.ko #0 doadump () at pcpu.h:244 244 __asm("movl %%fs:0,%0" : "=r" (td)); (kgdb) bt #0 doadump () at pcpu.h:244 #1 0xc091c313 in boot (howto=260) at /.../src/sys/kern/kern_shutdown.c:443 #2 0xc091c5fe in panic (fmt=) at /.../src/sys/kern/kern_shutdown.c:634 #3 0xc0d75193 in trap_fatal (frame=0xfcfb5a80, eva=17) at /.../src/sys/i386/i386/trap.c:1010 #4 0xc0d7529c in trap_pfault (frame=0xfcfb5a80, usermode=0, eva=17) at /.../src/sys/i386/i386/trap.c:872 #5 0xc0d7600a in trap (frame=0xfcfb5a80) at /.../src/sys/i386/i386/trap.c:546 #6 0xc0d5c2dc in calltrap () at /.../src/sys/i386/i386/exception.s:168 #7 0xc13cb9f4 in range_tree_vacate (rt=0xc83dc000, func=0, arg=0x0) at /.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c:364 #8 0xc13c91d9 in metaslab_sync (msp=0xc8309000, txg=21088308) at /.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c:1486 #9 0xc13eb280 in vdev_sync (vd=0xc7f69800, txg=Unhandled dwarf expression opcode 0x93 ) at /.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c:2274 #10 0xc13dded6 in spa_sync (spa=0xdf3bf000, txg=21088308) at /.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:6506 #11 0xc13e8835 in txg_sync_thread (arg=0xc7907400) at /.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:518 #12 0xc08ef767 in fork_exit (callout=0xc13e86f0 , arg=0xc7907400, frame=0xfcfb5d28) at /.../src/sys/kern/kern_fork.c:872 #13 0xc0d5c354 in fork_trampoline () at /.../src/sys/i386/i386/exception.s:275 (kgdb) list *0xc13cb9f4 0xc13cb9f4 is in range_tree_vacate (/.../src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c:364). 359 void *cookie = NULL; 360 361 ASSERT(MUTEX_HELD(rt->rt_lock)); 362 363 if (rt->rt_ops != NULL) 364 rt->rt_ops->rtop_vacate(rt, rt->rt_arg); 365 366 while ((rs = avl_destroy_nodes(&rt->rt_root, &cookie)) != NULL) { 367 if (func != NULL) 368 func(arg, rs->rs_start, rs->rs_end - rs->rs_start); I'm copying over the readonly import now and can probably play with this if you need more. Thanks.