From owner-freebsd-current@freebsd.org Wed Mar 2 12:18:22 2016 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB65AAC12A6 for ; Wed, 2 Mar 2016 12:18:22 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "thebighonker.lerctr.org", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AD01E1589 for ; Wed, 2 Mar 2016 12:18:22 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Content-Type:MIME-Version:Message-ID:Subject:To:From:Date; bh=67rzpL6mJGG+dspm3x1XW8XqITfTLLVIur7StwXBcr8=; b=n582u3lOyxpr55yRFfzLGm7M6O cVIXbyxNzRxub7ZGvnQJP1mckiHrYY05jA9R0tYNDKs7y57PEyPgAe3HOw+PUYffkMWOcsrGEIjML WUVZgtraL7h4yvGHBo/ABxebgoNqkkR0k19Jrv895iechNV693SBB5W5+v+kYXP0cZRM=; Received: from [2605:6000:ec17:203:ed2:92ff:fe96:98a3] (port=56585 helo=trivet.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.86 (FreeBSD)) (envelope-from ) id 1ab5jR-000D6L-8y for freebsd-current@freebsd.org; Wed, 02 Mar 2016 06:18:21 -0600 Date: Wed, 2 Mar 2016 06:18:19 -0600 From: Larry Rosenman To: freebsd-current@freebsd.org Subject: random ZFS panic... Message-ID: <20160302121819.GB1499@trivet.lerctr.org> Mail-Followup-To: freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Score: -1.0 (-) X-LERCTR-Spam-Score: -1.0 (-) X-Spam-Report: SpamScore (-1.0/5.0) ALL_TRUSTED=-1,SHORTCIRCUIT=-0.0001 X-LERCTR-Spam-Report: SpamScore (-1.0/5.0) ALL_TRUSTED=-1, SHORTCIRCUIT=-0.0001 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Mar 2016 12:18:23 -0000 Was rebooting my laptop, and got the following: trivet dumped core - see /var/crash/vmcore.0 Wed Mar 2 06:09:19 CST 2016 FreeBSD trivet 11.0-CURRENT FreeBSD 11.0-CURRENT #3 r296287: Tue Mar 1 19:13:37 CST 2016 root@trivet:/usr/obj/usr/src/sys/GENERIC amd64 panic: from debugger GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: <118>. <118>Terminated <118>Mar 2 06:07:22 trivet syslogd: exiting on signal 15 <5>wlan0: link state changed to DOWN Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...0 0 0 0 done All buffers synced. lock order reversal: 1st 0xfffff80012855d50 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1222 2nd 0xfffff800128557c8 zfs_gfs (zfs_gfs) @ /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/gfs.c:494 stack backtrace: #0 0xffffffff80a825a0 at witness_debugger+0x70 #1 0xffffffff80a824a1 at witness_checkorder+0xe71 #2 0xffffffff80a00a6b at __lockmgr_args+0xd3b #3 0xffffffff80ace2bc at vop_stdlock+0x3c #4 0xffffffff80fcb010 at VOP_LOCK1_APV+0x100 #5 0xffffffff80aeef9a at _vn_lock+0x9a #6 0xffffffff820c8b13 at gfs_file_create+0x73 #7 0xffffffff820c8bbd at gfs_dir_create+0x1d #8 0xffffffff82191f57 at zfsctl_mknode_snapdir+0x47 #9 0xffffffff820c9135 at gfs_dir_lookup+0x185 #10 0xffffffff820c961d at gfs_vop_lookup+0x1d #11 0xffffffff82190f75 at zfsctl_root_lookup+0xf5 #12 0xffffffff82191e13 at zfsctl_umount_snapshots+0x83 #13 0xffffffff821aacfb at zfs_umount+0x7b #14 0xffffffff80ad7c70 at dounmount+0x530 #15 0xffffffff80ae127b at vfs_unmountall+0x6b #16 0xffffffff80ac24f9 at bufshutdown+0x3b9 #17 0xffffffff80a26fe9 at kern_reboot+0x189 lock order reversal: 1st 0xfffff800125197c8 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1222 2nd 0xfffff8000ffbc240 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2498 stack backtrace: #0 0xffffffff80a825a0 at witness_debugger+0x70 #1 0xffffffff80a824a1 at witness_checkorder+0xe71 #2 0xffffffff80a00a6b at __lockmgr_args+0xd3b #3 0xffffffff80ace2bc at vop_stdlock+0x3c #4 0xffffffff80fcb010 at VOP_LOCK1_APV+0x100 #5 0xffffffff80aeef9a at _vn_lock+0x9a #6 0xffffffff80adf553 at vget+0x63 #7 0xffffffff808fcb4d at devfs_allocv+0xcd #8 0xffffffff808fc653 at devfs_root+0x43 #9 0xffffffff80ad7b8f at dounmount+0x44f #10 0xffffffff80ae12d4 at vfs_unmountall+0xc4 #11 0xffffffff80ac24f9 at bufshutdown+0x3b9 #12 0xffffffff80a26fe9 at kern_reboot+0x189 #13 0xffffffff80a26e03 at sys_reboot+0x3e3 #14 0xffffffff80e7a15b at amd64_syscall+0x2db #15 0xffffffff80e5926b at Xfast_syscall+0xfb Uptime: 7h20m6s Fatal trap 9: general protection fault while in kernel mode cpuid = 2; apic id = 02 instruction pointer = 0x20:0xffffffff8215369b stack pointer = 0x28:0xfffffe02329189b0 frame pointer = 0x28:0xfffffe02329189c0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (dbu_evict) Uptime: 7h20m7s Dumping 2338 out of 8050 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from /usr/lib/debug//boot/kernel/coretemp.ko.debug...done. done. Loaded symbols for /boot/kernel/coretemp.ko Reading symbols from /boot/kernel/ichsmb.ko...Reading symbols from /usr/lib/debug//boot/kernel/ichsmb.ko.debug...done. done. Loaded symbols for /boot/kernel/ichsmb.ko Reading symbols from /boot/kernel/smbus.ko...Reading symbols from /usr/lib/debug//boot/kernel/smbus.ko.debug...done. done. Loaded symbols for /boot/kernel/smbus.ko Reading symbols from /boot/kernel/hwpmc.ko...Reading symbols from /usr/lib/debug//boot/kernel/hwpmc.ko.debug...done. done. Loaded symbols for /boot/kernel/hwpmc.ko Reading symbols from /boot/kernel/iwn135fw.ko...Reading symbols from /usr/lib/debug//boot/kernel/iwn135fw.ko.debug...done. done. Loaded symbols for /boot/kernel/iwn135fw.ko Reading symbols from /boot/kernel/aesni.ko...Reading symbols from /usr/lib/debug//boot/kernel/aesni.ko.debug...done. done. Loaded symbols for /boot/kernel/aesni.ko Reading symbols from /boot/kernel/cryptodev.ko...Reading symbols from /usr/lib/debug//boot/kernel/cryptodev.ko.debug...done. done. Loaded symbols for /boot/kernel/cryptodev.ko Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtraceall.ko.debug...done. done. Loaded symbols for /boot/kernel/dtraceall.ko Reading symbols from /boot/kernel/profile.ko...Reading symbols from /usr/lib/debug//boot/kernel/profile.ko.debug...done. done. Loaded symbols for /boot/kernel/profile.ko Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtrace.ko.debug...done. done. Loaded symbols for /boot/kernel/dtrace.ko Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace_freebsd32.ko.debug...done. done. Loaded symbols for /boot/kernel/systrace_freebsd32.ko Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace.ko.debug...done. done. Loaded symbols for /boot/kernel/systrace.ko Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /usr/lib/debug//boot/kernel/sdt.ko.debug...done. done. Loaded symbols for /boot/kernel/sdt.ko Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from /usr/lib/debug//boot/kernel/fasttrap.ko.debug...done. done. Loaded symbols for /boot/kernel/fasttrap.ko Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /usr/lib/debug//boot/kernel/fbt.ko.debug...done. done. Loaded symbols for /boot/kernel/fbt.ko Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtnfscl.ko.debug...done. done. Loaded symbols for /boot/kernel/dtnfscl.ko Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtmalloc.ko.debug...done. done. Loaded symbols for /boot/kernel/dtmalloc.ko Reading symbols from /boot/kernel/cpuctl.ko...Reading symbols from /usr/lib/debug//boot/kernel/cpuctl.ko.debug...done. done. Loaded symbols for /boot/kernel/cpuctl.ko Reading symbols from /boot/kernel/cuse.ko...Reading symbols from /usr/lib/debug//boot/kernel/cuse.ko.debug...done. done. Loaded symbols for /boot/kernel/cuse.ko Reading symbols from /boot/kernel/ng_ubt.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_ubt.ko.debug...done. done. Loaded symbols for /boot/kernel/ng_ubt.ko Reading symbols from /boot/kernel/netgraph.ko...Reading symbols from /usr/lib/debug//boot/kernel/netgraph.ko.debug...done. done. Loaded symbols for /boot/kernel/netgraph.ko Reading symbols from /boot/kernel/ng_hci.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_hci.ko.debug...done. done. Loaded symbols for /boot/kernel/ng_hci.ko Reading symbols from /boot/kernel/ng_bluetooth.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_bluetooth.ko.debug...done. done. Loaded symbols for /boot/kernel/ng_bluetooth.ko Reading symbols from /boot/kernel/ng_l2cap.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_l2cap.ko.debug...done. done. Loaded symbols for /boot/kernel/ng_l2cap.ko Reading symbols from /boot/kernel/ng_btsocket.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_btsocket.ko.debug...done. done. Loaded symbols for /boot/kernel/ng_btsocket.ko Reading symbols from /boot/kernel/ng_socket.ko...Reading symbols from /usr/lib/debug//boot/kernel/ng_socket.ko.debug...done. done. Loaded symbols for /boot/kernel/ng_socket.ko #0 doadump (textdump=1) at pcpu.h:221 221 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=1) at pcpu.h:221 #1 0xffffffff80a27235 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:364 #2 0xffffffff80a2780b in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:757 #3 0xffffffff80a27853 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:688 #4 0xffffffff80383887 in db_panic (addr=, have_addr=false, count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:473 #5 0xffffffff80382ebe in db_command (cmd_table=0x0) at /usr/src/sys/ddb/db_command.c:440 #6 0xffffffff80382c54 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493 #7 0xffffffff803856eb in db_trap (type=, code=0) at /usr/src/sys/ddb/db_main.c:251 #8 0xffffffff80a65773 in kdb_trap (type=9, code=0, tf=) at /usr/src/sys/kern/subr_kdb.c:654 #9 0xffffffff80e79961 in trap_fatal (frame=0xfffffe0232918900, eva=) at /usr/src/sys/amd64/amd64/trap.c:836 #10 0xffffffff80e7963a in trap (frame=) at /usr/src/sys/amd64/amd64/trap.c:203 #11 0xffffffff80e58f87 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:234 #12 0xffffffff8215369b in txg_list_member (tl=0xfffff8001221c320, p=, txg=) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:850 #13 0xffffffff82112309 in dsl_dir_evict (dbu=0xfffff8000fedc400) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dir.c:145 #14 0xffffffff80a75e30 in taskqueue_run_locked (queue=0xfffff80004557800) at /usr/src/sys/kern/subr_taskqueue.c:430 #15 0xffffffff80a76958 in taskqueue_thread_loop (arg=) at /usr/src/sys/kern/subr_taskqueue.c:683 #16 0xffffffff809eb474 in fork_exit ( callout=0xffffffff80a768d0 , arg=0xfffff80006b7c170, frame=0xfffffe0232918ac0) at /usr/src/sys/kern/kern_fork.c:1034 #17 0xffffffff80e594be in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:609 #18 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) Ideas? Core IS available. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 7011 W Parmer Ln, Apt 1115, Austin, TX 78729-6961