Date: Wed, 20 Jul 2016 16:50:11 +0100 From: Steven Hartland <killing@multiplay.co.uk> To: freebsd-stable@freebsd.org Subject: Re: Panic on BETA1 in the ZFS subsystem Message-ID: <019db3bf-f4aa-6bbb-d191-01e8abc2e210@multiplay.co.uk> In-Reply-To: <6be60e94-cce7-6e97-0392-5b08d13d7945@denninger.net> References: <8f44bc09-1237-44d0-fe7a-7eb9cf4fe85b@denninger.net> <6be60e94-cce7-6e97-0392-5b08d13d7945@denninger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
The panic was due to stack exhaustion, why it was so deep not looked. On 20/07/2016 15:32, Karl Denninger wrote: > The panic occurred during a zfs send/receive operation for system > backup..... I've seen this one before, unfortunately, and it appears > that it's still there -- may be related to > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207464 > > On 7/20/2016 09:26, Karl Denninger wrote: >> Came out of the ZFS system; this machine is running *unpatched* (none of >> my changes are in the kernel); I have a good core dump if further trace >> information will help. >> >> Dump header from device: /dev/gpt/dump >> Architecture: amd64 >> Architecture Version: 2 >> Dump Length: 9582022656 >> Blocksize: 512 >> Dumptime: Wed Jul 20 09:10:31 2016 >> Hostname: NewFS.denninger.net >> Magic: FreeBSD Kernel Dump >> Version String: FreeBSD 11.0-BETA1 #0 r302489: Sat Jul 9 10:15:24 CDT >> 2016 >> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP >> Panic String: double fault >> Dump Parity: 445173880 >> Bounds: 4 >> Dump Status: good >> >> NewFS.denninger.net dumped core - see /var/crash/vmcore.4 >> >> Wed Jul 20 09:18:15 CDT 2016 >> >> FreeBSD NewFS.denninger.net 11.0-BETA1 FreeBSD 11.0-BETA1 #0 r302489: >> Sat Jul 9 10:15:24 CDT 2016 >> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 >> >> panic: double fault >> >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> >> Fatal double fault >> rip = 0xffffffff8220bc92 >> rsp = 0xfffffe0667f0ef30 >> rbp = 0xfffffe0667f0f3c0 >> cpuid = 14; apic id = 34 >> panic: double fault >> cpuid = 14 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe0649db8e30 >> vpanic() at vpanic+0x182/frame 0xfffffe0649db8eb0 >> panic() at panic+0x43/frame 0xfffffe0649db8f10 >> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649db8f30 >> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649db8f30 >> --- trap 0x17, rip = 0xffffffff8220bc92, rsp = 0xfffffe0667f0ef30, rbp = >> 0xfffffe0667f0f3c0 --- >> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x22/frame >> 0xfffffe0667f0f3c0 >> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe0667f0f400 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f430 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f480 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f4e0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f530 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f570 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f5a0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f5f0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f650 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f6a0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f6e0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f710 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f760 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f7c0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f810 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f850 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f880 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f8d0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f930 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f980 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f9c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f9f0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fa40 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0faa0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0faf0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fb30 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fb60 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fbb0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fc10 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fc60 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fca0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fcd0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fd20 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fd80 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fdd0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fe10 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fe40 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fe90 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fef0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0ff40 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0ff80 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0ffb0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10000 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10060 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f100b0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f100f0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10120 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10170 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f101d0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10220 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10260 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10290 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f102e0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10340 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10390 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f103d0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10400 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10450 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f104b0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10500 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10540 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10570 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f105c0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10620 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10670 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f106b0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f106e0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10730 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10790 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f107e0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10820 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10850 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f108a0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10900 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10950 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10990 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f109c0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10a10 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10a70 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10ac0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10b00 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10b30 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10b80 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10be0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10c30 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10c70 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10ca0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10cf0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10d50 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10da0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10de0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10e10 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10e60 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10ec0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10f10 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10f50 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10f80 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10fd0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11030 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11080 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f110c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f110f0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11140 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f111a0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f111f0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11230 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11260 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f112b0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11310 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11360 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f113a0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f113d0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11420 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11480 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f114d0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11510 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11540 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11590 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f115f0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11640 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11680 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f116b0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11700 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11760 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f117b0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f117f0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11820 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11870 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f118d0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11920 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11960 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11990 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f119e0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11a40 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11a90 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11ad0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11b00 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11b50 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11bb0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11c00 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11c40 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11c70 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11cc0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11d20 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11d70 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11db0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11de0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11e30 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11e90 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11ee0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11f20 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11f50 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11fa0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12000 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12050 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12090 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f120c0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12110 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12170 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f121c0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12200 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12230 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12280 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f122e0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12330 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12370 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f123a0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f123f0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12450 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f124a0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f124e0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12510 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12560 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f125c0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12610 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12650 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12680 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f126d0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12730 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12780 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f127c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f127f0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12840 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f128a0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f128f0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12930 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12960 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f129b0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12a10 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12a60 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12aa0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12ad0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12b20 >> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame >> 0xfffffe0667f12b80 >> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame >> 0xfffffe0667f12bb0 >> fork_exit() at fork_exit+0x85/frame 0xfffffe0667f12bf0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0667f12bf0 >> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >> Uptime: 10d21h36m8s >> Dumping 9138 out of 24531 >> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% >> >> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/zfs.ko >> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/opensolaris.ko >> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/geom_eli.ko >> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/geom_mirror.ko >> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/aesni.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/aesni.ko >> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/uhid.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/uhid.ko >> Reading symbols from /boot/kernel/ums.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ums.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ums.ko >> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ipfw.ko >> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ipfw_nat.ko >> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/libalias.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/libalias.ko >> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/daemon_saver.ko >> #0 doadump (textdump=1) at pcpu.h:221 >> 221 pcpu.h: No such file or directory. >> in pcpu.h >> (kgdb) #0 doadump (textdump=1) at pcpu.h:221 >> #1 0xffffffff80ae548e in kern_reboot (howto=260) >> at /usr/src/sys/kern/kern_shutdown.c:366 >> #2 0xffffffff80ae5a5b in vpanic (fmt=<value optimized out>, >> ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:759 >> #3 0xffffffff80ae5893 in panic (fmt=0x0) >> at /usr/src/sys/kern/kern_shutdown.c:690 >> #4 0xffffffff80fc9692 in dblfault_handler (frame=<value optimized out>) >> at /usr/src/sys/amd64/amd64/trap.c:868 >> #5 0xffffffff80fac15c in Xdblfault () >> at /usr/src/sys/amd64/amd64/exception.S:292 >> #6 0xffffffff8220bc92 in vdev_queue_io_to_issue (vq=0xfffff804d86833e0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:585 >> #7 0xffffffff8220c6e3 in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:881 >> #8 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8018859c000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #9 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #10 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8018859c000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #11 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #12 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #13 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801df1f43b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #14 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #15 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801df1f43b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #16 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #17 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #18 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8024583e770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #19 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #20 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8024583e770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #21 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #22 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #23 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802e12df770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #24 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #25 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802e12df770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #26 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #27 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #28 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004f81f3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #29 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #30 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004f81f3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #31 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #32 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #33 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8021bdf7b28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #34 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #35 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8021bdf7b28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #36 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #37 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #38 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80165851770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #39 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #40 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80165851770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #41 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #42 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #43 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8060de643b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #44 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #45 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8060de643b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #46 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #47 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #48 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800793c0770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #49 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #50 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800793c0770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #51 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #52 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #53 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8026917cb28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #54 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #55 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8026917cb28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #56 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #57 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #58 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8012408d3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #59 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #60 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8012408d3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #61 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #62 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #63 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802bfc9ab28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #64 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #65 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802bfc9ab28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #66 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #67 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #68 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802dda5db28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #69 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #70 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802dda5db28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #71 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #72 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #73 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800ae13d000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #74 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #75 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800ae13d000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #76 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #77 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #78 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804a70efb28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #79 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #80 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804a70efb28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #81 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #82 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #83 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80485f5f000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #84 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #85 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80485f5f000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #86 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #87 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #88 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800666fc3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #89 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #90 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800666fc3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #91 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #92 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #93 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8010da3ab28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #94 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #95 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8010da3ab28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #96 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #97 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #98 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80315f73770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #99 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #100 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80315f73770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #101 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #102 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #103 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff806011703b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #104 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #105 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff806011703b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #106 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #107 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #108 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805964f1770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #109 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #110 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff805964f1770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #111 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #112 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #113 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8023902f770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #114 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #115 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8023902f770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #116 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #117 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #118 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff803bc7e2b28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #119 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #120 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff803bc7e2b28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #121 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #122 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #123 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804dfbf43b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #124 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #125 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804dfbf43b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #126 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #127 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #128 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8005be4a3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #129 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #130 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8005be4a3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #131 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #132 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #133 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004ff79000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #134 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #135 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004ff79000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #136 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #137 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #138 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801aa2a2000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #139 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #140 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801aa2a2000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #141 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #142 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #143 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800592003b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #144 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #145 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800592003b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #146 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #147 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #148 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80202d11770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #149 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #150 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80202d11770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #151 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #152 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #153 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80392d3c3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #154 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #155 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80392d3c3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #156 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #157 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #158 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80550fdd000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #159 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #160 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80550fdd000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #161 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #162 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #163 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8055fd15770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #164 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #165 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8055fd15770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #166 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #167 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #168 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80576f99770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #169 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #170 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80576f99770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #171 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #172 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #173 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802269bf000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #174 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #175 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802269bf000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #176 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #177 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #178 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8003d7973b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #179 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #180 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8003d7973b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #181 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #182 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #183 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80108754770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #184 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #185 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80108754770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #186 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #187 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #188 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804cfdf7770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #189 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #190 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804cfdf7770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #191 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #192 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #193 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801469a13b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #194 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #195 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801469a13b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #196 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #197 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #198 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805dcd9d3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #199 0xffffffff82225d96 in zio_execute (zio=<value optimized out>) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #200 0xffffffff80b415aa in taskqueue_run_locked (queue=<value optimized >> out>) >> at /usr/src/sys/kern/subr_taskqueue.c:465 >> #201 0xffffffff80b424b8 in taskqueue_thread_loop (arg=<value optimized out>) >> at /usr/src/sys/kern/subr_taskqueue.c:719 >> #202 0xffffffff80a9ad85 in fork_exit ( >> callout=0xffffffff80b423d0 <taskqueue_thread_loop>, >> arg=0xfffff8037db02580, frame=0xfffffe0667f12c00) >> at /usr/src/sys/kern/kern_fork.c:1038 >> #203 0xffffffff80fac59e in fork_trampoline () >> at /usr/src/sys/amd64/amd64/exception.S:611 >> #204 0x0000000000000000 in ?? () >> Current language: auto; currently minimal >> (kgdb) >>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?019db3bf-f4aa-6bbb-d191-01e8abc2e210>