Date: Wed, 20 Jul 2016 10:56:25 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Panic on BETA1 in the ZFS subsystem Message-ID: <78eed415-a558-c4f9-a334-9a2039922861@denninger.net> In-Reply-To: <019db3bf-f4aa-6bbb-d191-01e8abc2e210@multiplay.co.uk> References: <8f44bc09-1237-44d0-fe7a-7eb9cf4fe85b@denninger.net> <6be60e94-cce7-6e97-0392-5b08d13d7945@denninger.net> <019db3bf-f4aa-6bbb-d191-01e8abc2e210@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms080007090109000601090106 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable An attempted repeat of the transfer panic'd the machine again immediately. I removed all of the snapshots on the root zfs and the copy ran to completion. This appears to be identical to the problem that was previously present where snapshots on the root filesystem sometimes are corrupted, with the corruption apparently occurring at the time when the snapshot is taken (race condition in the snapshot code?) and when touched later you panic the machine. Since zfs send "touches" everything in the incremental path (or all snapshots if taken as a full send) once that bad snapshot is there on the disk you're screwed on any attempt to copy that filesystem elsewhere (e.g. for backup purposes.) Note that when this situation arose in the past a scrub succeeds without errors, and the same was true here -- since the root filesystem is small a scrub is fast, and I ran one -- it said there were no problems. There was a bug against this (which I referenced) but it is marked closed/resolved and I *was* running that patch on 10.2 without a panic for quite a long time (before the patch my average time-before-boom-boom was about a week) -- so the root cause may be different in this case on 11-BETA1 but the outcome is the same. On 7/20/2016 10:50, Steven Hartland wrote: > The panic was due to stack exhaustion, why it was so deep not looked. > > On 20/07/2016 15:32, Karl Denninger wrote: >> The panic occurred during a zfs send/receive operation for system >> backup..... I've seen this one before, unfortunately, and it appears >> that it's still there -- may be related to >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D207464 >> >> On 7/20/2016 09:26, Karl Denninger wrote: >>> Came out of the ZFS system; this machine is running *unpatched* >>> (none of >>> my changes are in the kernel); I have a good core dump if further tra= ce >>> information will help. >>> >>> Dump header from device: /dev/gpt/dump >>> Architecture: amd64 >>> Architecture Version: 2 >>> Dump Length: 9582022656 >>> Blocksize: 512 >>> Dumptime: Wed Jul 20 09:10:31 2016 >>> Hostname: NewFS.denninger.net >>> Magic: FreeBSD Kernel Dump >>> Version String: FreeBSD 11.0-BETA1 #0 r302489: Sat Jul 9 >>> 10:15:24 CDT >>> 2016 >>> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP >>> Panic String: double fault >>> Dump Parity: 445173880 >>> Bounds: 4 >>> Dump Status: good >>> >>> NewFS.denninger.net dumped core - see /var/crash/vmcore.4 >>> >>> Wed Jul 20 09:18:15 CDT 2016 >>> >>> FreeBSD NewFS.denninger.net 11.0-BETA1 FreeBSD 11.0-BETA1 #0 r302489:= >>> Sat Jul 9 10:15:24 CDT 2016 >>> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 >>> >>> panic: double fault >>> >>> GNU gdb 6.1.1 [FreeBSD] >>> Copyright 2004 Free Software Foundation, Inc. >>> GDB is free software, covered by the GNU General Public License, and >>> you are >>> welcome to change it and/or distribute copies of it under certain >>> conditions. >>> Type "show copying" to see the conditions. >>> There is absolutely no warranty for GDB. Type "show warranty" for >>> details. >>> This GDB was configured as "amd64-marcel-freebsd"... >>> >>> Unread portion of the kernel message buffer: >>> >>> Fatal double fault >>> rip =3D 0xffffffff8220bc92 >>> rsp =3D 0xfffffe0667f0ef30 >>> rbp =3D 0xfffffe0667f0f3c0 >>> cpuid =3D 14; apic id =3D 34 >>> panic: double fault >>> cpuid =3D 14 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>> 0xfffffe0649db8e30 >>> vpanic() at vpanic+0x182/frame 0xfffffe0649db8eb0 >>> panic() at panic+0x43/frame 0xfffffe0649db8f10 >>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649db8f30 >>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649db8f30 >>> --- trap 0x17, rip =3D 0xffffffff8220bc92, rsp =3D 0xfffffe0667f0ef30= , >>> rbp =3D >>> 0xfffffe0667f0f3c0 --- >>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x22/frame >>> 0xfffffe0667f0f3c0 >>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame >>> 0xfffffe0667f0f400 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f430 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f480 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f4= e0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f530 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f0f570 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f5a0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f5f0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f6= 50 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f6a0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f0f6e0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f710 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f760 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f7= c0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f810 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f0f850 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f880 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f8d0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f9= 30 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f980 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f0f9c0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f9f0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fa40 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fa= a0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0faf0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f0fb30 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fb60 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fbb0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fc= 10 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fc60 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f0fca0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fcd0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fd20 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fd= 80 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fdd0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f0fe10 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fe40 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fe90 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fe= f0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0ff40 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f0ff80 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0ffb0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10000 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f100= 60 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f100b0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f100f0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10120 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10170 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f101= d0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10220 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f10260 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10290 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f102e0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f103= 40 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10390 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f103d0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10400 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10450 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f104= b0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10500 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f10540 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10570 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f105c0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f106= 20 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10670 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f106b0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f106e0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10730 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f107= 90 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f107e0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f10820 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10850 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f108a0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f109= 00 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10950 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f10990 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f109c0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10a10 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10a= 70 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10ac0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f10b00 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10b30 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10b80 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10b= e0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10c30 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f10c70 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10ca0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10cf0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10d= 50 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10da0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f10de0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10e10 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10e60 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10e= c0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10f10 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f10f50 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10f80 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10fd0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f110= 30 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11080 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f110c0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f110f0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11140 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f111= a0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f111f0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f11230 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11260 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f112b0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f113= 10 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11360 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f113a0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f113d0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11420 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f114= 80 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f114d0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f11510 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11540 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11590 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f115= f0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11640 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f11680 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f116b0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11700 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f117= 60 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f117b0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f117f0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11820 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11870 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f118= d0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11920 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f11960 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11990 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f119e0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11a= 40 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11a90 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f11ad0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11b00 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11b50 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11b= b0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11c00 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f11c40 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11c70 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11cc0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11d= 20 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11d70 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f11db0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11de0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11e30 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11e= 90 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11ee0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f11f20 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11f50 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11fa0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f120= 00 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12050 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f12090 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f120c0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12110 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f121= 70 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f121c0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f12200 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12230 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12280 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f122= e0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12330 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f12370 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f123a0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f123f0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f124= 50 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f124a0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f124e0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12510 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12560 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f125= c0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12610 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f12650 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12680 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f126d0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f127= 30 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12780 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f127c0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f127f0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12840 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f128= a0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f128f0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f12930 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12960 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f129b0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12a= 10 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12a60 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>> 0xfffffe0667f12aa0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12ad0 >>> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12b20 >>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame >>> 0xfffffe0667f12b80 >>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame >>> 0xfffffe0667f12bb0 >>> fork_exit() at fork_exit+0x85/frame 0xfffffe0667f12bf0 >>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0667f12bf0 >>> --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- >>> Uptime: 10d21h36m8s >>> Dumping 9138 out of 24531 >>> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% >>> >>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/zfs.ko >>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols fr= om >>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/opensolaris.ko >>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/geom_eli.ko >>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols fr= om >>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/geom_mirror.ko >>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/aesni.ko >>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/uhid.ko >>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/ums.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/ums.ko >>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/ipfw.ko >>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/ipfw_nat.ko >>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/libalias.ko >>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols >>> from >>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/daemon_saver.ko >>> #0 doadump (textdump=3D1) at pcpu.h:221 >>> 221 pcpu.h: No such file or directory. >>> in pcpu.h >>> (kgdb) #0 doadump (textdump=3D1) at pcpu.h:221 >>> #1 0xffffffff80ae548e in kern_reboot (howto=3D260) >>> at /usr/src/sys/kern/kern_shutdown.c:366 >>> #2 0xffffffff80ae5a5b in vpanic (fmt=3D<value optimized out>, >>> ap=3D<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c= :759 >>> #3 0xffffffff80ae5893 in panic (fmt=3D0x0) >>> at /usr/src/sys/kern/kern_shutdown.c:690 >>> #4 0xffffffff80fc9692 in dblfault_handler (frame=3D<value optimized >>> out>) >>> at /usr/src/sys/amd64/amd64/trap.c:868 >>> #5 0xffffffff80fac15c in Xdblfault () >>> at /usr/src/sys/amd64/amd64/exception.S:292 >>> #6 0xffffffff8220bc92 in vdev_queue_io_to_issue >>> (vq=3D0xfffff804d86833e0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 585 >>> >>> #7 0xffffffff8220c6e3 in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 881 >>> >>> #8 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8018859c000)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #9 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #10 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8018859c000= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #11 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #12 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #13 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff801df1f43b8)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #14 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #15 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff801df1f43b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #16 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #17 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #18 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8024583e770)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #19 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #20 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8024583e770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #21 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #22 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #23 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff802e12df770)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #24 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #25 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff802e12df770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #26 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #27 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #28 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8004f81f3b8)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #29 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #30 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8004f81f3b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #31 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #32 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #33 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8021bdf7b28)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #34 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #35 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8021bdf7b28= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #36 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #37 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #38 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff80165851770)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #39 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #40 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff80165851770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #41 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #42 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #43 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8060de643b8)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #44 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #45 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8060de643b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #46 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #47 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #48 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff800793c0770)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #49 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #50 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff800793c0770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #51 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #52 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #53 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8026917cb28)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #54 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #55 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8026917cb28= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #56 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #57 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #58 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8012408d3b8)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #59 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #60 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8012408d3b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #61 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #62 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #63 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff802bfc9ab28)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #64 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #65 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff802bfc9ab28= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #66 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #67 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #68 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff802dda5db28)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #69 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #70 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff802dda5db28= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #71 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #72 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #73 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff800ae13d000)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #74 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #75 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff800ae13d000= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #76 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #77 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #78 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff804a70efb28)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #79 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #80 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff804a70efb28= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #81 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #82 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #83 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff80485f5f000)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #84 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #85 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff80485f5f000= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #86 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #87 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #88 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff800666fc3b8)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #89 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #90 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff800666fc3b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #91 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #92 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #93 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8010da3ab28)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #94 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #95 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8010da3ab28= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #96 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #97 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #98 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff80315f73770)= >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #99 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #100 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff80315f7377= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #101 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #102 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #103 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff806011703b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #104 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #105 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff806011703b= 8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #106 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #107 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #108 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff805964f1770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #109 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #110 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff805964f177= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #111 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #112 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #113 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8023902f770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #114 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #115 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8023902f77= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #116 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #117 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #118 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff803bc7e2b28= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #119 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #120 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff803bc7e2b2= 8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #121 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #122 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #123 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff804dfbf43b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #124 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #125 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff804dfbf43b= 8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #126 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #127 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #128 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8005be4a3b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #129 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #130 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8005be4a3b= 8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #131 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #132 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #133 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8004ff79000= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #134 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #135 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8004ff7900= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #136 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #137 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #138 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff801aa2a2000= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #139 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #140 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff801aa2a200= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #141 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #142 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #143 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff800592003b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #144 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #145 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff800592003b= 8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #146 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #147 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #148 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff80202d11770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #149 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #150 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff80202d1177= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #151 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #152 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #153 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff80392d3c3b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #154 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #155 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff80392d3c3b= 8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #156 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #157 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #158 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff80550fdd000= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #159 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #160 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff80550fdd00= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #161 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #162 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #163 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8055fd15770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #164 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #165 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8055fd1577= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #166 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #167 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #168 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff80576f99770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #169 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #170 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff80576f9977= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #171 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #172 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #173 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff802269bf000= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #174 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #175 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff802269bf00= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #176 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #177 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #178 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff8003d7973b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #179 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #180 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8003d7973b= 8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #181 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #182 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #183 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff80108754770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #184 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #185 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff8010875477= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #186 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #187 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #188 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff804cfdf7770= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #189 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #190 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff804cfdf777= 0) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #191 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #192 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #193 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff801469a13b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #194 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #195 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff801469a13b= 8) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >>> #196 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #197 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized= >>> out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 887 >>> >>> #198 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff805dcd9d3b8= ) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >>> #199 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >>> #200 0xffffffff80b415aa in taskqueue_run_locked (queue=3D<value optim= ized >>> out>) >>> at /usr/src/sys/kern/subr_taskqueue.c:465 >>> #201 0xffffffff80b424b8 in taskqueue_thread_loop (arg=3D<value >>> optimized out>) >>> at /usr/src/sys/kern/subr_taskqueue.c:719 >>> #202 0xffffffff80a9ad85 in fork_exit ( >>> callout=3D0xffffffff80b423d0 <taskqueue_thread_loop>, >>> arg=3D0xfffff8037db02580, frame=3D0xfffffe0667f12c00) >>> at /usr/src/sys/kern/kern_fork.c:1038 >>> #203 0xffffffff80fac59e in fork_trampoline () >>> at /usr/src/sys/amd64/amd64/exception.S:611 >>> #204 0x0000000000000000 in ?? () >>> Current language: auto; currently minimal >>> (kgdb) >>> > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or= g" --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms080007090109000601090106 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA3MjAxNTU2MjVaME8GCSqGSIb3DQEJBDFCBEDN awk2WqFJQNkdiNE7IyeBCUnBlahr6v1xCmPC2o4nQOcJBjCs4HzA0FiU5/0g/SpPnc+YtzpG RY06kOotWh7MMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAayPVIEwO MNLPBoUD7yEceFupSvk6dlD6v1HgGg2Rh0RHswKxLNTLl7g20R1uEvH77Lyw1TpRM0BnJK2W t8PFfXjVQMdQb7SZWvwcZKlaK4+aNVTpu8CWBR8hOBLJks0rq0vwSEw8uJV2LBQ865glHHfQ unWssUo4EENx4+fdQA/Uneb2K7hR8I91Evo0pDVipL/CyAvCtUAe681nd8KPCaM9AMLsbPqp oWhPVYaV8QyQAa7JNNepiHcLZ2PEg35NOHqINU30vhEebsODIf0dPdCwvFiSWhY8PAq0isNt e1oVb63bYV0sGOr+5tNXCK8LIBhb6hQ6EkeoiyY+QBwQgaTl4a/JwVppRc5JZ6K0FvQku20A cAP21IbJ0oGqmPNtFYvzC6LxdktKYIcmXFlMazqSFhrTb0x8mIfMffrgWYD219g4zfa3vAPk HdIxMpzLJNtP5PMeZW0jQ0Kxwjm/9ZMgyls/RgzFhiS50/7etmcsoKdab3TphNqfqzA9IuNX uIp0l32ro+6H+gUnHNJoALkCWtjaSGGGvdpM/ulAldypwCj274ghwxPJgWvKPZqA1gnzJj2o S+/SfGLwg1p2QwU1wSAdFa2l27LNwB3oAiHm5o/CwGtapuVaPNtOOKDT8P0ltlDX/o5qLBaA 6mMDwn20ZyGycx01s+WoxKW2HwsAAAAAAAA= --------------ms080007090109000601090106--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?78eed415-a558-c4f9-a334-9a2039922861>