Date: Mon, 17 Oct 2016 13:08:40 -0500 From: Karl Denninger <karl@denninger.net> To: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE Message-ID: <0f58b11f-0bca-bc08-6f90-4e6e530f9956@denninger.net> In-Reply-To: <CANCZdfq5QCDNhLY5GOpmBoh5ONYy2VPteuaMhQ2=3v%2B0vcoM0g@mail.gmail.com> References: <3d4f25c9-a262-a373-ec7e-755325f8810b@denninger.net> <9adecd24-6659-0da5-5c05-d0d3957a2cb3@denninger.net> <CANCZdfq5QCDNhLY5GOpmBoh5ONYy2VPteuaMhQ2=3v%2B0vcoM0g@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms090900030904000100020607 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable The target (and devices that trigger this) are a pair of 4Gb 7200RPM SATA rotating rust drives (zmirror) with each provider geli-encrypted (that is, the actual devices used for the pool create are the .eli's) The machine generating the problem has both rotating rust devices *and* SSDs, so I can't simply shut TRIM off system-wide and call it a day as TRIM itself is heavily-used; both the boot/root pools and a Postgresql database pool are on SSDs, while several terabytes of lesser-used data is on a pool of Raidz2 that is made up of spinning rust. vfs.zfs.trim.max_interval: 1 vfs.zfs.trim.timeout: 30 vfs.zfs.trim.txg_delay: 32 vfs.zfs.trim.enabled: 1 vfs.zfs.vdev.trim_max_pending: 10000 vfs.zfs.vdev.trim_max_active: 64 vfs.zfs.vdev.trim_min_active: 1 vfs.zfs.vdev.trim_on_init: 1 kstat.zfs.misc.zio_trim.failed: 0 kstat.zfs.misc.zio_trim.unsupported: 1080 kstat.zfs.misc.zio_trim.success: 573768 kstat.zfs.misc.zio_trim.bytes: 28964282368 The machine in question has been up for ~3 hours now since the last panic, so obviously TRIM is being heavily used... The issue, once the problem has been created, is *portable* and it is not being caused by the SSD source drives. That is, once the machine panics if I remove the two disks that form the backup pool, physically move them to my sandbox machine, geli attach the drives and import the pool within seconds the second machine will panic in the identical fashion. It's possible (but have not proved) that if I were to reboot enough times the filesystem would eventually reach consistency with the removed snapshots all gone and the panics would stop, but I got a half-dozen of them sequentially this morning on my test machine so I'm not at all sure how many more I'd need to allow to run, or whether *any* of the removals committed before the panic (if not then the cycle of reboot/attach/panic would never end....) :-) Reducing trim_max_active (to 10, a quite-drastic reduction) did not stop the panics. What appears to be happening is that the removal of the datasets in question on a reasonably newly-imported pool, whether it occurs by the incremental zfs recv -Fudv or by zfs destroy -r from the command line, generates a large number of TRIM requests which are of course rejected by the providers as spinning rust does not support them. However the attempt to queue them generates a stack overflow and double-fault panic as a result, and since once the command is issued the filesystem now has the deletes pending and the consistent state is in fact with them gone, any attempt to reattach the drives with TRIM enabled can result in an immediate additional panic. I tried to work around this in my backup script by creating and then destroying a file on the backup volume, then sleeping for a few seconds before the backup actually commenced, in the hope that this would (1) trigger a TRIM attempt and (2) lead the system to recognize that the target volume cannot support TRIM and thus stop trying to do so (and thus not lead to the storm that exhausts the stack and panic.) That approach, however (see below), failed to prevent the problem. # # Now try to trigger TRIM so that we don't have a storm of them # echo "Attempting to disable TRIM on spinning rust...." mount -t zfs backup/no-trim /mnt dd if=3D/dev/random of=3D/mnt/kill-trim bs=3D128k count=3D2 echo "Performed 2 writes" sleep 2 rm /mnt/kill-trim echo "Performed delete of written file" sleep 5 umount /mnt echo "Unmounted temporary filesystem" sleep 2 echo "TRIM disable theoretically done...." On 10/17/2016 12:43, Warner Losh wrote: > what's your underlying media? > > Warner > > > On Mon, Oct 17, 2016 at 10:02 AM, Karl Denninger <karl@denninger.net> w= rote: >> Update from my test system: >> >> Setting vfs.zfs.vdev_trim_max_active to 10 (from default 64) does *not= * >> stop the panics. >> >> Setting vfs.zfs.vdev.trim.enabled =3D 0 (which requires a reboot) DOES= >> stop the panics. >> >> I am going to run a scrub on the pack, but I suspect the pack itself >> (now that I can actually mount it without the machine blowing up!) is = fine. >> >> THIS (OBVIOUSLY) NEEDS ATTENTION! >> >> On 10/17/2016 09:17, Karl Denninger wrote: >>> This is a situation I've had happen before, and reported -- it appear= ed >>> to be a kernel stack overflow, and it has gotten materially worse on >>> 11.0-STABLE. >>> >>> The issue occurs after some period of time (normally a week or so.) = The >>> system has a mirrored pair of large drives used for backup purposes t= o >>> which ZFS snapshots are written using a script that iterates over the= >>> system. >>> >>> The panic /only /happens when the root filesystem is being sent, and = it >>> appears that the panic itself is being triggered by an I/O pattern on= >>> the /backup /drive -- not the source drives. Zpool scrubs on the sou= rce >>> are clean; I am going to run one now on the backup, but in the past t= hat >>> has been clean as well. >>> >>> I now have a *repeatable* panic in that if I attempt a "zfs list -rt = all >>> backup" on the backup volume I get the below panic. A "zfs list" >>> does*__*not panic the system. >>> >>> The operating theory previously (after digging through the passed >>> structures in the dump) was that the ZFS system was attempting to iss= ue >>> TRIMs on a device that can't do them before the ZFS system realizes t= his >>> and stops asking (the backup volume is comprised of spinning rust) bu= t >>> the appearance of the panic now on the volume when I simply do a "zfs= >>> list -rt all backup" appears to negate that theory since no writes ar= e >>> performed by that operation, and thus no TRIM calls should be issued.= >>> >>> I can leave the backup volume in the state that causes this for a sho= rt >>> period of time in an attempt to find and fix this. >>> >>> >>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1 >>> >>> Mon Oct 17 09:02:33 CDT 2016 >>> >>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13 >>> r307318M: Fri Oct 14 09:23:46 CDT 2016 >>> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 >>> >>> panic: double fault >>> >>> GNU gdb 6.1.1 [FreeBSD] >>> Copyright 2004 Free Software Foundation, Inc. >>> GDB is free software, covered by the GNU General Public License, and = you are >>> welcome to change it and/or distribute copies of it under certain >>> conditions. >>> Type "show copying" to see the conditions. >>> There is absolutely no warranty for GDB. Type "show warranty" for de= tails. >>> This GDB was configured as "amd64-marcel-freebsd"... >>> >>> Unread portion of the kernel message buffer: >>> >>> Fatal double fault >>> rip =3D 0xffffffff8220d9ec >>> rsp =3D 0xfffffe066821f000 >>> rbp =3D 0xfffffe066821f020 >>> cpuid =3D 6; apic id =3D 14 >>> panic: double fault >>> cpuid =3D 6 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>> 0xfffffe0649d78e30 >>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0 >>> panic() at panic+0x43/frame 0xfffffe0649d78f10 >>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30 >>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30 >>> --- trap 0x17, rip =3D 0xffffffff8220d9ec, rsp =3D 0xfffffe066821f000= , rbp =3D >>> 0xfffffe066821f020 --- >>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020 >>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070 >>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame >>> 0xfffffe066821f530 >>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe066821f= 570 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f6= 50 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f= 6e0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f7= c0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f= 850 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821f9= 30 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f= 9c0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fa= a0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f= b30 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fc= 10 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f= ca0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fd= 80 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f= e10 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe066821fe= f0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe066821f= f80 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682200= 60 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= 0f0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682201= d0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= 260 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682203= 40 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= 3d0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682204= b0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= 540 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682206= 20 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= 6b0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682207= 90 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= 820 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682209= 00 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= 990 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220a= 70 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= b00 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220b= e0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= c70 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220d= 50 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= de0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668220e= c0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668220= f50 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682210= 30 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= 0c0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682211= a0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= 230 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682213= 10 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= 3a0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682214= 80 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= 510 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682215= f0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= 680 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682217= 60 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= 7f0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682218= d0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= 960 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221a= 40 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= ad0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221b= b0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= c40 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221d= 20 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= db0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668221e= 90 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668221= f20 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682220= 00 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222= 090 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682221= 70 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222= 200 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682222= e0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222= 370 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682224= 50 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222= 4e0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682225= c0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222= 650 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682227= 30 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222= 7c0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe06682228= a0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222= 930 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0 >>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0668222a= 10 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60 >>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0668222= aa0 >>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0 >>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20 >>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame >>> 0xfffffe0668222b80 >>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame >>> 0xfffffe0668222bb0 >>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0 >>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0 >>> --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- >>> Uptime: 4m29s >>> Dumping 1497 out of 24531 >>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% >>> >>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/zfs.ko >>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols fr= om >>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/opensolaris.ko >>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/geom_eli.ko >>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols fr= om >>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/geom_mirror.ko >>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/aesni.ko >>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/uhid.ko >>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/ums.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/ums.ko >>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/ipfw.ko >>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/ipfw_nat.ko >>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from >>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/libalias.ko >>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols f= rom >>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. >>> done. >>> Loaded symbols for /boot/kernel/daemon_saver.ko >>> #0 doadump (textdump=3D1) at pcpu.h:221 >>> 221 pcpu.h: No such file or directory. >>> in pcpu.h >>> (kgdb) #0 doadump (textdump=3D1) at pcpu.h:221 >>> #1 0xffffffff80ae9b6e in kern_reboot (howto=3D260) >>> at /usr/src/sys/kern/kern_shutdown.c:366 >>> #2 0xffffffff80aea13b in vpanic (fmt=3D<value optimized out>, >>> ap=3D<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:= 759 >>> #3 0xffffffff80ae9f73 in panic (fmt=3D0x0) >>> at /usr/src/sys/kern/kern_shutdown.c:690 >>> #4 0xffffffff80fd18f2 in dblfault_handler (frame=3D<value optimized = out>) >>> at /usr/src/sys/amd64/amd64/trap.c:868 >>> #5 0xffffffff80fb406c in Xdblfault () >>> at /usr/src/sys/amd64/amd64/exception.S:292 >>> #6 0xffffffff8220d9ec in avl_rotation (tree=3D0xfffff8004dc54510, >>> node=3D0xfffff8056e45cdd0, balance=3D2) >>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304 >>> #7 0xffffffff8220de58 in avl_remove (tree=3D<value optimized out>, >>> data=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818 >>> #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=3D<value optimiz= ed out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 439 >>> #9 0xffffffff822a2123 in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 907 >>> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e3f33d8)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #11 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e3f33d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #13 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e578000)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #16 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e578000= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #18 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ec3d8)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #21 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ec3d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #23 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e48a000)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #26 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e48a000= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #28 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e57ab88)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #31 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e57ab88= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #33 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69a3000)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #36 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69a3000= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #38 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5c8000)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #41 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5c8000= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #43 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23f9000)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #46 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23f9000= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #48 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5147b0)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #51 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5147b0= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #53 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e977000)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #56 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e977000= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #58 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90b3d8)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #61 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90b3d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #63 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b1160b88)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #66 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b1160b88= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #68 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4f53d8)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #71 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4f53d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #73 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8ebb88)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #76 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8ebb88= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #78 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8d9b88)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #81 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8d9b88= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #83 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21a03d8)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #86 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21a03d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #88 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e6903d8)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #91 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e6903d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #93 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a22dc7b0)= >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #96 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a22dc7b0= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #98 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized = out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5fb7b0= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #101 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #102 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5fb7b= 0) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #103 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ae3d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #106 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #107 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ae3d= 8) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #108 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a2131b88= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #111 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #112 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a2131b8= 8) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #113 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e80b7b0= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #116 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #117 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e80b7b= 0) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #118 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b14b9000= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #121 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #122 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b14b900= 0) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #123 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21fa3d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #126 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #127 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21fa3d= 8) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #128 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5333d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #131 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #132 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5333d= 8) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #133 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90ab88= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #136 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #137 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90ab8= 8) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #138 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4047b0= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #141 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #142 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4047b= 0) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #143 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e7083d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #146 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #147 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e7083d= 8) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #148 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23917b0= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #151 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #152 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23917b= 0) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #153 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e636000= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #156 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #157 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e63600= 0) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #158 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5afb88= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #161 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #162 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5afb8= 8) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #163 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69523d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #166 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #167 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69523d= 8) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #168 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e47c000= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #171 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #172 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e47c00= 0) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #173 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b17b23d8= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #176 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #177 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b17b23d= 8) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #178 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d367b0= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #181 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #182 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a6d367b= 0) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #183 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e99fb88= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #186 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #187 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e99fb8= 8) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #188 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5227b0= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #191 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #192 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5227b= 0) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3112 >>> #193 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=3D<value optimized= out>) >>> at >>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:= 913 >>> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d3c000= ) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 3137 >>> #196 0xffffffff822bbefd in zio_execute (zio=3D<value optimized out>) >>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:= 1651 >>> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=3D<value optim= ized >>> out>) >>> at /usr/src/sys/kern/subr_taskqueue.c:454 >>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=3D<value optimi= zed out>) >>> at /usr/src/sys/kern/subr_taskqueue.c:724 >>> #199 0xffffffff80a9f255 in fork_exit ( >>> callout=3D0xffffffff80b49a70 <taskqueue_thread_loop>, >>> arg=3D0xfffff8057caa72a0, frame=3D0xfffffe0668222c00) >>> at /usr/src/sys/kern/kern_fork.c:1040 >>> #200 0xffffffff80fb44ae in fork_trampoline () >>> at /usr/src/sys/amd64/amd64/exception.S:611 >>> #201 0x0000000000000000 in ?? () >>> Current language: auto; currently minimal >>> (kgdb) >>> >>> >> -- >> Karl Denninger >> karl@denninger.net <mailto:karl@denninger.net> >> /The Market Ticker/ >> /[S/MIME encrypted email preferred]/ --=20 Karl Denninger karl@denninger.net <mailto:karl@denninger.net> /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms090900030904000100020607 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjEwMTcxODA4NDBaME8GCSqGSIb3DQEJBDFCBEBt NG0at3W7rzXELJhd1+h0bFAjin4DKITneKa1eW5ZZn4Lwu5VJO43WZ4nKqDKMVpl4AS9eBJA zdDT9f+XD3FEMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAdSlvGYOL bRAL+b4UNX2MJ4SQDAVKcxjQPS6d4x7pb1fqa/MsuImSfpJCoBBG1EcRTaDbJgdEdDWvzJs0 ZmPO8kK/vWYQwGMG17xurCjGBwZk2od6kJAgfjNxmdwffmxJee3BfRmEfhJqsk23cdOT/I64 /vZilR8rfDhyL1Rwm2aE/D4dqdoIH+XE9C0d7Z9ukKmN9RiM6Q5oZKNkIJD3ltc10kEqMr4q hOiXeQ4g9atM3+ii5Y33mvCs8i+16v8jVNckfuC+5f/q0LUOzVsBQg9fu2gP6Fa5hODTbdxz OvLzPpqzDdQW0sM89JlWnpY1+onZDq1FzsO5MTpaoNhMDmsrb+GF/PPUhhq5C65OjnOoZQ6C tXOuq6EtT5iGyjyAI45hm1Bjp35du3Xu8OC0yKd7mxbNTYxCF2w8i5mxeo9A9kC1JWQe3MK6 LGz/xkAVIvMN5N0WPz5WVN8nxFWzjkw3HYuk3ji6M9FqrBeUDf7NXl6I8K85MRxY+ParTgCu S26KqPGOMhlIyTLj962LtHz1hDfSQ7eH1sdBVxqCSl4quaqncIMC/wgmW7a/ZsyEScL3lrDO +SjfGMlCYEhGa7eoo7VVgvTrVQjlEzShzxKxUXK5ntT3Db29cDyhro8FC0cHeIsEgJp0rE6l a9dFR8BgLPoPJrAIrQo4yUrwS00AAAAAAAA= --------------ms090900030904000100020607--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0f58b11f-0bca-bc08-6f90-4e6e530f9956>