From owner-freebsd-stable@freebsd.org Mon Oct 17 21:50:05 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41EF2C16749 for ; Mon, 17 Oct 2016 21:50:05 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (denninger.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 98BD2AF for ; Mon, 17 Oct 2016 21:50:04 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id 31D5F303394 for ; Mon, 17 Oct 2016 16:50:02 -0500 (CDT) Subject: Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE To: freebsd-stable@freebsd.org References: <3d4f25c9-a262-a373-ec7e-755325f8810b@denninger.net> <9adecd24-6659-0da5-5c05-d0d3957a2cb3@denninger.net> <0f58b11f-0bca-bc08-6f90-4e6e530f9956@denninger.net> <43a67287-f4f8-5d3e-6c5e-b3599c6adb4d@multiplay.co.uk> <76551fd6-0565-ee6c-b0f2-7d472ad6a4b3@denninger.net> <25ff3a3e-77a9-063b-e491-8d10a06e6ae2@multiplay.co.uk> <26e092b2-17c6-8744-5035-d0853d733870@denninger.net> From: Karl Denninger Message-ID: Date: Mon, 17 Oct 2016 16:50:00 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms090702050402030004080907" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Oct 2016 21:50:05 -0000 This is a cryptographically signed message in MIME format. --------------ms090702050402030004080907 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable I will make some effort on the sandbox machine to see if I can come up with a way to replicate this. I do have plenty of spare larger drives laying around that used to be in service and were obsolesced due to capacity -- but what I don't know if whether the system will misbehave if the source is all spinning rust. In other words: 1. Root filesystem is mirrored spinning rust (production is mirrored SSDs= ) 2. Backup is mirrored spinning rust (of approx the same size) 3. Set up auto-snapshot exactly as the production system has now (which the sandbox is NOT since I don't care about incremental recovery on that machine; it's a sandbox!) 4. Run a bunch of build-somethings (e.g. buildworlds, cross-build for the Pi2s I have here, etc) to generate a LOT of filesystem entropy across lots of snapshots. 5. Back that up. 6. Export the backup pool. 7. Re-import it and "zfs destroy -r" the backup filesystem. That is what got me in a reboot loop after the *first* panic; I was simply going to destroy the backup filesystem and re-run the backup, but as soon as I issued that zfs destroy the machine panic'd and as soon as I re-attached it after a reboot it panic'd again. Repeat until I set trim=3D0. But... if I CAN replicate it that still shouldn't be happening, and the system should *certainly* survive attempting to TRIM on a vdev that doesn't support TRIMs, even if the removal is for a large amount of space and/or files on the target, without blowing up. BTW I bet it isn't that rare -- if you're taking timed snapshots on an active filesystem (with lots of entropy) and then make the mistake of trying to remove those snapshots (as is the case with a zfs destroy -r or a zfs recv of an incremental copy that attempts to sync against a source) on a pool that has been imported before the system realizes that TRIM is unavailable on those vdevs. Noting this: Yes need to find some time to have a look at it, but given how rare this is and with TRIM being re-implemented upstream in a totally different manor I'm reticent to spend any real time on it. What's in-process in this regard, if you happen to have a reference? On 10/17/2016 16:40, Steven Hartland wrote: > Setting those values will only effect what's queued to the device not > what's actually outstanding. > > On 17/10/2016 21:22, Karl Denninger wrote: >> Since I cleared it (by setting TRIM off on the test machine, rebooting= , >> importing the pool and noting that it did not panic -- pulled drives, >> re-inserted into the production machine and ran backup routine -- all >> was normal) it may be a while before I see it again (a week or so is >> usual.) >> >> It appears to be related to entropy in the filesystem that comes up as= >> "eligible" to be removed from the backup volume, which (not >> surprisingly) tends to happen a few days after I do a new world build = or >> something similar (the daily and/or periodic snapshots roll off at abo= ut >> that point.) >> >> I don't happen to have a spare pair of high-performance SSDs I can sti= ck >> in the sandbox machine in an attempt to force the condition to assert >> itself in test, unfortunately. >> >> I *am* concerned that it's not "simple" stack exhaustion because setti= ng >> the max outstanding TRIMs on a per-vdev basis down quite-dramatically >> did *not* prevent it from happening -- and if it was simply stack dept= h >> related I would have expected that to put a stop to it. >> >> On 10/17/2016 15:16, Steven Hartland wrote: >>> Be good to confirm its not an infinite loop by giving it a good bump >>> first. >>> >>> On 17/10/2016 19:58, Karl Denninger wrote: >>>> I can certainly attempt setting that higher.... but is that not just= >>>> hiding the problem rather than addressing it? >>>> >>>> >>>> On 10/17/2016 13:54, Steven Hartland wrote: >>>>> You're hitting stack exhaustion, have you tried increasing the kern= el >>>>> stack pages? >>>>> It can be changed from /boot/loader.conf >>>>> kern.kstack_pages=3D"6" >>>>> >>>>> Default on amd64 is 4 IIRC >>>>> >>>>> On 17/10/2016 19:08, Karl Denninger wrote: >>>>>> The target (and devices that trigger this) are a pair of 4Gb 7200R= PM >>>>>> SATA rotating rust drives (zmirror) with each provider >>>>>> geli-encrypted >>>>>> (that is, the actual devices used for the pool create are the >>>>>> .eli's) >>>>>> >>>>>> The machine generating the problem has both rotating rust devices >>>>>> *and* >>>>>> SSDs, so I can't simply shut TRIM off system-wide and call it a >>>>>> day as >>>>>> TRIM itself is heavily-used; both the boot/root pools and a >>>>>> Postgresql >>>>>> database pool are on SSDs, while several terabytes of lesser-used >>>>>> data >>>>>> is on a pool of Raidz2 that is made up of spinning rust. >>>>> snip... >>>>>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1 >>>>>> >>>>>> Mon Oct 17 09:02:33 CDT 2016 >>>>>> >>>>>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13 >>>>>> r307318M: Fri Oct 14 09:23:46 CDT 2016 >>>>>> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 >>>>>> >>>>>> panic: double fault >>>>>> >>>>>> GNU gdb 6.1.1 [FreeBSD] >>>>>> Copyright 2004 Free Software Foundation, Inc. >>>>>> GDB is free software, covered by the GNU General Public License, a= nd >>>>>> you are >>>>>> welcome to change it and/or distribute copies of it under certain >>>>>> conditions. >>>>>> Type "show copying" to see the conditions. >>>>>> There is absolutely no warranty for GDB. Type "show warranty" for= >>>>>> details. >>>>>> This GDB was configured as "amd64-marcel-freebsd"... >>>>>> >>>>>> Unread portion of the kernel message buffer: >>>>>> >>>>>> Fatal double fault >>>>>> rip =3D 0xffffffff8220d9ec >>>>>> rsp =3D 0xfffffe066821f000 >>>>>> rbp =3D 0xfffffe066821f020 >>>>>> cpuid =3D 6; apic id =3D 14 >>>>>> panic: double fault >>>>>> cpuid =3D 6 >>>>>> KDB: stack backtrace: >>>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>>>>> 0xfffffe0649d78e30 >>>>>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0 >>>>>> panic() at panic+0x43/frame 0xfffffe0649d78f10 >>>>>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f= 30 >>>>>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30 >>>>>> --- trap 0x17, rip =3D 0xffffffff8220d9ec, rsp =3D 0xfffffe066821f= 000, >>>>>> rbp =3D >>>>>> 0xfffffe066821f020 --- >>>>>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020 >>>>>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070 >>>>>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame >>>>>> 0xfffffe066821f530 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame >>>>>> 0xfffffe066821f570 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5= a0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe066821f650 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe066821f6e0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f7= 10 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe066821f7c0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe066821f850 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f8= 80 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe066821f930 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe066821f9c0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9= f0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe066821faa0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe066821fb30 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb= 60 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe066821fc10 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe066821fca0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fc= d0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe066821fd80 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe066821fe10 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe= 40 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe066821fef0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe066821ff80 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ff= b0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668220060 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe06682200f0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682201= 20 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe06682201d0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668220260 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682202= 90 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668220340 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe06682203d0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682204= 00 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe06682204b0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668220540 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682205= 70 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668220620 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe06682206b0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206= e0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668220790 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668220820 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682208= 50 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668220900 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668220990 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209= c0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668220a70 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668220b00 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b= 30 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668220be0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668220c70 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220c= a0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668220d50 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668220de0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e= 10 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668220ec0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668220f50 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f= 80 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668221030 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe06682210c0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210= f0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe06682211a0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668221230 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682212= 60 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668221310 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe06682213a0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213= d0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668221480 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668221510 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682215= 40 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe06682215f0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668221680 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216= b0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668221760 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe06682217f0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682218= 20 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe06682218d0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668221960 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682219= 90 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668221a40 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668221ad0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b= 00 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668221bb0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668221c40 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c= 70 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668221d20 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668221db0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221d= e0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668221e90 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668221f20 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f= 50 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668222000 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668222090 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220= c0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668222170 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668222200 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682222= 30 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe06682222e0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668222370 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223= a0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668222450 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe06682224e0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682225= 10 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe06682225c0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668222650 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682226= 80 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668222730 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe06682227c0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227= f0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe06682228a0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668222930 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682229= 60 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0 >>>>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>>>> 0xfffffe0668222a10 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60 >>>>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>>>> 0xfffffe0668222aa0 >>>>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222a= d0 >>>>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20 >>>>>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame >>>>>> 0xfffffe0668222b80 >>>>>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame >>>>>> 0xfffffe0668222bb0 >>>>>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0 >>>>>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0 >>>>>> --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- >>>>>> Uptime: 4m29s >>>>>> Dumping 1497 out of 24531 >>>>>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% >>>>>> >>>>>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >>>>>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/zfs.ko >>>>>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols= >>>>>> from >>>>>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/opensolaris.ko >>>>>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols fr= om >>>>>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/geom_eli.ko >>>>>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols= >>>>>> from >>>>>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/geom_mirror.ko >>>>>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from >>>>>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/aesni.ko >>>>>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from >>>>>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/uhid.ko >>>>>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from >>>>>> /usr/lib/debug//boot/kernel/ums.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/ums.ko >>>>>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from >>>>>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/ipfw.ko >>>>>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols fr= om >>>>>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/ipfw_nat.ko >>>>>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols fr= om >>>>>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/libalias.ko >>>>>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbol= s >>>>>> from >>>>>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. >>>>>> done. >>>>>> Loaded symbols for /boot/kernel/daemon_saver.ko >>>>>> #0 doadump (textdump=3D1) at pcpu.h:221 >>>>>> 221 pcpu.h: No such file or directory. >>>>>> in pcpu.h >>>>>> (kgdb) #0 doadump (textdump=3D1) at pcpu.h:221 >>>>>> #1 0xffffffff80ae9b6e in kern_reboot (howto=3D260) >>>>>> at /usr/src/sys/kern/kern_shutdown.c:366 >>>>>> #2 0xffffffff80aea13b in vpanic (fmt=3D, >>>>>> ap=3D) at >>>>>> /usr/src/sys/kern/kern_shutdown.c:759 >>>>>> #3 0xffffffff80ae9f73 in panic (fmt=3D0x0) >>>>>> at /usr/src/sys/kern/kern_shutdown.c:690 >>>>>> #4 0xffffffff80fd18f2 in dblfault_handler (frame=3D>>>>> out>) >>>>>> at /usr/src/sys/amd64/amd64/trap.c:868 >>>>>> #5 0xffffffff80fb406c in Xdblfault () >>>>>> at /usr/src/sys/amd64/amd64/exception.S:292 >>>>>> #6 0xffffffff8220d9ec in avl_rotation (tree=3D0xfffff8004dc54510,= >>>>>> node=3D0xfffff8056e45cdd0, balance=3D2) >>>>>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:3= 04 >>>>>> #7 0xffffffff8220de58 in avl_remove (tree=3D= , >>>>>> data=3D) >>>>>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:8= 18 >>>>>> #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=3D>>>>> optimized >>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:439 >>>>>> >>>>>> >>>>>> #9 0xffffffff822a2123 in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:907 >>>>>> >>>>>> >>>>>> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e3f33= d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #11 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e3f3= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #13 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5780= 00) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #16 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e578= 000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #18 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ec3= d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #21 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ec= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #23 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e48a0= 00) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #26 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e48a= 000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #28 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e57ab= 88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #31 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e57a= b88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #33 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69a30= 00) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #36 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69a3= 000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #38 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5c80= 00) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #41 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5c8= 000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #43 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23f90= 00) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #46 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23f9= 000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #48 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5147= b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #51 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e514= 7b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #53 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e9770= 00) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #56 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e977= 000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #58 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90b3= d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #61 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90b= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #63 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b1160b= 88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #66 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b1160= b88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #68 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4f53= d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #71 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4f5= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #73 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8ebb= 88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #76 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8eb= b88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #78 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8d9b= 88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #81 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8d9= b88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #83 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21a03= d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #86 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21a0= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #88 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e6903= d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #91 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e690= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #93 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a22dc7= b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #96 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a22dc= 7b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #98 0xffffffff822bbefd in zio_execute (zio=3D= ) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5fb= 7b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #101 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #102 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e5fb7b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #103 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ae= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #106 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #107 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e5ae3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #108 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a2131= b88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #111 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #112 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff802a2131b88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #113 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e80b= 7b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #116 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #117 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e80b7b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #118 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b14b9= 000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #121 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #122 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff800b14b9000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #123 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21fa= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #126 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #127 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff802a21fa3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #128 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e533= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #131 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #132 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e5333d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #133 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90a= b88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #136 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #137 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e90ab88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #138 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e404= 7b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #141 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #142 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e4047b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #143 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e708= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #146 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #147 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e7083d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #148 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a2391= 7b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #151 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #152 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff802a23917b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #153 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e636= 000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #156 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #157 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e636000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #158 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5af= b88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #161 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #162 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e5afb88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #163 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6952= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #166 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #167 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff800a69523d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #168 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e47c= 000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #171 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #172 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e47c000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #173 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b17b2= 3d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #176 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #177 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff800b17b23d8) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #178 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d36= 7b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #181 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #182 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff800a6d367b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #183 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e99f= b88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #186 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #187 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e99fb88) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #188 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e522= 7b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #191 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #192 0xffffffff822bf75d in zio_vdev_io_start >>>>>> (zio=3D0xfffff8056e5227b0) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112= >>>>>> #193 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>>>> out>) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue= =2Ec:913 >>>>>> >>>>>> >>>>>> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d3c= 000) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137= >>>>>> #196 0xffffffff822bbefd in zio_execute (zio=3D) >>>>>> at >>>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651= >>>>>> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=3D>>>>> optimized >>>>>> out>) >>>>>> at /usr/src/sys/kern/subr_taskqueue.c:454 >>>>>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=3D>>>>> optimized out>) >>>>>> at /usr/src/sys/kern/subr_taskqueue.c:724 >>>>>> #199 0xffffffff80a9f255 in fork_exit ( >>>>>> callout=3D0xffffffff80b49a70 , >>>>>> arg=3D0xfffff8057caa72a0, frame=3D0xfffffe0668222c00) >>>>>> at /usr/src/sys/kern/kern_fork.c:1040 >>>>>> #200 0xffffffff80fb44ae in fork_trampoline () >>>>>> at /usr/src/sys/amd64/amd64/exception.S:611 >>>>>> #201 0x0000000000000000 in ?? () >>>>>> Current language: auto; currently minimal >>>>>> (kgdb) >>>>>> >>>>>> >>>>>>>> --=20 >>>>>>>> Karl Denninger >>>>>>>> karl@denninger.net >>>>>>>> /The Market Ticker/ >>>>>>>> /[S/MIME encrypted email preferred]/ >>>>> _______________________________________________ >>>>> freebsd-stable@freebsd.org mailing list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>>> To unsubscribe, send any mail to >>>>> "freebsd-stable-unsubscribe@freebsd.org" >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to >>> "freebsd-stable-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or= g" --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms090702050402030004080907 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjEwMTcyMTUwMDBaME8GCSqGSIb3DQEJBDFCBEDD 2raSvQ0MtHDlD4uLkgruGKirfquUn7LlpM/qO97f/qJNrx333Lof5XtbxK67zfk+J2xkVx1m OXPtobCMsrNLMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAZtlArLyl 91aIQgIxdD5WePp0kHB5nE7IpzE9Hu1sXIzzALqsIma1yktQodOh0Aql/FjOS8Y777fP2nbl WYPknpLKq6q6Ixfrj0Di8tAMKas4x8XXQH8jT0nRpVqmfWlL6TgyKkSFp68wCnUmoewOMGx8 SyD1L0UjH10WORA0uQ11m2YdqW8onHVF5kAxfl09vMVWzNgGLiLUbfUbx4eZPqLTAaVtvMfC Ar7NKgjueZe2p1DkzyBnipjo3pe1sfGolc5+Zleo4Jf/82Q7zxPYp8flIvWDRxGIRJk2LYsu U7SxNkW/pK+/H9oX4pHZ6nYxK6dwdIqzd0EoFMPF6A3i5U4i8O390RH5v7cvDLv8MgvONOpy 3AKJ9w+qx8sNb8+ZtxfjNu2N+2u3xunV0AiVqc//87HuiUpqzlUXL0nqMka3Ca79CsYi6rI7 WADXiV0hWix3L3KnD0+dg8pGPm5TRAaJzjXu0ckht+1J8orHiwYqCFalhCe/mEmeKgiHhRA5 idTcXsAald/PSYovs8pOEkqPycyV+7obKUFYyhB51kOhhENncvTpv5YLcMd24gB04Fq/101b AOZWjCbM44o1rBiNy1vGfE0rjYH8QhPEe/SbYv3qETlC1f3+CMEiLTnFZmvI1P5iFCoxtzfz g1mXteoG4MaGeuplvXeLlp/gP1EAAAAAAAA= --------------ms090702050402030004080907--