From owner-freebsd-stable@freebsd.org Mon Oct 17 20:22:49 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0148DAF4FEF for ; Mon, 17 Oct 2016 20:22:49 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (denninger.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 83F44831 for ; Mon, 17 Oct 2016 20:22:48 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id 1EEBC303058 for ; Mon, 17 Oct 2016 15:22:46 -0500 (CDT) Subject: Re: Repeatable panic on ZFS filesystem (used for backups); 11.0-STABLE To: freebsd-stable@freebsd.org References: <3d4f25c9-a262-a373-ec7e-755325f8810b@denninger.net> <9adecd24-6659-0da5-5c05-d0d3957a2cb3@denninger.net> <0f58b11f-0bca-bc08-6f90-4e6e530f9956@denninger.net> <43a67287-f4f8-5d3e-6c5e-b3599c6adb4d@multiplay.co.uk> <76551fd6-0565-ee6c-b0f2-7d472ad6a4b3@denninger.net> <25ff3a3e-77a9-063b-e491-8d10a06e6ae2@multiplay.co.uk> From: Karl Denninger Message-ID: <26e092b2-17c6-8744-5035-d0853d733870@denninger.net> Date: Mon, 17 Oct 2016 15:22:44 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <25ff3a3e-77a9-063b-e491-8d10a06e6ae2@multiplay.co.uk> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms020209010902040701010306" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Oct 2016 20:22:49 -0000 This is a cryptographically signed message in MIME format. --------------ms020209010902040701010306 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Since I cleared it (by setting TRIM off on the test machine, rebooting, importing the pool and noting that it did not panic -- pulled drives, re-inserted into the production machine and ran backup routine -- all was normal) it may be a while before I see it again (a week or so is usua= l.) It appears to be related to entropy in the filesystem that comes up as "eligible" to be removed from the backup volume, which (not surprisingly) tends to happen a few days after I do a new world build or something similar (the daily and/or periodic snapshots roll off at about that point.) I don't happen to have a spare pair of high-performance SSDs I can stick in the sandbox machine in an attempt to force the condition to assert itself in test, unfortunately. I *am* concerned that it's not "simple" stack exhaustion because setting the max outstanding TRIMs on a per-vdev basis down quite-dramatically did *not* prevent it from happening -- and if it was simply stack depth related I would have expected that to put a stop to it. On 10/17/2016 15:16, Steven Hartland wrote: > Be good to confirm its not an infinite loop by giving it a good bump > first. > > On 17/10/2016 19:58, Karl Denninger wrote: >> I can certainly attempt setting that higher.... but is that not just >> hiding the problem rather than addressing it? >> >> >> On 10/17/2016 13:54, Steven Hartland wrote: >>> You're hitting stack exhaustion, have you tried increasing the kernel= >>> stack pages? >>> It can be changed from /boot/loader.conf >>> kern.kstack_pages=3D"6" >>> >>> Default on amd64 is 4 IIRC >>> >>> On 17/10/2016 19:08, Karl Denninger wrote: >>>> The target (and devices that trigger this) are a pair of 4Gb 7200RPM= >>>> SATA rotating rust drives (zmirror) with each provider geli-encrypte= d >>>> (that is, the actual devices used for the pool create are the .eli's= ) >>>> >>>> The machine generating the problem has both rotating rust devices >>>> *and* >>>> SSDs, so I can't simply shut TRIM off system-wide and call it a day = as >>>> TRIM itself is heavily-used; both the boot/root pools and a Postgres= ql >>>> database pool are on SSDs, while several terabytes of lesser-used da= ta >>>> is on a pool of Raidz2 that is made up of spinning rust. >>> snip... >>>> NewFS.denninger.net dumped core - see /var/crash/vmcore.1 >>>> >>>> Mon Oct 17 09:02:33 CDT 2016 >>>> >>>> FreeBSD NewFS.denninger.net 11.0-STABLE FreeBSD 11.0-STABLE #13 >>>> r307318M: Fri Oct 14 09:23:46 CDT 2016 >>>> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 >>>> >>>> panic: double fault >>>> >>>> GNU gdb 6.1.1 [FreeBSD] >>>> Copyright 2004 Free Software Foundation, Inc. >>>> GDB is free software, covered by the GNU General Public License, and= >>>> you are >>>> welcome to change it and/or distribute copies of it under certain >>>> conditions. >>>> Type "show copying" to see the conditions. >>>> There is absolutely no warranty for GDB. Type "show warranty" for >>>> details. >>>> This GDB was configured as "amd64-marcel-freebsd"... >>>> >>>> Unread portion of the kernel message buffer: >>>> >>>> Fatal double fault >>>> rip =3D 0xffffffff8220d9ec >>>> rsp =3D 0xfffffe066821f000 >>>> rbp =3D 0xfffffe066821f020 >>>> cpuid =3D 6; apic id =3D 14 >>>> panic: double fault >>>> cpuid =3D 6 >>>> KDB: stack backtrace: >>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >>>> 0xfffffe0649d78e30 >>>> vpanic() at vpanic+0x182/frame 0xfffffe0649d78eb0 >>>> panic() at panic+0x43/frame 0xfffffe0649d78f10 >>>> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649d78f30= >>>> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649d78f30 >>>> --- trap 0x17, rip =3D 0xffffffff8220d9ec, rsp =3D 0xfffffe066821f00= 0, >>>> rbp =3D >>>> 0xfffffe066821f020 --- >>>> avl_rotation() at avl_rotation+0xc/frame 0xfffffe066821f020 >>>> avl_remove() at avl_remove+0x1c8/frame 0xfffffe066821f070 >>>> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x87f/frame >>>> 0xfffffe066821f530 >>>> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame >>>> 0xfffffe066821f570 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f5a0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f5f0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe066821f650 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f6a0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe066821f6e0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f710= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f760 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe066821f7c0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f810 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe066821f850 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f880= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f8d0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe066821f930 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821f980 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe066821f9c0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821f9f0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fa40 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe066821faa0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821faf0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe066821fb30 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fb60= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fbb0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe066821fc10 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fc60 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe066821fca0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fcd0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fd20 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe066821fd80 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fdd0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe066821fe10 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821fe40= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821fe90 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe066821fef0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe066821ff40 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe066821ff80 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe066821ffb0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220000 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668220060 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682200b0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe06682200f0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220120= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220170 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe06682201d0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220220 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668220260 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220290= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682202e0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668220340 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220390 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe06682203d0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220400= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220450 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe06682204b0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220500 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668220540 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220570= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682205c0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668220620 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220670 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe06682206b0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682206e0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220730 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668220790 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682207e0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668220820 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220850= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682208a0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668220900 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220950 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668220990 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682209c0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220a10 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668220a70 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220ac0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668220b00 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220b30= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220b80 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668220be0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220c30 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668220c70 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220ca0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220cf0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668220d50 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220da0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668220de0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220e10= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220e60 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668220ec0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220f10 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668220f50 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668220f80= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668220fd0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668221030 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221080 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe06682210c0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682210f0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221140 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe06682211a0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682211f0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668221230 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221260= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682212b0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668221310 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221360 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe06682213a0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682213d0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221420 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668221480 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682214d0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668221510 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221540= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221590 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe06682215f0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221640 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668221680 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682216b0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221700 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668221760 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682217b0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe06682217f0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221820= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221870 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe06682218d0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221920 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668221960 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221990= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682219e0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668221a40 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221a90 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668221ad0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221b00= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221b50 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668221bb0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221c00 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668221c40 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221c70= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221cc0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668221d20 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221d70 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668221db0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221de0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221e30 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668221e90 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221ee0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668221f20 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668221f50= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668221fa0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668222000 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222050 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668222090 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682220c0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222110 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668222170 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682221c0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668222200 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222230= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222280 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe06682222e0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222330 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668222370 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682223a0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682223f0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668222450 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682224a0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe06682224e0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222510= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222560 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe06682225c0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222610 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668222650 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222680= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682226d0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668222730 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222780 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe06682227c0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe06682227f0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222840 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe06682228a0 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682228f0 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668222930 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222960= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe06682229b0 >>>> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame >>>> 0xfffffe0668222a10 >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222a60 >>>> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame >>>> 0xfffffe0668222aa0 >>>> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0668222ad0= >>>> zio_execute() at zio_execute+0x23d/frame 0xfffffe0668222b20 >>>> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame >>>> 0xfffffe0668222b80 >>>> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame >>>> 0xfffffe0668222bb0 >>>> fork_exit() at fork_exit+0x85/frame 0xfffffe0668222bf0 >>>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0668222bf0 >>>> --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- >>>> Uptime: 4m29s >>>> Dumping 1497 out of 24531 >>>> MB:..2%..11%..21%..31%..41%..51%..61%..71%..81%..91% >>>> >>>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >>>> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/zfs.ko >>>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols >>>> from >>>> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/opensolaris.ko >>>> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from= >>>> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/geom_eli.ko >>>> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols >>>> from >>>> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/geom_mirror.ko >>>> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from >>>> /usr/lib/debug//boot/kernel/aesni.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/aesni.ko >>>> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from >>>> /usr/lib/debug//boot/kernel/uhid.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/uhid.ko >>>> Reading symbols from /boot/kernel/ums.ko...Reading symbols from >>>> /usr/lib/debug//boot/kernel/ums.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/ums.ko >>>> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from >>>> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/ipfw.ko >>>> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from= >>>> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/ipfw_nat.ko >>>> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from= >>>> /usr/lib/debug//boot/kernel/libalias.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/libalias.ko >>>> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols >>>> from >>>> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. >>>> done. >>>> Loaded symbols for /boot/kernel/daemon_saver.ko >>>> #0 doadump (textdump=3D1) at pcpu.h:221 >>>> 221 pcpu.h: No such file or directory. >>>> in pcpu.h >>>> (kgdb) #0 doadump (textdump=3D1) at pcpu.h:221 >>>> #1 0xffffffff80ae9b6e in kern_reboot (howto=3D260) >>>> at /usr/src/sys/kern/kern_shutdown.c:366 >>>> #2 0xffffffff80aea13b in vpanic (fmt=3D, >>>> ap=3D) at >>>> /usr/src/sys/kern/kern_shutdown.c:759 >>>> #3 0xffffffff80ae9f73 in panic (fmt=3D0x0) >>>> at /usr/src/sys/kern/kern_shutdown.c:690 >>>> #4 0xffffffff80fd18f2 in dblfault_handler (frame=3D>>> out>) >>>> at /usr/src/sys/amd64/amd64/trap.c:868 >>>> #5 0xffffffff80fb406c in Xdblfault () >>>> at /usr/src/sys/amd64/amd64/exception.S:292 >>>> #6 0xffffffff8220d9ec in avl_rotation (tree=3D0xfffff8004dc54510, >>>> node=3D0xfffff8056e45cdd0, balance=3D2) >>>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:304 >>>> #7 0xffffffff8220de58 in avl_remove (tree=3D, >>>> data=3D) >>>> at /usr/src/sys/cddl/contrib/opensolaris/common/avl/avl.c:818 >>>> #8 0xffffffff822a1eff in vdev_queue_io_to_issue (vq=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :439 >>>> >>>> #9 0xffffffff822a2123 in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :907 >>>> >>>> #10 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e3f33d8= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #11 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #12 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e3f33d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #13 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #14 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #15 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e578000= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #16 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #17 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e57800= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #18 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #19 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #20 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ec3d8= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #21 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #22 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ec3d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #23 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #24 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #25 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e48a000= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #26 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #27 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e48a00= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #28 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #29 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #30 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e57ab88= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #31 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #32 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e57ab8= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #33 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #34 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #35 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69a3000= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #36 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #37 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69a300= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #38 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #39 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #40 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5c8000= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #41 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #42 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5c800= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #43 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #44 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #45 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23f9000= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #46 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #47 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23f900= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #48 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #49 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #50 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5147b0= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #51 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #52 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5147b= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #53 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #54 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #55 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e977000= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #56 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #57 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e97700= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #58 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #59 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #60 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90b3d8= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #61 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #62 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90b3d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #63 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #64 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #65 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b1160b88= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #66 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #67 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b1160b8= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #68 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #69 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #70 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4f53d8= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #71 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #72 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4f53d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #73 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #74 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #75 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8ebb88= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #76 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #77 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8ebb8= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #78 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #79 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #80 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e8d9b88= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #81 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #82 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e8d9b8= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #83 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #84 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #85 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21a03d8= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #86 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #87 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21a03d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #88 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #89 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #90 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e6903d8= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #91 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #92 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e6903d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #93 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #94 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #95 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a22dc7b0= ) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #96 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #97 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a22dc7b= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #98 0xffffffff822bbefd in zio_execute (zio=3D) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #99 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #100 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5fb7b= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #101 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #102 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5fb7= b0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #103 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #104 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #105 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5ae3d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #106 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #107 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5ae3= d8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #108 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #109 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #110 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a2131b8= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #111 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #112 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a2131b= 88) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #113 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #114 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #115 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e80b7b= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #116 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #117 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e80b7= b0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #118 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #119 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #120 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b14b900= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #121 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #122 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b14b90= 00) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #123 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #124 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #125 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a21fa3d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #126 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #127 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a21fa3= d8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #128 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #129 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #130 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5333d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #131 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #132 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5333= d8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #133 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #134 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #135 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e90ab8= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #136 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #137 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e90ab= 88) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #138 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #139 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #140 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e4047b= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #141 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #142 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e4047= b0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #143 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #144 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #145 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e7083d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #146 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #147 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e7083= d8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #148 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #149 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #150 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff802a23917b= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #151 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #152 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff802a23917= b0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #153 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #154 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #155 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e63600= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #156 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #157 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e6360= 00) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #158 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #159 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #160 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5afb8= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #161 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #162 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5afb= 88) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #163 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #164 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #165 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a69523d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #166 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #167 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a69523= d8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #168 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #169 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #170 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e47c00= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #171 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #172 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e47c0= 00) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #173 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #174 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #175 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800b17b23d= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #176 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #177 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800b17b23= d8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #178 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #179 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #180 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d367b= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #181 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #182 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff800a6d367= b0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #183 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #184 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #185 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e99fb8= 8) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #186 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #187 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e99fb= 88) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #188 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #189 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #190 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff8056e5227b= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #191 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #192 0xffffffff822bf75d in zio_vdev_io_start (zio=3D0xfffff8056e5227= b0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3112 >>>> #193 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #194 0xffffffff822a216d in vdev_queue_io_done (zio=3D>>> out>) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c= :913 >>>> >>>> #195 0xffffffff822bf919 in zio_vdev_io_done (zio=3D0xfffff800a6d3c00= 0) >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:3137 >>>> #196 0xffffffff822bbefd in zio_execute (zio=3D)= >>>> at >>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1651 >>>> #197 0xffffffff80b4895a in taskqueue_run_locked (queue=3D>>> optimized >>>> out>) >>>> at /usr/src/sys/kern/subr_taskqueue.c:454 >>>> #198 0xffffffff80b49b58 in taskqueue_thread_loop (arg=3D>>> optimized out>) >>>> at /usr/src/sys/kern/subr_taskqueue.c:724 >>>> #199 0xffffffff80a9f255 in fork_exit ( >>>> callout=3D0xffffffff80b49a70 , >>>> arg=3D0xfffff8057caa72a0, frame=3D0xfffffe0668222c00) >>>> at /usr/src/sys/kern/kern_fork.c:1040 >>>> #200 0xffffffff80fb44ae in fork_trampoline () >>>> at /usr/src/sys/amd64/amd64/exception.S:611 >>>> #201 0x0000000000000000 in ?? () >>>> Current language: auto; currently minimal >>>> (kgdb) >>>> >>>> >>>>>> --=20 >>>>>> Karl Denninger >>>>>> karl@denninger.net >>>>>> /The Market Ticker/ >>>>>> /[S/MIME encrypted email preferred]/ >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to >>> "freebsd-stable-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or= g" --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms020209010902040701010306 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjEwMTcyMDIyNDRaME8GCSqGSIb3DQEJBDFCBEBD SxPkkVliNyJn5MJyl2y0Dd6m5ZhzAGpkqyjAImE47rnEdCi0PrUs4RmErcgrsRegUIapZoy0 PspD3M7ml3alMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAiXOqnbPF eFhqqO1oWvvvGhgRNUTxfwz33KiFTwW974YH40XzSAoMmgUVyq0GADp87h18R2snGkkWfa0a 24K7tVHAeGhoQvvMFLsaVeaSj9+ajSvP//4/Tl1u+8qPLynjyCmVkub3ORZCjFbgrE70vklm 57M1ylMjvFZQkOPU2U3UlQytiCrJbs8XVVdJfser0ckNkk81ZMq8ubZ/Ojx+8+dFK4Vn8p1E uXaKewjpylySMBqXF7U3Aua8mnMdOTYUSlqdAvM6ooL9NPcwdIfaCckAgIhY0PJvprIAJcmw 7q4oqCqyOXhZfGiS1fKpyMjD2x9C4crk5mTo4RtHnGw9tA5lRo3e/cYLwQPvp7ii1NlmGRHk CwWt4h9l8XAIX4G16CwHRFJtQ46ECjU/JGk3NBUQlPdLeGYeZUTKnOjc2X2nJfsNszlZPbHq rGF35En9MfhgmGb2W6DIei9fXcxgib+Xgt9J1YgPFQqMMHgesl5DW7hGpFx8MJuqaDyZBDv5 ZqzDN38DE4SOlSXYtn2eJYdpDt5mMM9cNNNUGRl3Lbc3J9DaIWucAAZCvI5uA4N1tKuYbaOe wOIu49GLec4SS+8KIsxKgrOp8ZOxboqRcq6odTMCO/7R1jAQP7tOFI+qXHG4Pm0CTPUNW7Rv fEpdbICwKGvtGRuCnX9WZnBvu30AAAAAAAA= --------------ms020209010902040701010306--