From owner-freebsd-stable@freebsd.org Wed Jul 20 15:50:10 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F14DB9E725 for ; Wed, 20 Jul 2016 15:50:10 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x22a.google.com (mail-wm0-x22a.google.com [IPv6:2a00:1450:400c:c09::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E9C3D114B for ; Wed, 20 Jul 2016 15:50:09 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-wm0-x22a.google.com with SMTP id i5so75166472wmg.0 for ; Wed, 20 Jul 2016 08:50:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to; bh=80MEbWGZcZ3Jw7w3vKXXnagGQ6ppjHb27pslzV5txGw=; b=WhpJpTQgeiih8flyiA3sDey794HOAKIj6W6Zw8tsJ7lsp88llCqfAtWkOJNYkFj1bT ygeY9sgeBeqGqWZSY1XcdzaRe8HzO3kQ8SrJ+7XGHBYUPwzvKdDuQecV9FWznk5tj0Yk PDG6YgGEuKAlgrqG31VETErz4rb94exXZIsKHxCPba9+zP+P4yzRR5Tz2IfqyLKjgJga tdxveZC5lYVnlqXKMpZA1ax8k7feeHyiDQqPppiDpG+0PQZend9eZwJ1l7iKgADxkABw G94pur7PW3/SEObCU/kgbu2Z3g6/nk0ILW7M7WhSMys8m+OrDrUVy9pNuEc46olpBKtt Y4jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to; bh=80MEbWGZcZ3Jw7w3vKXXnagGQ6ppjHb27pslzV5txGw=; b=WusVQ+feU+wLY3YRwSKkF6g/bE/5Sm5SPp9CXqUb8C1uh+OV23PVVST035X7U6edlF Uiyklo3NrTo6/WrXwMuWXxZ8MDCKoeln10gKSbBPh8PKBcw1M08T5wTD9IX44o1VCuP4 swv2SXz2VXFsuq1xD/eg7oYCYWpTDaLJAvxY1YL0tWKG2D4E6968NW6qv+VzIg3qvczv gaabRU0F+WJHSOkCzgSPU5hlEw5gkuhB7s7+U2dzhTrlf8RReb2LyJue8aSmpvEdy7hk i+1GyMBnEheL45kxBLrSDOi9BitENuXJ7STwsX56v2iGbaOSfYWz8y+pgIh3qRZGsx/w ELGg== X-Gm-Message-State: ALyK8tJztIkTRHANBW/j4srxpwuRzEqdNiv8Ps9WGAMycdlbskZxaFLKY5XU1Ikmfw5pbdvE X-Received: by 10.194.142.111 with SMTP id rv15mr2055507wjb.75.1469029806873; Wed, 20 Jul 2016 08:50:06 -0700 (PDT) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id qq6sm1887392wjc.34.2016.07.20.08.50.05 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Jul 2016 08:50:05 -0700 (PDT) Subject: Re: Panic on BETA1 in the ZFS subsystem To: freebsd-stable@freebsd.org References: <8f44bc09-1237-44d0-fe7a-7eb9cf4fe85b@denninger.net> <6be60e94-cce7-6e97-0392-5b08d13d7945@denninger.net> From: Steven Hartland Message-ID: <019db3bf-f4aa-6bbb-d191-01e8abc2e210@multiplay.co.uk> Date: Wed, 20 Jul 2016 16:50:11 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <6be60e94-cce7-6e97-0392-5b08d13d7945@denninger.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 15:50:10 -0000 The panic was due to stack exhaustion, why it was so deep not looked. On 20/07/2016 15:32, Karl Denninger wrote: > The panic occurred during a zfs send/receive operation for system > backup..... I've seen this one before, unfortunately, and it appears > that it's still there -- may be related to > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207464 > > On 7/20/2016 09:26, Karl Denninger wrote: >> Came out of the ZFS system; this machine is running *unpatched* (none of >> my changes are in the kernel); I have a good core dump if further trace >> information will help. >> >> Dump header from device: /dev/gpt/dump >> Architecture: amd64 >> Architecture Version: 2 >> Dump Length: 9582022656 >> Blocksize: 512 >> Dumptime: Wed Jul 20 09:10:31 2016 >> Hostname: NewFS.denninger.net >> Magic: FreeBSD Kernel Dump >> Version String: FreeBSD 11.0-BETA1 #0 r302489: Sat Jul 9 10:15:24 CDT >> 2016 >> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP >> Panic String: double fault >> Dump Parity: 445173880 >> Bounds: 4 >> Dump Status: good >> >> NewFS.denninger.net dumped core - see /var/crash/vmcore.4 >> >> Wed Jul 20 09:18:15 CDT 2016 >> >> FreeBSD NewFS.denninger.net 11.0-BETA1 FreeBSD 11.0-BETA1 #0 r302489: >> Sat Jul 9 10:15:24 CDT 2016 >> karl@NewFS.denninger.net:/usr/obj/usr/src/sys/KSD-SMP amd64 >> >> panic: double fault >> >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> >> Fatal double fault >> rip = 0xffffffff8220bc92 >> rsp = 0xfffffe0667f0ef30 >> rbp = 0xfffffe0667f0f3c0 >> cpuid = 14; apic id = 34 >> panic: double fault >> cpuid = 14 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe0649db8e30 >> vpanic() at vpanic+0x182/frame 0xfffffe0649db8eb0 >> panic() at panic+0x43/frame 0xfffffe0649db8f10 >> dblfault_handler() at dblfault_handler+0xa2/frame 0xfffffe0649db8f30 >> Xdblfault() at Xdblfault+0xac/frame 0xfffffe0649db8f30 >> --- trap 0x17, rip = 0xffffffff8220bc92, rsp = 0xfffffe0667f0ef30, rbp = >> 0xfffffe0667f0f3c0 --- >> vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x22/frame >> 0xfffffe0667f0f3c0 >> vdev_queue_io_done() at vdev_queue_io_done+0x83/frame 0xfffffe0667f0f400 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f430 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f480 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f4e0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f530 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f570 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f5a0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f5f0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f650 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f6a0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f6e0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f710 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f760 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f7c0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f810 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f850 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f880 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f8d0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0f930 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0f980 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0f9c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0f9f0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fa40 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0faa0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0faf0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fb30 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fb60 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fbb0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fc10 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fc60 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fca0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fcd0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fd20 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fd80 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fdd0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0fe10 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0fe40 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0fe90 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f0fef0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f0ff40 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f0ff80 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f0ffb0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10000 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10060 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f100b0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f100f0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10120 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10170 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f101d0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10220 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10260 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10290 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f102e0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10340 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10390 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f103d0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10400 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10450 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f104b0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10500 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10540 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10570 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f105c0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10620 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10670 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f106b0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f106e0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10730 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10790 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f107e0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10820 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10850 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f108a0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10900 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10950 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10990 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f109c0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10a10 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10a70 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10ac0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10b00 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10b30 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10b80 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10be0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10c30 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10c70 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10ca0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10cf0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10d50 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10da0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10de0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10e10 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10e60 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f10ec0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10f10 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f10f50 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f10f80 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f10fd0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11030 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11080 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f110c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f110f0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11140 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f111a0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f111f0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11230 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11260 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f112b0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11310 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11360 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f113a0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f113d0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11420 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11480 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f114d0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11510 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11540 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11590 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f115f0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11640 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11680 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f116b0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11700 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11760 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f117b0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f117f0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11820 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11870 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f118d0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11920 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11960 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11990 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f119e0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11a40 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11a90 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11ad0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11b00 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11b50 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11bb0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11c00 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11c40 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11c70 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11cc0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11d20 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11d70 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11db0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11de0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11e30 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f11e90 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11ee0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f11f20 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f11f50 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f11fa0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12000 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12050 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12090 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f120c0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12110 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12170 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f121c0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12200 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12230 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12280 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f122e0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12330 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12370 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f123a0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f123f0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12450 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f124a0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f124e0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12510 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12560 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f125c0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12610 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12650 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12680 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f126d0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12730 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12780 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f127c0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f127f0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12840 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f128a0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f128f0 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12930 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12960 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f129b0 >> zio_vdev_io_start() at zio_vdev_io_start+0x34d/frame 0xfffffe0667f12a10 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12a60 >> vdev_queue_io_done() at vdev_queue_io_done+0xcd/frame 0xfffffe0667f12aa0 >> zio_vdev_io_done() at zio_vdev_io_done+0xd9/frame 0xfffffe0667f12ad0 >> zio_execute() at zio_execute+0x236/frame 0xfffffe0667f12b20 >> taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame >> 0xfffffe0667f12b80 >> taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame >> 0xfffffe0667f12bb0 >> fork_exit() at fork_exit+0x85/frame 0xfffffe0667f12bf0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0667f12bf0 >> --- trap 0, rip = 0, rsp = 0, rbp = 0 --- >> Uptime: 10d21h36m8s >> Dumping 9138 out of 24531 >> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% >> >> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/zfs.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/zfs.ko >> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/opensolaris.ko >> Reading symbols from /boot/kernel/geom_eli.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/geom_eli.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/geom_eli.ko >> Reading symbols from /boot/kernel/geom_mirror.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/geom_mirror.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/geom_mirror.ko >> Reading symbols from /boot/kernel/aesni.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/aesni.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/aesni.ko >> Reading symbols from /boot/kernel/uhid.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/uhid.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/uhid.ko >> Reading symbols from /boot/kernel/ums.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ums.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ums.ko >> Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ipfw.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ipfw.ko >> Reading symbols from /boot/kernel/ipfw_nat.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/ipfw_nat.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/ipfw_nat.ko >> Reading symbols from /boot/kernel/libalias.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/libalias.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/libalias.ko >> Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from >> /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. >> done. >> Loaded symbols for /boot/kernel/daemon_saver.ko >> #0 doadump (textdump=1) at pcpu.h:221 >> 221 pcpu.h: No such file or directory. >> in pcpu.h >> (kgdb) #0 doadump (textdump=1) at pcpu.h:221 >> #1 0xffffffff80ae548e in kern_reboot (howto=260) >> at /usr/src/sys/kern/kern_shutdown.c:366 >> #2 0xffffffff80ae5a5b in vpanic (fmt=, >> ap=) at /usr/src/sys/kern/kern_shutdown.c:759 >> #3 0xffffffff80ae5893 in panic (fmt=0x0) >> at /usr/src/sys/kern/kern_shutdown.c:690 >> #4 0xffffffff80fc9692 in dblfault_handler (frame=) >> at /usr/src/sys/amd64/amd64/trap.c:868 >> #5 0xffffffff80fac15c in Xdblfault () >> at /usr/src/sys/amd64/amd64/exception.S:292 >> #6 0xffffffff8220bc92 in vdev_queue_io_to_issue (vq=0xfffff804d86833e0) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:585 >> #7 0xffffffff8220c6e3 in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:881 >> #8 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8018859c000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #9 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #10 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8018859c000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #11 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #12 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #13 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801df1f43b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #14 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #15 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801df1f43b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #16 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #17 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #18 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8024583e770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #19 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #20 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8024583e770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #21 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #22 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #23 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802e12df770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #24 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #25 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802e12df770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #26 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #27 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #28 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004f81f3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #29 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #30 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004f81f3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #31 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #32 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #33 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8021bdf7b28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #34 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #35 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8021bdf7b28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #36 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #37 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #38 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80165851770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #39 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #40 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80165851770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #41 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #42 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #43 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8060de643b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #44 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #45 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8060de643b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #46 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #47 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #48 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800793c0770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #49 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #50 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800793c0770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #51 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #52 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #53 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8026917cb28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #54 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #55 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8026917cb28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #56 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #57 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #58 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8012408d3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #59 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #60 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8012408d3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #61 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #62 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #63 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802bfc9ab28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #64 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #65 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802bfc9ab28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #66 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #67 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #68 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802dda5db28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #69 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #70 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802dda5db28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #71 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #72 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #73 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800ae13d000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #74 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #75 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800ae13d000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #76 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #77 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #78 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804a70efb28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #79 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #80 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804a70efb28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #81 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #82 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #83 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80485f5f000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #84 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #85 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80485f5f000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #86 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #87 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #88 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800666fc3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #89 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #90 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800666fc3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #91 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #92 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #93 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8010da3ab28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #94 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #95 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8010da3ab28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #96 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #97 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #98 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80315f73770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #99 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #100 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80315f73770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #101 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #102 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #103 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff806011703b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #104 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #105 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff806011703b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #106 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #107 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #108 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805964f1770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #109 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #110 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff805964f1770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #111 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #112 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #113 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8023902f770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #114 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #115 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8023902f770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #116 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #117 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #118 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff803bc7e2b28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #119 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #120 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff803bc7e2b28) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #121 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #122 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #123 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804dfbf43b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #124 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #125 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804dfbf43b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #126 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #127 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #128 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8005be4a3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #129 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #130 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8005be4a3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #131 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #132 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #133 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8004ff79000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #134 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #135 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8004ff79000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #136 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #137 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #138 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801aa2a2000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #139 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #140 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801aa2a2000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #141 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #142 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #143 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff800592003b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #144 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #145 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff800592003b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #146 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #147 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #148 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80202d11770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #149 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #150 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80202d11770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #151 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #152 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #153 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80392d3c3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #154 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #155 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80392d3c3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #156 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #157 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #158 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80550fdd000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #159 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #160 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80550fdd000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #161 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #162 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #163 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8055fd15770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #164 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #165 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8055fd15770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #166 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #167 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #168 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80576f99770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #169 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #170 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80576f99770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #171 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #172 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #173 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff802269bf000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #174 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #175 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff802269bf000) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #176 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #177 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #178 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff8003d7973b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #179 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #180 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff8003d7973b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #181 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #182 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #183 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff80108754770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #184 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #185 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80108754770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #186 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #187 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #188 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804cfdf7770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #189 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #190 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804cfdf7770) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #191 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #192 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #193 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801469a13b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #194 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #195 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801469a13b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870 >> #196 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #197 0xffffffff8220c72d in vdev_queue_io_done (zio=) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887 >> #198 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805dcd9d3b8) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895 >> #199 0xffffffff82225d96 in zio_execute (zio=) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556 >> #200 0xffffffff80b415aa in taskqueue_run_locked (queue=> out>) >> at /usr/src/sys/kern/subr_taskqueue.c:465 >> #201 0xffffffff80b424b8 in taskqueue_thread_loop (arg=) >> at /usr/src/sys/kern/subr_taskqueue.c:719 >> #202 0xffffffff80a9ad85 in fork_exit ( >> callout=0xffffffff80b423d0 , >> arg=0xfffff8037db02580, frame=0xfffffe0667f12c00) >> at /usr/src/sys/kern/kern_fork.c:1038 >> #203 0xffffffff80fac59e in fork_trampoline () >> at /usr/src/sys/amd64/amd64/exception.S:611 >> #204 0x0000000000000000 in ?? () >> Current language: auto; currently minimal >> (kgdb) >>