From owner-freebsd-stable@freebsd.org Wed Jul 20 21:54:06 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 88F6AB9F698 for ; Wed, 20 Jul 2016 21:54:06 +0000 (UTC) (envelope-from karl@denninger.net) Received: from mail.denninger.net (denninger.net [70.169.168.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 315CF1446 for ; Wed, 20 Jul 2016 21:54:05 +0000 (UTC) (envelope-from karl@denninger.net) Received: from [192.168.1.40] (Karl-Desktop.Denninger.net [192.168.1.40]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.denninger.net (Postfix) with ESMTPSA id 6D54489353 for ; Wed, 20 Jul 2016 16:54:03 -0500 (CDT) Subject: Re: Panic on BETA1 in the ZFS subsystem To: freebsd-stable@freebsd.org References: <8f44bc09-1237-44d0-fe7a-7eb9cf4fe85b@denninger.net> <54e5974c-312e-c33c-ab83-9e1148618ddc@FreeBSD.org> From: Karl Denninger Message-ID: <97cf5283-683b-83fd-c484-18c14973b065@denninger.net> Date: Wed, 20 Jul 2016 16:54:03 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <54e5974c-312e-c33c-ab83-9e1148618ddc@FreeBSD.org> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms000502000101080107030506" X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Jul 2016 21:54:06 -0000 This is a cryptographically signed message in MIME format. --------------ms000502000101080107030506 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 7/20/2016 16:32, Andriy Gapon wrote: > On 20/07/2016 17:26, Karl Denninger wrote: >> #185 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff80108754770= ) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2= 870 >> #186 0xffffffff82225d96 in zio_execute (zio=3D) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1= 556 >> #187 0xffffffff8220c72d in vdev_queue_io_done (zio=3D) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:8= 87 >> #188 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff804cfdf7770)= >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2= 895 >> #189 0xffffffff82225d96 in zio_execute (zio=3D) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1= 556 >> #190 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff804cfdf7770= ) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2= 870 >> #191 0xffffffff82225d96 in zio_execute (zio=3D) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1= 556 >> #192 0xffffffff8220c72d in vdev_queue_io_done (zio=3D) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:8= 87 >> #193 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff801469a13b8)= >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2= 895 >> #194 0xffffffff82225d96 in zio_execute (zio=3D) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1= 556 >> #195 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff801469a13b8= ) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2= 870 >> #196 0xffffffff82225d96 in zio_execute (zio=3D) >> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1= 556 >> #197 0xffffffff8220c72d in vdev_queue_io_done (zio=3D) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:8= 87 >> #198 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff805dcd9d3b8)= > I see how this pseudo-recursion can happen in theory, but not why it ha= s > happened. It would interesting to see 3 or 4 of zio-s printed, > including the top-most one at 0xfffff805dcd9d3b8. > > Something like: > > set print pretty > p *zio > p *zio->io_vd > p *zio->io_vd->vdev_ops > > in several frames. > There was a clone from a previous boot environment embedded in the snapshot list, but I don't believe it was (directly) the cause as several backups have taken place since it was taken, and none resulted in the panic. I had this problem some time ago in which on the root pool/filesystem, and ONLY on the root, I would get a panic after several days uptime *and* the disk structure itself was ok but the incore was not. It would manifest as a snapshot you could "cd" into but not see anything once inside, and an attempt to destroy it would fail. If you attempted a zfs send that touched that snapshot you got a panic. The PR that I referenced above apparently was related, and the patch that came out of it appears to have stopped it on 10.2, as once I put that on I didn't have it happen again. It was consistent enough that I actually killed automated backups on the machines involved, as it would nail you without warning on a roughly a one-week interval. That appears to be roughly where I am again.... about one week into running BETA1, and "boom". But this one IS different in that this time an immediate repeat (which used to succeed as the disk was fine) resulted in an immediate second panic. I've removed the snapshots on that filesystem (on the theory that the previous error wasn't really fixed but rather papered over) and run a backup, which succeeded. If I get another one I'll know it's a new probl= em. Let me see what I can come up with on some frame dumps. Here's the top one... (kgdb) p *zio $5 =3D { io_bookmark =3D { zb_objset =3D 0, zb_object =3D 0, zb_level =3D 0, zb_blkid =3D 0 }, io_prop =3D { zp_checksum =3D ZIO_CHECKSUM_INHERIT, zp_compress =3D ZIO_COMPRESS_INHERIT, zp_type =3D DMU_OT_NONE, zp_level =3D 0 '\0', zp_copies =3D 0 '\0', zp_dedup =3D 0, zp_dedup_verify =3D 0, zp_nopwrite =3D 0 }, io_type =3D ZIO_TYPE_FREE, io_child_type =3D ZIO_CHILD_VDEV, io_cmd =3D 0, io_priority =3D ZIO_PRIORITY_TRIM, io_reexecute =3D 0 '\0', io_state =3D 0xfffff805dcd9d405 "\001", io_txg =3D 0, io_spa =3D 0xfffff801c29d1000, io_bp =3D 0x0, io_bp_override =3D 0x0, io_bp_copy =3D { blk_dva =3D 0xfffff805dcd9d428, blk_prop =3D 0, blk_pad =3D 0xfffff805dcd9d460, blk_phys_birth =3D 0, blk_birth =3D 0, blk_fill =3D 0, blk_cksum =3D { zc_word =3D 0xfffff805dcd9d488 } }, io_parent_list =3D { list_size =3D 48, list_offset =3D 16, list_head =3D { list_next =3D 0xfffff8042bf35340, list_prev =3D 0xfffff8042bf35340 } }, io_child_list =3D { list_size =3D 48, list_offset =3D 32, list_head =3D { list_next =3D 0xfffff805dcd9d4d8, list_prev =3D 0xfffff805dcd9d4d8 } }, io_walk_link =3D 0x0, io_logical =3D 0x0, io_transform_stack =3D 0x0, io_ready =3D 0, io_physdone =3D 0, io_done =3D 0, io_private =3D 0x0, io_prev_space_delta =3D 0, io_bp_orig =3D { blk_dva =3D 0xfffff805dcd9d528, blk_prop =3D 0, blk_pad =3D 0xfffff805dcd9d560, blk_phys_birth =3D 0, blk_birth =3D 0, blk_fill =3D 0, blk_cksum =3D { zc_word =3D 0xfffff805dcd9d588 } }, io_data =3D 0x0, io_orig_data =3D 0x0, io_size =3D 8192, io_orig_size =3D 8192, io_vd =3D 0xfffff804d8683000, io_vsd =3D 0x0, io_vsd_ops =3D 0x0, io_offset =3D 897895682048, io_timestamp =3D 941768555692856, io_target_timestamp =3D 0, io_queue_node =3D { avl_child =3D 0xfffff805dcd9d5f8, avl_pcb =3D 18446735278553267561 }, io_offset_node =3D { avl_child =3D 0xfffff805dcd9d610, avl_pcb =3D 0 }, io_flags =3D 789633, io_stage =3D ZIO_STAGE_VDEV_IO_DONE, io_pipeline =3D 3080192, io_orig_flags =3D 525441, io_orig_stage =3D ZIO_STAGE_OPEN, io_orig_pipeline =3D 3080192, io_error =3D 45, io_child_error =3D 0xfffff805dcd9d644, io_children =3D 0xfffff805dcd9d658, io_child_count =3D 0, io_phys_children =3D 0, io_parent_count =3D 1, io_stall =3D 0x0, io_gang_leader =3D 0x0, io_gang_tree =3D 0x0, io_executor =3D 0xfffff80339d9d500, io_waiter =3D 0x0, io_lock =3D { lock_object =3D { lo_name =3D 0xffffffff822bf23e "zio->io_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0 }, sx_lock =3D 1 }, io_cv =3D { cv_description =3D 0xffffffff822bf24c "zio->io_cv", cv_waiters =3D 0 }, io_cksum_report =3D 0x0, io_ena =3D 0, io_tqent =3D { tqent_task =3D { ta_link =3D { stqe_next =3D 0xfffff8016b2d0ad0 }, ta_pending =3D 0, ta_priority =3D 0, ta_func =3D 0xffffffff82185a60 , ta_context =3D 0xfffff805dcd9d718 }, tqent_func =3D 0xffffffff82225b60 , tqent_arg =3D 0xfffff805dcd9d3b8 }, io_trim_node =3D { avl_child =3D 0xfffff805dcd9d748, avl_pcb =3D 0 }, io_trim_link =3D { list_next =3D 0x0, list_prev =3D 0x0 } } (kgdb) p zio->io_vd $6 =3D { vdev_id =3D 2, vdev_guid =3D 13112130218511473524, vdev_guid_sum =3D 13112130218511473524, vdev_orig_guid =3D 0, vdev_asize =3D 4000782221312, vdev_min_asize =3D 3985729650688, vdev_max_asize =3D 4000782221312, vdev_ashift =3D 12, vdev_logical_ashift =3D 12, vdev_physical_ashift =3D 0, vdev_state =3D 7, vdev_prevstate =3D 0, vdev_ops =3D 0xffffffff822d2200, vdev_spa =3D 0xfffff801c29d1000, vdev_tsd =3D 0xfffff801638b2780, vdev_name_vp =3D 0x0, vdev_devid_vp =3D 0x0, vdev_top =3D 0xfffff803462c1800, vdev_parent =3D 0xfffff803462c1800, vdev_child =3D 0x0, vdev_children =3D 0, vdev_stat =3D { vs_timestamp =3D 941737635691796, vs_state =3D 0, vs_aux =3D 0, vs_alloc =3D 0, vs_space =3D 0, vs_dspace =3D 0, vs_rsize =3D 0, vs_esize =3D 0, vs_ops =3D 0xfffff804d86830e8, vs_bytes =3D 0xfffff804d8683118, vs_read_errors =3D 0, vs_write_errors =3D 0, vs_checksum_errors =3D 0, vs_self_healed =3D 0, vs_scan_removing =3D 0, vs_scan_processed =3D 0, vs_configured_ashift =3D 0, vs_logical_ashift =3D 0, vs_physical_ashift =3D 0, vs_fragmentation =3D 0 }, vdev_expanding =3D 0, vdev_reopening =3D 0, vdev_open_error =3D 0, vdev_open_thread =3D 0x0, vdev_crtxg =3D 4, vdev_ms_array =3D 0, vdev_ms_shift =3D 0, vdev_ms_count =3D 0, vdev_mg =3D 0x0, vdev_ms =3D 0x0, vdev_ms_list =3D { tl_lock =3D { lock_object =3D { lo_name =3D 0xffffffff822bbd27 "tl->tl_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0 }, sx_lock =3D 1 }, tl_offset =3D 808, tl_head =3D 0xfffff804d8683208 }, vdev_dtl_list =3D { tl_lock =3D { lock_object =3D { lo_name =3D 0xffffffff822bbd27 "tl->tl_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0 }, sx_lock =3D 1 }, tl_offset =3D 784, tl_head =3D 0xfffff804d8683250 }, vdev_txg_node =3D { tn_next =3D 0xfffff804d8683270, tn_member =3D 0xfffff804d8683290 "" }, vdev_remove_wanted =3D 0, vdev_probe_wanted =3D 0, vdev_config_dirty_node =3D { list_next =3D 0x0, list_prev =3D 0x0 }, vdev_state_dirty_node =3D { list_next =3D 0x0, list_prev =3D 0x0 }, vdev_deflate_ratio =3D 0, vdev_islog =3D 0, vdev_removing =3D 0, vdev_ishole =3D 0, vdev_top_zap =3D 0, vdev_dtl =3D 0xfffff804d86832e8, vdev_dtl_sm =3D 0xfffff80455b00200, vdev_dtl_node =3D { tn_next =3D 0xfffff804d8683310, tn_member =3D 0xfffff804d8683330 "" }, vdev_dtl_object =3D 65, vdev_psize =3D 4000786939904, vdev_wholedisk =3D 1, vdev_offline =3D 0, vdev_faulted =3D 0, vdev_degraded =3D 0, vdev_removed =3D 0, vdev_resilver_txg =3D 0, vdev_nparity =3D 0, vdev_path =3D 0xfffff802bd5acbe0 "/dev/label/backup42.eli", vdev_devid =3D 0x0, vdev_physpath =3D 0xfffff803be782ea0 "/dev/label/backup42.eli", vdev_fru =3D 0x0, vdev_not_present =3D 0, vdev_unspare =3D 0, vdev_nowritecache =3D 0, vdev_notrim =3D 1, vdev_checkremove =3D 0, vdev_forcefault =3D 0, vdev_splitting =3D 0, vdev_delayed_close =3D 0, vdev_tmpoffline =3D 0, vdev_detached =3D 0, vdev_cant_read =3D 0, vdev_cant_write =3D 0, vdev_isspare =3D 0, vdev_isl2cache =3D 0, vdev_queue =3D { vq_vdev =3D 0xfffff804d8683000, vq_class =3D 0xfffff804d86833e8, vq_active_tree =3D { avl_root =3D 0xfffff80290a71240, avl_compar =3D 0xffffffff8220b8a0 , avl_offset =3D 576, avl_numnodes =3D 64, avl_size =3D 952 }, vq_read_offset_tree =3D { avl_root =3D 0x0, avl_compar =3D 0xffffffff8220b8a0 , avl_offset =3D 600, avl_numnodes =3D 0, avl_size =3D 952 }, vq_write_offset_tree =3D { avl_root =3D 0x0, avl_compar =3D 0xffffffff8220b8a0 , avl_offset =3D 600, avl_numnodes =3D 0, avl_size =3D 952 }, vq_last_offset =3D 897918853120, vq_io_complete_ts =3D 941768555692856, vq_lock =3D { lock_object =3D { lo_name =3D 0xffffffff822bcc85 "vq->vq_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0 }, sx_lock =3D 18446735291472008448 }, vq_lastoffset =3D 33829150720 }, vdev_cache =3D { vc_offset_tree =3D { avl_root =3D 0x0, avl_compar =3D 0xffffffff82208310 , avl_offset =3D 24, avl_numnodes =3D 0, avl_size =3D 88 }, vc_lastused_tree =3D { avl_root =3D 0x0, avl_compar =3D 0xffffffff82208330 , avl_offset =3D 48, avl_numnodes =3D 0, avl_size =3D 88 }, vc_lock =3D { lock_object =3D { lo_name =3D 0xffffffff822bc53d "vc->vc_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0 }, sx_lock =3D 1 } } (kgdb) p *zio->io_vd->vdev_ops $7 =3D { vdev_op_open =3D 0xffffffff8225c940 , vdev_op_close =3D 0xffffffff8225d1b0 , vdev_op_asize =3D 0xffffffff82201670 , vdev_op_io_start =3D 0xffffffff8225d2a0 , vdev_op_io_done =3D 0xffffffff8225d470 , vdev_op_state_change =3D 0, vdev_op_hold =3D 0xffffffff8225d480 , vdev_op_rele =3D 0xffffffff8225d490 , vdev_op_type =3D 0xffffffff822d2240 "disk", vdev_op_leaf =3D 1 } (kgdb) frame 195 #195 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff801469a13b8) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870= 2870 vd->vdev_ops->vdev_op_io_start(zio); $8 =3D { io_bookmark =3D { zb_objset =3D 0, zb_object =3D 0, zb_level =3D 0, zb_blkid =3D 0 }, io_prop =3D { zp_checksum =3D ZIO_CHECKSUM_INHERIT, zp_compress =3D ZIO_COMPRESS_INHERIT, zp_type =3D DMU_OT_NONE, zp_level =3D 0 '\0', zp_copies =3D 0 '\0', zp_dedup =3D 0, zp_dedup_verify =3D 0, zp_nopwrite =3D 0 }, io_type =3D ZIO_TYPE_FREE, io_child_type =3D ZIO_CHILD_VDEV, io_cmd =3D 0, io_priority =3D ZIO_PRIORITY_TRIM, io_reexecute =3D 0 '\0', io_state =3D 0xfffff801469a1405 "\001", io_txg =3D 0, io_spa =3D 0xfffff801c29d1000, io_bp =3D 0x0, io_bp_override =3D 0x0, io_bp_copy =3D { blk_dva =3D 0xfffff801469a1428, blk_prop =3D 0, blk_pad =3D 0xfffff801469a1460, blk_phys_birth =3D 0, blk_birth =3D 0, blk_fill =3D 0, blk_cksum =3D { zc_word =3D 0xfffff801469a1488 } }, io_parent_list =3D { list_size =3D 48, list_offset =3D 16, list_head =3D { list_next =3D 0xfffff803c203a1c0, list_prev =3D 0xfffff803c203a1c0 } }, io_child_list =3D { list_size =3D 48, list_offset =3D 32, list_head =3D { list_next =3D 0xfffff801469a14d8, list_prev =3D 0xfffff801469a14d8 } }, io_walk_link =3D 0x0, io_logical =3D 0x0, io_transform_stack =3D 0x0, io_ready =3D 0, io_physdone =3D 0, io_done =3D 0, io_private =3D 0x0, io_prev_space_delta =3D 0, io_bp_orig =3D { blk_dva =3D 0xfffff801469a1528, blk_prop =3D 0, blk_pad =3D 0xfffff801469a1560, blk_phys_birth =3D 0, blk_birth =3D 0, blk_fill =3D 0, blk_cksum =3D { zc_word =3D 0xfffff801469a1588 } }, io_data =3D 0x0, io_orig_data =3D 0x0, io_size =3D 4096, io_orig_size =3D 4096, io_vd =3D 0xfffff804d8683000, io_vsd =3D 0x0, io_vsd_ops =3D 0x0, io_offset =3D 898323513344, io_timestamp =3D 941768555692856, io_target_timestamp =3D 0, io_queue_node =3D { avl_child =3D 0xfffff801469a15f8, avl_pcb =3D 18446735288905504249 }, io_offset_node =3D { avl_child =3D 0xfffff801469a1610, avl_pcb =3D 0 }, io_flags =3D 789633, io_stage =3D ZIO_STAGE_VDEV_IO_DONE, io_pipeline =3D 3080192, io_orig_flags =3D 525441, io_orig_stage =3D ZIO_STAGE_OPEN, io_orig_pipeline =3D 3080192, io_error =3D 45, io_child_error =3D 0xfffff801469a1644, io_children =3D 0xfffff801469a1658, io_child_count =3D 0, io_phys_children =3D 0, io_parent_count =3D 1, io_stall =3D 0x0, io_gang_leader =3D 0x0, io_gang_tree =3D 0x0, io_executor =3D 0xfffff80339d9d500, io_waiter =3D 0x0, io_lock =3D { lock_object =3D { lo_name =3D 0xffffffff822bf23e "zio->io_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0 }, sx_lock =3D 1 }, io_cv =3D { cv_description =3D 0xffffffff822bf24c "zio->io_cv", cv_waiters =3D 0 }, io_cksum_report =3D 0x0, io_ena =3D 0, io_tqent =3D { tqent_task =3D { ta_link =3D { stqe_next =3D 0x0 }, ta_pending =3D 0, ta_priority =3D 0, ta_func =3D 0, ta_context =3D 0x0 }, tqent_func =3D 0, tqent_arg =3D 0x0 }, io_trim_node =3D { avl_child =3D 0xfffff801469a1748, avl_pcb =3D 0 }, io_trim_link =3D { list_next =3D 0x0, list_prev =3D 0x0 } } (kgdb) p *zio->io_vd $9 =3D { vdev_id =3D 2, vdev_guid =3D 13112130218511473524, vdev_guid_sum =3D 13112130218511473524, vdev_orig_guid =3D 0, vdev_asize =3D 4000782221312, vdev_min_asize =3D 3985729650688, vdev_max_asize =3D 4000782221312, vdev_ashift =3D 12, vdev_logical_ashift =3D 12, vdev_physical_ashift =3D 0, vdev_state =3D 7, vdev_prevstate =3D 0, vdev_ops =3D 0xffffffff822d2200, vdev_spa =3D 0xfffff801c29d1000, vdev_tsd =3D 0xfffff801638b2780, vdev_name_vp =3D 0x0, vdev_devid_vp =3D 0x0, vdev_top =3D 0xfffff803462c1800, vdev_parent =3D 0xfffff803462c1800, vdev_child =3D 0x0, vdev_children =3D 0, vdev_stat =3D { vs_timestamp =3D 941737635691796, vs_state =3D 0, vs_aux =3D 0, vs_alloc =3D 0, vs_space =3D 0, vs_dspace =3D 0, vs_rsize =3D 0, vs_esize =3D 0, vs_ops =3D 0xfffff804d86830e8, vs_bytes =3D 0xfffff804d8683118, vs_read_errors =3D 0, vs_write_errors =3D 0, vs_checksum_errors =3D 0, vs_self_healed =3D 0, vs_scan_removing =3D 0, vs_scan_processed =3D 0, vs_configured_ashift =3D 0, vs_logical_ashift =3D 0, vs_physical_ashift =3D 0, vs_fragmentation =3D 0 }, vdev_expanding =3D 0, vdev_reopening =3D 0, vdev_open_error =3D 0, vdev_open_thread =3D 0x0, vdev_crtxg =3D 4, vdev_ms_array =3D 0, vdev_ms_shift =3D 0, vdev_ms_count =3D 0, vdev_mg =3D 0x0, vdev_ms =3D 0x0, vdev_ms_list =3D { tl_lock =3D { lock_object =3D { lo_name =3D 0xffffffff822bbd27 "tl->tl_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0 }, sx_lock =3D 1 }, tl_offset =3D 808, tl_head =3D 0xfffff804d8683208 }, vdev_dtl_list =3D { tl_lock =3D { lock_object =3D { lo_name =3D 0xffffffff822bbd27 "tl->tl_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0 }, sx_lock =3D 1 }, tl_offset =3D 784, tl_head =3D 0xfffff804d8683250 }, vdev_txg_node =3D { tn_next =3D 0xfffff804d8683270, tn_member =3D 0xfffff804d8683290 "" }, vdev_remove_wanted =3D 0, vdev_probe_wanted =3D 0, vdev_config_dirty_node =3D { list_next =3D 0x0, list_prev =3D 0x0 }, vdev_state_dirty_node =3D { list_next =3D 0x0, list_prev =3D 0x0 }, vdev_deflate_ratio =3D 0, vdev_islog =3D 0, vdev_removing =3D 0, vdev_ishole =3D 0, vdev_top_zap =3D 0, vdev_dtl =3D 0xfffff804d86832e8, vdev_dtl_sm =3D 0xfffff80455b00200, vdev_dtl_node =3D { tn_next =3D 0xfffff804d8683310, tn_member =3D 0xfffff804d8683330 "" }, vdev_dtl_object =3D 65, vdev_psize =3D 4000786939904, vdev_wholedisk =3D 1, vdev_offline =3D 0, vdev_faulted =3D 0, vdev_degraded =3D 0, vdev_removed =3D 0, vdev_resilver_txg =3D 0, vdev_nparity =3D 0, vdev_path =3D 0xfffff802bd5acbe0 "/dev/label/backup42.eli", vdev_devid =3D 0x0, vdev_physpath =3D 0xfffff803be782ea0 "/dev/label/backup42.eli", vdev_fru =3D 0x0, vdev_not_present =3D 0, vdev_unspare =3D 0, vdev_nowritecache =3D 0, vdev_notrim =3D 1, vdev_checkremove =3D 0, vdev_forcefault =3D 0, vdev_splitting =3D 0, vdev_delayed_close =3D 0, vdev_tmpoffline =3D 0, vdev_detached =3D 0, vdev_cant_read =3D 0, vdev_cant_write =3D 0, vdev_isspare =3D 0, vdev_isl2cache =3D 0, vdev_queue =3D { vq_vdev =3D 0xfffff804d8683000, vq_class =3D 0xfffff804d86833e8, vq_active_tree =3D { avl_root =3D 0xfffff80290a71240, avl_compar =3D 0xffffffff8220b8a0 , avl_offset =3D 576, avl_numnodes =3D 64, avl_size =3D 952 }, vq_read_offset_tree =3D { avl_root =3D 0x0, avl_compar =3D 0xffffffff8220b8a0 , avl_offset =3D 600, avl_numnodes =3D 0, avl_size =3D 952 }, vq_write_offset_tree =3D { avl_root =3D 0x0, avl_compar =3D 0xffffffff8220b8a0 , avl_offset =3D 600, avl_numnodes =3D 0, avl_size =3D 952 }, vq_last_offset =3D 897918853120, vq_io_complete_ts =3D 941768555692856, vq_lock =3D { lock_object =3D { lo_name =3D 0xffffffff822bcc85 "vq->vq_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0 }, sx_lock =3D 18446735291472008448 }, vq_lastoffset =3D 33829150720 }, vdev_cache =3D { vc_offset_tree =3D { avl_root =3D 0x0, avl_compar =3D 0xffffffff82208310 , avl_offset =3D 24, avl_numnodes =3D 0, avl_size =3D 88 }, vc_lastused_tree =3D { avl_root =3D 0x0, avl_compar =3D 0xffffffff82208330 , avl_offset =3D 48, avl_numnodes =3D 0, avl_size =3D 88 }, vc_lock =3D { lock_object =3D { lo_name =3D 0xffffffff822bc53d "vc->vc_lock", lo_flags =3D 40960000, lo_data =3D 0, lo_witness =3D 0x0 }, sx_lock =3D 1 } } (kgdb) p *zio->io_vd->vdev_ops $10 =3D { vdev_op_open =3D 0xffffffff8225c940 , vdev_op_close =3D 0xffffffff8225d1b0 , vdev_op_asize =3D 0xffffffff82201670 , vdev_op_io_start =3D 0xffffffff8225d2a0 , vdev_op_io_done =3D 0xffffffff8225d470 , vdev_op_state_change =3D 0, vdev_op_hold =3D 0xffffffff8225d480 , vdev_op_rele =3D 0xffffffff8225d490 , vdev_op_type =3D 0xffffffff822d2240 "disk", vdev_op_leaf =3D 1 } (kgdb) There's two.... How many more 'ya want? As you can see they're big! :) --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms000502000101080107030506 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp 3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5 vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/ o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6 eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+ JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ 3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0 FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG 1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5 c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA3MjAyMTU0MDNaME8GCSqGSIb3DQEJBDFCBEAf /DQGozH1edogymv1KenZatl7I/yg5HQOn/0oYOVAx/Y3YHkIYcmixn9aSFvDZ06GdImWCHDD olXFTHLIIuHIMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1 ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG 9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAGtE3Y34l LDmCA3bAov3FP/i6fydhSUC5NQ4SS1hrOon//1Hilmbt4Zs2JdKZGxOTt3jeitXxB9eKxWcV M01cbeHdgDCQ6KFH98jxS7sx/KTT+E48L1gZ9tbUmT93mKAHsL2DvnhN233ZZC2Oih3u+VXw XABv1F1b7l5L81vDHee66et3wpe6yI/JfbRZR4cNIU0HEQAlsd7tYS4yexIXeAPFIwgRZ+8a wQNLMg4W+9jUOFwYijJ7Tht/lIKPhoOfDXeFnh6Xr4LQO/ljbnOiStQ9Jn0qm1+8S4VW0rQF pI4+pUjlZoSIdPbZS1+edn+hD6NrZzdRP8mB+JYUUSQ7T/yGLhts4Ja6mTNi0fDxoboYpUGL 286X09dI29Q+EUMSXIDlbEsDFK8LKQbE3j4+FjcypvER1QLbFhZvXT+zuaTFhYgHTCB7a3V4 SS/yYgmK5V3+hbktKqu8HinHQyRmVZr+BiwmKZ/K4A/caAAPf6R4IsLWDuHDLOySznSNOnYy dW/wRnlksZdD07M3shRUEcxFsggoa8fz6RmRsvoM4ZM6tEMTEO09tCB8L0AoEL6uYyrMx8pT bAPvWGQT1nYhrhRxX8XtC9CX8VeGlPybjHTGMyqL1ES2AoMYqh0tWvn9IQ3YSocne4OMcWnk oMs4kkf+gahed/JRRq8QZKDO17QAAAAAAAA= --------------ms000502000101080107030506--