Date: Wed, 20 Jul 2016 16:54:03 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Panic on BETA1 in the ZFS subsystem Message-ID: <97cf5283-683b-83fd-c484-18c14973b065@denninger.net> In-Reply-To: <54e5974c-312e-c33c-ab83-9e1148618ddc@FreeBSD.org> References: <8f44bc09-1237-44d0-fe7a-7eb9cf4fe85b@denninger.net> <54e5974c-312e-c33c-ab83-9e1148618ddc@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format.
--------------ms000502000101080107030506
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
On 7/20/2016 16:32, Andriy Gapon wrote:
> On 20/07/2016 17:26, Karl Denninger wrote:
>> #185 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff80108754770=
)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2=
870
>> #186 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1=
556
>> #187 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized =
out>)
>> at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:8=
87
>> #188 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff804cfdf7770)=
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2=
895
>> #189 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1=
556
>> #190 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff804cfdf7770=
)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2=
870
>> #191 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1=
556
>> #192 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized =
out>)
>> at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:8=
87
>> #193 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff801469a13b8)=
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2=
895
>> #194 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1=
556
>> #195 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff801469a13b8=
)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2=
870
>> #196 0xffffffff82225d96 in zio_execute (zio=3D<value optimized out>)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1=
556
>> #197 0xffffffff8220c72d in vdev_queue_io_done (zio=3D<value optimized =
out>)
>> at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:8=
87
>> #198 0xffffffff82229719 in zio_vdev_io_done (zio=3D0xfffff805dcd9d3b8)=
> I see how this pseudo-recursion can happen in theory, but not why it ha=
s
> happened. It would interesting to see 3 or 4 of zio-s printed,
> including the top-most one at 0xfffff805dcd9d3b8.
>
> Something like:
>
> set print pretty
> p *zio
> p *zio->io_vd
> p *zio->io_vd->vdev_ops
>
> in several frames.
>
There was a clone from a previous boot environment embedded in the
snapshot list, but I don't believe it was (directly) the cause as
several backups have taken place since it was taken, and none resulted
in the panic.
I had this problem some time ago in which on the root pool/filesystem,
and ONLY on the root, I would get a panic after several days uptime
*and* the disk structure itself was ok but the incore was not. It would
manifest as a snapshot you could "cd" into but not see anything once
inside, and an attempt to destroy it would fail. If you attempted a zfs
send that touched that snapshot you got a panic.
The PR that I referenced above apparently was related, and the patch
that came out of it appears to have stopped it on 10.2, as once I put
that on I didn't have it happen again. It was consistent enough that I
actually killed automated backups on the machines involved, as it would
nail you without warning on a roughly a one-week interval.
That appears to be roughly where I am again.... about one week into
running BETA1, and "boom". But this one IS different in that this time
an immediate repeat (which used to succeed as the disk was fine)
resulted in an immediate second panic.
I've removed the snapshots on that filesystem (on the theory that the
previous error wasn't really fixed but rather papered over) and run a
backup, which succeeded. If I get another one I'll know it's a new probl=
em.
Let me see what I can come up with on some frame dumps.
Here's the top one...
(kgdb) p *zio
$5 =3D {
io_bookmark =3D {
zb_objset =3D 0,
zb_object =3D 0,
zb_level =3D 0,
zb_blkid =3D 0
},
io_prop =3D {
zp_checksum =3D ZIO_CHECKSUM_INHERIT,
zp_compress =3D ZIO_COMPRESS_INHERIT,
zp_type =3D DMU_OT_NONE,
zp_level =3D 0 '\0',
zp_copies =3D 0 '\0',
zp_dedup =3D 0,
zp_dedup_verify =3D 0,
zp_nopwrite =3D 0
},
io_type =3D ZIO_TYPE_FREE,
io_child_type =3D ZIO_CHILD_VDEV,
io_cmd =3D 0,
io_priority =3D ZIO_PRIORITY_TRIM,
io_reexecute =3D 0 '\0',
io_state =3D 0xfffff805dcd9d405 "\001",
io_txg =3D 0,
io_spa =3D 0xfffff801c29d1000,
io_bp =3D 0x0,
io_bp_override =3D 0x0,
io_bp_copy =3D {
blk_dva =3D 0xfffff805dcd9d428,
blk_prop =3D 0,
blk_pad =3D 0xfffff805dcd9d460,
blk_phys_birth =3D 0,
blk_birth =3D 0,
blk_fill =3D 0,
blk_cksum =3D {
zc_word =3D 0xfffff805dcd9d488
}
},
io_parent_list =3D {
list_size =3D 48,
list_offset =3D 16,
list_head =3D {
list_next =3D 0xfffff8042bf35340,
list_prev =3D 0xfffff8042bf35340
}
},
io_child_list =3D {
list_size =3D 48,
list_offset =3D 32,
list_head =3D {
list_next =3D 0xfffff805dcd9d4d8,
list_prev =3D 0xfffff805dcd9d4d8
}
},
io_walk_link =3D 0x0,
io_logical =3D 0x0,
io_transform_stack =3D 0x0,
io_ready =3D 0,
io_physdone =3D 0,
io_done =3D 0,
io_private =3D 0x0,
io_prev_space_delta =3D 0,
io_bp_orig =3D {
blk_dva =3D 0xfffff805dcd9d528,
blk_prop =3D 0,
blk_pad =3D 0xfffff805dcd9d560,
blk_phys_birth =3D 0,
blk_birth =3D 0,
blk_fill =3D 0,
blk_cksum =3D {
zc_word =3D 0xfffff805dcd9d588
}
},
io_data =3D 0x0,
io_orig_data =3D 0x0,
io_size =3D 8192,
io_orig_size =3D 8192,
io_vd =3D 0xfffff804d8683000,
io_vsd =3D 0x0,
io_vsd_ops =3D 0x0,
io_offset =3D 897895682048,
io_timestamp =3D 941768555692856,
io_target_timestamp =3D 0,
io_queue_node =3D {
avl_child =3D 0xfffff805dcd9d5f8,
avl_pcb =3D 18446735278553267561
},
io_offset_node =3D {
avl_child =3D 0xfffff805dcd9d610,
avl_pcb =3D 0
},
io_flags =3D 789633,
io_stage =3D ZIO_STAGE_VDEV_IO_DONE,
io_pipeline =3D 3080192,
io_orig_flags =3D 525441,
io_orig_stage =3D ZIO_STAGE_OPEN,
io_orig_pipeline =3D 3080192,
io_error =3D 45,
io_child_error =3D 0xfffff805dcd9d644,
io_children =3D 0xfffff805dcd9d658,
io_child_count =3D 0,
io_phys_children =3D 0,
io_parent_count =3D 1,
io_stall =3D 0x0,
io_gang_leader =3D 0x0,
io_gang_tree =3D 0x0,
io_executor =3D 0xfffff80339d9d500,
io_waiter =3D 0x0,
io_lock =3D {
lock_object =3D {
lo_name =3D 0xffffffff822bf23e "zio->io_lock",
lo_flags =3D 40960000,
lo_data =3D 0,
lo_witness =3D 0x0
},
sx_lock =3D 1
},
io_cv =3D {
cv_description =3D 0xffffffff822bf24c "zio->io_cv",
cv_waiters =3D 0
},
io_cksum_report =3D 0x0,
io_ena =3D 0,
io_tqent =3D {
tqent_task =3D {
ta_link =3D {
stqe_next =3D 0xfffff8016b2d0ad0
},
ta_pending =3D 0,
ta_priority =3D 0,
ta_func =3D 0xffffffff82185a60 <taskq_run_ent>,
ta_context =3D 0xfffff805dcd9d718
},
tqent_func =3D 0xffffffff82225b60 <zio_execute>,
tqent_arg =3D 0xfffff805dcd9d3b8
},
io_trim_node =3D {
avl_child =3D 0xfffff805dcd9d748,
avl_pcb =3D 0
},
io_trim_link =3D {
list_next =3D 0x0,
list_prev =3D 0x0
}
}
(kgdb) p zio->io_vd
$6 =3D {
vdev_id =3D 2,
vdev_guid =3D 13112130218511473524,
vdev_guid_sum =3D 13112130218511473524,
vdev_orig_guid =3D 0,
vdev_asize =3D 4000782221312,
vdev_min_asize =3D 3985729650688,
vdev_max_asize =3D 4000782221312,
vdev_ashift =3D 12,
vdev_logical_ashift =3D 12,
vdev_physical_ashift =3D 0,
vdev_state =3D 7,
vdev_prevstate =3D 0,
vdev_ops =3D 0xffffffff822d2200,
vdev_spa =3D 0xfffff801c29d1000,
vdev_tsd =3D 0xfffff801638b2780,
vdev_name_vp =3D 0x0,
vdev_devid_vp =3D 0x0,
vdev_top =3D 0xfffff803462c1800,
vdev_parent =3D 0xfffff803462c1800,
vdev_child =3D 0x0,
vdev_children =3D 0,
vdev_stat =3D {
vs_timestamp =3D 941737635691796,
vs_state =3D 0,
vs_aux =3D 0,
vs_alloc =3D 0,
vs_space =3D 0,
vs_dspace =3D 0,
vs_rsize =3D 0,
vs_esize =3D 0,
vs_ops =3D 0xfffff804d86830e8,
vs_bytes =3D 0xfffff804d8683118,
vs_read_errors =3D 0,
vs_write_errors =3D 0,
vs_checksum_errors =3D 0,
vs_self_healed =3D 0,
vs_scan_removing =3D 0,
vs_scan_processed =3D 0,
vs_configured_ashift =3D 0,
vs_logical_ashift =3D 0,
vs_physical_ashift =3D 0,
vs_fragmentation =3D 0
},
vdev_expanding =3D 0,
vdev_reopening =3D 0,
vdev_open_error =3D 0,
vdev_open_thread =3D 0x0,
vdev_crtxg =3D 4,
vdev_ms_array =3D 0,
vdev_ms_shift =3D 0,
vdev_ms_count =3D 0,
vdev_mg =3D 0x0,
vdev_ms =3D 0x0,
vdev_ms_list =3D {
tl_lock =3D {
lock_object =3D {
lo_name =3D 0xffffffff822bbd27 "tl->tl_lock",
lo_flags =3D 40960000,
lo_data =3D 0,
lo_witness =3D 0x0
},
sx_lock =3D 1
},
tl_offset =3D 808,
tl_head =3D 0xfffff804d8683208
},
vdev_dtl_list =3D {
tl_lock =3D {
lock_object =3D {
lo_name =3D 0xffffffff822bbd27 "tl->tl_lock",
lo_flags =3D 40960000,
lo_data =3D 0,
lo_witness =3D 0x0
},
sx_lock =3D 1
},
tl_offset =3D 784,
tl_head =3D 0xfffff804d8683250
},
vdev_txg_node =3D {
tn_next =3D 0xfffff804d8683270,
tn_member =3D 0xfffff804d8683290 ""
},
vdev_remove_wanted =3D 0,
vdev_probe_wanted =3D 0,
vdev_config_dirty_node =3D {
list_next =3D 0x0,
list_prev =3D 0x0
},
vdev_state_dirty_node =3D {
list_next =3D 0x0,
list_prev =3D 0x0
},
vdev_deflate_ratio =3D 0,
vdev_islog =3D 0,
vdev_removing =3D 0,
vdev_ishole =3D 0,
vdev_top_zap =3D 0,
vdev_dtl =3D 0xfffff804d86832e8,
vdev_dtl_sm =3D 0xfffff80455b00200,
vdev_dtl_node =3D {
tn_next =3D 0xfffff804d8683310,
tn_member =3D 0xfffff804d8683330 ""
},
vdev_dtl_object =3D 65,
vdev_psize =3D 4000786939904,
vdev_wholedisk =3D 1,
vdev_offline =3D 0,
vdev_faulted =3D 0,
vdev_degraded =3D 0,
vdev_removed =3D 0,
vdev_resilver_txg =3D 0,
vdev_nparity =3D 0,
vdev_path =3D 0xfffff802bd5acbe0 "/dev/label/backup42.eli",
vdev_devid =3D 0x0,
vdev_physpath =3D 0xfffff803be782ea0 "/dev/label/backup42.eli",
vdev_fru =3D 0x0,
vdev_not_present =3D 0,
vdev_unspare =3D 0,
vdev_nowritecache =3D 0,
vdev_notrim =3D 1,
vdev_checkremove =3D 0,
vdev_forcefault =3D 0,
vdev_splitting =3D 0,
vdev_delayed_close =3D 0,
vdev_tmpoffline =3D 0,
vdev_detached =3D 0,
vdev_cant_read =3D 0,
vdev_cant_write =3D 0,
vdev_isspare =3D 0,
vdev_isl2cache =3D 0,
vdev_queue =3D {
vq_vdev =3D 0xfffff804d8683000,
vq_class =3D 0xfffff804d86833e8,
vq_active_tree =3D {
avl_root =3D 0xfffff80290a71240,
avl_compar =3D 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset =3D 576,
avl_numnodes =3D 64,
avl_size =3D 952
},
vq_read_offset_tree =3D {
avl_root =3D 0x0,
avl_compar =3D 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset =3D 600,
avl_numnodes =3D 0,
avl_size =3D 952
},
vq_write_offset_tree =3D {
avl_root =3D 0x0,
avl_compar =3D 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset =3D 600,
avl_numnodes =3D 0,
avl_size =3D 952
},
vq_last_offset =3D 897918853120,
vq_io_complete_ts =3D 941768555692856,
vq_lock =3D {
lock_object =3D {
lo_name =3D 0xffffffff822bcc85 "vq->vq_lock",
lo_flags =3D 40960000,
lo_data =3D 0,
lo_witness =3D 0x0
},
sx_lock =3D 18446735291472008448
},
vq_lastoffset =3D 33829150720
},
vdev_cache =3D {
vc_offset_tree =3D {
avl_root =3D 0x0,
avl_compar =3D 0xffffffff82208310 <vdev_cache_offset_compare>,
avl_offset =3D 24,
avl_numnodes =3D 0,
avl_size =3D 88
},
vc_lastused_tree =3D {
avl_root =3D 0x0,
avl_compar =3D 0xffffffff82208330 <vdev_cache_lastused_compare>,
avl_offset =3D 48,
avl_numnodes =3D 0,
avl_size =3D 88
},
vc_lock =3D {
lock_object =3D {
lo_name =3D 0xffffffff822bc53d "vc->vc_lock",
lo_flags =3D 40960000,
lo_data =3D 0,
lo_witness =3D 0x0
},
sx_lock =3D 1
}
}
(kgdb) p *zio->io_vd->vdev_ops
$7 =3D {
vdev_op_open =3D 0xffffffff8225c940 <vdev_geom_open>,
vdev_op_close =3D 0xffffffff8225d1b0 <vdev_geom_close>,
vdev_op_asize =3D 0xffffffff82201670 <vdev_default_asize>,
vdev_op_io_start =3D 0xffffffff8225d2a0 <vdev_geom_io_start>,
vdev_op_io_done =3D 0xffffffff8225d470 <vdev_geom_io_done>,
vdev_op_state_change =3D 0,
vdev_op_hold =3D 0xffffffff8225d480 <vdev_geom_hold>,
vdev_op_rele =3D 0xffffffff8225d490 <vdev_geom_rele>,
vdev_op_type =3D 0xffffffff822d2240 "disk",
vdev_op_leaf =3D 1
}
(kgdb) frame 195
#195 0xffffffff8222955d in zio_vdev_io_start (zio=3D0xfffff801469a13b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870=
2870 vd->vdev_ops->vdev_op_io_start(zio);
$8 =3D {
io_bookmark =3D {
zb_objset =3D 0,
zb_object =3D 0,
zb_level =3D 0,
zb_blkid =3D 0
},
io_prop =3D {
zp_checksum =3D ZIO_CHECKSUM_INHERIT,
zp_compress =3D ZIO_COMPRESS_INHERIT,
zp_type =3D DMU_OT_NONE,
zp_level =3D 0 '\0',
zp_copies =3D 0 '\0',
zp_dedup =3D 0,
zp_dedup_verify =3D 0,
zp_nopwrite =3D 0
},
io_type =3D ZIO_TYPE_FREE,
io_child_type =3D ZIO_CHILD_VDEV,
io_cmd =3D 0,
io_priority =3D ZIO_PRIORITY_TRIM,
io_reexecute =3D 0 '\0',
io_state =3D 0xfffff801469a1405 "\001",
io_txg =3D 0,
io_spa =3D 0xfffff801c29d1000,
io_bp =3D 0x0,
io_bp_override =3D 0x0,
io_bp_copy =3D {
blk_dva =3D 0xfffff801469a1428,
blk_prop =3D 0,
blk_pad =3D 0xfffff801469a1460,
blk_phys_birth =3D 0,
blk_birth =3D 0,
blk_fill =3D 0,
blk_cksum =3D {
zc_word =3D 0xfffff801469a1488
}
},
io_parent_list =3D {
list_size =3D 48,
list_offset =3D 16,
list_head =3D {
list_next =3D 0xfffff803c203a1c0,
list_prev =3D 0xfffff803c203a1c0
}
},
io_child_list =3D {
list_size =3D 48,
list_offset =3D 32,
list_head =3D {
list_next =3D 0xfffff801469a14d8,
list_prev =3D 0xfffff801469a14d8
}
},
io_walk_link =3D 0x0,
io_logical =3D 0x0,
io_transform_stack =3D 0x0,
io_ready =3D 0,
io_physdone =3D 0,
io_done =3D 0,
io_private =3D 0x0,
io_prev_space_delta =3D 0,
io_bp_orig =3D {
blk_dva =3D 0xfffff801469a1528,
blk_prop =3D 0,
blk_pad =3D 0xfffff801469a1560,
blk_phys_birth =3D 0,
blk_birth =3D 0,
blk_fill =3D 0,
blk_cksum =3D {
zc_word =3D 0xfffff801469a1588
}
},
io_data =3D 0x0,
io_orig_data =3D 0x0,
io_size =3D 4096,
io_orig_size =3D 4096,
io_vd =3D 0xfffff804d8683000,
io_vsd =3D 0x0,
io_vsd_ops =3D 0x0,
io_offset =3D 898323513344,
io_timestamp =3D 941768555692856,
io_target_timestamp =3D 0,
io_queue_node =3D {
avl_child =3D 0xfffff801469a15f8,
avl_pcb =3D 18446735288905504249
},
io_offset_node =3D {
avl_child =3D 0xfffff801469a1610,
avl_pcb =3D 0
},
io_flags =3D 789633,
io_stage =3D ZIO_STAGE_VDEV_IO_DONE,
io_pipeline =3D 3080192,
io_orig_flags =3D 525441,
io_orig_stage =3D ZIO_STAGE_OPEN,
io_orig_pipeline =3D 3080192,
io_error =3D 45,
io_child_error =3D 0xfffff801469a1644,
io_children =3D 0xfffff801469a1658,
io_child_count =3D 0,
io_phys_children =3D 0,
io_parent_count =3D 1,
io_stall =3D 0x0,
io_gang_leader =3D 0x0,
io_gang_tree =3D 0x0,
io_executor =3D 0xfffff80339d9d500,
io_waiter =3D 0x0,
io_lock =3D {
lock_object =3D {
lo_name =3D 0xffffffff822bf23e "zio->io_lock",
lo_flags =3D 40960000,
lo_data =3D 0,
lo_witness =3D 0x0
},
sx_lock =3D 1
},
io_cv =3D {
cv_description =3D 0xffffffff822bf24c "zio->io_cv",
cv_waiters =3D 0
},
io_cksum_report =3D 0x0,
io_ena =3D 0,
io_tqent =3D {
tqent_task =3D {
ta_link =3D {
stqe_next =3D 0x0
},
ta_pending =3D 0,
ta_priority =3D 0,
ta_func =3D 0,
ta_context =3D 0x0
},
tqent_func =3D 0,
tqent_arg =3D 0x0
},
io_trim_node =3D {
avl_child =3D 0xfffff801469a1748,
avl_pcb =3D 0
},
io_trim_link =3D {
list_next =3D 0x0,
list_prev =3D 0x0
}
}
(kgdb) p *zio->io_vd
$9 =3D {
vdev_id =3D 2,
vdev_guid =3D 13112130218511473524,
vdev_guid_sum =3D 13112130218511473524,
vdev_orig_guid =3D 0,
vdev_asize =3D 4000782221312,
vdev_min_asize =3D 3985729650688,
vdev_max_asize =3D 4000782221312,
vdev_ashift =3D 12,
vdev_logical_ashift =3D 12,
vdev_physical_ashift =3D 0,
vdev_state =3D 7,
vdev_prevstate =3D 0,
vdev_ops =3D 0xffffffff822d2200,
vdev_spa =3D 0xfffff801c29d1000,
vdev_tsd =3D 0xfffff801638b2780,
vdev_name_vp =3D 0x0,
vdev_devid_vp =3D 0x0,
vdev_top =3D 0xfffff803462c1800,
vdev_parent =3D 0xfffff803462c1800,
vdev_child =3D 0x0,
vdev_children =3D 0,
vdev_stat =3D {
vs_timestamp =3D 941737635691796,
vs_state =3D 0,
vs_aux =3D 0,
vs_alloc =3D 0,
vs_space =3D 0,
vs_dspace =3D 0,
vs_rsize =3D 0,
vs_esize =3D 0,
vs_ops =3D 0xfffff804d86830e8,
vs_bytes =3D 0xfffff804d8683118,
vs_read_errors =3D 0,
vs_write_errors =3D 0,
vs_checksum_errors =3D 0,
vs_self_healed =3D 0,
vs_scan_removing =3D 0,
vs_scan_processed =3D 0,
vs_configured_ashift =3D 0,
vs_logical_ashift =3D 0,
vs_physical_ashift =3D 0,
vs_fragmentation =3D 0
},
vdev_expanding =3D 0,
vdev_reopening =3D 0,
vdev_open_error =3D 0,
vdev_open_thread =3D 0x0,
vdev_crtxg =3D 4,
vdev_ms_array =3D 0,
vdev_ms_shift =3D 0,
vdev_ms_count =3D 0,
vdev_mg =3D 0x0,
vdev_ms =3D 0x0,
vdev_ms_list =3D {
tl_lock =3D {
lock_object =3D {
lo_name =3D 0xffffffff822bbd27 "tl->tl_lock",
lo_flags =3D 40960000,
lo_data =3D 0,
lo_witness =3D 0x0
},
sx_lock =3D 1
},
tl_offset =3D 808,
tl_head =3D 0xfffff804d8683208
},
vdev_dtl_list =3D {
tl_lock =3D {
lock_object =3D {
lo_name =3D 0xffffffff822bbd27 "tl->tl_lock",
lo_flags =3D 40960000,
lo_data =3D 0,
lo_witness =3D 0x0
},
sx_lock =3D 1
},
tl_offset =3D 784,
tl_head =3D 0xfffff804d8683250
},
vdev_txg_node =3D {
tn_next =3D 0xfffff804d8683270,
tn_member =3D 0xfffff804d8683290 ""
},
vdev_remove_wanted =3D 0,
vdev_probe_wanted =3D 0,
vdev_config_dirty_node =3D {
list_next =3D 0x0,
list_prev =3D 0x0
},
vdev_state_dirty_node =3D {
list_next =3D 0x0,
list_prev =3D 0x0
},
vdev_deflate_ratio =3D 0,
vdev_islog =3D 0,
vdev_removing =3D 0,
vdev_ishole =3D 0,
vdev_top_zap =3D 0,
vdev_dtl =3D 0xfffff804d86832e8,
vdev_dtl_sm =3D 0xfffff80455b00200,
vdev_dtl_node =3D {
tn_next =3D 0xfffff804d8683310,
tn_member =3D 0xfffff804d8683330 ""
},
vdev_dtl_object =3D 65,
vdev_psize =3D 4000786939904,
vdev_wholedisk =3D 1,
vdev_offline =3D 0,
vdev_faulted =3D 0,
vdev_degraded =3D 0,
vdev_removed =3D 0,
vdev_resilver_txg =3D 0,
vdev_nparity =3D 0,
vdev_path =3D 0xfffff802bd5acbe0 "/dev/label/backup42.eli",
vdev_devid =3D 0x0,
vdev_physpath =3D 0xfffff803be782ea0 "/dev/label/backup42.eli",
vdev_fru =3D 0x0,
vdev_not_present =3D 0,
vdev_unspare =3D 0,
vdev_nowritecache =3D 0,
vdev_notrim =3D 1,
vdev_checkremove =3D 0,
vdev_forcefault =3D 0,
vdev_splitting =3D 0,
vdev_delayed_close =3D 0,
vdev_tmpoffline =3D 0,
vdev_detached =3D 0,
vdev_cant_read =3D 0,
vdev_cant_write =3D 0,
vdev_isspare =3D 0,
vdev_isl2cache =3D 0,
vdev_queue =3D {
vq_vdev =3D 0xfffff804d8683000,
vq_class =3D 0xfffff804d86833e8,
vq_active_tree =3D {
avl_root =3D 0xfffff80290a71240,
avl_compar =3D 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset =3D 576,
avl_numnodes =3D 64,
avl_size =3D 952
},
vq_read_offset_tree =3D {
avl_root =3D 0x0,
avl_compar =3D 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset =3D 600,
avl_numnodes =3D 0,
avl_size =3D 952
},
vq_write_offset_tree =3D {
avl_root =3D 0x0,
avl_compar =3D 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset =3D 600,
avl_numnodes =3D 0,
avl_size =3D 952
},
vq_last_offset =3D 897918853120,
vq_io_complete_ts =3D 941768555692856,
vq_lock =3D {
lock_object =3D {
lo_name =3D 0xffffffff822bcc85 "vq->vq_lock",
lo_flags =3D 40960000,
lo_data =3D 0,
lo_witness =3D 0x0
},
sx_lock =3D 18446735291472008448
},
vq_lastoffset =3D 33829150720
},
vdev_cache =3D {
vc_offset_tree =3D {
avl_root =3D 0x0,
avl_compar =3D 0xffffffff82208310 <vdev_cache_offset_compare>,
avl_offset =3D 24,
avl_numnodes =3D 0,
avl_size =3D 88
},
vc_lastused_tree =3D {
avl_root =3D 0x0,
avl_compar =3D 0xffffffff82208330 <vdev_cache_lastused_compare>,
avl_offset =3D 48,
avl_numnodes =3D 0,
avl_size =3D 88
},
vc_lock =3D {
lock_object =3D {
lo_name =3D 0xffffffff822bc53d "vc->vc_lock",
lo_flags =3D 40960000,
lo_data =3D 0,
lo_witness =3D 0x0
},
sx_lock =3D 1
}
}
(kgdb) p *zio->io_vd->vdev_ops
$10 =3D {
vdev_op_open =3D 0xffffffff8225c940 <vdev_geom_open>,
vdev_op_close =3D 0xffffffff8225d1b0 <vdev_geom_close>,
vdev_op_asize =3D 0xffffffff82201670 <vdev_default_asize>,
vdev_op_io_start =3D 0xffffffff8225d2a0 <vdev_geom_io_start>,
vdev_op_io_done =3D 0xffffffff8225d470 <vdev_geom_io_done>,
vdev_op_state_change =3D 0,
vdev_op_hold =3D 0xffffffff8225d480 <vdev_geom_hold>,
vdev_op_rele =3D 0xffffffff8225d490 <vdev_geom_rele>,
vdev_op_type =3D 0xffffffff822d2240 "disk",
vdev_op_leaf =3D 1
}
(kgdb)
There's two....
How many more 'ya want? As you can see they're big! :)
--=20
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
--------------ms000502000101080107030506
Content-Type: application/pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC
Bl8wggZbMIIEQ6ADAgECAgEpMA0GCSqGSIb3DQEBCwUAMIGQMQswCQYDVQQGEwJVUzEQMA4G
A1UECBMHRmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3Rl
bXMgTExDMRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhND
dWRhIFN5c3RlbXMgTExDIENBMB4XDTE1MDQyMTAyMjE1OVoXDTIwMDQxOTAyMjE1OVowWjEL
MAkGA1UEBhMCVVMxEDAOBgNVBAgTB0Zsb3JpZGExGTAXBgNVBAoTEEN1ZGEgU3lzdGVtcyBM
TEMxHjAcBgNVBAMTFUthcmwgRGVubmluZ2VyIChPQ1NQKTCCAiIwDQYJKoZIhvcNAQEBBQAD
ggIPADCCAgoCggIBALmEWPhAdphrWd4K5VTvE5pxL3blRQPyGF3ApjUjgtavqU1Y8pbI3Byg
XDj2/Uz9Si8XVj/kNbKEjkRh5SsNvx3Fc0oQ1uVjyCq7zC/kctF7yLzQbvWnU4grAPZ3IuAp
3/fFxIVaXpxEdKmyZAVDhk9az+IgHH43rdJRIMzxJ5vqQMb+n2EjadVqiGPbtG9aZEImlq7f
IYDTnKyToi23PAnkPwwT+q1IkI2DTvf2jzWrhLR5DTX0fUYC0nxlHWbjgpiapyJWtR7K2YQO
aevQb/3vN9gSojT2h+cBem7QIj6U69rEYcEDvPyCMXEV9VcXdcmW42LSRsPvZcBHFkWAJqMZ
Myiz4kumaP+s+cIDaXitR/szoqDKGSHM4CPAZV9Yh8asvxQL5uDxz5wvLPgS5yS8K/o7zDR5
vNkMCyfYQuR6PAJxVOk5Arqvj9lfP3JSVapwbr01CoWDBkpuJlKfpQIEeC/pcCBKknllbMYq
yHBO2TipLyO5Ocd1nhN/nOsO+C+j31lQHfOMRZaPQykXVPWG5BbhWT7ttX4vy5hOW6yJgeT/
o3apynlp1cEavkQRS8uJHoQszF6KIrQMID/JfySWvVQ4ksnfzwB2lRomrdrwnQ4eG/HBS+0l
eozwOJNDIBlAP+hLe8A5oWZgooIIK/SulUAsfI6Sgd8dTZTTYmlhAgMBAAGjgfQwgfEwNwYI
KwYBBQUHAQEEKzApMCcGCCsGAQUFBzABhhtodHRwOi8vY3VkYXN5c3RlbXMubmV0Ojg4ODgw
CQYDVR0TBAIwADARBglghkgBhvhCAQEEBAMCBaAwCwYDVR0PBAQDAgXgMCwGCWCGSAGG+EIB
DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQUxRyULenJaFwX
RtT79aNmIB/u5VkwHwYDVR0jBBgwFoAUJHGbnYV9/N3dvbDKkpQDofrTbTUwHQYDVR0RBBYw
FIESa2FybEBkZW5uaW5nZXIubmV0MA0GCSqGSIb3DQEBCwUAA4ICAQBPf3cYtmKowmGIYsm6
eBinJu7QVWvxi1vqnBz3KE+HapqoIZS8/PolB/hwiY0UAE1RsjBJ7yEjihVRwummSBvkoOyf
G30uPn4yg4vbJkR9lTz8d21fPshWETa6DBh2jx2Qf13LZpr3Pj2fTtlu6xMYKzg7cSDgd2bO
sJGH/rcvva9Spkx5Vfq0RyOrYph9boshRN3D4tbWgBAcX9POdXCVfJONDxhfBuPHsJ6vEmPb
An+XL5Yl26XYFPiODQ+Qbk44Ot1kt9s7oS3dVUrh92Qv0G3J3DF+Vt6C15nED+f+bk4gScu+
JHT7RjEmfa18GT8DcT//D1zEke1Ymhb41JH+GyZchDRWtjxsS5OBFMzrju7d264zJUFtX7iJ
3xvpKN7VcZKNtB6dLShj3v/XDsQVQWXmR/1YKWZ93C3LpRs2Y5nYdn6gEOpL/WfQFThtfnat
HNc7fNs5vjotaYpBl5H8+VCautKbGOs219uQbhGZLYTv6okuKcY8W+4EJEtK0xB08vqr9Jd0
FS9MGjQE++GWo+5eQxFt6nUENHbVYnsr6bYPQsZH0CRNycgTG9MwY/UIXOf4W034UpR82TBG
1LiMsYfb8ahQJhs3wdf1nzipIjRwoZKT1vGXh/cj3gwSr64GfenURBxaFZA5O1acOZUjPrRT
n3ci4McYW/0WVVA3lDGCBRMwggUPAgEBMIGWMIGQMQswCQYDVQQGEwJVUzEQMA4GA1UECBMH
RmxvcmlkYTESMBAGA1UEBxMJTmljZXZpbGxlMRkwFwYDVQQKExBDdWRhIFN5c3RlbXMgTExD
MRwwGgYDVQQDExNDdWRhIFN5c3RlbXMgTExDIENBMSIwIAYJKoZIhvcNAQkBFhNDdWRhIFN5
c3RlbXMgTExDIENBAgEpMA0GCWCGSAFlAwQCAwUAoIICTTAYBgkqhkiG9w0BCQMxCwYJKoZI
hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNjA3MjAyMTU0MDNaME8GCSqGSIb3DQEJBDFCBEAf
/DQGozH1edogymv1KenZatl7I/yg5HQOn/0oYOVAx/Y3YHkIYcmixn9aSFvDZ06GdImWCHDD
olXFTHLIIuHIMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAK
BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI
KoZIhvcNAwICASgwgacGCSsGAQQBgjcQBDGBmTCBljCBkDELMAkGA1UEBhMCVVMxEDAOBgNV
BAgTB0Zsb3JpZGExEjAQBgNVBAcTCU5pY2V2aWxsZTEZMBcGA1UEChMQQ3VkYSBTeXN0ZW1z
IExMQzEcMBoGA1UEAxMTQ3VkYSBTeXN0ZW1zIExMQyBDQTEiMCAGCSqGSIb3DQEJARYTQ3Vk
YSBTeXN0ZW1zIExMQyBDQQIBKTCBqQYLKoZIhvcNAQkQAgsxgZmggZYwgZAxCzAJBgNVBAYT
AlVTMRAwDgYDVQQIEwdGbG9yaWRhMRIwEAYDVQQHEwlOaWNldmlsbGUxGTAXBgNVBAoTEEN1
ZGEgU3lzdGVtcyBMTEMxHDAaBgNVBAMTE0N1ZGEgU3lzdGVtcyBMTEMgQ0ExIjAgBgkqhkiG
9w0BCQEWE0N1ZGEgU3lzdGVtcyBMTEMgQ0ECASkwDQYJKoZIhvcNAQEBBQAEggIAGtE3Y34l
LDmCA3bAov3FP/i6fydhSUC5NQ4SS1hrOon//1Hilmbt4Zs2JdKZGxOTt3jeitXxB9eKxWcV
M01cbeHdgDCQ6KFH98jxS7sx/KTT+E48L1gZ9tbUmT93mKAHsL2DvnhN233ZZC2Oih3u+VXw
XABv1F1b7l5L81vDHee66et3wpe6yI/JfbRZR4cNIU0HEQAlsd7tYS4yexIXeAPFIwgRZ+8a
wQNLMg4W+9jUOFwYijJ7Tht/lIKPhoOfDXeFnh6Xr4LQO/ljbnOiStQ9Jn0qm1+8S4VW0rQF
pI4+pUjlZoSIdPbZS1+edn+hD6NrZzdRP8mB+JYUUSQ7T/yGLhts4Ja6mTNi0fDxoboYpUGL
286X09dI29Q+EUMSXIDlbEsDFK8LKQbE3j4+FjcypvER1QLbFhZvXT+zuaTFhYgHTCB7a3V4
SS/yYgmK5V3+hbktKqu8HinHQyRmVZr+BiwmKZ/K4A/caAAPf6R4IsLWDuHDLOySznSNOnYy
dW/wRnlksZdD07M3shRUEcxFsggoa8fz6RmRsvoM4ZM6tEMTEO09tCB8L0AoEL6uYyrMx8pT
bAPvWGQT1nYhrhRxX8XtC9CX8VeGlPybjHTGMyqL1ES2AoMYqh0tWvn9IQ3YSocne4OMcWnk
oMs4kkf+gahed/JRRq8QZKDO17QAAAAAAAA=
--------------ms000502000101080107030506--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?97cf5283-683b-83fd-c484-18c14973b065>