Date: Wed, 20 Jul 2016 16:54:03 -0500 From: Karl Denninger <karl@denninger.net> To: freebsd-stable@freebsd.org Subject: Re: Panic on BETA1 in the ZFS subsystem Message-ID: <97cf5283-683b-83fd-c484-18c14973b065@denninger.net> In-Reply-To: <54e5974c-312e-c33c-ab83-9e1148618ddc@FreeBSD.org> References: <8f44bc09-1237-44d0-fe7a-7eb9cf4fe85b@denninger.net> <54e5974c-312e-c33c-ab83-9e1148618ddc@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
On 7/20/2016 16:32, Andriy Gapon wrote:
> On 20/07/2016 17:26, Karl Denninger wrote:
>> #185 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff80108754770)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #186 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #187 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
>> at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #188 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff804cfdf7770)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #189 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #190 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff804cfdf7770)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #191 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #192 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
>> at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #193 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff801469a13b8)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2895
>> #194 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #195 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801469a13b8)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
>> #196 0xffffffff82225d96 in zio_execute (zio=<value optimized out>)
>> at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1556
>> #197 0xffffffff8220c72d in vdev_queue_io_done (zio=<value optimized out>)
>> at
>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:887
>> #198 0xffffffff82229719 in zio_vdev_io_done (zio=0xfffff805dcd9d3b8)
> I see how this pseudo-recursion can happen in theory, but not why it has
> happened. It would interesting to see 3 or 4 of zio-s printed,
> including the top-most one at 0xfffff805dcd9d3b8.
>
> Something like:
>
> set print pretty
> p *zio
> p *zio->io_vd
> p *zio->io_vd->vdev_ops
>
> in several frames.
>
There was a clone from a previous boot environment embedded in the
snapshot list, but I don't believe it was (directly) the cause as
several backups have taken place since it was taken, and none resulted
in the panic.
I had this problem some time ago in which on the root pool/filesystem,
and ONLY on the root, I would get a panic after several days uptime
*and* the disk structure itself was ok but the incore was not. It would
manifest as a snapshot you could "cd" into but not see anything once
inside, and an attempt to destroy it would fail. If you attempted a zfs
send that touched that snapshot you got a panic.
The PR that I referenced above apparently was related, and the patch
that came out of it appears to have stopped it on 10.2, as once I put
that on I didn't have it happen again. It was consistent enough that I
actually killed automated backups on the machines involved, as it would
nail you without warning on a roughly a one-week interval.
That appears to be roughly where I am again.... about one week into
running BETA1, and "boom". But this one IS different in that this time
an immediate repeat (which used to succeed as the disk was fine)
resulted in an immediate second panic.
I've removed the snapshots on that filesystem (on the theory that the
previous error wasn't really fixed but rather papered over) and run a
backup, which succeeded. If I get another one I'll know it's a new problem.
Let me see what I can come up with on some frame dumps.
Here's the top one...
(kgdb) p *zio
$5 = {
io_bookmark = {
zb_objset = 0,
zb_object = 0,
zb_level = 0,
zb_blkid = 0
},
io_prop = {
zp_checksum = ZIO_CHECKSUM_INHERIT,
zp_compress = ZIO_COMPRESS_INHERIT,
zp_type = DMU_OT_NONE,
zp_level = 0 '\0',
zp_copies = 0 '\0',
zp_dedup = 0,
zp_dedup_verify = 0,
zp_nopwrite = 0
},
io_type = ZIO_TYPE_FREE,
io_child_type = ZIO_CHILD_VDEV,
io_cmd = 0,
io_priority = ZIO_PRIORITY_TRIM,
io_reexecute = 0 '\0',
io_state = 0xfffff805dcd9d405 "\001",
io_txg = 0,
io_spa = 0xfffff801c29d1000,
io_bp = 0x0,
io_bp_override = 0x0,
io_bp_copy = {
blk_dva = 0xfffff805dcd9d428,
blk_prop = 0,
blk_pad = 0xfffff805dcd9d460,
blk_phys_birth = 0,
blk_birth = 0,
blk_fill = 0,
blk_cksum = {
zc_word = 0xfffff805dcd9d488
}
},
io_parent_list = {
list_size = 48,
list_offset = 16,
list_head = {
list_next = 0xfffff8042bf35340,
list_prev = 0xfffff8042bf35340
}
},
io_child_list = {
list_size = 48,
list_offset = 32,
list_head = {
list_next = 0xfffff805dcd9d4d8,
list_prev = 0xfffff805dcd9d4d8
}
},
io_walk_link = 0x0,
io_logical = 0x0,
io_transform_stack = 0x0,
io_ready = 0,
io_physdone = 0,
io_done = 0,
io_private = 0x0,
io_prev_space_delta = 0,
io_bp_orig = {
blk_dva = 0xfffff805dcd9d528,
blk_prop = 0,
blk_pad = 0xfffff805dcd9d560,
blk_phys_birth = 0,
blk_birth = 0,
blk_fill = 0,
blk_cksum = {
zc_word = 0xfffff805dcd9d588
}
},
io_data = 0x0,
io_orig_data = 0x0,
io_size = 8192,
io_orig_size = 8192,
io_vd = 0xfffff804d8683000,
io_vsd = 0x0,
io_vsd_ops = 0x0,
io_offset = 897895682048,
io_timestamp = 941768555692856,
io_target_timestamp = 0,
io_queue_node = {
avl_child = 0xfffff805dcd9d5f8,
avl_pcb = 18446735278553267561
},
io_offset_node = {
avl_child = 0xfffff805dcd9d610,
avl_pcb = 0
},
io_flags = 789633,
io_stage = ZIO_STAGE_VDEV_IO_DONE,
io_pipeline = 3080192,
io_orig_flags = 525441,
io_orig_stage = ZIO_STAGE_OPEN,
io_orig_pipeline = 3080192,
io_error = 45,
io_child_error = 0xfffff805dcd9d644,
io_children = 0xfffff805dcd9d658,
io_child_count = 0,
io_phys_children = 0,
io_parent_count = 1,
io_stall = 0x0,
io_gang_leader = 0x0,
io_gang_tree = 0x0,
io_executor = 0xfffff80339d9d500,
io_waiter = 0x0,
io_lock = {
lock_object = {
lo_name = 0xffffffff822bf23e "zio->io_lock",
lo_flags = 40960000,
lo_data = 0,
lo_witness = 0x0
},
sx_lock = 1
},
io_cv = {
cv_description = 0xffffffff822bf24c "zio->io_cv",
cv_waiters = 0
},
io_cksum_report = 0x0,
io_ena = 0,
io_tqent = {
tqent_task = {
ta_link = {
stqe_next = 0xfffff8016b2d0ad0
},
ta_pending = 0,
ta_priority = 0,
ta_func = 0xffffffff82185a60 <taskq_run_ent>,
ta_context = 0xfffff805dcd9d718
},
tqent_func = 0xffffffff82225b60 <zio_execute>,
tqent_arg = 0xfffff805dcd9d3b8
},
io_trim_node = {
avl_child = 0xfffff805dcd9d748,
avl_pcb = 0
},
io_trim_link = {
list_next = 0x0,
list_prev = 0x0
}
}
(kgdb) p zio->io_vd
$6 = {
vdev_id = 2,
vdev_guid = 13112130218511473524,
vdev_guid_sum = 13112130218511473524,
vdev_orig_guid = 0,
vdev_asize = 4000782221312,
vdev_min_asize = 3985729650688,
vdev_max_asize = 4000782221312,
vdev_ashift = 12,
vdev_logical_ashift = 12,
vdev_physical_ashift = 0,
vdev_state = 7,
vdev_prevstate = 0,
vdev_ops = 0xffffffff822d2200,
vdev_spa = 0xfffff801c29d1000,
vdev_tsd = 0xfffff801638b2780,
vdev_name_vp = 0x0,
vdev_devid_vp = 0x0,
vdev_top = 0xfffff803462c1800,
vdev_parent = 0xfffff803462c1800,
vdev_child = 0x0,
vdev_children = 0,
vdev_stat = {
vs_timestamp = 941737635691796,
vs_state = 0,
vs_aux = 0,
vs_alloc = 0,
vs_space = 0,
vs_dspace = 0,
vs_rsize = 0,
vs_esize = 0,
vs_ops = 0xfffff804d86830e8,
vs_bytes = 0xfffff804d8683118,
vs_read_errors = 0,
vs_write_errors = 0,
vs_checksum_errors = 0,
vs_self_healed = 0,
vs_scan_removing = 0,
vs_scan_processed = 0,
vs_configured_ashift = 0,
vs_logical_ashift = 0,
vs_physical_ashift = 0,
vs_fragmentation = 0
},
vdev_expanding = 0,
vdev_reopening = 0,
vdev_open_error = 0,
vdev_open_thread = 0x0,
vdev_crtxg = 4,
vdev_ms_array = 0,
vdev_ms_shift = 0,
vdev_ms_count = 0,
vdev_mg = 0x0,
vdev_ms = 0x0,
vdev_ms_list = {
tl_lock = {
lock_object = {
lo_name = 0xffffffff822bbd27 "tl->tl_lock",
lo_flags = 40960000,
lo_data = 0,
lo_witness = 0x0
},
sx_lock = 1
},
tl_offset = 808,
tl_head = 0xfffff804d8683208
},
vdev_dtl_list = {
tl_lock = {
lock_object = {
lo_name = 0xffffffff822bbd27 "tl->tl_lock",
lo_flags = 40960000,
lo_data = 0,
lo_witness = 0x0
},
sx_lock = 1
},
tl_offset = 784,
tl_head = 0xfffff804d8683250
},
vdev_txg_node = {
tn_next = 0xfffff804d8683270,
tn_member = 0xfffff804d8683290 ""
},
vdev_remove_wanted = 0,
vdev_probe_wanted = 0,
vdev_config_dirty_node = {
list_next = 0x0,
list_prev = 0x0
},
vdev_state_dirty_node = {
list_next = 0x0,
list_prev = 0x0
},
vdev_deflate_ratio = 0,
vdev_islog = 0,
vdev_removing = 0,
vdev_ishole = 0,
vdev_top_zap = 0,
vdev_dtl = 0xfffff804d86832e8,
vdev_dtl_sm = 0xfffff80455b00200,
vdev_dtl_node = {
tn_next = 0xfffff804d8683310,
tn_member = 0xfffff804d8683330 ""
},
vdev_dtl_object = 65,
vdev_psize = 4000786939904,
vdev_wholedisk = 1,
vdev_offline = 0,
vdev_faulted = 0,
vdev_degraded = 0,
vdev_removed = 0,
vdev_resilver_txg = 0,
vdev_nparity = 0,
vdev_path = 0xfffff802bd5acbe0 "/dev/label/backup42.eli",
vdev_devid = 0x0,
vdev_physpath = 0xfffff803be782ea0 "/dev/label/backup42.eli",
vdev_fru = 0x0,
vdev_not_present = 0,
vdev_unspare = 0,
vdev_nowritecache = 0,
vdev_notrim = 1,
vdev_checkremove = 0,
vdev_forcefault = 0,
vdev_splitting = 0,
vdev_delayed_close = 0,
vdev_tmpoffline = 0,
vdev_detached = 0,
vdev_cant_read = 0,
vdev_cant_write = 0,
vdev_isspare = 0,
vdev_isl2cache = 0,
vdev_queue = {
vq_vdev = 0xfffff804d8683000,
vq_class = 0xfffff804d86833e8,
vq_active_tree = {
avl_root = 0xfffff80290a71240,
avl_compar = 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset = 576,
avl_numnodes = 64,
avl_size = 952
},
vq_read_offset_tree = {
avl_root = 0x0,
avl_compar = 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset = 600,
avl_numnodes = 0,
avl_size = 952
},
vq_write_offset_tree = {
avl_root = 0x0,
avl_compar = 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset = 600,
avl_numnodes = 0,
avl_size = 952
},
vq_last_offset = 897918853120,
vq_io_complete_ts = 941768555692856,
vq_lock = {
lock_object = {
lo_name = 0xffffffff822bcc85 "vq->vq_lock",
lo_flags = 40960000,
lo_data = 0,
lo_witness = 0x0
},
sx_lock = 18446735291472008448
},
vq_lastoffset = 33829150720
},
vdev_cache = {
vc_offset_tree = {
avl_root = 0x0,
avl_compar = 0xffffffff82208310 <vdev_cache_offset_compare>,
avl_offset = 24,
avl_numnodes = 0,
avl_size = 88
},
vc_lastused_tree = {
avl_root = 0x0,
avl_compar = 0xffffffff82208330 <vdev_cache_lastused_compare>,
avl_offset = 48,
avl_numnodes = 0,
avl_size = 88
},
vc_lock = {
lock_object = {
lo_name = 0xffffffff822bc53d "vc->vc_lock",
lo_flags = 40960000,
lo_data = 0,
lo_witness = 0x0
},
sx_lock = 1
}
}
(kgdb) p *zio->io_vd->vdev_ops
$7 = {
vdev_op_open = 0xffffffff8225c940 <vdev_geom_open>,
vdev_op_close = 0xffffffff8225d1b0 <vdev_geom_close>,
vdev_op_asize = 0xffffffff82201670 <vdev_default_asize>,
vdev_op_io_start = 0xffffffff8225d2a0 <vdev_geom_io_start>,
vdev_op_io_done = 0xffffffff8225d470 <vdev_geom_io_done>,
vdev_op_state_change = 0,
vdev_op_hold = 0xffffffff8225d480 <vdev_geom_hold>,
vdev_op_rele = 0xffffffff8225d490 <vdev_geom_rele>,
vdev_op_type = 0xffffffff822d2240 "disk",
vdev_op_leaf = 1
}
(kgdb) frame 195
#195 0xffffffff8222955d in zio_vdev_io_start (zio=0xfffff801469a13b8)
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2870
2870 vd->vdev_ops->vdev_op_io_start(zio);
$8 = {
io_bookmark = {
zb_objset = 0,
zb_object = 0,
zb_level = 0,
zb_blkid = 0
},
io_prop = {
zp_checksum = ZIO_CHECKSUM_INHERIT,
zp_compress = ZIO_COMPRESS_INHERIT,
zp_type = DMU_OT_NONE,
zp_level = 0 '\0',
zp_copies = 0 '\0',
zp_dedup = 0,
zp_dedup_verify = 0,
zp_nopwrite = 0
},
io_type = ZIO_TYPE_FREE,
io_child_type = ZIO_CHILD_VDEV,
io_cmd = 0,
io_priority = ZIO_PRIORITY_TRIM,
io_reexecute = 0 '\0',
io_state = 0xfffff801469a1405 "\001",
io_txg = 0,
io_spa = 0xfffff801c29d1000,
io_bp = 0x0,
io_bp_override = 0x0,
io_bp_copy = {
blk_dva = 0xfffff801469a1428,
blk_prop = 0,
blk_pad = 0xfffff801469a1460,
blk_phys_birth = 0,
blk_birth = 0,
blk_fill = 0,
blk_cksum = {
zc_word = 0xfffff801469a1488
}
},
io_parent_list = {
list_size = 48,
list_offset = 16,
list_head = {
list_next = 0xfffff803c203a1c0,
list_prev = 0xfffff803c203a1c0
}
},
io_child_list = {
list_size = 48,
list_offset = 32,
list_head = {
list_next = 0xfffff801469a14d8,
list_prev = 0xfffff801469a14d8
}
},
io_walk_link = 0x0,
io_logical = 0x0,
io_transform_stack = 0x0,
io_ready = 0,
io_physdone = 0,
io_done = 0,
io_private = 0x0,
io_prev_space_delta = 0,
io_bp_orig = {
blk_dva = 0xfffff801469a1528,
blk_prop = 0,
blk_pad = 0xfffff801469a1560,
blk_phys_birth = 0,
blk_birth = 0,
blk_fill = 0,
blk_cksum = {
zc_word = 0xfffff801469a1588
}
},
io_data = 0x0,
io_orig_data = 0x0,
io_size = 4096,
io_orig_size = 4096,
io_vd = 0xfffff804d8683000,
io_vsd = 0x0,
io_vsd_ops = 0x0,
io_offset = 898323513344,
io_timestamp = 941768555692856,
io_target_timestamp = 0,
io_queue_node = {
avl_child = 0xfffff801469a15f8,
avl_pcb = 18446735288905504249
},
io_offset_node = {
avl_child = 0xfffff801469a1610,
avl_pcb = 0
},
io_flags = 789633,
io_stage = ZIO_STAGE_VDEV_IO_DONE,
io_pipeline = 3080192,
io_orig_flags = 525441,
io_orig_stage = ZIO_STAGE_OPEN,
io_orig_pipeline = 3080192,
io_error = 45,
io_child_error = 0xfffff801469a1644,
io_children = 0xfffff801469a1658,
io_child_count = 0,
io_phys_children = 0,
io_parent_count = 1,
io_stall = 0x0,
io_gang_leader = 0x0,
io_gang_tree = 0x0,
io_executor = 0xfffff80339d9d500,
io_waiter = 0x0,
io_lock = {
lock_object = {
lo_name = 0xffffffff822bf23e "zio->io_lock",
lo_flags = 40960000,
lo_data = 0,
lo_witness = 0x0
},
sx_lock = 1
},
io_cv = {
cv_description = 0xffffffff822bf24c "zio->io_cv",
cv_waiters = 0
},
io_cksum_report = 0x0,
io_ena = 0,
io_tqent = {
tqent_task = {
ta_link = {
stqe_next = 0x0
},
ta_pending = 0,
ta_priority = 0,
ta_func = 0,
ta_context = 0x0
},
tqent_func = 0,
tqent_arg = 0x0
},
io_trim_node = {
avl_child = 0xfffff801469a1748,
avl_pcb = 0
},
io_trim_link = {
list_next = 0x0,
list_prev = 0x0
}
}
(kgdb) p *zio->io_vd
$9 = {
vdev_id = 2,
vdev_guid = 13112130218511473524,
vdev_guid_sum = 13112130218511473524,
vdev_orig_guid = 0,
vdev_asize = 4000782221312,
vdev_min_asize = 3985729650688,
vdev_max_asize = 4000782221312,
vdev_ashift = 12,
vdev_logical_ashift = 12,
vdev_physical_ashift = 0,
vdev_state = 7,
vdev_prevstate = 0,
vdev_ops = 0xffffffff822d2200,
vdev_spa = 0xfffff801c29d1000,
vdev_tsd = 0xfffff801638b2780,
vdev_name_vp = 0x0,
vdev_devid_vp = 0x0,
vdev_top = 0xfffff803462c1800,
vdev_parent = 0xfffff803462c1800,
vdev_child = 0x0,
vdev_children = 0,
vdev_stat = {
vs_timestamp = 941737635691796,
vs_state = 0,
vs_aux = 0,
vs_alloc = 0,
vs_space = 0,
vs_dspace = 0,
vs_rsize = 0,
vs_esize = 0,
vs_ops = 0xfffff804d86830e8,
vs_bytes = 0xfffff804d8683118,
vs_read_errors = 0,
vs_write_errors = 0,
vs_checksum_errors = 0,
vs_self_healed = 0,
vs_scan_removing = 0,
vs_scan_processed = 0,
vs_configured_ashift = 0,
vs_logical_ashift = 0,
vs_physical_ashift = 0,
vs_fragmentation = 0
},
vdev_expanding = 0,
vdev_reopening = 0,
vdev_open_error = 0,
vdev_open_thread = 0x0,
vdev_crtxg = 4,
vdev_ms_array = 0,
vdev_ms_shift = 0,
vdev_ms_count = 0,
vdev_mg = 0x0,
vdev_ms = 0x0,
vdev_ms_list = {
tl_lock = {
lock_object = {
lo_name = 0xffffffff822bbd27 "tl->tl_lock",
lo_flags = 40960000,
lo_data = 0,
lo_witness = 0x0
},
sx_lock = 1
},
tl_offset = 808,
tl_head = 0xfffff804d8683208
},
vdev_dtl_list = {
tl_lock = {
lock_object = {
lo_name = 0xffffffff822bbd27 "tl->tl_lock",
lo_flags = 40960000,
lo_data = 0,
lo_witness = 0x0
},
sx_lock = 1
},
tl_offset = 784,
tl_head = 0xfffff804d8683250
},
vdev_txg_node = {
tn_next = 0xfffff804d8683270,
tn_member = 0xfffff804d8683290 ""
},
vdev_remove_wanted = 0,
vdev_probe_wanted = 0,
vdev_config_dirty_node = {
list_next = 0x0,
list_prev = 0x0
},
vdev_state_dirty_node = {
list_next = 0x0,
list_prev = 0x0
},
vdev_deflate_ratio = 0,
vdev_islog = 0,
vdev_removing = 0,
vdev_ishole = 0,
vdev_top_zap = 0,
vdev_dtl = 0xfffff804d86832e8,
vdev_dtl_sm = 0xfffff80455b00200,
vdev_dtl_node = {
tn_next = 0xfffff804d8683310,
tn_member = 0xfffff804d8683330 ""
},
vdev_dtl_object = 65,
vdev_psize = 4000786939904,
vdev_wholedisk = 1,
vdev_offline = 0,
vdev_faulted = 0,
vdev_degraded = 0,
vdev_removed = 0,
vdev_resilver_txg = 0,
vdev_nparity = 0,
vdev_path = 0xfffff802bd5acbe0 "/dev/label/backup42.eli",
vdev_devid = 0x0,
vdev_physpath = 0xfffff803be782ea0 "/dev/label/backup42.eli",
vdev_fru = 0x0,
vdev_not_present = 0,
vdev_unspare = 0,
vdev_nowritecache = 0,
vdev_notrim = 1,
vdev_checkremove = 0,
vdev_forcefault = 0,
vdev_splitting = 0,
vdev_delayed_close = 0,
vdev_tmpoffline = 0,
vdev_detached = 0,
vdev_cant_read = 0,
vdev_cant_write = 0,
vdev_isspare = 0,
vdev_isl2cache = 0,
vdev_queue = {
vq_vdev = 0xfffff804d8683000,
vq_class = 0xfffff804d86833e8,
vq_active_tree = {
avl_root = 0xfffff80290a71240,
avl_compar = 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset = 576,
avl_numnodes = 64,
avl_size = 952
},
vq_read_offset_tree = {
avl_root = 0x0,
avl_compar = 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset = 600,
avl_numnodes = 0,
avl_size = 952
},
vq_write_offset_tree = {
avl_root = 0x0,
avl_compar = 0xffffffff8220b8a0 <vdev_queue_offset_compare>,
avl_offset = 600,
avl_numnodes = 0,
avl_size = 952
},
vq_last_offset = 897918853120,
vq_io_complete_ts = 941768555692856,
vq_lock = {
lock_object = {
lo_name = 0xffffffff822bcc85 "vq->vq_lock",
lo_flags = 40960000,
lo_data = 0,
lo_witness = 0x0
},
sx_lock = 18446735291472008448
},
vq_lastoffset = 33829150720
},
vdev_cache = {
vc_offset_tree = {
avl_root = 0x0,
avl_compar = 0xffffffff82208310 <vdev_cache_offset_compare>,
avl_offset = 24,
avl_numnodes = 0,
avl_size = 88
},
vc_lastused_tree = {
avl_root = 0x0,
avl_compar = 0xffffffff82208330 <vdev_cache_lastused_compare>,
avl_offset = 48,
avl_numnodes = 0,
avl_size = 88
},
vc_lock = {
lock_object = {
lo_name = 0xffffffff822bc53d "vc->vc_lock",
lo_flags = 40960000,
lo_data = 0,
lo_witness = 0x0
},
sx_lock = 1
}
}
(kgdb) p *zio->io_vd->vdev_ops
$10 = {
vdev_op_open = 0xffffffff8225c940 <vdev_geom_open>,
vdev_op_close = 0xffffffff8225d1b0 <vdev_geom_close>,
vdev_op_asize = 0xffffffff82201670 <vdev_default_asize>,
vdev_op_io_start = 0xffffffff8225d2a0 <vdev_geom_io_start>,
vdev_op_io_done = 0xffffffff8225d470 <vdev_geom_io_done>,
vdev_op_state_change = 0,
vdev_op_hold = 0xffffffff8225d480 <vdev_geom_hold>,
vdev_op_rele = 0xffffffff8225d490 <vdev_geom_rele>,
vdev_op_type = 0xffffffff822d2240 "disk",
vdev_op_leaf = 1
}
(kgdb)
There's two....
How many more 'ya want? As you can see they're big! :)
--
Karl Denninger
karl@denninger.net <mailto:karl@denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
[-- Attachment #2 --]
0 *H
010
`He 0 *H
_0[0C)0
*H
01 0 UUS10UFlorida10U Niceville10U
Cuda Systems LLC1 0UCuda Systems LLC CA1"0 *H
Cuda Systems LLC CA0
150421022159Z
200419022159Z0Z1 0 UUS10UFlorida10U
Cuda Systems LLC1 0 UKarl Denninger (OCSP)0"0
*H
0
X@vkY
Tq/vE]5#֯MX \8LJ/V?5Da+
sJc*/r{ȼnS+ w")ąZ^DtdCOZ ~7Q '@a#ijc۴oZdB&!Ӝ-< ? HN5y
5}F|e f"V لio74zn">a1qWuɖbFeGE&3(KhixG3!#e_XƬ Ϝ/,$+;4y 'Bz<qT9_?rRUpn5
Jn&Rx/p Jyel*pN8/#9u/YP EC)TY>~/˘N[vyiDKˉ ,^" ?$T8 v& K%z8C @?K{9f`+@,| Mbia 007++0)0'+0http://cudasystems.net:88880 U 0 0 `HB0 U 0, `HB
OpenSSL Generated Certificate0 U -h\Ff Y0 U #0$q}ݽʒm50 U 0karl@denninger.net0
*H
Owbabɺx&Uk[ (Oj!%p MQ0I!#QH}.>~2&D}<wm_>V6 v ]f>=Nn+8;q wfΰ/RLyUG#b}n!Dր _up|_ǰc/%ۥ
nN8:d;-UJd/m1~VނיnN I˾$tF1&}|?q?\đXԑ&\4V<lKۮ3%Am_(q -(cAeGX)f}-˥6cv~Kg8m~v ;|9:-iAPқ6ېn-.)<[$KJtt/L4ᖣ^Cmu4vb{+BG$M0c\[MR|0FԸP&78"4p# }D Z9;V9#>Sw"[UP710001 0 UUS10UFlorida10U Niceville10U
Cuda Systems LLC1 0UCuda Systems LLC CA1"0 *H
Cuda Systems LLC CA)0
`He M0 *H
1 *H
0 *H
1
160720215403Z0O *H
1B@ 41y k)j{#t(`@7`yaɢZH[gNtpâULr"0l *H
1_0]0 `He*0 `He0
*H
0*H
0
*H
@0+0
*H
(0 +71001 0 UUS10UFlorida10U Niceville10U
Cuda Systems LLC1 0UCuda Systems LLC CA1"0 *H
Cuda Systems LLC CA)0 *H
101 0 UUS10UFlorida10U Niceville10U
Cuda Systems LLC1 0UCuda Systems LLC CA1"0 *H
Cuda Systems LLC CA)0
*H
7c~%,9v?'aI@5KXk:Qf6%ҙxފg3M\m݀0GK1N</Xԙ?wxM}d- U\ o][^K[ wȏ}YG
!M %a.2{x#gK28\2{N
w ;cnsJ=&}*_KVҴ>HftK_vkg7Q?ɁQ$;O.l3bAΗH>C\lK )>>72o]?ŅL {kuxI/b ]-* )C$fU,&)h x",t:v2uoFydCӳ7TE(k :C= |/@(c*SlXdv!q_ ЗWt3*D -Z!
J'{qi8G^wQFd״
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?97cf5283-683b-83fd-c484-18c14973b065>