Date: Mon, 21 Jul 2014 07:24:27 +0200 From: Florian Smeets <flo@smeets.im> To: Steven Hartland <killing@multiplay.co.uk>, Larry Rosenman <ler@lerctr.org> Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: [ZFS][PANIC] Solaris Assert/zio.c:2548 Message-ID: <53CCA40B.10503@smeets.im> In-Reply-To: <4A0B8A1798484B3DB4DCD924A2DFEF22@multiplay.co.uk> References: <20140720140350.GA8498@borg.lerctr.org> <8D84B82C674B495DBF4951E0EE0D7117@multiplay.co.uk> <21dbecad27074fe34610bc587e6d0764@thebighonker.lerctr.org> <DE8C75117E794B8A83C7B2BAA417CD27@multiplay.co.uk> <E89CAE635D1E4BEDB421ABA5C04171E3@multiplay.co.uk> <05c8bccf4faf5005881fbd2b22f35428@thebighonker.lerctr.org> <4A0B8A1798484B3DB4DCD924A2DFEF22@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--rp9sFLx0TfVQoplMJscgoqQj9Iqaw3u1w
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
On 21/07/14 01:46, Steven Hartland wrote:
> ----- Original Message ----- From: "Larry Rosenman" <ler@lerctr.org>
> To: "Steven Hartland" <killing@multiplay.co.uk>
> Cc: <freebsd-fs@freebsd.org>; <freebsd-current@freebsd.org>
> Sent: Monday, July 21, 2014 12:22 AM
> Subject: Re: [ZFS][PANIC] Solaris Assert/zio.c:2548
>=20
>=20
>> On 2014-07-20 18:21, Steven Hartland wrote:
>>> Can you try reverting r265321 and see if you still see the
>>> same crash?
>>>
>>> Regards
>>> Steve
>> I'll do the revert, but it's been a ONE TIME hit.
>>
>> There was a followup to mine with a reproducible poudriere crash like
>> mine.
>=20
> If you don't have a reproducable senario I'd hold off.
>=20
> Florian, is yours reproducable and can you send me
> a pretty print of the crashing zio?
>=20
My backtrace looks a little different.
panic: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), file:
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/zio.c,
line: 2874
cpuid =3D 3
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
0xfffffe00002e97f0
kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe00002e98a0
vpanic() at vpanic+0x126/frame 0xfffffe00002e98e0
panic() at panic+0x43/frame 0xfffffe00002e9940
assfail() at assfail+0x1d/frame 0xfffffe00002e9950
zio_vdev_io_assess() at zio_vdev_io_assess+0x2e8/frame 0xfffffe00002e9980=
zio_execute() at zio_execute+0x1e9/frame 0xfffffe00002e99e0
taskqueue_run_locked() at taskqueue_run_locked+0xf0/frame 0xfffffe00002e9=
a40
taskqueue_thread_loop() at taskqueue_thread_loop+0x9b/frame
0xfffffe00002e9a70
fork_exit() at fork_exit+0x84/frame 0xfffffe00002e9ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00002e9ab0
--- trap 0, rip =3D 0, rsp =3D 0xfffffe00002e9b70, rbp =3D 0 ---
KDB: enter: panic
(kgdb) where
#0 doadump (textdump=3D-2125462752) at pcpu.h:219
#1 0xffffffff80347655 in db_fncall (dummy1=3D<value optimized out>,
dummy2=3D<value optimized out>, dummy3=3D<value optimized out>,
dummy4=3D<value optimized out>) at /usr/src/sys/ddb/db_command.c:578
#2 0xffffffff8034733d in db_command (cmd_table=3D0x0)
at /usr/src/sys/ddb/db_command.c:449
#3 0xffffffff803470b4 in db_command_loop ()
at /usr/src/sys/ddb/db_command.c:502
#4 0xffffffff80349a90 in db_trap (type=3D<value optimized out>, code=3D0=
)
at /usr/src/sys/ddb/db_main.c:231
#5 0xffffffff80944159 in kdb_trap (type=3D3, code=3D0, tf=3D<value optim=
ized
out>)
at /usr/src/sys/kern/subr_kdb.c:654
#6 0xffffffff80d1e532 in trap (frame=3D0xfffffe00002e97d0)
at /usr/src/sys/amd64/amd64/trap.c:542
#7 0xffffffff80d01202 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:231
#8 0xffffffff809438be in kdb_enter (why=3D0xffffffff80f9ce38 "panic",
msg=3D<value optimized out>) at cpufunc.h:63
#9 0xffffffff8090bb66 in vpanic (fmt=3D<value optimized out>,
ap=3D<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:737
#10 0xffffffff8090bbd3 in panic (fmt=3D0xffffffff815a59a0 "\004")
at /usr/src/sys/kern/kern_shutdown.c:673
#11 0xffffffff81fb821d in assfail (a=3D<value optimized out>,
---Type <return> to continue, or q <return> to quit---
f=3D<value optimized out>, l=3D<value optimized out>)
at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81=
#12 0xffffffff81eca848 in zio_vdev_io_assess (ziop=3D<value optimized out=
>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/zio.c:2874
#13 0xffffffff81ec58b9 in zio_execute (zio=3D0xfffff801a8abc398)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/zio.c:1416
#14 0xffffffff80954150 in taskqueue_run_locked (queue=3D0xfffff80009249b0=
0)
at /usr/src/sys/kern/subr_taskqueue.c:356
#15 0xffffffff80954c1b in taskqueue_thread_loop (arg=3D<value optimized o=
ut>)
at /usr/src/sys/kern/subr_taskqueue.c:623
#16 0xffffffff808d9834 in fork_exit (
callout=3D0xffffffff80954b80 <taskqueue_thread_loop>,
arg=3D0xfffff80003dfeed0, frame=3D0xfffffe00002e9ac0)
at /usr/src/sys/kern/kern_fork.c:977
#17 0xffffffff80d0173e in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:605
#18 0x0000000000000000 in ?? ()
(kgdb) frame 12
#12 0xffffffff81eca848 in zio_vdev_io_assess (ziop=3D<value optimized out=
>)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs=
/zio.c:2874
2874 ASSERT(!(zio->io_flags & ZIO_FLAG_DELEGATED));
(kgdb) print zio
$3 =3D (zio_t *) 0xfffff801a8abc398
(kgdb) print *zio
$4 =3D {io_bookmark =3D {zb_objset =3D 4339, zb_object =3D 327827, zb_lev=
el =3D 0,
zb_blkid =3D 0}, io_prop =3D {zp_checksum =3D ZIO_CHECKSUM_INHERIT,
zp_compress =3D ZIO_COMPRESS_INHERIT, zp_type =3D DMU_OT_NONE,
zp_level =3D 0 '\0', zp_copies =3D 0 '\0', zp_dedup =3D 0, zp_dedup_v=
erify
=3D 0,
zp_nopwrite =3D 0}, io_type =3D ZIO_TYPE_WRITE,
io_child_type =3D ZIO_CHILD_VDEV, io_cmd =3D 0,
io_priority =3D ZIO_PRIORITY_ASYNC_WRITE, io_reexecute =3D 0 '\0',
io_state =3D "\001", io_txg =3D 1312558, io_spa =3D 0xfffffe00022e6000,=
io_bp =3D 0xfffffe000a94a640, io_bp_override =3D 0x0, io_bp_copy =3D
{blk_dva =3D {{
dva_word =3D {1, 58754170}}, {dva_word =3D {1, 69614673}}, {dva_w=
ord
=3D {0,
0}}}, blk_prop =3D 9229009297394892802, blk_pad =3D {0, 0},
blk_phys_birth =3D 0, blk_birth =3D 1312558, blk_fill =3D 1, blk_cksu=
m =3D {
zc_word =3D {72684358009, 6982033287555, 350329209490535,
12175142665158025}}}, io_parent_list =3D {list_size =3D 48,
list_offset =3D 16, list_head =3D {list_next =3D 0xfffff800092e7520,
list_prev =3D 0xfffff800092e7520}}, io_child_list =3D {list_size =3D=
48,
list_offset =3D 32, list_head =3D {list_next =3D 0xfffff801a8abc4b8,
list_prev =3D 0xfffff801a8abc4b8}}, io_walk_link =3D 0x0,
io_logical =3D 0xfffff801a8cb7730, io_transform_stack =3D 0x0, io_ready=
=3D 0,
io_physdone =3D 0xffffffff81e34ee0 <arc_write_physdone>,
io_done =3D 0xffffffff81ea7ea0 <vdev_mirror_child_done>,
io_private =3D 0xfffff80027266d18, io_prev_space_delta =3D 0, io_bp_ori=
g =3D {
blk_dva =3D {{dva_word =3D {1, 58754170}}, {dva_word =3D {1, 69614673=
}}, {
---Type <return> to continue, or q <return> to quit---
dva_word =3D {0, 0}}}, blk_prop =3D 9229009297394892802, blk_pad =
=3D
{0, 0},
blk_phys_birth =3D 0, blk_birth =3D 1312558, blk_fill =3D 1, blk_cksu=
m =3D {
zc_word =3D {72684358009, 6982033287555, 350329209490535,
12175142665158025}}}, io_data =3D 0xfffff8017b0b3000,
io_orig_data =3D 0xfffff8017b0b3000, io_size =3D 512, io_orig_size =3D =
512,
io_vd =3D 0xfffff8000935f800, io_vsd =3D 0x0, io_vsd_ops =3D 0x0,
io_offset =3D 30086329344, io_timestamp =3D 76492632312, io_queue_node =
=3D {
avl_child =3D {0x0, 0x0}, avl_pcb =3D 18446735284737973505},
io_flags =3D 269224064, io_stage =3D ZIO_STAGE_VDEV_IO_ASSESS,
io_pipeline =3D 3014656, io_orig_flags =3D 524416,
io_orig_stage =3D ZIO_STAGE_READY, io_orig_pipeline =3D 3014656, io_err=
or =3D 0,
io_child_error =3D {0, 0, 0, 0}, io_children =3D {{0, 0}, {0, 0}, {0, 0=
}, {0,
0}}, io_child_count =3D 0, io_phys_children =3D 0, io_parent_count =
=3D 1,
io_stall =3D 0x0, io_gang_leader =3D 0x0, io_gang_tree =3D 0x0,
io_executor =3D 0xfffff80009265490, io_waiter =3D 0x0, io_lock =3D
{lock_object =3D {
lo_name =3D 0xffffffff81f763a5 "zio->io_lock", lo_flags =3D 4096000=
0,
lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, io_cv =3D {
cv_description =3D 0xffffffff81f763b3 "zio->io_cv", cv_waiters =3D 0}=
,
io_cksum_report =3D 0x0, io_ena =3D 0, io_tqent =3D {tqent_task =3D {ta=
_link =3D {
stqe_next =3D 0x0}, ta_pending =3D 0, ta_priority =3D 0,
ta_func =3D 0xffffffff81e1dd60 <taskq_run_ent>,
ta_context =3D 0xfffff801a8abc6d8},
tqent_func =3D 0xffffffff81ec56d0 <zio_execute>,
---Type <return> to continue, or q <return> to quit---
tqent_arg =3D 0xfffff801a8abc398}, io_trim_node =3D {avl_child =3D {0=
x0, 0x0},
avl_pcb =3D 0}, io_trim_link =3D {list_next =3D 0x0, list_prev =3D 0x=
0}}
I'm not an expert with kgdb. If you need anything else let me know what
you need.
Florian
--rp9sFLx0TfVQoplMJscgoqQj9Iqaw3u1w
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - https://gpgtools.org
iQIcBAEBCgAGBQJTzKQMAAoJEOcFPfn/hvB2+QIQAN0ut0ahLdyhD527Ecz3fskG
YGZ/hW8R6pKPBTv9ZceXfAwgH5uJ4cF2yqXL9Mrj+hYQqcltDPhSwyyHXwIrkLig
4RrajKRBGF4ybYEZYzTwnYTUVCvv8EtmAeZm1Q1N3vQ5FAEw7cwNZzCEumZawMue
ZmbVI+id1LvqALORBuX3HH3HcEB8QeJ2rWJboP+n9+ttxuPJMv69K6HPwxVEgRQ+
MIY0P+BVLRLX9Q5/Hl8q4YDJwlL3NB74jlNT46wyHcfLVWxc5FlGILXwsW0IlGbl
ybEPQHlhgutSibQsuUY8bbxkhR4Xn+NISIv8tG6eH5aLE8OHDIyWiBvOoHm7HEfl
hXdkA1UAoX5QzrhMmlvMnWoH72SPwlOh/1cpHb4FnfY5APcWkaURDCflx1CuXATQ
fYaGHkrHHoT5K+h+ADwu4+W6nrx8rAC3f2vNjJrgvz3kwSsLVUBBIpoHy5pkwfkA
bm8+iNhwBXyV6cs4Fyv8OsEAyN7EUd/7kxIeBoj5ogxcPUN1oiA+FXruMMx+wdTA
vMSKIOV51NhSaFsppUwy+uQaLI3FZWvMshC1DVHDR12DX0i86F3X0MHyu3QUZqcz
TFIoywoF2LtVYuzYCoewfyyCBOe3/1LkXCoRt9uxhzMOwOv9HVs8WQ0pbUN5RcY7
UY7ZUh0QMItMtlILk/c4
=d1HU
-----END PGP SIGNATURE-----
--rp9sFLx0TfVQoplMJscgoqQj9Iqaw3u1w--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?53CCA40B.10503>