From owner-freebsd-fs@FreeBSD.ORG Mon Jul 21 05:24:34 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D5C141A5; Mon, 21 Jul 2014 05:24:34 +0000 (UTC) Received: from mail-out.smeets.im (mail-out.smeets.im [IPv6:2a01:4f8:160:918a::25:11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7E1732546; Mon, 21 Jul 2014 05:24:34 +0000 (UTC) Received: from mail.smeets.im (mail.smeets.im [IPv6:2a01:4f8:160:918a::25:3]) by mail-out.smeets.im (Postfix) with ESMTP id 3C74918B; Mon, 21 Jul 2014 07:24:32 +0200 (CEST) Received: from amavis.smeets.im (amavis.smeets.im [IPv6:2a01:4f8:160:918a::aa:4]) by mail.smeets.im (Postfix) with ESMTP id B9C0D892BD; Mon, 21 Jul 2014 07:24:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at smeets.im Received: from mail.smeets.im ([IPv6:2a01:4f8:160:918a::25:3]) by amavis.smeets.im (amavis.smeets.im [IPv6:2a01:4f8:160:918a::aa:4]) (amavisd-new, port 10025) with ESMTP id KmrFO1qGupFZ; Mon, 21 Jul 2014 07:24:31 +0200 (CEST) Received: from nibbler-osx.local (unknown [IPv6:2001:4dd0:fd65:d00d:fdfe:c788:e9be:e820]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.smeets.im (Postfix) with ESMTPSA id C5FD7892A7; Mon, 21 Jul 2014 07:24:30 +0200 (CEST) Message-ID: <53CCA40B.10503@smeets.im> Date: Mon, 21 Jul 2014 07:24:27 +0200 From: Florian Smeets User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: Steven Hartland , Larry Rosenman Subject: Re: [ZFS][PANIC] Solaris Assert/zio.c:2548 References: <20140720140350.GA8498@borg.lerctr.org> <8D84B82C674B495DBF4951E0EE0D7117@multiplay.co.uk> <21dbecad27074fe34610bc587e6d0764@thebighonker.lerctr.org> <05c8bccf4faf5005881fbd2b22f35428@thebighonker.lerctr.org> <4A0B8A1798484B3DB4DCD924A2DFEF22@multiplay.co.uk> In-Reply-To: <4A0B8A1798484B3DB4DCD924A2DFEF22@multiplay.co.uk> Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="rp9sFLx0TfVQoplMJscgoqQj9Iqaw3u1w" Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 05:24:34 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --rp9sFLx0TfVQoplMJscgoqQj9Iqaw3u1w Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 21/07/14 01:46, Steven Hartland wrote: > ----- Original Message ----- From: "Larry Rosenman" > To: "Steven Hartland" > Cc: ; > Sent: Monday, July 21, 2014 12:22 AM > Subject: Re: [ZFS][PANIC] Solaris Assert/zio.c:2548 >=20 >=20 >> On 2014-07-20 18:21, Steven Hartland wrote: >>> Can you try reverting r265321 and see if you still see the >>> same crash? >>> >>> Regards >>> Steve >> I'll do the revert, but it's been a ONE TIME hit. >> >> There was a followup to mine with a reproducible poudriere crash like >> mine. >=20 > If you don't have a reproducable senario I'd hold off. >=20 > Florian, is yours reproducable and can you send me > a pretty print of the crashing zio? >=20 My backtrace looks a little different. panic: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs= /zio.c, line: 2874 cpuid =3D 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00002e97f0 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe00002e98a0 vpanic() at vpanic+0x126/frame 0xfffffe00002e98e0 panic() at panic+0x43/frame 0xfffffe00002e9940 assfail() at assfail+0x1d/frame 0xfffffe00002e9950 zio_vdev_io_assess() at zio_vdev_io_assess+0x2e8/frame 0xfffffe00002e9980= zio_execute() at zio_execute+0x1e9/frame 0xfffffe00002e99e0 taskqueue_run_locked() at taskqueue_run_locked+0xf0/frame 0xfffffe00002e9= a40 taskqueue_thread_loop() at taskqueue_thread_loop+0x9b/frame 0xfffffe00002e9a70 fork_exit() at fork_exit+0x84/frame 0xfffffe00002e9ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00002e9ab0 --- trap 0, rip =3D 0, rsp =3D 0xfffffe00002e9b70, rbp =3D 0 --- KDB: enter: panic (kgdb) where #0 doadump (textdump=3D-2125462752) at pcpu.h:219 #1 0xffffffff80347655 in db_fncall (dummy1=3D, dummy2=3D, dummy3=3D, dummy4=3D) at /usr/src/sys/ddb/db_command.c:578 #2 0xffffffff8034733d in db_command (cmd_table=3D0x0) at /usr/src/sys/ddb/db_command.c:449 #3 0xffffffff803470b4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:502 #4 0xffffffff80349a90 in db_trap (type=3D, code=3D0= ) at /usr/src/sys/ddb/db_main.c:231 #5 0xffffffff80944159 in kdb_trap (type=3D3, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 #6 0xffffffff80d1e532 in trap (frame=3D0xfffffe00002e97d0) at /usr/src/sys/amd64/amd64/trap.c:542 #7 0xffffffff80d01202 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:231 #8 0xffffffff809438be in kdb_enter (why=3D0xffffffff80f9ce38 "panic", msg=3D) at cpufunc.h:63 #9 0xffffffff8090bb66 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:737 #10 0xffffffff8090bbd3 in panic (fmt=3D0xffffffff815a59a0 "\004") at /usr/src/sys/kern/kern_shutdown.c:673 #11 0xffffffff81fb821d in assfail (a=3D, ---Type to continue, or q to quit--- f=3D, l=3D) at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81= #12 0xffffffff81eca848 in zio_vdev_io_assess (ziop=3D) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs= /zio.c:2874 #13 0xffffffff81ec58b9 in zio_execute (zio=3D0xfffff801a8abc398) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs= /zio.c:1416 #14 0xffffffff80954150 in taskqueue_run_locked (queue=3D0xfffff80009249b0= 0) at /usr/src/sys/kern/subr_taskqueue.c:356 #15 0xffffffff80954c1b in taskqueue_thread_loop (arg=3D) at /usr/src/sys/kern/subr_taskqueue.c:623 #16 0xffffffff808d9834 in fork_exit ( callout=3D0xffffffff80954b80 , arg=3D0xfffff80003dfeed0, frame=3D0xfffffe00002e9ac0) at /usr/src/sys/kern/kern_fork.c:977 #17 0xffffffff80d0173e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:605 #18 0x0000000000000000 in ?? () (kgdb) frame 12 #12 0xffffffff81eca848 in zio_vdev_io_assess (ziop=3D) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs= /zio.c:2874 2874 ASSERT(!(zio->io_flags & ZIO_FLAG_DELEGATED)); (kgdb) print zio $3 =3D (zio_t *) 0xfffff801a8abc398 (kgdb) print *zio $4 =3D {io_bookmark =3D {zb_objset =3D 4339, zb_object =3D 327827, zb_lev= el =3D 0, zb_blkid =3D 0}, io_prop =3D {zp_checksum =3D ZIO_CHECKSUM_INHERIT, zp_compress =3D ZIO_COMPRESS_INHERIT, zp_type =3D DMU_OT_NONE, zp_level =3D 0 '\0', zp_copies =3D 0 '\0', zp_dedup =3D 0, zp_dedup_v= erify =3D 0, zp_nopwrite =3D 0}, io_type =3D ZIO_TYPE_WRITE, io_child_type =3D ZIO_CHILD_VDEV, io_cmd =3D 0, io_priority =3D ZIO_PRIORITY_ASYNC_WRITE, io_reexecute =3D 0 '\0', io_state =3D "\001", io_txg =3D 1312558, io_spa =3D 0xfffffe00022e6000,= io_bp =3D 0xfffffe000a94a640, io_bp_override =3D 0x0, io_bp_copy =3D {blk_dva =3D {{ dva_word =3D {1, 58754170}}, {dva_word =3D {1, 69614673}}, {dva_w= ord =3D {0, 0}}}, blk_prop =3D 9229009297394892802, blk_pad =3D {0, 0}, blk_phys_birth =3D 0, blk_birth =3D 1312558, blk_fill =3D 1, blk_cksu= m =3D { zc_word =3D {72684358009, 6982033287555, 350329209490535, 12175142665158025}}}, io_parent_list =3D {list_size =3D 48, list_offset =3D 16, list_head =3D {list_next =3D 0xfffff800092e7520, list_prev =3D 0xfffff800092e7520}}, io_child_list =3D {list_size =3D= 48, list_offset =3D 32, list_head =3D {list_next =3D 0xfffff801a8abc4b8, list_prev =3D 0xfffff801a8abc4b8}}, io_walk_link =3D 0x0, io_logical =3D 0xfffff801a8cb7730, io_transform_stack =3D 0x0, io_ready= =3D 0, io_physdone =3D 0xffffffff81e34ee0 , io_done =3D 0xffffffff81ea7ea0 , io_private =3D 0xfffff80027266d18, io_prev_space_delta =3D 0, io_bp_ori= g =3D { blk_dva =3D {{dva_word =3D {1, 58754170}}, {dva_word =3D {1, 69614673= }}, { ---Type to continue, or q to quit--- dva_word =3D {0, 0}}}, blk_prop =3D 9229009297394892802, blk_pad = =3D {0, 0}, blk_phys_birth =3D 0, blk_birth =3D 1312558, blk_fill =3D 1, blk_cksu= m =3D { zc_word =3D {72684358009, 6982033287555, 350329209490535, 12175142665158025}}}, io_data =3D 0xfffff8017b0b3000, io_orig_data =3D 0xfffff8017b0b3000, io_size =3D 512, io_orig_size =3D = 512, io_vd =3D 0xfffff8000935f800, io_vsd =3D 0x0, io_vsd_ops =3D 0x0, io_offset =3D 30086329344, io_timestamp =3D 76492632312, io_queue_node = =3D { avl_child =3D {0x0, 0x0}, avl_pcb =3D 18446735284737973505}, io_flags =3D 269224064, io_stage =3D ZIO_STAGE_VDEV_IO_ASSESS, io_pipeline =3D 3014656, io_orig_flags =3D 524416, io_orig_stage =3D ZIO_STAGE_READY, io_orig_pipeline =3D 3014656, io_err= or =3D 0, io_child_error =3D {0, 0, 0, 0}, io_children =3D {{0, 0}, {0, 0}, {0, 0= }, {0, 0}}, io_child_count =3D 0, io_phys_children =3D 0, io_parent_count = =3D 1, io_stall =3D 0x0, io_gang_leader =3D 0x0, io_gang_tree =3D 0x0, io_executor =3D 0xfffff80009265490, io_waiter =3D 0x0, io_lock =3D {lock_object =3D { lo_name =3D 0xffffffff81f763a5 "zio->io_lock", lo_flags =3D 4096000= 0, lo_data =3D 0, lo_witness =3D 0x0}, sx_lock =3D 1}, io_cv =3D { cv_description =3D 0xffffffff81f763b3 "zio->io_cv", cv_waiters =3D 0}= , io_cksum_report =3D 0x0, io_ena =3D 0, io_tqent =3D {tqent_task =3D {ta= _link =3D { stqe_next =3D 0x0}, ta_pending =3D 0, ta_priority =3D 0, ta_func =3D 0xffffffff81e1dd60 , ta_context =3D 0xfffff801a8abc6d8}, tqent_func =3D 0xffffffff81ec56d0 , ---Type to continue, or q to quit--- tqent_arg =3D 0xfffff801a8abc398}, io_trim_node =3D {avl_child =3D {0= x0, 0x0}, avl_pcb =3D 0}, io_trim_link =3D {list_next =3D 0x0, list_prev =3D 0x= 0}} I'm not an expert with kgdb. If you need anything else let me know what you need. Florian --rp9sFLx0TfVQoplMJscgoqQj9Iqaw3u1w Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJTzKQMAAoJEOcFPfn/hvB2+QIQAN0ut0ahLdyhD527Ecz3fskG YGZ/hW8R6pKPBTv9ZceXfAwgH5uJ4cF2yqXL9Mrj+hYQqcltDPhSwyyHXwIrkLig 4RrajKRBGF4ybYEZYzTwnYTUVCvv8EtmAeZm1Q1N3vQ5FAEw7cwNZzCEumZawMue ZmbVI+id1LvqALORBuX3HH3HcEB8QeJ2rWJboP+n9+ttxuPJMv69K6HPwxVEgRQ+ MIY0P+BVLRLX9Q5/Hl8q4YDJwlL3NB74jlNT46wyHcfLVWxc5FlGILXwsW0IlGbl ybEPQHlhgutSibQsuUY8bbxkhR4Xn+NISIv8tG6eH5aLE8OHDIyWiBvOoHm7HEfl hXdkA1UAoX5QzrhMmlvMnWoH72SPwlOh/1cpHb4FnfY5APcWkaURDCflx1CuXATQ fYaGHkrHHoT5K+h+ADwu4+W6nrx8rAC3f2vNjJrgvz3kwSsLVUBBIpoHy5pkwfkA bm8+iNhwBXyV6cs4Fyv8OsEAyN7EUd/7kxIeBoj5ogxcPUN1oiA+FXruMMx+wdTA vMSKIOV51NhSaFsppUwy+uQaLI3FZWvMshC1DVHDR12DX0i86F3X0MHyu3QUZqcz TFIoywoF2LtVYuzYCoewfyyCBOe3/1LkXCoRt9uxhzMOwOv9HVs8WQ0pbUN5RcY7 UY7ZUh0QMItMtlILk/c4 =d1HU -----END PGP SIGNATURE----- --rp9sFLx0TfVQoplMJscgoqQj9Iqaw3u1w--