From owner-freebsd-bugs@freebsd.org Thu Jan 10 19:22:23 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CBA741495FD2 for ; Thu, 10 Jan 2019 19:22:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 608116AD45 for ; Thu, 10 Jan 2019 19:22:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 23EDA1495FD1; Thu, 10 Jan 2019 19:22:23 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F2B931495FD0 for ; Thu, 10 Jan 2019 19:22:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 18F0B6AD38 for ; Thu, 10 Jan 2019 19:22:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5DEBA126B for ; Thu, 10 Jan 2019 19:22:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x0AJMLq8056785 for ; Thu, 10 Jan 2019 19:22:21 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x0AJMLY5056784 for bugs@FreeBSD.org; Thu, 10 Jan 2019 19:22:21 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 234296] FreeBSD 12.0-STABLE r342216 Fatal trap 12 Date: Thu, 10 Jan 2019 19:22:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: markj@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jan 2019 19:22:24 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D234296 --- Comment #7 from Mark Johnston --- I looked at the kernel dumps. In all three cases, we crashed while process= ing a callout that had been mostly zeroed out. However, in all cases, at offset 0x10 into the callout there is a pointer in the direct map. Following that pointer leads me to the middle of a ZFS vnode: (kgdb) frame 9 #9 0xffffffff805dd328 in callout_process (now=3D91167878320709) at /usr/src/sys/kern/kern_timeout.c:510 510 LIST_REMOVE(tmp, c_links.le= ); (kgdb) p/x *cc->cc_callwheel[firstb & callwheelmask].lh_first $21 =3D { c_links =3D { le =3D { le_next =3D 0x0,=20 le_prev =3D 0x0 },=20 sle =3D { sle_next =3D 0x0 },=20 tqe =3D { tqe_next =3D 0x0,=20 tqe_prev =3D 0x0 } },=20 c_time =3D 0xfffff803773850c0,=20 c_precision =3D 0x0,=20 c_arg =3D 0x0,=20 c_func =3D 0x0,=20 c_lock =3D 0x0,=20 c_flags =3D 0x0,=20 c_iflags =3D 0x0,=20 c_cpu =3D 0x0 } (kgdb) x/32g 0xfffff803773850c0 0xfffff803773850c0: 0x0000000000000000 0xfffff8000a161de0 0xfffff803773850d0: 0xffffffff809bf965 0x0000000005230000 0xfffff803773850e0: 0x0000000000000000 0x0000000000000001 0xfffff803773850f0: 0xffffffff80c96170 0xfffff80463b90100 0xfffff80377385100: 0x0000000000000000 0x0000000000000000 0xfffff80377385110: 0xfffff80377385000 0x0000000000000000 0xfffff80377385120: 0xfffff80377385118 0x0000000000000000 0xfffff80377385130: 0x0000000000000000 0x0000000000000000 0xfffff80377385140: 0xfffff80377385138 0x0000000000000000 0xfffff80377385150: 0x0000000000000000 0x0000000000000000 0xfffff80377385160: 0x0000000300000000 0x0000000000020000 0xfffff80377385170: 0x0000000000000000 0x0000000000000000 0xfffff80377385180: 0x0000000000000000 0x0000000000000000 0xfffff80377385190: 0xfffff80377385188 0x0000000000000000 0xfffff803773851a0: 0x0000000000000000 0x0000000000000000 0xfffff803773851b0: 0x0000000000000000 0x0000000200000000 0xffffffff809bf965 is a pointer to the string "bufobj interlock". So the pointer in this callout is to the v_actfreelist member of a vnode. Indeed, 0xfffff803773850c0 & ~0xff is a pointer to a ZFS vnode. So, the pointer in the callout structure is probably the prev pointer, which means the callout structure is actually embedded in a vnode. (kgdb) p *(struct vnode * )((uintptr_t)$23 & ~0xff) $24 =3D {=20 v_tag =3D 0x0,=20=20=20 v_op =3D 0x0,=20=20=20=20=20=20 v_data =3D 0x0, v_mount =3D 0xfffff8000a3fb000, v_nmntvnodes =3D { tqe_next =3D 0x0,=20=20 tqe_prev =3D 0x0 },=20=20=20=20=20=20=20=20=20=20=20=20=20=20 ... v_vnlock =3D 0x0, v_actfreelist =3D { tqe_next =3D 0x0, tqe_prev =3D 0xfffff803773850c0 }, ... v_iflag =3D 0, v_vflag =3D 0, v_mflag =3D 0, v_writecount =3D 0, v_hash =3D 0, v_type =3D VMARKER } So the only non-zero fields are those of a marker vnode initialized in __mnt_vnode_first_active(). Those vnodes are allocated using malloc(), not the vnode zone. --=20 You are receiving this mail because: You are the assignee for the bug.=