From owner-freebsd-net@freebsd.org Sun Feb 21 18:06:27 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E073755D39D for ; Sun, 21 Feb 2021 18:06:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DkCtW5qntz3Dg3 for ; Sun, 21 Feb 2021 18:06:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id C81AB55D39C; Sun, 21 Feb 2021 18:06:27 +0000 (UTC) Delivered-To: net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C7E2B55D274 for ; Sun, 21 Feb 2021 18:06:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DkCtW5D0gz3Djr for ; Sun, 21 Feb 2021 18:06:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A65DA16B6A for ; Sun, 21 Feb 2021 18:06:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 11LI6RIW028075 for ; Sun, 21 Feb 2021 18:06:27 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 11LI6Rmg028073 for net@FreeBSD.org; Sun, 21 Feb 2021 18:06:27 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 253587] iflib (?): reproducible mbuf-related crashes Date: Sun, 21 Feb 2021 18:06:27 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: spambox@haruhiism.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Feb 2021 18:06:27 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253587 --- Comment #5 from Kamigishi Rei --- Update: this happens with maxthreads=3D1 as well. Does not happen inside a = VM. With an INVARIANTS kernel I can reproduce this reliably by initiating a zfs send over SSH through this host acting as a router (4 crashes out of 4 send attempts). Out of these 4 crashes, three were the same KASSERT: panic: Assertion m->m_nextpkt =3D=3D NULL failed at /usr/src/sys/net/iflib.= c:3638 cpuid =3D 2 time =3D 1613930234 KDB: stack backtrace: #0 0xffffffff807fcfe5 at kdb_backtrace+0x65 #1 0xffffffff807b2cd1 at vpanic+0x181 #2 0xffffffff807b2aa3 at panic+0x43 #3 0xffffffff808ec3a1 at iflib_completed_tx_reclaim+0x2d1 #4 0xffffffff808eb780 at iflib_txq_drain+0x60 #5 0xffffffff808f2dfe at drain_ring_lockless+0x9e #6 0xffffffff808f2b93 at ifmp_ring_enqueue+0x313 #7 0xffffffff808f1520 at iflib_if_transmit+0xa0 #8 0xffffffff808d0418 at bridge_enqueue+0xc8 #9 0xffffffff808d26c4 at bridge_output+0x134 #10 0xffffffff808d73af at ether_output+0x63f #11 0xffffffff8097480b at ip6_forward+0x95b #12 0xffffffff80976084 at ip6_input+0xf04 #13 0xffffffff808f4491 at netisr_dispatch_src+0xb1 #14 0xffffffff808d76be at ether_demux+0x17e #15 0xffffffff808d8d4c at ether_nh_input+0x40c #16 0xffffffff808f4491 at netisr_dispatch_src+0xb1 #17 0xffffffff808d7bb1 at ether_input+0xa1 Uptime: 1m36s Dumping 402 out of 4051 MB:..4%..12%..24%..32%..44%..52%..64%..72%..84%..92% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(stru= ct pcpu, (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown= .c:399 #2 0xffffffff807b28fb in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:486 #3 0xffffffff807b2d40 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff807b2aa3 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:843 #5 0xffffffff808ec3a1 in iflib_tx_desc_free (txq=3D, n=3D) at /usr/src/sys/net/iflib.c:3638 #6 iflib_completed_tx_reclaim (txq=3D, txq@entry=3D0xfffffe0063088000, thresh=3D) at /usr/src/sys/net/iflib.c:3680 #7 0xffffffff808eb780 in iflib_txq_drain (r=3D0xfffffe0063094000, r@entry= =3D, cidx=3D718, cidx@entry=3D, pidx=3D719, pidx@entry=3D) at /usr/src/sys/net/iflib.c:3744 #8 0xffffffff808f2dfe in drain_ring_lockless (r=3D, os=3D..= ., prev=3D0, budget=3D) at /usr/src/sys/net/mp_ring.c:187 #9 0xffffffff808f2b93 in ifmp_ring_enqueue (r=3D0xfffffe0063094000, items=3D, items@entry=3D0xfffffe0007f924e8, n=3D, n@entry=3D1, budget=3D, budget@entry=3D32, abdicate=3D, abdicate@entry=3D0) at /usr/src/sys/net/mp_ring.c:470 #10 0xffffffff808f1520 in iflib_if_transmit (ifp=3D, m=3D0xfffff80015f48000) at /usr/src/sys/net/iflib.c:4135 #11 0xffffffff808d0418 in bridge_enqueue (sc=3Dsc@entry=3D0xfffff80015aa0c0= 0, dst_ifp=3Ddst_ifp@entry=3D0xfffff80002647800, m=3D, m@entry=3D0xfffff80015f48000) at /usr/src/sys/net/if_bridge.c:1983 #12 0xffffffff808d26c4 in bridge_output (ifp=3D, ifp@entry= =3D, m=3D0xfffff80015f48000, m@entry= =3D, sa=3D, sa@entry=3D, rt=3D, rt@entry=3D) at /usr/src/sys/net/if_bridge.c:2145 #13 0xffffffff808d73af in ether_output (ifp=3D0xfffff80002647800, m=3D, dst=3D0xfffffe0007f92670, ro=3D) at /usr/src/sys/net/if_ethersubr.c:414 #14 0xffffffff8097480b in ip6_forward (m=3D, srcrt=3Dsrcrt@ent= ry=3D0) at /usr/src/sys/netinet6/ip6_forward.c:387 #15 0xffffffff80976084 in ip6_input (m=3D, m@entry=3D) at /usr/src/sys/netinet6/ip6_input.c:896 #16 0xffffffff808f4491 in netisr_dispatch_src (proto=3D6, source=3Dsource@e= ntry=3D0, m=3D0xfffff80023e49900) at /usr/src/sys/net/netisr.c:1143 #17 0xffffffff808f47df in netisr_dispatch (proto=3D, m=3D) at /usr/src/sys/net/netisr.c:1234 #18 0xffffffff808d76be in ether_demux (ifp=3Difp@entry=3D0xfffff800026cb800, m=3D) at /usr/src/sys/net/if_ethersubr.c:923 #19 0xffffffff808d8d4c in ether_input_internal (ifp=3D0xfffff800026cb800, m=3D) at /usr/src/sys/net/if_ethersubr.c:709 #20 ether_nh_input (m=3D, m@entry=3D) at /usr/src/sys/net/if_ethersubr.c:739 #21 0xffffffff808f4491 in netisr_dispatch_src (proto=3Dproto@entry=3D5, source=3Dsource@entry=3D0, m=3Dm@entry=3D0xfffff80023e49900) at /usr/src/sys/net/netisr.c:1143 #22 0xffffffff808f47df in netisr_dispatch (proto=3D, proto@ent= ry=3D5, m=3D, m@entry=3D0xfffff80023e49900) at /usr/src/sys/net/netisr= .c:1234 #23 0xffffffff808d7bb1 in ether_input (ifp=3D0xfffff800026cb800, m=3D0xfffff80023e49900) at /usr/src/sys/net/if_ethersubr.c:830 #24 0xffffffff808f0556 in iflib_rxeof (rxq=3D, rxq@entry=3D0xfffff800026cb000, budget=3D) at /usr/src/sys/net/iflib.c:3008 #25 0xffffffff808ea0ca in _task_fn_rx (context=3D0xfffff800026cb000) at /usr/src/sys/net/iflib.c:3951 #26 0xffffffff807fb977 in gtaskqueue_run_locked (queue=3Dqueue@entry=3D0xfffff80002423300) at /usr/src/sys/kern/subr_gtaskqueue.c:371 #27 0xffffffff807fb774 in gtaskqueue_thread_loop (arg=3Darg@entry=3D0xfffffe0008d54038) at /usr/src/sys/kern/subr_gtaskqueue= .c:547 #28 0xffffffff8076efb0 in fork_exit (callout=3D0xffffffff807fb6e0 , arg=3D0xfffffe0008d54038, frame=3D0xfffffe0007f92= c00) at /usr/src/sys/kern/kern_fork.c:1069 #29 4th crash: panic: m_dup: no mbuf packet header! cpuid =3D 1 time =3D 1613919472 KDB: stack backtrace: #0 0xffffffff807fcfe5 at kdb_backtrace+0x65 #1 0xffffffff807b2cd1 at vpanic+0x181 #2 0xffffffff807b2aa3 at panic+0x43 #3 0xffffffff80842981 at m_dup+0x351 #4 0xffffffff808ec610 at iflib_encap+0x210 #5 0xffffffff808ebb39 at iflib_txq_drain+0x419 #6 0xffffffff808f2dfe at drain_ring_lockless+0x9e #7 0xffffffff808f2b93 at ifmp_ring_enqueue+0x313 #8 0xffffffff808f1520 at iflib_if_transmit+0xa0 #9 0xffffffff808d0418 at bridge_enqueue+0xc8 #10 0xffffffff808d26c4 at bridge_output+0x134 #11 0xffffffff808d73af at ether_output+0x63f #12 0xffffffff8097480b at ip6_forward+0x95b #13 0xffffffff80976084 at ip6_input+0xf04 #14 0xffffffff808f4491 at netisr_dispatch_src+0xb1 #15 0xffffffff808d76be at ether_demux+0x17e #16 0xffffffff808d8d4c at ether_nh_input+0x40c #17 0xffffffff808f4491 at netisr_dispatch_src+0xb1 Uptime: 3m59s Dumping 409 out of 4051 MB:..4%..12%..24%..32%..43%..51%..63%..71%..83%..94% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(stru= ct pcpu, (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown= .c:399 #2 0xffffffff807b28fb in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:486 #3 0xffffffff807b2d40 in vpanic (fmt=3D, ap=3D) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff807b2aa3 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:843 #5 0xffffffff80842981 in m_dup (m=3D, how=3D1) at /usr/src/sys/kern/uipc_mbuf.c:733 #6 0xffffffff808ec610 in iflib_parse_header (txq=3D0xfffffe006302ea40, pi=3D0xfffffe0007f47338, mp=3D0xfffffe006304f7f8) at /usr/src/sys/net/iflib= .c:3138 #7 iflib_encap (txq=3Dtxq@entry=3D0xfffffe006302ea40, m_headp=3Dm_headp@entry=3D0xfffffe006304f7f8) at /usr/src/sys/net/iflib.c:3= 464 #8 0xffffffff808ebb39 in iflib_txq_drain (r=3D, r@entry=3D<= error reading variable: value is not available>, cidx=3D, cidx@entry=3D, pidx=3D0, pidx@entry=3D) at /usr/src/sys/net/iflib.c:3801 #9 0xffffffff808f2dfe in drain_ring_lockless (r=3D, os=3D..= ., prev=3D0, budget=3D) at /usr/src/sys/net/mp_ring.c:187 #10 0xffffffff808f2b93 in ifmp_ring_enqueue (r=3D0xfffffe006304c000, items=3D, items@entry=3D0xfffffe0007f474e8, n=3D, n@entry=3D1, budget=3D, budget@entry=3D32, abdicate=3D, abdicate@entry=3D0) at /usr/src/sys/net/mp_ring.c:470 #11 0xffffffff808f1520 in iflib_if_transmit (ifp=3D, m=3D0xfffff800586f9000) at /usr/src/sys/net/iflib.c:4135 #12 0xffffffff808d0418 in bridge_enqueue (sc=3Dsc@entry=3D0xfffff80016b54c0= 0, dst_ifp=3Ddst_ifp@entry=3D0xfffff80002456800, m=3D, m@entry=3D0xfffff800586f9000) at /usr/src/sys/net/if_bridge.c:1983 #13 0xffffffff808d26c4 in bridge_output (ifp=3D, ifp@entry= =3D, m=3D0xfffff800586f9000, m@entry= =3D, sa=3D, sa@entry=3D, rt=3D, rt@entry=3D) at /usr/src/sys/net/if_bridge.c:2145 #14 0xffffffff808d73af in ether_output (ifp=3D0xfffff80002456800, m=3D, dst=3D0xfffffe0007f47670, ro=3D) at /usr/src/sys/net/if_ethersubr.c:414 #15 0xffffffff8097480b in ip6_forward (m=3D, srcrt=3Dsrcrt@ent= ry=3D0) at /usr/src/sys/netinet6/ip6_forward.c:387 #16 0xffffffff80976084 in ip6_input (m=3D, m@entry=3D) at /usr/src/sys/netinet6/ip6_input.c:896 #17 0xffffffff808f4491 in netisr_dispatch_src (proto=3D6, source=3Dsource@e= ntry=3D0, m=3D0xfffff80016ed7600) at /usr/src/sys/net/netisr.c:1143 #18 0xffffffff808f47df in netisr_dispatch (proto=3D, m=3D) at /usr/src/sys/net/netisr.c:1234 #19 0xffffffff808d76be in ether_demux (ifp=3Difp@entry=3D0xfffff80002480800, m=3D) at /usr/src/sys/net/if_ethersubr.c:923 #20 0xffffffff808d8d4c in ether_input_internal (ifp=3D0xfffff80002480800, m=3D) at /usr/src/sys/net/if_ethersubr.c:709 #21 ether_nh_input (m=3D, m@entry=3D) at /usr/src/sys/net/if_ethersubr.c:739 #22 0xffffffff808f4491 in netisr_dispatch_src (proto=3Dproto@entry=3D5, source=3Dsource@entry=3D0, m=3Dm@entry=3D0xfffff80016ed7600) at /usr/src/sys/net/netisr.c:1143 #23 0xffffffff808f47df in netisr_dispatch (proto=3D, proto@ent= ry=3D5, m=3D, m@entry=3D0xfffff80016ed7600) at /usr/src/sys/net/netisr= .c:1234 #24 0xffffffff808d7bb1 in ether_input (ifp=3D0xfffff80002480800, m=3D0xfffff80016ed7600) at /usr/src/sys/net/if_ethersubr.c:830 #25 0xffffffff808f0556 in iflib_rxeof (rxq=3D, rxq@entry=3D0xfffff80002480300, budget=3D) at /usr/src/sys/net/iflib.c:3008 #26 0xffffffff808ea0ca in _task_fn_rx (context=3D0xfffff80002480300) at /usr/src/sys/net/iflib.c:3951 #27 0xffffffff807fb977 in gtaskqueue_run_locked (queue=3Dqueue@entry=3D0xfffff80002422500) at /usr/src/sys/kern/subr_gtaskqueue.c:371 #28 0xffffffff807fb774 in gtaskqueue_thread_loop (arg=3Darg@entry=3D0xfffffe0008d54020) at /usr/src/sys/kern/subr_gtaskqueue= .c:547 #29 0xffffffff8076efb0 in fork_exit (callout=3D0xffffffff807fb6e0 , arg=3D0xfffffe0008d54020, frame=3D0xfffffe0007f47= c00) at /usr/src/sys/kern/kern_fork.c:1069 #30 --=20 You are receiving this mail because: You are the assignee for the bug.=