Date: Sun, 21 Feb 2021 18:06:27 +0000 From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 253587] iflib (?): reproducible mbuf-related crashes Message-ID: <bug-253587-7501-TtXVuvFqYj@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-253587-7501@https.bugs.freebsd.org/bugzilla/> References: <bug-253587-7501@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253587 --- Comment #5 from Kamigishi Rei <spambox@haruhiism.net> --- Update: this happens with maxthreads=3D1 as well. Does not happen inside a = VM. With an INVARIANTS kernel I can reproduce this reliably by initiating a zfs send over SSH through this host acting as a router (4 crashes out of 4 send attempts). Out of these 4 crashes, three were the same KASSERT: panic: Assertion m->m_nextpkt =3D=3D NULL failed at /usr/src/sys/net/iflib.= c:3638 cpuid =3D 2 time =3D 1613930234 KDB: stack backtrace: #0 0xffffffff807fcfe5 at kdb_backtrace+0x65 #1 0xffffffff807b2cd1 at vpanic+0x181 #2 0xffffffff807b2aa3 at panic+0x43 #3 0xffffffff808ec3a1 at iflib_completed_tx_reclaim+0x2d1 #4 0xffffffff808eb780 at iflib_txq_drain+0x60 #5 0xffffffff808f2dfe at drain_ring_lockless+0x9e #6 0xffffffff808f2b93 at ifmp_ring_enqueue+0x313 #7 0xffffffff808f1520 at iflib_if_transmit+0xa0 #8 0xffffffff808d0418 at bridge_enqueue+0xc8 #9 0xffffffff808d26c4 at bridge_output+0x134 #10 0xffffffff808d73af at ether_output+0x63f #11 0xffffffff8097480b at ip6_forward+0x95b #12 0xffffffff80976084 at ip6_input+0xf04 #13 0xffffffff808f4491 at netisr_dispatch_src+0xb1 #14 0xffffffff808d76be at ether_demux+0x17e #15 0xffffffff808d8d4c at ether_nh_input+0x40c #16 0xffffffff808f4491 at netisr_dispatch_src+0xb1 #17 0xffffffff808d7bb1 at ether_input+0xa1 Uptime: 1m36s Dumping 402 out of 4051 MB:..4%..12%..24%..32%..44%..52%..64%..72%..84%..92% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(stru= ct pcpu, (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D<optimized out>) at /usr/src/sys/kern/kern_shutdown= .c:399 #2 0xffffffff807b28fb in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:486 #3 0xffffffff807b2d40 in vpanic (fmt=3D<optimized out>, ap=3D<optimized ou= t>) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff807b2aa3 in panic (fmt=3D<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:843 #5 0xffffffff808ec3a1 in iflib_tx_desc_free (txq=3D<optimized out>, n=3D<o= ptimized out>) at /usr/src/sys/net/iflib.c:3638 #6 iflib_completed_tx_reclaim (txq=3D<optimized out>, txq@entry=3D0xfffffe0063088000, thresh=3D<optimized out>) at /usr/src/sys/net/iflib.c:3680 #7 0xffffffff808eb780 in iflib_txq_drain (r=3D0xfffffe0063094000, r@entry= =3D<error reading variable: value is not available>, cidx=3D718, cidx@entry=3D<error = reading variable: value is not available>, pidx=3D719, pidx@entry=3D<error reading variable: value is not available>) at /usr/src/sys/net/iflib.c:3744 #8 0xffffffff808f2dfe in drain_ring_lockless (r=3D<optimized out>, os=3D..= ., prev=3D0, budget=3D<optimized out>) at /usr/src/sys/net/mp_ring.c:187 #9 0xffffffff808f2b93 in ifmp_ring_enqueue (r=3D0xfffffe0063094000, items=3D<optimized out>, items@entry=3D0xfffffe0007f924e8, n=3D<optimized o= ut>, n@entry=3D1, budget=3D<optimized out>, budget@entry=3D32, abdicate=3D<optim= ized out>, abdicate@entry=3D0) at /usr/src/sys/net/mp_ring.c:470 #10 0xffffffff808f1520 in iflib_if_transmit (ifp=3D<optimized out>, m=3D0xfffff80015f48000) at /usr/src/sys/net/iflib.c:4135 #11 0xffffffff808d0418 in bridge_enqueue (sc=3Dsc@entry=3D0xfffff80015aa0c0= 0, dst_ifp=3Ddst_ifp@entry=3D0xfffff80002647800, m=3D<unavailable>, m@entry=3D0xfffff80015f48000) at /usr/src/sys/net/if_bridge.c:1983 #12 0xffffffff808d26c4 in bridge_output (ifp=3D<optimized out>, ifp@entry= =3D<error reading variable: value is not available>, m=3D0xfffff80015f48000, m@entry= =3D<error reading variable: value is not available>, sa=3D<unavailable>, sa@entry=3D<error reading variable: value is not available>, rt=3D<unavailable>, rt@entry=3D<error reading variable: value is not availa= ble>) at /usr/src/sys/net/if_bridge.c:2145 #13 0xffffffff808d73af in ether_output (ifp=3D0xfffff80002647800, m=3D<unavailable>, dst=3D0xfffffe0007f92670, ro=3D<optimized out>) at /usr/src/sys/net/if_ethersubr.c:414 #14 0xffffffff8097480b in ip6_forward (m=3D<unavailable>, srcrt=3Dsrcrt@ent= ry=3D0) at /usr/src/sys/netinet6/ip6_forward.c:387 #15 0xffffffff80976084 in ip6_input (m=3D<unavailable>, m@entry=3D<error re= ading variable: value is not available>) at /usr/src/sys/netinet6/ip6_input.c:896 #16 0xffffffff808f4491 in netisr_dispatch_src (proto=3D6, source=3Dsource@e= ntry=3D0, m=3D0xfffff80023e49900) at /usr/src/sys/net/netisr.c:1143 #17 0xffffffff808f47df in netisr_dispatch (proto=3D<unavailable>, m=3D<unavailable>) at /usr/src/sys/net/netisr.c:1234 #18 0xffffffff808d76be in ether_demux (ifp=3Difp@entry=3D0xfffff800026cb800, m=3D<unavailable>) at /usr/src/sys/net/if_ethersubr.c:923 #19 0xffffffff808d8d4c in ether_input_internal (ifp=3D0xfffff800026cb800, m=3D<unavailable>) at /usr/src/sys/net/if_ethersubr.c:709 #20 ether_nh_input (m=3D<optimized out>, m@entry=3D<error reading variable:= value is not available>) at /usr/src/sys/net/if_ethersubr.c:739 #21 0xffffffff808f4491 in netisr_dispatch_src (proto=3Dproto@entry=3D5, source=3Dsource@entry=3D0, m=3Dm@entry=3D0xfffff80023e49900) at /usr/src/sys/net/netisr.c:1143 #22 0xffffffff808f47df in netisr_dispatch (proto=3D<unavailable>, proto@ent= ry=3D5, m=3D<unavailable>, m@entry=3D0xfffff80023e49900) at /usr/src/sys/net/netisr= .c:1234 #23 0xffffffff808d7bb1 in ether_input (ifp=3D0xfffff800026cb800, m=3D0xfffff80023e49900) at /usr/src/sys/net/if_ethersubr.c:830 #24 0xffffffff808f0556 in iflib_rxeof (rxq=3D<optimized out>, rxq@entry=3D0xfffff800026cb000, budget=3D<optimized out>) at /usr/src/sys/net/iflib.c:3008 #25 0xffffffff808ea0ca in _task_fn_rx (context=3D0xfffff800026cb000) at /usr/src/sys/net/iflib.c:3951 #26 0xffffffff807fb977 in gtaskqueue_run_locked (queue=3Dqueue@entry=3D0xfffff80002423300) at /usr/src/sys/kern/subr_gtaskqueue.c:371 #27 0xffffffff807fb774 in gtaskqueue_thread_loop (arg=3Darg@entry=3D0xfffffe0008d54038) at /usr/src/sys/kern/subr_gtaskqueue= .c:547 #28 0xffffffff8076efb0 in fork_exit (callout=3D0xffffffff807fb6e0 <gtaskqueue_thread_loop>, arg=3D0xfffffe0008d54038, frame=3D0xfffffe0007f92= c00) at /usr/src/sys/kern/kern_fork.c:1069 #29 <signal handler called> 4th crash: panic: m_dup: no mbuf packet header! cpuid =3D 1 time =3D 1613919472 KDB: stack backtrace: #0 0xffffffff807fcfe5 at kdb_backtrace+0x65 #1 0xffffffff807b2cd1 at vpanic+0x181 #2 0xffffffff807b2aa3 at panic+0x43 #3 0xffffffff80842981 at m_dup+0x351 #4 0xffffffff808ec610 at iflib_encap+0x210 #5 0xffffffff808ebb39 at iflib_txq_drain+0x419 #6 0xffffffff808f2dfe at drain_ring_lockless+0x9e #7 0xffffffff808f2b93 at ifmp_ring_enqueue+0x313 #8 0xffffffff808f1520 at iflib_if_transmit+0xa0 #9 0xffffffff808d0418 at bridge_enqueue+0xc8 #10 0xffffffff808d26c4 at bridge_output+0x134 #11 0xffffffff808d73af at ether_output+0x63f #12 0xffffffff8097480b at ip6_forward+0x95b #13 0xffffffff80976084 at ip6_input+0xf04 #14 0xffffffff808f4491 at netisr_dispatch_src+0xb1 #15 0xffffffff808d76be at ether_demux+0x17e #16 0xffffffff808d8d4c at ether_nh_input+0x40c #17 0xffffffff808f4491 at netisr_dispatch_src+0xb1 Uptime: 3m59s Dumping 409 out of 4051 MB:..4%..12%..24%..32%..43%..51%..63%..71%..83%..94% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(stru= ct pcpu, (kgdb) bt #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D<optimized out>) at /usr/src/sys/kern/kern_shutdown= .c:399 #2 0xffffffff807b28fb in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:486 #3 0xffffffff807b2d40 in vpanic (fmt=3D<optimized out>, ap=3D<optimized ou= t>) at /usr/src/sys/kern/kern_shutdown.c:919 #4 0xffffffff807b2aa3 in panic (fmt=3D<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:843 #5 0xffffffff80842981 in m_dup (m=3D<optimized out>, how=3D1) at /usr/src/sys/kern/uipc_mbuf.c:733 #6 0xffffffff808ec610 in iflib_parse_header (txq=3D0xfffffe006302ea40, pi=3D0xfffffe0007f47338, mp=3D0xfffffe006304f7f8) at /usr/src/sys/net/iflib= .c:3138 #7 iflib_encap (txq=3Dtxq@entry=3D0xfffffe006302ea40, m_headp=3Dm_headp@entry=3D0xfffffe006304f7f8) at /usr/src/sys/net/iflib.c:3= 464 #8 0xffffffff808ebb39 in iflib_txq_drain (r=3D<optimized out>, r@entry=3D<= error reading variable: value is not available>, cidx=3D<optimized out>, cidx@entry=3D<error reading variable: value is not available>, pidx=3D0, pidx@entry=3D<error reading variable: value is not available>) at /usr/src/sys/net/iflib.c:3801 #9 0xffffffff808f2dfe in drain_ring_lockless (r=3D<optimized out>, os=3D..= ., prev=3D0, budget=3D<optimized out>) at /usr/src/sys/net/mp_ring.c:187 #10 0xffffffff808f2b93 in ifmp_ring_enqueue (r=3D0xfffffe006304c000, items=3D<optimized out>, items@entry=3D0xfffffe0007f474e8, n=3D<optimized o= ut>, n@entry=3D1, budget=3D<optimized out>, budget@entry=3D32, abdicate=3D<optim= ized out>, abdicate@entry=3D0) at /usr/src/sys/net/mp_ring.c:470 #11 0xffffffff808f1520 in iflib_if_transmit (ifp=3D<optimized out>, m=3D0xfffff800586f9000) at /usr/src/sys/net/iflib.c:4135 #12 0xffffffff808d0418 in bridge_enqueue (sc=3Dsc@entry=3D0xfffff80016b54c0= 0, dst_ifp=3Ddst_ifp@entry=3D0xfffff80002456800, m=3D<unavailable>, m@entry=3D0xfffff800586f9000) at /usr/src/sys/net/if_bridge.c:1983 #13 0xffffffff808d26c4 in bridge_output (ifp=3D<optimized out>, ifp@entry= =3D<error reading variable: value is not available>, m=3D0xfffff800586f9000, m@entry= =3D<error reading variable: value is not available>, sa=3D<unavailable>, sa@entry=3D<error reading variable: value is not available>, rt=3D<unavailable>, rt@entry=3D<error reading variable: value is not availa= ble>) at /usr/src/sys/net/if_bridge.c:2145 #14 0xffffffff808d73af in ether_output (ifp=3D0xfffff80002456800, m=3D<unavailable>, dst=3D0xfffffe0007f47670, ro=3D<optimized out>) at /usr/src/sys/net/if_ethersubr.c:414 #15 0xffffffff8097480b in ip6_forward (m=3D<unavailable>, srcrt=3Dsrcrt@ent= ry=3D0) at /usr/src/sys/netinet6/ip6_forward.c:387 #16 0xffffffff80976084 in ip6_input (m=3D<unavailable>, m@entry=3D<error re= ading variable: value is not available>) at /usr/src/sys/netinet6/ip6_input.c:896 #17 0xffffffff808f4491 in netisr_dispatch_src (proto=3D6, source=3Dsource@e= ntry=3D0, m=3D0xfffff80016ed7600) at /usr/src/sys/net/netisr.c:1143 #18 0xffffffff808f47df in netisr_dispatch (proto=3D<unavailable>, m=3D<unavailable>) at /usr/src/sys/net/netisr.c:1234 #19 0xffffffff808d76be in ether_demux (ifp=3Difp@entry=3D0xfffff80002480800, m=3D<unavailable>) at /usr/src/sys/net/if_ethersubr.c:923 #20 0xffffffff808d8d4c in ether_input_internal (ifp=3D0xfffff80002480800, m=3D<unavailable>) at /usr/src/sys/net/if_ethersubr.c:709 #21 ether_nh_input (m=3D<optimized out>, m@entry=3D<error reading variable:= value is not available>) at /usr/src/sys/net/if_ethersubr.c:739 #22 0xffffffff808f4491 in netisr_dispatch_src (proto=3Dproto@entry=3D5, source=3Dsource@entry=3D0, m=3Dm@entry=3D0xfffff80016ed7600) at /usr/src/sys/net/netisr.c:1143 #23 0xffffffff808f47df in netisr_dispatch (proto=3D<unavailable>, proto@ent= ry=3D5, m=3D<unavailable>, m@entry=3D0xfffff80016ed7600) at /usr/src/sys/net/netisr= .c:1234 #24 0xffffffff808d7bb1 in ether_input (ifp=3D0xfffff80002480800, m=3D0xfffff80016ed7600) at /usr/src/sys/net/if_ethersubr.c:830 #25 0xffffffff808f0556 in iflib_rxeof (rxq=3D<optimized out>, rxq@entry=3D0xfffff80002480300, budget=3D<optimized out>) at /usr/src/sys/net/iflib.c:3008 #26 0xffffffff808ea0ca in _task_fn_rx (context=3D0xfffff80002480300) at /usr/src/sys/net/iflib.c:3951 #27 0xffffffff807fb977 in gtaskqueue_run_locked (queue=3Dqueue@entry=3D0xfffff80002422500) at /usr/src/sys/kern/subr_gtaskqueue.c:371 #28 0xffffffff807fb774 in gtaskqueue_thread_loop (arg=3Darg@entry=3D0xfffffe0008d54020) at /usr/src/sys/kern/subr_gtaskqueue= .c:547 #29 0xffffffff8076efb0 in fork_exit (callout=3D0xffffffff807fb6e0 <gtaskqueue_thread_loop>, arg=3D0xfffffe0008d54020, frame=3D0xfffffe0007f47= c00) at /usr/src/sys/kern/kern_fork.c:1069 #30 <signal handler called> --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-253587-7501-TtXVuvFqYj>