Date: Mon, 12 Sep 2022 14:10:44 +0200 From: Dmitry Salychev <dsl@FreeBSD.org> To: bob prohaska <fbsd@www.zefox.net> Cc: Mark Johnston <markj@freebsd.org>, Andrew Turner <andrew@fubar.geek.nz>, Ronald Klop <ronald-lists@klop.ws>, Mark Millard <marklmi@yahoo.com>, freebsd-arm@freebsd.org, freebsd-current@freebsd.org Subject: Re: panic: data abort in critical section or under mutex (was: Re: panic: Unknown kernel exception 0 esr_el1 2000000 (on 14-CURRENT/aarch64 Feb 28)) Message-ID: <86czc0eotc.fsf@peasant.tower.home> In-Reply-To: <20220308154204.GA37265@www.zefox.net> References: <C2F96211-0180-45DA-872F-52358D9ED35B.ref@yahoo.com> <C2F96211-0180-45DA-872F-52358D9ED35B@yahoo.com> <1800459695.1.1646649539521@mailrelay> <132978150.92.1646660769467@mailrelay> <YiYhIQXl1sd4cOVS@nuc> <3374E0F8-D712-4ED0-A62B-B6924FC8A5E2@fubar.geek.nz> <YiY2jmD97leKev0F@nuc> <20220308154204.GA37265@www.zefox.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--=-=-=
Content-Type: text/plain
Content-Disposition: attachment; filename=dpaa2_panics.txt
(kgdb) bt
#0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36
#1 kdb_enter (why=<optimized out>, msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:508
#2 0xffff000000460268 in vpanic (fmt=<optimized out>, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967
#3 0xffff000000460018 in panic (fmt=0x12 <error: Cannot access memory at address 0x12>) at /usr/src/sys/kern/kern_shutdown.c:903
#4 0xffff00000077c05c in do_el1h_sync (td=0xffffa000006d8000, frame=0xffff0000b23a24c0) at /usr/src/sys/arm64/arm64/trap.c:532
#5 <signal handler called>
#6 0xffff000000760d40 in arm_gic_v3_intr (arg=0xffffa000021ab000) at /usr/src/sys/arm64/arm64/gic_v3.c:505
#7 0xffff00000074be4c in intr_irq_handler (tf=0xffff0000b23a26e0) at /usr/src/sys/kern/subr_intr.c:327
#8 0xffff00000074be4c in intr_irq_handler (tf=0xffffa000006d8000) at /usr/src/sys/kern/subr_intr.c:327
#9 <signal handler called>
#10 cpu_idle (busy=<optimized out>) at /usr/src/sys/arm64/arm64/machdep.c:257
#11 0xffff000000491ab4 in sched_idletd (dummy=dummy@entry=0x0) at /usr/src/sys/kern/sched_ule.c:3070
#12 0xffff000000416440 in fork_exit (callout=0xffff00000049163c <sched_idletd>, arg=0x0, frame=0xffff0000b23a2990) at /usr/src/sys/kern/kern_fork.c:1102
#13 <signal handler called>
---
(kgdb) bt
#0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36
#1 kdb_enter (why=<optimized out>, msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:508
#2 0xffff000000460268 in vpanic (fmt=<optimized out>, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967
#3 0xffff000000460018 in panic (fmt=0x12 <error: Cannot access memory at address 0x12>) at /usr/src/sys/kern/kern_shutdown.c:903
#4 0xffff00000077c05c in do_el1h_sync (td=0xffffa000024b5600, frame=0xffff0000bcda29c0) at /usr/src/sys/arm64/arm64/trap.c:532
#5 <signal handler called>
#6 0xffff0000004ced5c in witness_lock (lock=lock@entry=0xffffa0000056ca20, flags=8, file=file@entry=0xffff00000093011e "/usr/src/sys/dev/dpaa2/dpaa2_swp.c", line=line@entry=795) at /usr/src/sys/kern/subr_witness.c:1505
#7 0xffff00000043a3a8 in __mtx_lock_flags (c=0xffffa0000056ca38, opts=0, file=0xffff00000093011e "/usr/src/sys/dev/dpaa2/dpaa2_swp.c", line=795) at /usr/src/sys/kern/kern_mutex.c:290
#8 0xffff0000007d60a8 in dpaa2_swp_enq_mult (swp=swp@entry=0xffffa0000056ca00, ed=ed@entry=0xffff0000bcda2c70, fd=fd@entry=0xffff0000bcda2df8, flags=flags@entry=0xffff0000bcda2c6c, frames_n=frames_n@entry=1) at /usr/src/sys/dev/dpaa2/dpaa2_swp.c:795
#9 0xffff0000007d445c in dpaa2_io_enq_multiple_fq (iodev=<optimized out>, fqid=<optimized out>, fd=0xffff00000093011e, frames_n=1473863424) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:343
#10 0xffff0000007d7e3c in DPAA2_SWP_ENQ_MULTIPLE_FQ (fqid=205, fd=0xffff0000bcda2df8, frames_n=1, dev=<optimized out>) at ./dpaa2_swp_if.h:45
#11 dpaa2_ni_tx (sc=<optimized out>, tx=<optimized out>, m=<optimized out>) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2717
#12 dpaa2_ni_transmit (ifp=<optimized out>, m=<optimized out>) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2374
#13 0xffff0000005878bc in ether_output_frame (ifp=ifp@entry=0xffffa00002550800, m=0x8, m@entry=0xffffa00032552900) at /usr/src/sys/net/if_ethersubr.c:513
#14 0xffff0000005876f4 in ether_output (ifp=0xffffa00002550800, m=0xffffa00032552900, dst=0xffffa00028010b68, ro=0xffffa00028010b48) at /usr/src/sys/net/if_ethersubr.c:436
#15 0xffff0000005dff2c in ip_output (m=0xffff00000093011e, m@entry=0xffffa00032552900, opt=<optimized out>, ro=0xffffa00028010b48, flags=<optimized out>, imo=0x0, inp=0xffffa000280109f0) at /usr/src/sys/netinet/ip_output.c:793
#16 0xffff0000005f7204 in tcp_default_output (tp=0xffff000111f72590) at /usr/src/sys/netinet/tcp_output.c:1542
#17 0xffff0000005f0ca4 in tcp_output (tp=0xffff000111f72590) at /usr/src/sys/netinet/tcp_var.h:407
#18 tcp_do_segment (m=0xffffa08004b26d00, th=0xffff00011d8ec0e2, so=0xffff000111f27d80, tp=0xffff000111f72590, drop_hdrlen=52, tlen=<optimized out>, iptos=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:3294
#19 0xffff0000005ee094 in tcp_input_with_port (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>, port=0) at /usr/src/sys/netinet/tcp_input.c:1425
#20 0xffff0000005eee70 in tcp_input (mp=0xffffa0000056ca20, offp=0x8, proto=9634078) at /usr/src/sys/netinet/tcp_input.c:1520
#21 0xffff0000005dbed4 in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:838
#22 0xffff0000005a4110 in netisr_dispatch_src (proto=1, source=0, m=0xffffa08004b26d00) at /usr/src/sys/net/netisr.c:1153
#23 0xffff0000005a44c0 in netisr_dispatch (proto=5687840, m=0xffff00000093011e) at /usr/src/sys/net/netisr.c:1244
#24 0xffff000000587a64 in ether_demux (ifp=ifp@entry=0xffffa00002550800, m=0x8) at /usr/src/sys/net/if_ethersubr.c:925
#25 0xffff0000005890e0 in ether_input_internal (ifp=0xffffa00002550800, m=0x8) at /usr/src/sys/net/if_ethersubr.c:711
#26 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:741
#27 0xffff0000005a4110 in netisr_dispatch_src (proto=proto@entry=5, source=0, m=m@entry=0xffffa08004b26d00) at /usr/src/sys/net/netisr.c:1153
#28 0xffff0000005a44c0 in netisr_dispatch (proto=5687840, proto@entry=5, m=0xffff00000093011e, m@entry=0xffffa08004b26d00) at /usr/src/sys/net/netisr.c:1244
#29 0xffff000000587f24 in ether_input (ifp=0xffffa00002550800, m=0xffffa08004b26d00) at /usr/src/sys/net/if_ethersubr.c:832
#30 0xffff0000007dc3c8 in dpaa2_ni_rx (chan=0xffff0000be8f5000, fq=<optimized out>, fd=0xffff0000befb1360) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2870
#31 0xffff0000007dbf40 in dpaa2_ni_consume_frames (chan=0xffff0000be8f5000, src=<optimized out>, consumed=<optimized out>) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2763
#32 dpaa2_ni_poll (arg=0xffff0000be8f5000) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2633
#33 0xffff0000007d4ac8 in dpaa2_io_intr (arg=0xffffa000006c7c00) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:540
#34 0xffff000000419f54 in intr_event_execute_handlers (ie=0xffffa00000576200, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1205
#35 ithread_execute_handlers (ie=0xffffa00000576200, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1218
#36 ithread_loop (arg=<optimized out>, arg@entry=0xffffa000006c0a00) at /usr/src/sys/kern/kern_intr.c:1306
#37 0xffff000000416440 in fork_exit (callout=0xffff000000419cac <ithread_loop>, arg=0xffffa000006c0a00, frame=0xffff0000bcda3990) at /usr/src/sys/kern/kern_fork.c:1102
#38 <signal handler called>
---
(kgdb) bt
#0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36
#1 kdb_enter (why=<optimized out>, msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:508
#2 0xffff000000460268 in vpanic (fmt=<optimized out>, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967
#3 0xffff000000460018 in panic (fmt=0x12 <error: Cannot access memory at address 0x12>) at /usr/src/sys/kern/kern_shutdown.c:903
#4 0xffff00000077c05c in do_el1h_sync (td=0xffffa000024b5600, frame=0xffff0000bcda2db0) at /usr/src/sys/arm64/arm64/trap.c:532
#5 <signal handler called>
#6 0xffff0000005dee5c in ip_output (m=<optimized out>, m@entry=0xffffa00007036700, opt=<optimized out>, ro=0xffffa0006d1f5e98, flags=0, imo=0x0, inp=0xffffa0006d1f5d40) at /usr/src/sys/sys/mbuf.h:1533
#7 0xffff0000005f7204 in tcp_default_output (tp=0xffff00011e090450) at /usr/src/sys/netinet/tcp_output.c:1542
#8 0xffff0000005f0ca4 in tcp_output (tp=0xffff00011e090450) at /usr/src/sys/netinet/tcp_var.h:407
#9 tcp_do_segment (m=0xffffa00007036700, th=0xffff00011d71e0e2, so=0xffff00011cd8d680, tp=0xffff00011e090450, drop_hdrlen=52, tlen=<optimized out>, iptos=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:3294
#10 0xffff0000005ee094 in tcp_input_with_port (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>, port=0) at /usr/src/sys/netinet/tcp_input.c:1425
#11 0xffff0000005eee70 in tcp_input (mp=0xffffa00007036770, offp=0x1, proto=9323951) at /usr/src/sys/netinet/tcp_input.c:1520
#12 0xffff0000005dbed4 in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:838
#13 0xffff0000005a4110 in netisr_dispatch_src (proto=1, source=0, m=0xffffa00007036700) at /usr/src/sys/net/netisr.c:1153
#14 0xffff0000005a44c0 in netisr_dispatch (proto=117663600, m=0xffff0000008e45af) at /usr/src/sys/net/netisr.c:1244
#15 0xffff000000587a64 in ether_demux (ifp=ifp@entry=0xffffa00002550800, m=0x1) at /usr/src/sys/net/if_ethersubr.c:925
#16 0xffff0000005890e0 in ether_input_internal (ifp=0xffffa00002550800, m=0x1) at /usr/src/sys/net/if_ethersubr.c:711
#17 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:741
#18 0xffff0000005a4110 in netisr_dispatch_src (proto=proto@entry=5, source=0, m=m@entry=0xffffa00007036700) at /usr/src/sys/net/netisr.c:1153
#19 0xffff0000005a44c0 in netisr_dispatch (proto=117663600, proto@entry=5, m=0xffff0000008e45af, m@entry=0xffffa00007036700) at /usr/src/sys/net/netisr.c:1244
#20 0xffff000000587f24 in ether_input (ifp=0xffffa00002550800, m=0xffffa00007036700) at /usr/src/sys/net/if_ethersubr.c:832
#21 0xffff0000007dc3c8 in dpaa2_ni_rx (chan=0xffff0000be8f5000, fq=<optimized out>, fd=0xffff0000befb1260) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2870
#22 0xffff0000007dbf40 in dpaa2_ni_consume_frames (chan=0xffff0000be8f5000, src=<optimized out>, consumed=<optimized out>) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2763
#23 dpaa2_ni_poll (arg=0xffff0000be8f5000) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2633
#24 0xffff0000007d4ac8 in dpaa2_io_intr (arg=0xffffa000006c7c00) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:540
#25 0xffff000000419f54 in intr_event_execute_handlers (ie=0xffffa00000576200, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1205
#26 ithread_execute_handlers (ie=0xffffa00000576200, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1218
#27 ithread_loop (arg=<optimized out>, arg@entry=0xffffa000006c0a00) at /usr/src/sys/kern/kern_intr.c:1306
#28 0xffff000000416440 in fork_exit (callout=0xffff000000419cac <ithread_loop>, arg=0xffffa000006c0a00, frame=0xffff0000bcda3990) at /usr/src/sys/kern/kern_fork.c:1102
#29 <signal handler called>
---
(kgdb) bt
#0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36
#1 kdb_enter (why=<optimized out>, msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:508
#2 0xffff000000460268 in vpanic (fmt=<optimized out>, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967
#3 0xffff000000460018 in panic (fmt=0x12 <error: Cannot access memory at address 0x12>) at /usr/src/sys/kern/kern_shutdown.c:903
#4 0xffff00000077c05c in do_el1h_sync (td=0xffffa08070a06600, frame=0xffff000111df33d0) at /usr/src/sys/arm64/arm64/trap.c:532
#5 <signal handler called>
#6 0xffff0000004ced5c in witness_lock (lock=lock@entry=0xffff00011d2c2420, flags=8, file=file@entry=0xffff000000860324 "/usr/src/sys/kern/uipc_socket.c", line=line@entry=2240) at /usr/src/sys/kern/subr_witness.c:1505
#7 0xffff00000043a3a8 in __mtx_lock_flags (c=0xffff00011d2c2438, opts=0, file=0xffff000000860324 "/usr/src/sys/kern/uipc_socket.c", line=2240) at /usr/src/sys/kern/kern_mutex.c:290
#8 0xffff000000508f54 in soreceive_generic (so=0xffff00011d2c2200, psa=0x0, uio=<optimized out>, mp0=<optimized out>, controlp=0x0, flagsp=<optimized out>) at /usr/src/sys/kern/uipc_socket.c:2240
#9 0xffff00000050a57c in soreceive (so=0xffff00011d2c2420, psa=0x8, uio=0xffff000000860324, mp0=0x8c0, controlp=0xffffa08157d8c800, flagsp=0x10) at /usr/src/sys/kern/uipc_socket.c:2839
#10 0xffff0000004d37f4 in fo_read (fp=0xffffa0002f5215f0, uio=0xffff000111df3780, active_cred=0xffff000000860324, flags=0, td=0xffffa08070a06600) at /usr/src/sys/sys/file.h:341
#11 dofileread (td=td@entry=0xffffa08070a06600, fd=fd@entry=5, fp=0xffffa0002f5215f0, auio=auio@entry=0xffff000111df3780, offset=<optimized out>, flags=0) at /usr/src/sys/kern/sys_generic.c:369
#12 0xffff0000004d3460 in kern_readv (td=0xffffa08070a06600, fd=5, auio=auio@entry=0xffff000111df3780) at /usr/src/sys/kern/sys_generic.c:290
#13 0xffff0000004d3400 in sys_read (td=0xffff00011d2c2420, uap=<optimized out>) at /usr/src/sys/kern/sys_generic.c:206
#14 0xffff00000077c8c4 in syscallenter (td=0xffffa08070a06600) at /usr/src/sys/arm64/arm64/../../kern/subr_syscall.c:189
#15 svc_handler (td=0xffffa08070a06600, frame=<optimized out>) at /usr/src/sys/arm64/arm64/trap.c:199
#16 do_el0_sync (td=0xffffa08070a06600, frame=<optimized out>) at /usr/src/sys/arm64/arm64/trap.c:603
#17 <signal handler called>
#18 0x00000000859c9ef0 in ?? ()
#19 0x0000000088c00000 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
---
(kgdb) bt
#0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36
#1 kdb_enter (why=<optimized out>, msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:508
#2 0xffff000000460268 in vpanic (fmt=<optimized out>, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967
#3 0xffff000000460018 in panic (fmt=0x12 <error: Cannot access memory at address 0x12>) at /usr/src/sys/kern/kern_shutdown.c:903
#4 0xffff00000077c05c in do_el1h_sync (td=0xffffa000022b3000, frame=0xffff0000bcd79f90) at /usr/src/sys/arm64/arm64/trap.c:532
#5 <signal handler called>
#6 0xffff0000004ced5c in witness_lock (lock=0xffff000000e31b00 <cc_cpu>, flags=8, file=0xffff0000008e30ac "/usr/src/sys/kern/kern_timeout.c", line=582) at /usr/src/sys/kern/subr_witness.c:1505
#7 0xffff00000047d4ec in callout_lock (c=0xffff0001121792c0) at /usr/src/sys/kern/kern_timeout.c:582
#8 callout_reset_sbt_on (c=0xffff0001121792c0, sbt=<optimized out>, prec=<optimized out>, ftn=0xffff00000047d4ec <callout_reset_sbt_on+204>, arg=0xffff000112179000, cpu=0, flags=256) at /usr/src/sys/kern/kern_timeout.c:962
#9 0xffff0000005f0480 in tcp_do_segment (m=0xffffa08089e89c00, th=0xffff00011c8150e2, so=0xffff00011cd40b00, tp=<optimized out>, drop_hdrlen=52, tlen=<optimized out>, iptos=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:2913
#10 0xffff0000005ee094 in tcp_input_with_port (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>, port=0) at /usr/src/sys/netinet/tcp_input.c:1425
#11 0xffff0000005eee70 in tcp_input (mp=0xffff000000e31b00 <cc_cpu>, offp=0x8, proto=9318572) at /usr/src/sys/netinet/tcp_input.c:1520
#12 0xffff0000005dbed4 in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:838
#13 0xffff0000005a4110 in netisr_dispatch_src (proto=1, source=0, m=0xffffa08089e89c00) at /usr/src/sys/net/netisr.c:1153
#14 0xffff0000005a44c0 in netisr_dispatch (proto=14883584, m=0xffff0000008e30ac) at /usr/src/sys/net/netisr.c:1244
#15 0xffff000000587a64 in ether_demux (ifp=ifp@entry=0xffffa00002350800, m=0x8) at /usr/src/sys/net/if_ethersubr.c:925
#16 0xffff0000005890e0 in ether_input_internal (ifp=0xffffa00002350800, m=0x8) at /usr/src/sys/net/if_ethersubr.c:711
#17 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:741
#18 0xffff0000005a4110 in netisr_dispatch_src (proto=proto@entry=5, source=0, m=m@entry=0xffffa08089e89c00) at /usr/src/sys/net/netisr.c:1153
#19 0xffff0000005a44c0 in netisr_dispatch (proto=14883584, proto@entry=5, m=0xffff0000008e30ac, m@entry=0xffffa08089e89c00) at /usr/src/sys/net/netisr.c:1244
#20 0xffff000000587f24 in ether_input (ifp=0xffffa00002350800, m=0xffffa08089e89c00) at /usr/src/sys/net/if_ethersubr.c:832
#21 0xffff0000007dc3c8 in dpaa2_ni_rx (chan=0xffff0000bf737000, fq=<optimized out>, fd=0xffff0000bfdf30a0) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2870
#22 0xffff0000007dbf40 in dpaa2_ni_consume_frames (chan=0xffff0000bf737000, src=<optimized out>, consumed=<optimized out>) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2763
#23 dpaa2_ni_poll (arg=0xffff0000bf737000) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2633
#24 0xffff0000007d4ac8 in dpaa2_io_intr (arg=0xffffa000006c6180) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:540
#25 0xffff000000419f54 in intr_event_execute_handlers (ie=0xffffa00000576500, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1205
#26 ithread_execute_handlers (ie=0xffffa00000576500, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1218
#27 ithread_loop (arg=<optimized out>, arg@entry=0xffffa000006c0a60) at /usr/src/sys/kern/kern_intr.c:1306
#28 0xffff000000416440 in fork_exit (callout=0xffff000000419cac <ithread_loop>, arg=0xffffa000006c0a60, frame=0xffff0000bcd7a990) at /usr/src/sys/kern/kern_fork.c:1102
#29 <signal handler called>
---
(kgdb) bt
#0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36
#1 kdb_enter (why=<optimized out>, msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:508
#2 0xffff00000045d214 in vpanic (fmt=<optimized out>, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967
#3 0xffff00000045cfc4 in panic (fmt=0x12 <error: Cannot access memory at address 0x12>) at /usr/src/sys/kern/kern_shutdown.c:903
#4 0xffff00000077385c in do_el1h_sync (td=0xffffa00001d61000, frame=0xffff0000bc7b26a0) at /usr/src/sys/arm64/arm64/trap.c:532
#5 <signal handler called>
#6 sleepq_switch (wchan=0xffffa00000724780, wchan@entry=0x0, pri=0, pri@entry=7489536) at /usr/src/sys/kern/subr_sleepqueue.c:613
#7 0xffff0000004b8994 in sleepq_wait (wchan=<optimized out>, pri=<optimized out>) at /usr/src/sys/kern/subr_sleepqueue.c:660
#8 0xffff000000469134 in _sleep (ident=0xffffa00000724780, lock=0xffffa00000724800, priority=0, wmesg=0xffff000000885cee "-", sbt=0, pr=0, flags=256) at /usr/src/sys/kern/kern_synch.c:225
#9 0xffff0000004bed64 in TQ_SLEEP (tq=0xffffa00000724780, p=0xffffa00000724780, wm=<optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:126
#10 taskqueue_thread_loop (arg=arg@entry=0xffff0000bd74e5a0) at /usr/src/sys/kern/subr_taskqueue.c:834
#11 0xffff000000413568 in fork_exit (callout=0xffff0000004beca8 <taskqueue_thread_loop>, arg=0xffff0000bd74e5a0, frame=0xffff0000bc7b2990) at /usr/src/sys/kern/kern_fork.c:1102
#12 <signal handler called>
---
(kgdb) bt
#0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36
#1 kdb_enter (why=<optimized out>, msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:508
#2 0xffff00000045d214 in vpanic (fmt=<optimized out>, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967
#3 0xffff00000045cfc4 in panic (fmt=0x12 <error: Cannot access memory at address 0x12>) at /usr/src/sys/kern/kern_shutdown.c:903
#4 0xffff00000077385c in do_el1h_sync (td=0xffffa00001ab8600, frame=0xffff0000bc7a2f50) at /usr/src/sys/arm64/arm64/trap.c:532
#5 <signal handler called>
#6 mb_dtor_mbuf (mem=0xffffa0001d418800, size=256, arg=0x0) at /usr/src/sys/kern/kern_mbuf.c:681
#7 0xffff000000703e4c in item_dtor (zone=zone@entry=0xffff000041b5f000, item=item@entry=0xffffa0001d418800, size=256, udata=udata@entry=0x0, skip=<optimized out>) at /usr/src/sys/vm/uma_core.c:3488
#8 0xffff000000702a08 in uma_zfree_arg (zone=0xffff000041b5f000, item=item@entry=0xffffa0001d418800, udata=0x0) at /usr/src/sys/vm/uma_core.c:4493
#9 0xffff0000004332cc in mb_free_ext (m=m@entry=0xffffa0001d418800) at /usr/src/sys/kern/kern_mbuf.c:1212
#10 0xffff000000431c2c in m_free (m=0xffffa0001d418800) at /usr/src/sys/sys/mbuf.h:1523
#11 0xffff00000043303c in m_freem (mb=0xffff000000432194 <mb_dtor_mbuf>, mb@entry=0xffff00011d95f590) at /usr/src/sys/kern/kern_mbuf.c:1571
#12 0xffff0000005012fc in sbdrop (sb=sb@entry=0xffff00011c921310, len=<optimized out>, len@entry=2896) at /usr/src/sys/kern/uipc_sockbuf.c:1654
#13 0xffff0000005ebfec in tcp_do_segment (m=0xffffa08148514300, th=0xffff00011bbee0e2, so=0xffff00011c921000, tp=0xffff00011d95f590, drop_hdrlen=52, tlen=<optimized out>, iptos=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:1850
#14 0xffff0000005e829c in tcp_input_with_port (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>, port=0) at /usr/src/sys/netinet/tcp_input.c:1425
#15 0xffff0000005e9078 in tcp_input (mp=0xffffa0001d418800, offp=0x100, proto=0) at /usr/src/sys/netinet/tcp_input.c:1520
#16 0xffff0000005d60dc in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:838
#17 0xffff00000059e3b0 in netisr_dispatch_src (proto=1, source=0, m=0xffffa08148514300) at /usr/src/sys/net/netisr.c:1153
#18 0xffff00000059e760 in netisr_dispatch (proto=490833920, m=0x0) at /usr/src/sys/net/netisr.c:1244
#19 0xffff000000581dcc in ether_demux (ifp=ifp@entry=0xffffa00001b57800, m=0x100) at /usr/src/sys/net/if_ethersubr.c:925
#20 0xffff000000583430 in ether_input_internal (ifp=0xffffa00001b57800, m=0x100) at /usr/src/sys/net/if_ethersubr.c:711
#21 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:741
#22 0xffff00000059e3b0 in netisr_dispatch_src (proto=proto@entry=5, source=0, m=m@entry=0xffffa08148514300) at /usr/src/sys/net/netisr.c:1153
#23 0xffff00000059e760 in netisr_dispatch (proto=490833920, proto@entry=5, m=0x0, m@entry=0xffffa08148514300) at /usr/src/sys/net/netisr.c:1244
#24 0xffff00000058228c in ether_input (ifp=0xffffa00001b57800, m=0xffffa08148514300) at /usr/src/sys/net/if_ethersubr.c:832
#25 0xffff0000007d3bc4 in dpaa2_ni_rx (chan=0xffff0000be1d1000, fq=<optimized out>, fd=0xffff0000be88d060) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2879
#26 0xffff0000007d37a8 in dpaa2_ni_consume_frames (chan=0xffff0000be1d1000, src=<optimized out>, consumed=<optimized out>) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2770
#27 dpaa2_ni_poll (arg=0xffff0000be1d1000) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2633
#28 0xffff0000007cc2a8 in dpaa2_io_intr (arg=0xffffa000006bfc00) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:540
#29 0xffff00000041707c in intr_event_execute_handlers (ie=0xffffa0000056e200, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1205
#30 ithread_execute_handlers (ie=0xffffa0000056e200, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1218
#31 ithread_loop (arg=<optimized out>, arg@entry=0xffffa000006b8a00) at /usr/src/sys/kern/kern_intr.c:1306
#32 0xffff000000413568 in fork_exit (callout=0xffff000000416dd4 <ithread_loop>, arg=0xffffa000006b8a00, frame=0xffff0000bc7a3990) at /usr/src/sys/kern/kern_fork.c:1102
#33 <signal handler called>
---
Undefined instruction: f9401bf9
x0: 0
x1: ffffa0000073ae50
x2: 1
x3: 0
x4: 0
x5: 1
x6: 0
x7: ffff0000bc795c8c (_end + bb7e4c8c)
x8: 9de11000
x9: 0
x10: 0
x11: 1ff
x12: 5a8
x13: 42
x14: 9ac68042
x15: 1
x16: 10000
x17: 83cdd00
x18: ffff0000bc795c50 (_end + bb7e4c50)
x19: 1d
x20: ffff0000bdebfc18 (_end + bcf0ec18)
x21: ffffa00043275a00
x22: ffff0000bd478000 (_end + bc4c7000)
x23: ffff0000bdebfc20 (_end + bcf0ec20)
x24: ffff0000bdebf1a0 (_end + bcf0e1a0)
x25: ffff0000c0211000 (_end + bf260000)
x26: ffff0000bde0e000 (_end + bce5d000)
x27: ffff0000bde0e000 (_end + bce5d000)
x28: ffff0000bdebfc08 (_end + bcf0ec08)
x29: ffff0000bc795e20 (_end + bb7e4e20)
sp: ffff0000bc795c50
lr: ffff0000007cf584 (dpaa2_ni_transmit + 2bc)
elr: ffff00000049c5c0 (bus_dmamap_load + 114)
spsr: 80000045
far: 129a54403538
panic: Unknown kernel exception 0 esr_el1 2000000
cpuid = 3
time = 1662915749
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
vpanic() at vpanic+0x13c
panic() at panic+0x44
do_el1h_sync() at do_el1h_sync+0x194
handle_el1h_sync() at handle_el1h_sync+0x10
--- exception, esr 0x2000000
bus_dmamap_load() at bus_dmamap_load+0x114
ether_output_frame() at ether_output_frame+0xd4
ether_output() at ether_output+0x640
ip_output() at ip_output+0x1264
tcp_default_output() at tcp_default_output+0x1eb4
tcp_do_segment() at tcp_do_segment+0x1da4
tcp_input_with_port() at tcp_input_with_port+0xc00
tcp_input() at tcp_input+0xc
ip_input() at ip_input+0x30c
netisr_dispatch_src() at netisr_dispatch_src+0xe0
ether_demux() at ether_demux+0x174
ether_nh_input() at ether_nh_input+0x3ec
netisr_dispatch_src() at netisr_dispatch_src+0xe0
ether_input() at ether_input+0x80
dpaa2_ni_rx() at dpaa2_ni_rx+0x174
dpaa2_ni_poll() at dpaa2_ni_poll+0xb8
dpaa2_io_intr() at dpaa2_io_intr+0x13c
ithread_loop() at ithread_loop+0x2a4
fork_exit() at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x14
KDB: enter: panic
[ thread pid 12 tid 100104 ]
Stopped at kdb_enter+0x44: undefined f900027f
---
Undefined instruction: a8c17bfd
x0: ffffa00001ab6000
x1: ffff0000bc78aaa0 (_end + bb7d9aa0)
x2: ffff0000418f4300 (_end + 40943300)
x3: 8e4
x4: ffff0000bc78aaa0 (_end + bb7d9aa0)
x5: 0
x6: 0
x7: ffff0000bc78a5dc (_end + bb7d95dc)
x8: 0
x9: 1
x10: 2710
x11: 7ff8d402
x12: 7ff8a9fb
x13: 2af8
x14: 2a07
x15: 0
x16: 1
x17: 100
x18: ffff0000bc78a790 (_end + bb7d9790)
x19: ffffa00001ab6000
x20: ffffa000006d0000
x21: ffff0000418f4300 (_end + 40943300)
x22: ffff000000bb3000 (sdt_vfs_vop_vop_spare3_return + 58)
x23: ffff000000e5ea58 (group + 40)
x24: 0
x25: ffff0000bc78a7b8 (_end + bb7d97b8)
x26: ffff000000e00000 (thread0_st + 380)
x27: ffff000000e55000 (kdb_why + 0)
x28: ffff000000c3ee80 (pcpu_entry_tdq + 0)
x29: ffff0000bc78a790 (_end + bb7d9790)
sp: ffff0000bc78a790
lr: ffff0000007734b4 (cpu_switch + 60)
elr: ffff000000775540 (vfp_save_state + e0)
spsr: c5
far: 8649a000
panic: Unknown kernel exception 0 esr_el1 2000000
cpuid = 7
time = 1662916027
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
vpanic() at vpanic+0x13c
panic() at panic+0x44
do_el1h_sync() at do_el1h_sync+0x194
handle_el1h_sync() at handle_el1h_sync+0x10
--- exception, esr 0x2000000
vfp_save_state() at vfp_save_state+0xe0
cpu_switch() at cpu_switch+0x5c
mi_switch() at mi_switch+0x184
ithread_loop() at ithread_loop+0x9c
fork_exit() at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x14
KDB: enter: panic
[ thread pid 12 tid 100100 ]
Stopped at kdb_enter+0x44: undefined f900027f
---
panic: mutex not owned at /usr/src/sys/kern/uipc_socket.c:2187
cpuid = 7
time = 1662916786
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
vpanic() at vpanic+0x13c
panic() at panic+0x44
__mtx_assert() at __mtx_assert+0xf0
soreceive_generic() at soreceive_generic+0x54c
soreceive() at soreceive+0x44
dofileread() at dofileread+0x7c
kern_readv() at kern_readv+0x50
sys_read() at sys_read+0x80
do_el0_sync() at do_el0_sync+0x54c
handle_el0_sync() at handle_el0_sync+0x40
--- exception, esr 0x56000000
KDB: enter: panic
[ thread pid 1991 tid 100180 ]
Stopped at kdb_enter+0x44: undefined f900027f
---
Undefined instruction: b9805668
x0: 0
x1: ffffa00001ab8600
x2: ffff0000008998ad (do_execve.fexecv_proc_title + 10764)
x3: 7e
x4: 6d
x5: cb
x6: ffff0000bc7a2cd8 (_end + bb7f1cd8)
x7: ffff0000bc7a2cd4 (_end + bb7f1cd4)
x8: ffffa00001ab8600
x9: 2
x10: 0
x11: ffff000000ead678 (w_locklistdata + 3ff30)
x12: 3
x13: 2
x14: 10000
x15: 1
x16: 10000
x17: f5fa4dfb
x18: ffff0000bc7a2bc0 (_end + bb7f1bc0)
x19: ffffa0001e44a400
x20: ffff0000bc7a2cd4 (_end + bb7f1cd4)
x21: ffff0000bc7a2cd8 (_end + bb7f1cd8)
x22: ffffa0003b0379a4
x23: 42
x24: ffffa00001b5f400
x25: ffffa0003b037962
x26: 6c0
x27: ffffa0003b038022
x28: 962
x29: ffff0000bc7a2be0 (_end + bb7f1be0)
sp: ffff0000bc7a2bc0
lr: ffff00000074da20 (bounce_bus_dmamap_load_buffer + d4)
elr: ffff00000074db1c (bounce_bus_dmamap_load_buffer + 1d0)
spsr: 20000045
far: 82f3fdac
panic: Unknown kernel exception 0 esr_el1 2000000
cpuid = 4
time = 1662917050
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
vpanic() at vpanic+0x13c
panic() at panic+0x44
do_el1h_sync() at do_el1h_sync+0x194
handle_el1h_sync() at handle_el1h_sync+0x10
--- exception, esr 0x2000000
bounce_bus_dmamap_load_buffer() at bounce_bus_dmamap_load_buffer+0x1d0
bus_dmamap_load_mbuf_sg() at bus_dmamap_load_mbuf_sg+0x88
dpaa2_ni_transmit() at dpaa2_ni_transmit+0x190
ether_output_frame() at ether_output_frame+0xd4
ether_output() at ether_output+0x640
ip_output() at ip_output+0x1264
tcp_default_output() at tcp_default_output+0x1eb4
tcp_output() at tcp_output+0x38
tcp_do_segment() at tcp_do_segment+0x28d8
tcp_input_with_port() at tcp_input_with_port+0xc00
tcp_input() at tcp_input+0xc
ip_input() at ip_input+0x30c
netisr_dispatch_src() at netisr_dispatch_src+0xe0
ether_demux() at ether_demux+0x174
ether_nh_input() at ether_nh_input+0x3ec
netisr_dispatch_src() at netisr_dispatch_src+0xe0
ether_input() at ether_input+0x80
dpaa2_ni_rx() at dpaa2_ni_rx+0x174
dpaa2_ni_poll() at dpaa2_ni_poll+0xb8
dpaa2_io_intr() at dpaa2_io_intr+0x13c
ithread_loop() at ithread_loop+0x2a4
fork_exit() at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x14
KDB: enter: panic
[ thread pid 12 tid 100103 ]
Stopped at kdb_enter+0x44: undefined f900027f
---
Fatal data abort:
x0: ffffa00001ab6000
x1: ffff0000bc785fb8 (_end + bb7d4fb8)
x2: ffff000000928fa8 (console_pausestr + 24fd1)
x3: 7e
x4: 7e
x5: 60
x6: d28938770a080101
x7: fc7aaec8d2893877
x8: 0
x9: 2
x10: 2
x11: ffff000000eada30 (w_locklistdata + 402e8)
x12: 3
x13: 2
x14: 0
x15: 1
x16: 0
x17: 28
x18: ffff0000bc786030 (_end + bb7d5030)
x19: ffff00011c52ecf0
x20: ffff00011bb4a680
x21: ffff00011c52ed10
x22: ffffa0812fc9cd00
x23: ffff00011c52ed08
x24: 5a8
x25: ffffa0812fc9cd84
x26: ffff000000857a08 (cam_status_table + 4b60)
x27: ffff000000bb3000 (sdt_vfs_vop_vop_spare3_return + 58)
x28: 1
x29: ffff0000bc786190 (_end + bb7d5190)
sp: ffff0000bc786030
lr: ffff0000005f126c (tcp_default_output + 1d18)
elr: fffefffffc7f2b70
spsr: 20000045
far: fffefffffc7f2b70
esr: 86000004
panic: vm_fault failed: fffefffffc7f2b70 error 1
cpuid = 7
time = 1662983268
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_trace_self_wrapper() at db_trace_self_wrapper+0x30
vpanic() at vpanic+0x13c
panic() at panic+0x44
data_abort() at data_abort+0x2ec
handle_el1h_sync() at handle_el1h_sync+0x10
--- exception, esr 0x86000004
(null)() at 0xfffffffffc7f2b70
tcp_do_segment() at tcp_do_segment+0x1da4
tcp_input_with_port() at tcp_input_with_port+0xc00
tcp_input() at tcp_input+0xc
ip_input() at ip_input+0x30c
netisr_dispatch_src() at netisr_dispatch_src+0xe0
ether_demux() at ether_demux+0x174
ether_nh_input() at ether_nh_input+0x3ec
netisr_dispatch_src() at netisr_dispatch_src+0xe0
ether_input() at ether_input+0x80
dpaa2_ni_rx() at dpaa2_ni_rx+0x174
dpaa2_ni_poll() at dpaa2_ni_poll+0x140
dpaa2_io_intr() at dpaa2_io_intr+0x13c
ithread_loop() at ithread_loop+0x2a4
fork_exit() at fork_exit+0x74
fork_trampoline() at fork_trampoline+0x14
KDB: enter: panic
[ thread pid 12 tid 100100 ]
Stopped at kdb_enter+0x44: undefined f900027f
db>
---
(kgdb) bt
#0 breakpoint () at /usr/src/sys/arm64/include/cpufunc.h:36
#1 kdb_enter (why=<optimized out>, msg=<optimized out>) at /usr/src/sys/kern/subr_kdb.c:508
#2 0xffff00000045d214 in vpanic (fmt=<optimized out>, ap=...) at /usr/src/sys/kern/kern_shutdown.c:967
#3 0xffff00000045cfc4 in panic (fmt=0x12 <error: Cannot access memory at address 0x12>) at /usr/src/sys/kern/kern_shutdown.c:903
#4 0xffff00000077385c in do_el1h_sync (td=0xffffa00001ab6000, frame=0xffff0000bc785a30) at /usr/src/sys/arm64/arm64/trap.c:532
#5 <signal handler called>
#6 bounce_bus_dmamap_load_buffer (dmat=0xffffa00001b5f400, map=0xffffa0001e4b7800, buf=0xffffa0005200ea62, buflen=66, pmap=0xffff000000fa1050 <kernel_pmap_store>,
flags=<optimized out>, segs=0xffff0000bc785cd8, segp=0xffff0000bc785cd4) at /usr/src/sys/arm64/arm64/busdma_bounce.c:868
#7 0xffff00000049c7c4 in _bus_dmamap_load_buffer (dmat=0xffffa00001b5f400, map=0xffffa0001e4b7800, buf=0xffff0000008998ad, buflen=126,
pmap=0xffff000000fa1050 <kernel_pmap_store>, flags=2049, segs=0xffff0000bc785cd8, segp=0xffff0000bc785cd4) at /usr/src/sys/arm64/include/bus_dma.h:129
#8 _bus_dmamap_load_mbuf_sg (m0=<optimized out>, nsegs=0xffff0000bc785cd4, flags=1, dmat=<optimized out>, map=<optimized out>, segs=<optimized out>)
at /usr/src/sys/kern/subr_bus_dma.c:252
#9 bus_dmamap_load_mbuf_sg (dmat=0xffffa00001b5f400, map=0xffffa0001e4b7800, m0=<optimized out>, m0@entry=0xffffa0005200ea00, segs=segs@entry=0xffff0000bc785cd8,
nsegs=nsegs@entry=0xffff0000bc785cd4, flags=flags@entry=1) at /usr/src/sys/kern/subr_bus_dma.c:530
#10 0xffff0000007cf45c in dpaa2_ni_tx (sc=0xffff0000bd478000, tx=0xffff0000bf0c51a0, m=0xffffa0005200ea00) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2683
#11 dpaa2_ni_transmit (ifp=<optimized out>, m=0xffffa0005200ea00) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2374
#12 0xffff000000581c24 in ether_output_frame (ifp=ifp@entry=0xffffa00001b57800, m=0xffffa00001ab6000, m@entry=0xffffa0005200ea00)
at /usr/src/sys/net/if_ethersubr.c:513
#13 0xffff000000581a5c in ether_output (ifp=0xffffa00001b57800, m=0xffffa0005200ea00, dst=0xffffa000495734c8, ro=0xffffa000495734a8)
at /usr/src/sys/net/if_ethersubr.c:436
#14 0xffff0000005da134 in ip_output (m=0xffff0000008998ad, m@entry=0xffffa0005200ea00, opt=<optimized out>, ro=0xffffa000495734a8, flags=<optimized out>, imo=0x0,
inp=0xffffa00049573350) at /usr/src/sys/netinet/ip_output.c:793
#15 0xffff0000005f140c in tcp_default_output (tp=0xffff0000fe9fc000) at /usr/src/sys/netinet/tcp_output.c:1542
#16 0xffff0000005ec720 in tcp_output (tp=tp@entry=0xffff0000fe9fc000) at /usr/src/sys/netinet/tcp_var.h:407
#17 0xffff0000005eb9e0 in tcp_do_segment (m=<optimized out>, th=<optimized out>, so=0xffff00011c7d9680, tp=0xffff0000fe9fc000, drop_hdrlen=52, tlen=<optimized out>,
iptos=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:1963
#18 0xffff0000005e829c in tcp_input_with_port (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>, port=0) at /usr/src/sys/netinet/tcp_input.c:1425
#19 0xffff0000005e9078 in tcp_input (mp=0x0, offp=0xffffa00001ab6000, proto=9017517) at /usr/src/sys/netinet/tcp_input.c:1520
#20 0xffff0000005d60dc in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:838
#21 0xffff00000059e3b0 in netisr_dispatch_src (proto=1, source=0, m=0xffffa00052009c00) at /usr/src/sys/net/netisr.c:1153
#22 0xffff00000059e760 in netisr_dispatch (proto=0, m=0xffff0000008998ad) at /usr/src/sys/net/netisr.c:1244
#23 0xffff000000581dcc in ether_demux (ifp=ifp@entry=0xffffa00001b57800, m=0xffffa00001ab6000) at /usr/src/sys/net/if_ethersubr.c:925
#24 0xffff000000583430 in ether_input_internal (ifp=0xffffa00001b57800, m=0xffffa00001ab6000) at /usr/src/sys/net/if_ethersubr.c:711
#25 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:741
#26 0xffff00000059e3b0 in netisr_dispatch_src (proto=proto@entry=5, source=0, m=m@entry=0xffffa00052009c00) at /usr/src/sys/net/netisr.c:1153
#27 0xffff00000059e760 in netisr_dispatch (proto=0, proto@entry=5, m=0xffff0000008998ad, m@entry=0xffffa00052009c00) at /usr/src/sys/net/netisr.c:1244
#28 0xffff00000058228c in ether_input (ifp=0xffffa00001b57800, m=0xffffa00052009c00) at /usr/src/sys/net/if_ethersubr.c:832
#29 0xffff0000007d3be4 in dpaa2_ni_rx (chan=0xffff0000bf014000, fq=<optimized out>, fd=0xffff0000bf6d00a0) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2879
#30 0xffff0000007d37c8 in dpaa2_ni_consume_frames (chan=0xffff0000bf014000, src=<optimized out>, consumed=<optimized out>) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2770
#31 dpaa2_ni_poll (arg=0xffff0000bf014000) at /usr/src/sys/dev/dpaa2/dpaa2_ni.c:2633
#32 0xffff0000007cc2c8 in dpaa2_io_intr (arg=0xffffa000006be180) at /usr/src/sys/dev/dpaa2/dpaa2_io.c:540
--Type <RET> for more, q to quit, c to continue without paging--
#33 0xffff00000041707c in intr_event_execute_handlers (ie=0xffffa0000056e500, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1205
#34 ithread_execute_handlers (ie=0xffffa0000056e500, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1218
#35 ithread_loop (arg=<optimized out>, arg@entry=0xffffa000006b8a60) at /usr/src/sys/kern/kern_intr.c:1306
#36 0xffff000000413568 in fork_exit (callout=0xffff000000416dd4 <ithread_loop>, arg=0xffffa000006b8a60, frame=0xffff0000bc786990) at /usr/src/sys/kern/kern_fork.c:1102
#37 <signal handler called>
(kgdb)
--=-=-=
Content-Type: text/plain
Hi,
It seems that the recent 14-CURRENT/aarch64 (866e021) with DPAA2 drivers
panics under network throughtput stress test in random places with
unknown kernel exception 0 esr_el1 2000000 on Ten64 board (based on
NXP's LS1088A, Cortex-A53), but the same code doesn't panic on HoneyComb
(NXP LX2160A, Cortex-A72) even after ~10h long tests.
I've gathered some stack backtraces from ddb and kgdb (attached).
Panic itself can easily be reproduced after several minutes from the
start of the test. I've tried to change PCPU_PTR macro to use get_pcpu
again (as discussed in the thread earlier), but it didn't help.
If you want to get your hands dirty, DPAA2 stuff I'm using is at
https://github.com/mcusim/freebsd-src/tree/lx2160acex7-exp (branch is
lx2160acex7-exp!)
Any ideas or places to check would be really helpful.
bob prohaska <fbsd@www.zefox.net> writes:
> On Mon, Mar 07, 2022 at 11:45:02AM -0500, Mark Johnston wrote:
>> On Mon, Mar 07, 2022 at 04:25:22PM +0000, Andrew Turner wrote:
>> >
>> > > On 7 Mar 2022, at 15:13, Mark Johnston <markj@freebsd.org> wrote:
>> > > ...
>> > > A (the?) problem is that the compiler is treating "pc" as an alias
>> > > for x18, but the rmlock code assumes that the pcpu pointer is loaded
>> > > once, as it dereferences "pc" outside of the critical section. On
>> > > arm64, if a context switch occurs between the store at _rm_rlock+144 and
>> > > the load at +152, and the thread is migrated to another CPU, then we'll
>> > > end up using the wrong CPU ID in the rm->rm_writecpus test.
>> > >
>> > > I suspect the problem is unique to arm64 as its get_pcpu()
>> > > implementation is different from the others in that it doesn't use
>> > > volatile-qualified inline assembly. This has been the case since
>> > > https://cgit.freebsd.org/src/commit/?id=63c858a04d56529eddbddf85ad04fc8e99e73762
>> > > <https://cgit.freebsd.org/src/commit/?id=63c858a04d56529eddbddf85ad04fc8e99e73762>;
>> > > .
>> > >
>> > > I haven't been able to reproduce any crashes running poudriere in an
>> > > arm64 AWS instance, though. Could you please try the patch below and
>> > > confirm whether it fixes your panics? I verified that the apparent
>> > > problem described above is gone with the patch.
>> >
>> > Alternatively (or additionally) we could do something like the following. There are only a few MI users of get_pcpu with the main place being in rm locks.
>> >
>> > diff --git a/sys/arm64/include/pcpu.h b/sys/arm64/include/pcpu.h
>> > index 09f6361c651c..59b890e5c2ea 100644
>> > --- a/sys/arm64/include/pcpu.h
>> > +++ b/sys/arm64/include/pcpu.h
>> > @@ -58,7 +58,14 @@ struct pcpu;
>> >
>> > register struct pcpu *pcpup __asm ("x18");
>> >
>> > -#define get_pcpu() pcpup
>> > +static inline struct pcpu *
>> > +get_pcpu(void)
>> > +{
>> > + struct pcpu *pcpu;
>> > +
>> > + __asm __volatile("mov %0, x18" : "=&r"(pcpu));
>> > + return (pcpu);
>> > +}
>> >
>> > static inline struct thread *
>> > get_curthread(void)
>>
>> Indeed, I think this is probably the best solution.
>
> Just for fun I tried the patch on a Pi3 running -current, updated a day or two
> prior. The patch applied, compiled and seemed to run acceptably, but when I
> left a -j2 -DWITH_META_MODE buildworld running it crashed overnight, reporting
>
>
> login: panic: rm_rlock: recursed on non-recursive rmlock sysctl lock @ /usr/src/sys/kern/kern_sysctl.c:193
>
> cpuid = 0
> time = 1646720264
> KDB: stack backtrace:
> db_trace_self() at db_trace_self
> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
> vpanic() at vpanic+0x174
> panic() at panic+0x44
> _rm_rlock_debug() at _rm_rlock_debug+0x214
> sysctl_root_handler_locked() at sysctl_root_handler_locked+0x140
> sysctl_root() at sysctl_root+0x1ac
> userland_sysctl() at userland_sysctl+0x140
> sys___sysctl() at sys___sysctl+0x68
> do_el0_sync() at do_el0_sync+0x520
> handle_el0_sync() at handle_el0_sync+0x40
> --- exception, esr 0x56000000
> KDB: enter: panic
> [ thread pid 869 tid 100091 ]
> Stopped at kdb_enter+0x44: undefined f902011f
>
>
> I tried typing bt at the debugger prompt but got no more output.
>
> I've put the buildworld log file at
> http://www.zefox.net/~fbsd/rpi3/crashes/20220307/
>
> Hope this is of some use....
>
> bob prohaska
--
Dmitry Salychev
--=-=-=--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?86czc0eotc.fsf>