Date: Thu, 31 Aug 2023 13:48:57 -0500 From: Jeremy <jeremy.m.cox@gmail.com> To: freebsd-wireless@freebsd.org Subject: rtw88 kernel panic with 14 stable and recent 13.2 stable Message-ID: <CAPH9q1pBL0rOwNoYoFvH6RvAHWjAn6MHgtL4=7cq9f-1RsJMGQ@mail.gmail.com>
index | next in thread | raw e-mail
[-- Attachment #1 --] Hi all, After attempting a clean install of the 14 stable snapshot my rtw88 wireless card stopped working and the kernel panicked. It did connect to the wifi router properly and got an IP address, but any attempt to fetch distfiles or do a pkg bootstrap resulted in a kernel panic. So I attempted a clean install of a recent 13.2 stable snapshot (from August 25 2023) and it did the same thing. When I did a fresh install of 13.2 release, it started working again. I'm using a desktop with a Ryzen 4600G CPU and an RT8822CE wireless pci card. The kernel panics with a custom built kernelor the GENERIC kernel. The kernel panic: FreeBSD riotskates 13.2-STABLE FreeBSD 13.2-STABLE #0: Wed Aug 30 18:06:54 CDT 2023 root@riotskates:/usr/obj/usr/src/amd64.amd64/sys/VENUS amd64 panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe010666b000 GNU gdb (GDB) 13.1 [GDB v13.1 for FreeBSD] Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd13.2". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /boot/kernel/kernel... Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... Unread portion of the kernel message buffer: panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe010666b000 cpuid = 8 time = 1693437565 KDB: stack backtrace: #0 0xffffffff8078181b at kdb_backtrace+0x6b #1 0xffffffff80734c02 at vpanic+0x152 #2 0xffffffff80734aa3 at panic+0x43 #3 0xffffffff80a33b1a at vm_fault+0x12ea #4 0xffffffff80a32751 at vm_fault_trap+0xb1 #5 0xffffffff80adee11 at trap_pfault+0x1f1 #6 0xffffffff80ab8618 at calltrap+0x8 #7 0xffffffff809954ad at linux_work_fn+0xed #8 0xffffffff80796687 at taskqueue_run_locked+0x187 #9 0xffffffff80797983 at taskqueue_thread_loop+0xc3 #10 0xffffffff806f10e2 at fork_exit+0x82 #11 0xffffffff80ab968e at fork_trampoline+0xe Uptime: 1m36s Dumping 701 out of 15684 MB:..3%..12%..21%..32%..42%..51%..62%..71%..83%..92% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=<optimized out>) at /usr/src/sys/kern/kern_shutdown.c:396 #2 0xffffffff807347bf in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:484 #3 0xffffffff80734c6f in vpanic (fmt=<optimized out>, ap=ap@entry=0xfffffe01025e6b10) at /usr/src/sys/kern/kern_shutdown.c:923 #4 0xffffffff80734aa3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:847 #5 0xffffffff80a33b1a in vm_fault_lookup (fs=0xfffffe01025e6b80) at /usr/src/sys/vm/vm_fault.c:842 #6 vm_fault (map=<optimized out>, vaddr=vaddr@entry=18446741879088656384, fault_type=2 '\002', fault_flags=fault_flags@entry=0, m_hold=m_hold@entry=0x0) at /usr/src/sys/vm/vm_fault.c:1477 #7 0xffffffff80a32751 in vm_fault_trap (map=<optimized out>, vaddr=vaddr@entry=18446741879088656392, fault_type=<optimized out>, fault_flags=fault_flags@entry=0, signo=0x0, ucode=0x0) at /usr/src/sys/vm/vm_fault.c:662 #8 0xffffffff80adee11 in trap_pfault (frame=0xfffffe01025e6d00, usermode=false, signo=<unavailable>, ucode=<unavailable>) at /usr/src/sys/amd64/amd64/trap.c:846 #9 <signal handler called> #10 __skb_unlink (skb=0xfffffe0106e9f000, head=0xfffffe0104bfd0c0) at /usr/src/sys/compat/linuxkpi/common/include/linux/skbuff.h:616 #11 skb_unlink (skb=0xfffffe0106e9f000, head=0xfffffe0104bfd0c0) at /usr/src/sys/compat/linuxkpi/common/include/linux/skbuff.h:624 #12 rtw_c2h_work (work=<optimized out>) at /usr/src/sys/contrib/dev/rtw88/main.c:276 #13 0xffffffff809954ad in linux_work_fn (context=0xfffffe0104bfd0f8, pending=<optimized out>) at /usr/src/sys/compat/linuxkpi/common/src/linux_work.c:299 #14 0xffffffff80796687 in taskqueue_run_locked ( queue=queue@entry=0xfffff80006678600) at /usr/src/sys/kern/subr_taskqueue.c:514 #15 0xffffffff80797983 in taskqueue_thread_loop (arg=<optimized out>) at /usr/src/sys/kern/subr_taskqueue.c:826 #16 0xffffffff806f10e2 in fork_exit ( callout=0xffffffff807978c0 <taskqueue_thread_loop>, arg=0xfffff80001a78640, frame=0xfffffe01025e6f40) at /usr/src/sys/kern/kern_fork.c:1094 #17 <signal handler called> #18 0x4b061c04e8080000 in ?? () Backtrace stopped: Cannot access memory at address 0x6213040a04e8 (kgdb) The only things that jumped out at me were the __skb_unlink and skb_unlink in frames 10 and 11. Frame 12 mentions something about rtw_c2h_work, but the file main.c was last modified before 13.2 release came out. The other culprit, /usr/src/sys/compat/linuxkpicommon/include/linux/skbuff.h has been modified more recently. So I reverted stable back to commit 1b33afb7e88b2e36db2083c488de41dbe097ef49 on stable 13 and rebuilt and reinstalled it, and the wireless card seems to work again. I am able to use pkg update and fetch distfiles without a kernel panic. Essentially on the stable 13 branch commits: de40bc6f3d6bf8ab1a4b546630cc847a3b8c5113 LinuxKPI: skbuff.h: fix -Warray-bounds warnings and b30d9f2e5204b475cf79150542b758c3dab14e70 LinuxKPI: skbuff.h: add more (skeleton) functions used by wireless drivers were reverted (along with everything else after June 25th 2023) and it seems to work. I'm no kernel expert and there were quite a few commits for linux emulation and linuxkpi between then and now. But at least the wireless card is working again. BTW, I have to manually download these sources from github and copy them to a thumb drive because "git clone"ing a branch doesn't seem to work at all, it just downloads for 30 seconds and just sits and does nothing after that. Thanks for your time, Jeremy Cox ReplyForward [-- Attachment #2 --] <div dir="ltr"><div class="gmail-adn gmail-ads" style="border-left:none;padding:0px;display:flex;font-family:"Google Sans",Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div class="gmail-gs" style="margin:0px;padding:0px 0px 20px;width:1504px"><div class="gmail-"><div id="gmail-:1qi" class="gmail-ii gmail-gt" style="direction:ltr;margin:8px 0px 0px;padding:0px;font-size:0.875rem"><div id="gmail-:1qj" class="gmail-a3s gmail-aiL" style="font-variant-numeric:normal;font-variant-east-asian:normal;font-variant-alternates:normal;font-kerning:auto;font-feature-settings:normal;font-stretch:normal;font-size:small;line-height:1.5;font-family:Arial,Helvetica,sans-serif;overflow:hidden"><div dir="ltr">Hi all,<div><br></div><div>After attempting a clean install of the 14 stable snapshot my rtw88 wireless card stopped working and the kernel panicked. It did connect to the wifi router properly and got an IP address, but any attempt to fetch distfiles or do a pkg bootstrap resulted in a kernel panic. So I attempted a clean install of a recent 13.2 stable snapshot (from August 25 2023) and it did the same thing.</div><div><br></div><div>When I did a fresh install of 13.2 release, it started working again. I'm using a desktop with a Ryzen 4600G CPU and an RT8822CE wireless pci card. The kernel panics with a custom built kernelor the GENERIC kernel. </div><div><br></div><div>The kernel panic:</div><div><br></div><div>FreeBSD riotskates 13.2-STABLE FreeBSD 13.2-STABLE #0: Wed Aug 30 18:06:54 CDT 2023 root@riotskates:/usr/obj/usr/src/amd64.amd64/sys/VENUS amd64<br><br>panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe010666b000<br><br>GNU gdb (GDB) 13.1 [GDB v13.1 for FreeBSD]<br>Copyright (C) 2023 Free Software Foundation, Inc.<br>License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>><br>This is free software: you are free to change and redistribute it.<br>There is NO WARRANTY, to the extent permitted by law.<br>Type "show copying" and "show warranty" for details.<br>This GDB was configured as "x86_64-portbld-freebsd13.2".<br>Type "show configuration" for configuration details.<br>For bug reporting instructions, please see:<br><<a href="https://www.gnu.org/software/gdb/bugs/" target="_blank">https://www.gnu.org/software/gdb/bugs/</a>>.<br>Find the GDB manual and other documentation resources online at:<br> <<a href="http://www.gnu.org/software/gdb/documentation/" target="_blank">http://www.gnu.org/software/gdb/documentation/</a>>.<br><br>For help, type "help".<br>Type "apropos word" to search for commands related to "word"...<br>Reading symbols from /boot/kernel/kernel...<br>Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...<br><br>Unread portion of the kernel message buffer:<br>panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe010666b000<br>cpuid = 8<br>time = 1693437565<br>KDB: stack backtrace:<br>#0 0xffffffff8078181b at kdb_backtrace+0x6b<br>#1 0xffffffff80734c02 at vpanic+0x152<br>#2 0xffffffff80734aa3 at panic+0x43<br>#3 0xffffffff80a33b1a at vm_fault+0x12ea<br>#4 0xffffffff80a32751 at vm_fault_trap+0xb1<br>#5 0xffffffff80adee11 at trap_pfault+0x1f1<br>#6 0xffffffff80ab8618 at calltrap+0x8<br>#7 0xffffffff809954ad at linux_work_fn+0xed<br>#8 0xffffffff80796687 at taskqueue_run_locked+0x187<br>#9 0xffffffff80797983 at taskqueue_thread_loop+0xc3<br>#10 0xffffffff806f10e2 at fork_exit+0x82<br>#11 0xffffffff80ab968e at fork_trampoline+0xe<br>Uptime: 1m36s<br>Dumping 701 out of 15684 MB:..3%..12%..21%..32%..42%..51%..62%..71%..83%..92%<br><br>__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55<br>55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,<br>(kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55<br>#1 doadump (textdump=<optimized out>)<br> at /usr/src/sys/kern/kern_shutdown.c:396<br>#2 0xffffffff807347bf in kern_reboot (howto=260)<br> at /usr/src/sys/kern/kern_shutdown.c:484<br>#3 0xffffffff80734c6f in vpanic (fmt=<optimized out>,<br> ap=ap@entry=0xfffffe01025e6b10) at /usr/src/sys/kern/kern_shutdown.c:923<br>#4 0xffffffff80734aa3 in panic (fmt=<unavailable>)<br> at /usr/src/sys/kern/kern_shutdown.c:847<br>#5 0xffffffff80a33b1a in vm_fault_lookup (fs=0xfffffe01025e6b80)<br> at /usr/src/sys/vm/vm_fault.c:842<br>#6 vm_fault (map=<optimized out>, vaddr=vaddr@entry=18446741879088656384,<br> fault_type=2 '\002', fault_flags=fault_flags@entry=0,<br> m_hold=m_hold@entry=0x0) at /usr/src/sys/vm/vm_fault.c:1477<br>#7 0xffffffff80a32751 in vm_fault_trap (map=<optimized out>,<br> vaddr=vaddr@entry=18446741879088656392, fault_type=<optimized out>,<br> fault_flags=fault_flags@entry=0, signo=0x0, ucode=0x0)<br> at /usr/src/sys/vm/vm_fault.c:662<br>#8 0xffffffff80adee11 in trap_pfault (frame=0xfffffe01025e6d00,<br> usermode=false, signo=<unavailable>, ucode=<unavailable>)<br> at /usr/src/sys/amd64/amd64/trap.c:846<br>#9 <signal handler called><br>#10 __skb_unlink (skb=0xfffffe0106e9f000, head=0xfffffe0104bfd0c0)<br> at /usr/src/sys/compat/linuxkpi/common/include/linux/skbuff.h:616<br>#11 skb_unlink (skb=0xfffffe0106e9f000, head=0xfffffe0104bfd0c0)<br> at /usr/src/sys/compat/linuxkpi/common/include/linux/skbuff.h:624<br>#12 rtw_c2h_work (work=<optimized out>)<br> at /usr/src/sys/contrib/dev/rtw88/main.c:276<br>#13 0xffffffff809954ad in linux_work_fn (context=0xfffffe0104bfd0f8,<br> pending=<optimized out>)<br> at /usr/src/sys/compat/linuxkpi/common/src/linux_work.c:299<br>#14 0xffffffff80796687 in taskqueue_run_locked (<br> queue=queue@entry=0xfffff80006678600)<br> at /usr/src/sys/kern/subr_taskqueue.c:514<br>#15 0xffffffff80797983 in taskqueue_thread_loop (arg=<optimized out>)<br> at /usr/src/sys/kern/subr_taskqueue.c:826<br>#16 0xffffffff806f10e2 in fork_exit (<br> callout=0xffffffff807978c0 <taskqueue_thread_loop>,<br> arg=0xfffff80001a78640, frame=0xfffffe01025e6f40)<br> at /usr/src/sys/kern/kern_fork.c:1094<br>#17 <signal handler called><br>#18 0x4b061c04e8080000 in ?? ()<br>Backtrace stopped: Cannot access memory at address 0x6213040a04e8<br>(kgdb)</div><div><br></div><div>The only things that jumped out at me were the __skb_unlink and skb_unlink in frames 10 and 11. Frame 12 mentions something about rtw_c2h_work, but the file main.c was last modified before 13.2 release came out. The other culprit, /usr/src/sys/compat/linuxkpicommon/include/linux/skbuff.h has been modified more recently. So I reverted stable back to commit 1b33afb7e88b2e36db2083c488de41dbe097ef49 on stable 13 and rebuilt and reinstalled it, and the wireless card seems to work again. I am able to use pkg update and fetch distfiles without a kernel panic. <br></div><div><br></div><div>Essentially on the stable 13 branch commits:</div><div><br></div><div>de40bc6f3d6bf8ab1a4b546630cc847a3b8c5113 <font face="arial black, sans-serif"> </font><span style="background-color:rgba(234,238,242,0.5);color:rgb(31,35,40)"><font face="arial, sans-serif">LinuxKPI: skbuff.h: fix -Warray-bounds warnings</font></span><span style="background-color:rgba(234,238,242,0.5);color:rgb(31,35,40);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI","Noto Sans",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji";font-size:16px;font-weight:600"> </span><span style="background-color:rgba(234,238,242,0.5);color:rgb(31,35,40);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI","Noto Sans",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji";font-weight:600">and</span><br></div><div><span style="background-color:rgba(234,238,242,0.5);color:rgb(31,35,40);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI","Noto Sans",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji";font-size:16px;font-weight:600"><br></span></div><div>b30d9f2e5204b475cf79150542b758c3dab14e70 <font face="arial, sans-serif"><span style="color:rgb(31,35,40)">LinuxKPI: skbuff.h: add more (skeleton) functions used by wireless dr</span><span style="color:rgb(31,35,40)">ivers</span></font></div><div><font face="arial, sans-serif"><span style="color:rgb(31,35,40)"><br></span></font></div><div><font face="arial, sans-serif"><span style="color:rgb(31,35,40)">were reverted (along with everything else after June 25th 2023) and it seems to work. I'm no kernel expert and there were quite a few commits for linux emulation and linuxkpi between then and now. But at least the wireless card is working again. BTW, I have to manually download these sources from github and copy them to a thumb drive because "git clone"ing a branch doesn't seem to work at all, it just downloads for 30 seconds and just sits and does nothing after that. </span></font></div><div><font face="arial, sans-serif"><span style="color:rgb(31,35,40)"><br></span></font></div><div><font color="#1f2328" face="arial, sans-serif">Thanks for your time, </font></div><div><font color="#1f2328" face="arial, sans-serif">Jeremy Cox</font></div></div><div class="gmail-yj6qo"></div><div class="gmail-adL"></div></div></div><div class="gmail-hi" style="padding:0px;width:auto;background:rgb(242,242,242);margin:0px;border-bottom-left-radius:1px;border-bottom-right-radius:1px"></div></div></div><div class="gmail-ajx" style="clear:both"></div></div><div class="gmail-gA gmail-gt gmail-acV" style="font-size:0.875rem;padding:0px;width:auto;border-bottom-left-radius:0px;border-bottom-right-radius:0px;border-top:none;margin:0px;background-image:initial;background-position:initial;background-size:initial;background-repeat:initial;background-origin:initial;background-clip:initial;font-family:"Google Sans",Roboto,RobotoDraft,Helvetica,Arial,sans-serif"><div class="gmail-gB gmail-xu" style="border-top:0px;padding:0px"><div class="gmail-ip gmail-iq" style="clear:both;margin:0px;padding:16px 0px;border-top:none"><div id="gmail-:1qh"><table class="gmail-cf gmail-wS" role="presentation" style="border-collapse:collapse"><tbody><tr><td class="gmail-amq" style="padding:0px 16px;vertical-align:top;width:44px"><img id="gmail-:mr_13" name=":mr" src="https://lh3.googleusercontent.com/a/AAcHTtf_UHbvtZSoZZJ4l4JAGrPp6FgLvu0zabWH2ij2rmh4Qg=s40-p" class="gmail-ajn gmail-bofPge" style="width: 40px; height: 40px; border-radius: 50%; display: block;"></td><td class="gmail-amr" style="padding:0px;width:1504px"><div class="gmail-nr gmail-wR" style="box-sizing:border-box;border-radius:1px;padding:0px;border:none;margin:0px"><div class="gmail-amn" style="color:inherit;height:auto;padding:0px;display:flex;line-height:20px"><span id="gmail-:1qz" role="link" tabindex="0" class="gmail-ams gmail-bkH" style="border:1px solid rgb(116,119,117);display:inline-flex;font-size:0.875rem;background:none;border-radius:18px;box-sizing:border-box;color:rgb(68,71,70);height:36px;outline:none;padding:0px 16px 0px 12px;min-width:104px;margin-right:8px">Reply</span><span id="gmail-:1qm" role="link" tabindex="0" class="gmail-ams gmail-bkG" style="border:1px solid rgb(116,119,117);display:inline-flex;font-size:0.875rem;background:none;border-radius:18px;box-sizing:border-box;color:rgb(68,71,70);height:36px;outline:none;padding:0px 16px 0px 12px;min-width:104px;margin-right:8px">Forward</span></div></div></td></tr></tbody></table></div></div></div></div></div>help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPH9q1pBL0rOwNoYoFvH6RvAHWjAn6MHgtL4=7cq9f-1RsJMGQ>