Date: Wed, 02 Apr 2025 05:23:22 +0000 From: "Steven Harms (High-Security Mail)" <sgharms@stevengharms.com> To: "freebsd-current@freebsd.org" <freebsd-current@FreeBSD.org> Subject: Logged Blank at kernel panic (?) (WAS: Re: Kernel panic on HEAD (93cf4310 2025-03-28)) Message-ID: <1M48yy09C3z85rk72AgFB_MYJ28ODexPUbUwlhRoxSGCLVdQVRmPzgOz_j_Oqj-u-suIcu-BjBwyVNBUh88xY6b3e_1Nd9v2EoOunXivtO0=@stevengharms.com>
index | next in thread | raw e-mail
[-- Attachment #1 --] With some additional insights I can add: - I can boot in single user on 15.0-CURRENT (Thanks Dave C. for the idea). We're definitely getting to running the rc's and it's something that breaks about the time /etc/rc.d/devmatch starts - At present there are no module loads in /etc/rc.conf: kld_list is commented-out - It seems like the cut off is when the kernel starts working with the 'hms' subsystem (see below) I was able to get the panic output from messages -- but there's no "here's where we failed," AFAICT. Have I gotten all the useful debugging messages at this point? Thanks for help. Hopefully useful log data are below. Steven === PANIC === Apr 2 04:16:27 freebsd kernel: iichid0: <DELL073B:00 06CB:7E7E I2C HID device> at addr 0x2c irq 51 on iicbus0 Apr 2 04:16:27 freebsd kernel: hidbus0: <HID bus> on iichid0 Apr 2 04:16:27 freebsd kernel: Apr 2 04:16:27 freebsd syslogd: last message repeated 1 times Apr 2 04:16:27 freebsd kernel: Fatal trap 12: page fault while in kernel mode Apr 2 04:16:27 freebsd kernel: cpuid = 0; apic id = 00 Apr 2 04:16:27 freebsd kernel: fault virtual address = 0x0 Apr 2 04:16:27 freebsd kernel: fault code = supervisor read instruction, page not present Apr 2 04:16:27 freebsd kernel: instruction pointer = 0x20:0x0 Apr 2 04:16:27 freebsd kernel: stack pointer = 0x28:0xfffffe0068dfce38 Apr 2 04:16:27 freebsd kernel: frame pointer = 0x28:0xfffffe0068dfce60 Apr 2 04:16:27 freebsd kernel: code segment = base 0x0, limit 0xfffff, type 0x1b Apr 2 04:16:27 freebsd kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 Apr 2 04:16:27 freebsd kernel: processor eflags = interrupt enabled, resume, IOPL = 0 - I'm surprised by the "blank like" repeated 1 times - This portion of the trace didn't really help me, but these call stacks seemed interesting too Interesting stack data? Apr 2 04:16:27 freebsd kernel: KDB: stack backtrace: Apr 2 04:16:27 freebsd kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0068dfcb60 Apr 2 04:16:27 freebsd kernel: vpanic() at vpanic+0x136/frame 0xfffffe0068dfcc90 Apr 2 04:16:27 freebsd kernel: panic() at panic+0x43/frame 0xfffffe0068dfccf0 Apr 2 04:16:27 freebsd kernel: trap_pfault() at trap_pfault+0x466/frame 0xfffffe0068dfcd60 Apr 2 04:16:27 freebsd kernel: calltrap() at calltrap+0x8/frame 0xfffffe0068dfcd60 Apr 2 04:16:27 freebsd kernel: --- trap 0xc, rip = 0, rsp = 0xfffffe0068dfce38, rbp = 0xfffffe0068dfce60 --- Apr 2 04:16:27 freebsd kernel: ??() at 0/frame 0xfffffe0068dfce60 Apr 2 04:16:27 freebsd kernel: ithread_loop() at ithread_loop+0x266/frame 0xfffffe0068dfcef0 Apr 2 04:16:27 freebsd kernel: fork_exit() at fork_exit+0x82/frame 0xfffffe0068dfcf30 Apr 2 04:16:27 freebsd kernel: fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0068dfcf30 Apr 2 04:16:27 freebsd kernel: --- trap 0, rip = 0, rsp = 0x1, rbp = 0x35013fc5ea44 ---Apr 2 04:16:27 freebsd kernel: KDB: enter: panic ==== Using my old, stable kernel:=== Here's what comes next: Mar 16 21:43:21 freebsd kernel: iichid0: <DELL073B:00 06CB:7E7E I2C HID device> at addr 0x2c irq 51 on iicbus0 Mar 16 21:43:21 freebsd kernel: hidbus0: <HID bus> on iichid0 Mar 16 21:43:21 freebsd kernel: hms0: <DELL073B:00 06CB:7E7E Mouse> on hidbus0 Mar 16 21:43:21 freebsd kernel: hms0: 2 buttons and [XY] coordinates ID=2 Mar 16 21:43:21 freebsd kernel: hmt0: <DELL073B:00 06CB:7E7E TouchPad> on hidbus0 Mar 16 21:43:21 freebsd kernel: hconf0: <DELL073B:00 06CB:7E7E Configuration> on hidbus0 Mar 16 21:43:21 freebsd kernel: hmt0: Multitouch touchpad with 0 external buttons, click-pad Mar 16 21:43:21 freebsd kernel: hmt0: 5 contacts with [C] properties. Report range [0:0] - [1229:749] Mar 16 21:43:21 freebsd kernel: wlan0: link state changed to UP Mar 16 21:43:21 freebsd kernel: ugen0.3: <vendor 0x8087 product 0x0a2a> at usbus0 --- Public Key: 22BE39E2FA68D8BA8DC4B43A55A16D8CE2B036DE Messages from this account are considered the best-secured and most reliable. Send information regarding health, wealth, or requiring higher standards of security to this address. Sent with [Proton Mail](https://proton.me/mail/home) secure email. On Saturday, March 29th, 2025 at 8:46 PM, Steven Harms (High-Security Mail) <sgharms@stevengharms.com> wrote: > Hey folks, > > I tried building current from source and when I boot, I get a panic after the final Autoloading module (pchtherm, in my case) loads. Comparing to the stable 14.2 kernel, the next step is for "Setting up harvesting," "Feeding entropy," and starting wpa_supplicant. > > To be clear, this commit is not the thing that broke things. > > In pursuit of getting more data, I'm trying to isolate two questions: > > - Why am I not getting any panic dump artifacts? > - Did I built current from HEAD properly? > > In effort to get more data, I'm following: https://docs.freebsd.org/en/books/developers-handbook/kerneldebug/ > > - Confirm that /etc/rc.conf has dumpdev set to AUTO > - Both /etc/fstab and swap info confirm that /dev/ada0p3 is configured for swap. It is also the only device. Therefore I expect dumped to write to that device > - Yet despite the kernel panicking, there's nothing in that directory except file "minfree" which says "2048" in it. > - I manually set dumpdir to "/var/crash" which has 750 permissions and try to boot the new kernel; panic; boot into stable kernel and...nothing in /var/crash except minfree > > Now, I'm still very new to building CURRENT, but I've been following: > > https://docs.freebsd.org/en/books/handbook/cutting-edge/#updating-src-quick-start > > # etcupdate extract (1) > # etcupdate diff (2) > # git pull -C /usr/src (1) > check /usr/src/UPDATING (2) > # cd /usr/src (3) > # make -j4 buildworld (4) > # make -j4 kernel (5)# shutdown -r now (6) > > And am after the shutdown -r now step: > > If my process for kernel and world build is correct, then I have found a panic. How can I help/debug? > > Thanks! > > Steven > > --- > > Public Key: 22BE39E2FA68D8BA8DC4B43A55A16D8CE2B036DE > > Messages from this account are considered the best-secured and most reliable. Send information regarding health, wealth, or requiring higher standards of security to this address. > > Sent with [Proton Mail](https://proton.me/mail/home) secure email. [-- Attachment #2 --] <div style="font-family: Arial, sans-serif; font-size: 14px;">With some additional insights I can add:<br><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><ol data-editing-info="{"orderedStyleType":1,"unorderedStyleType":1}" style="margin-top: 0px; margin-bottom: 0px;" data-listchain="__List_Chain_225"><li style="list-style-type: "1. ";"><span>I can boot in single user on 15.0-CURRENT (Thanks Dave C. for the idea). We're definitely getting to running the rc's and it's something that breaks about the time /etc/rc.d/devmatch starts</span></li><ol style="margin-top: 0px; margin-bottom: 0px; list-style-type: lower-alpha;"><li style=""><span>At present there are no module loads in /etc/rc.conf: kld_list is commented-out</span></li><li style=""><span>It <i>seems</i> like the cut off is when the kernel starts working with the 'hms' subsystem (see below)</span></li></ol></ol><div><span>I was able to get the panic output from messages -- but there's no "here's where we failed," AFAICT. Have I gotten all the useful debugging messages at this point?<br><br></span><div><span>Thanks for help. Hopefully useful log data are below.</span></div><div><span><br></span></div><div><span>Steven</span></div></div><div><br></div><div>=== PANIC ===<br><br><span>Apr 2 04:16:27 freebsd kernel: iichid0: <DELL073B:00 06CB:7E7E I2C HID device> at addr 0x2c irq 51 on iicbus0</span><div><span>Apr 2 04:16:27 freebsd kernel: hidbus0: <HID bus> on iichid0</span></div><div><span>Apr 2 04:16:27 freebsd kernel: </span></div><div><span>Apr 2 04:16:27 freebsd syslogd: last message repeated 1 times</span></div><div><span>Apr 2 04:16:27 freebsd kernel: Fatal trap 12: page fault while in kernel mode</span></div><div><span>Apr 2 04:16:27 freebsd kernel: cpuid = 0; apic id = 00</span></div><div><span>Apr 2 04:16:27 freebsd kernel: fault virtual address = 0x0</span></div><div><span>Apr 2 04:16:27 freebsd kernel: fault code = supervisor read instruction, page not present</span></div><div><span>Apr 2 04:16:27 freebsd kernel: instruction pointer = 0x20:0x0</span></div><div><span>Apr 2 04:16:27 freebsd kernel: stack pointer = 0x28:0xfffffe0068dfce38</span></div><div><span>Apr 2 04:16:27 freebsd kernel: frame pointer = 0x28:0xfffffe0068dfce60</span></div><div><span>Apr 2 04:16:27 freebsd kernel: code segment = base 0x0, limit 0xfffff, type 0x1b</span></div><div><span>Apr 2 04:16:27 freebsd kernel: = DPL 0, pres 1, long 1, def32 0, gran 1</span></div><div><span>Apr 2 04:16:27 freebsd kernel: processor eflags = interrupt enabled, resume, IOPL = 0</span></div><span></span></div><div><br></div><div><ol data-editing-info="{"orderedStyleType":1,"unorderedStyleType":1}" style="margin-top: 0px; margin-bottom: 0px;" data-listchain="__List_Chain_226"><li style="list-style-type: "1. ";"><span>I'm surprised by the "blank like" repeated 1 times</span></li><li style="list-style-type: "2. ";"><span>This portion of the trace didn't really help me, but these call stacks seemed interesting too</span></li></ol><div><br></div><div><span>Interesting stack data?</span><div><br></div><div><span>Apr 2 04:16:27 freebsd kernel: KDB: stack backtrace:</span><div><span>Apr 2 04:16:27 freebsd kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0068dfcb60</span></div><div><span>Apr 2 04:16:27 freebsd kernel: vpanic() at vpanic+0x136/frame 0xfffffe0068dfcc90</span></div><div><span>Apr 2 04:16:27 freebsd kernel: panic() at panic+0x43/frame 0xfffffe0068dfccf0</span></div><div><span>Apr 2 04:16:27 freebsd kernel: trap_pfault() at trap_pfault+0x466/frame 0xfffffe0068dfcd60</span></div><div><span>Apr 2 04:16:27 freebsd kernel: calltrap() at calltrap+0x8/frame 0xfffffe0068dfcd60</span></div><div><span>Apr 2 04:16:27 freebsd kernel: --- trap 0xc, rip = 0, rsp = 0xfffffe0068dfce38, rbp = 0xfffffe0068dfce60 ---</span></div><div><span>Apr 2 04:16:27 freebsd kernel: ??() at 0/frame 0xfffffe0068dfce60</span></div><div><span>Apr 2 04:16:27 freebsd kernel: ithread_loop() at ithread_loop+0x266/frame 0xfffffe0068dfcef0</span></div><div><span>Apr 2 04:16:27 freebsd kernel: fork_exit() at fork_exit+0x82/frame 0xfffffe0068dfcf30</span></div><div><span>Apr 2 04:16:27 freebsd kernel: fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0068dfcf30</span></div><div><span>Apr 2 04:16:27 freebsd kernel: --- trap 0, rip = 0, rsp = 0x1, rbp = 0x35013fc5ea44 ---</span></div><span><span>Apr 2 04:16:27 freebsd kernel: KDB: enter: panic</span></span><span></span></div><br></div><div><br></div><div>==== Using my old, stable kernel:===</div><div><br></div><div>Here's what comes next:</div><div><br></div><div><span>Mar 16 21:43:21 freebsd kernel: iichid0: <DELL073B:00 06CB:7E7E I2C HID device> at addr 0x2c irq 51 on iicbus0</span><div><span>Mar 16 21:43:21 freebsd kernel: hidbus0: <HID bus> on iichid0</span></div><div><span>Mar 16 21:43:21 freebsd kernel: hms0: <DELL073B:00 06CB:7E7E Mouse> on hidbus0</span></div><div><span>Mar 16 21:43:21 freebsd kernel: hms0: 2 buttons and [XY] coordinates ID=2</span></div><div><span>Mar 16 21:43:21 freebsd kernel: hmt0: <DELL073B:00 06CB:7E7E TouchPad> on hidbus0</span></div><div><span>Mar 16 21:43:21 freebsd kernel: hconf0: <DELL073B:00 06CB:7E7E Configuration> on hidbus0</span></div><div><span>Mar 16 21:43:21 freebsd kernel: hmt0: Multitouch touchpad with 0 external buttons, click-pad</span></div><div><span>Mar 16 21:43:21 freebsd kernel: hmt0: 5 contacts with [C] properties. Report range [0:0] - [1229:749]</span></div><div><span>Mar 16 21:43:21 freebsd kernel: wlan0: link state changed to UP</span></div><div><span>Mar 16 21:43:21 freebsd kernel: ugen0.3: <vendor 0x8087 product 0x0a2a> at usbus0</span></div><span></span><br></div><div><br></div></div></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div> <div class="protonmail_signature_block" style="font-family: Arial, sans-serif; font-size: 14px;"> <div class="protonmail_signature_block-user"> <div>---<br></div><div><br></div><div style="font-family: Arial, sans-serif; font-size: 14px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Public Key: <a title="22BE39E2FA68D8BA8DC4B43A55A16D8CE2B036DE" href="https://22BE39E2FA68D8BA8DC4B43A55A16D8CE2B036DE"><span>22BE39E2FA68D8BA8DC4B43A55A16D8CE2B036DE</span></a><br></div><div style="font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34);"><br></div><div>Messages from this account are considered the best-secured and most reliable. Send information regarding health, wealth, or requiring higher standards of security to this address.<br></div> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div> <div class="protonmail_signature_block-proton"> Sent with <a target="_blank" href="https://proton.me/mail/home">Proton Mail</a> secure email. </div> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div class="protonmail_quote"> On Saturday, March 29th, 2025 at 8:46 PM, Steven Harms (High-Security Mail) <sgharms@stevengharms.com> wrote:<br> <blockquote class="protonmail_quote" type="cite"> <div style="font-family: Arial, sans-serif; font-size: 14px;">Hey folks,</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">I tried building current from source and when I boot, I get a panic after the final Autoloading module (pchtherm, in my case) loads. Comparing to the stable 14.2 kernel, the next step is for "Setting up harvesting," "Feeding entropy," and starting wpa_supplicant.</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">To be clear, this commit is not the thing that broke things.</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">In pursuit of getting more data, I'm trying to isolate two questions:</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><ol data-listchain="__List_Chain_221" style="margin-top: 0px; margin-bottom: 0px;" data-editing-info="{"orderedStyleType":1,"unorderedStyleType":1}"><li style="list-style-type: "1. ";">Why am I not getting any panic dump artifacts?</li><li style="list-style-type: "2. ";">Did I built current from HEAD properly?</li></ol></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">In effort to get more data, I'm following: <span><a href="https://docs.freebsd.org/en/books/developers-handbook/kerneldebug/" rel="noreferrer nofollow noopener" target="_blank">https://docs.freebsd.org/en/books/developers-handbook/kerneldebug/</a></span></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div><ol data-listchain="__List_Chain_222" style="margin-top: 0px; margin-bottom: 0px;" data-editing-info="{"orderedStyleType":1,"unorderedStyleType":1}"><li style="list-style-type: "1. ";"><font face="Arial, sans-serif">Confirm that /etc/rc.conf has dumpdev set to AUTO</font></li><li style="list-style-type: "2. ";"><font face="Arial, sans-serif">Both /etc/fstab and swap info confirm that /dev/ada0p3 is configured for swap. It is also the only device. Therefore I expect dumped to write to that device</font></li><li style="list-style-type: "3. ";"><font face="Arial, sans-serif">Yet despite the kernel panicking, there's nothing in that directory except file "minfree" which says "2048" in it.</font></li><li style="list-style-type: "4. ";"><font face="Arial, sans-serif">I manually set dumpdir to "/var/crash" which has 750 permissions and try to boot the new kernel; panic; boot into stable kernel and...nothing in /var/crash except <code>minfree</code></font></li></ol></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Now, I'm still very new to building CURRENT, but I've been following:</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span><a href="https://docs.freebsd.org/en/books/handbook/cutting-edge/#updating-src-quick-start" rel="noreferrer nofollow noopener" target="_blank">https://docs.freebsd.org/en/books/handbook/cutting-edge/#updating-src-quick-start</a></span><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><span># etcupdate extract (1)</span><div><span># etcupdate diff (2)</span></div><div><span># git pull -C /usr/src (1)</span></div><div><span>check /usr/src/UPDATING (2)</span></div><div><span># cd /usr/src (3)</span></div><div><span># make -j4 buildworld (4)</span></div><div><span># make -j4 kernel (5)</span></div><span># shutdown -r now (6)</span><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">And am after the <code>shutdown -r now</code> step:</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">If my process for kernel and world build is correct, then I have found a panic. How can I help/debug?</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Thanks!</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div><div style="font-family: Arial, sans-serif; font-size: 14px;">Steven</div><div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div> <div style="font-family: Arial, sans-serif; font-size: 14px;" class="protonmail_signature_block"> <div class="protonmail_signature_block-user"> <div>---<br></div><div><br></div><div style="font-family: Arial, sans-serif; font-size: 14px; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">Public Key: <a href="https://22BE39E2FA68D8BA8DC4B43A55A16D8CE2B036DE" title="22BE39E2FA68D8BA8DC4B43A55A16D8CE2B036DE" target="_blank" rel="noreferrer nofollow noopener"><span>22BE39E2FA68D8BA8DC4B43A55A16D8CE2B036DE</span></a><br></div><div style="font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Arial, Helvetica, sans-serif; color: rgb(34, 34, 34);"><br></div><div>Messages from this account are considered the best-secured and most reliable. Send information regarding health, wealth, or requiring higher standards of security to this address.<br></div> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br></div> <div class="protonmail_signature_block-proton"> Sent with <a href="https://proton.me/mail/home" target="_blank" rel="noreferrer nofollow noopener">Proton Mail</a> secure email. </div> </div> </blockquote><br> </div>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1M48yy09C3z85rk72AgFB_MYJ28ODexPUbUwlhRoxSGCLVdQVRmPzgOz_j_Oqj-u-suIcu-BjBwyVNBUh88xY6b3e_1Nd9v2EoOunXivtO0=>