Date: Thu, 29 Jun 2023 15:03:18 +0000 From: Sha Faisal <sfaisal@juniper.net> To: "andrew@fubar.geek.nz" <andrew@fubar.geek.nz> Cc: "freebsd-arm@FreeBSD.org" <freebsd-arm@freebsd.org>, Steve Kiernan <stevek@juniper.net>, Mahesh Vardhamanaiah <maheshmv@juniper.net>, Amit Verma <amitv@juniper.net> Subject: Re: Kernel crash before login prompt Message-ID: <BYAPR05MB6677AED54B7EA054E7B8C170B725A@BYAPR05MB6677.namprd05.prod.outlook.com> In-Reply-To: <d12ce5cd719f9e1c2f4c5f53b89e0265@fubar.geek.nz> References: <BYAPR05MB66779D6947C5BE371C8EA3B3B724A@BYAPR05MB6677.namprd05.prod.outlook.com> <BYAPR05MB6677A86EDDAEDBE5E0DD6266B724A@BYAPR05MB6677.namprd05.prod.outlook.com> <E03D720A-B18B-45C6-9BD6-7CA6FDA83560@fubar.geek.nz> <BYAPR05MB6677E63414F88873FF17174BB724A@BYAPR05MB6677.namprd05.prod.outlook.com> <6537067F-670A-4B52-87FA-FF2DBAD68120@fubar.geek.nz> <BYAPR05MB667764EF552E113B08FD1222B725A@BYAPR05MB6677.namprd05.prod.outlook.com> <d12ce5cd719f9e1c2f4c5f53b89e0265@fubar.geek.nz>
next in thread | previous in thread | raw e-mail | index | archive | help
--_000_BYAPR05MB6677AED54B7EA054E7B8C170B725ABYAPR05MB6677namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hello Andrew, The addr2line for the address points to, [sfaisal@svl-bsdx-02 /b/sfaisal/stable_main/src]$ addr2line -e ../obj/arm64= /juniper/kernels/JNPR-ARM64-WIT/kernel/kernel.full -f 0xffff0000002244dc sched_add /b/sfaisal/stable_main/src/sys/kern/sched_ule.c:1309 Sched_setcpu is the exact function which calls TDQ_LOCK(__mtx_lock_spin_fla= gs) , and sched_setcpu is an inline called from sched_add() Regards Faisal Juniper Business Use Only From: andrew@fubar.geek.nz <andrew@fubar.geek.nz> Date: Thursday, 29 June 2023 at 20:17 To: Sha Faisal <sfaisal@juniper.net> Cc: freebsd-arm@FreeBSD.org <freebsd-arm@freebsd.org>, Steve Kiernan <steve= k@juniper.net>, Mahesh Vardhamanaiah <maheshmv@juniper.net> Subject: Re: Kernel crash before login prompt [External Email. Be cautious of content] Can you find out which function 0xffff0000002244dc is from (i.e. the caller of __mtx_lock_spin_flags)? It looks like it is in sched_pickcpu, but it also looks like sched_pickcpu should return before calling __mtx_lock_spin_flags based on the boot log. Andrew On 2023-06-29 14:26, Sha Faisal wrote: > Below are the values you asked for > > Just after dpcpu_init() in initarm() > > --------In initarm and pcpup->pc_dynamic =3D 0x2886480 > > --------In initarm and dpcpu_off[0] =3D 0x2886480 > > --------In initarm and DPCPU_START =3D 0xffff000000681b80 > > --------In initarm and DPCPU_STOP =3D 0xffff000000687300 > > --------In initarm and DPCPU_BYTES =3D 0x5780 > > --------In initarm and DPCPU_SIZE =3D 0x6000 > > --------In initarm and PAGE_SIZE =3D 0x1000 > > --------In initarm and pcpu =3D 0xffff00000095bd80 > > --------In initarm and pcpu_entry_timerstate is 0xffff00000068ac80---- > > > --------In initarm and __start_set_pcpu is 0---- > > --------In initarm and pcpu_entry_tdq is 0xffff00000068ac80---- > > ---<<BOOT>>--- > > Below are after it crashes on db prompt: > > db> p __start_set_pcpu > ffff000000681b80 > db> p __stop_set_pcpu > ffff000000687300 > db> show dpcpu_off > dpcpu_off[ 0] =3D 0x2886480 (+ DPCPU_START =3D 0xffff000002f08000) > > db> p pcpu_entry_tdq > > ffff000000681c80 > > Regards > > Faisal > > Juniper Business Use Only > From: owner-freebsd-arm@freebsd.org <owner-freebsd-arm@freebsd.org> > on behalf of Andrew Turner <andrew@fubar.geek.nz> > Date: Thursday, 29 June 2023 at 15:27 > To: Sha Faisal <sfaisal@juniper.net> > Cc: freebsd-arm@FreeBSD.org <freebsd-arm@FreeBSD.org>, Steve Kiernan > <stevek@juniper.net>, Mahesh Vardhamanaiah <maheshmv@juniper.net> > Subject: Re: Kernel crash before login prompt > > [External Email. Be cautious of content] > > It looks like that exception is due to DPCPU_ID_PTR calculating an > invalid pointer. > > Can you print the contents of the dpcpu_off array, and the addresses > of __start_set_pcpu and pcpu_entry_tdq. > > Andrew > >> On 28 Jun 2023, at 19:45, Sha Faisal <sfaisal@juniper.net> wrote: >> >> Hello Andrew, >> >> Attached is the boot log, also at the end I have pasted the remote >> gdb backtrace >> >> Regards >> >> faisal >> >> Juniper Business Use Only >> >> From: Andrew Turner <andrew@fubar.geek.nz> >> Date: Wednesday, 28 June 2023 at 20:47 >> To: Sha Faisal <sfaisal@juniper.net> >> Cc: freebsd-arm@FreeBSD.org <freebsd-arm@FreeBSD.org>, Steve Kiernan >> <stevek@juniper.net>, Mahesh Vardhamanaiah <maheshmv@juniper.net> >> Subject: Re: Kernel crash before login prompt >> >> [External Email. Be cautious of content] >> >>> On 28 Jun 2023, at 13:18, Sha Faisal <sfaisal@juniper.net> wrote: >>> >>> Hello FreeBSD team, >>> >>> We have upgraded our FreeBSD kernel from 12 to main branch and >>> with latest main branch one of our boxes with >>> >>> Broadcom iProc Northstar2 BCM5871X SoC (Quad-core ARM Cortex-A57 >>> 64-bit 2GHz ARMv8-A CPU architecture) is crashing at >>> mtx_spin_lock_flags(), with data_abort exception. >>> >>> Sometimes the exception is raised at DPCPCU_PTR() functions. >>> >>> Has anyone tried to boot freeBSD14 on the above mentioned BCM SoC? >>> >>> >>> Please share your inputs and comment >> >> I don=92t know of any issues with either of these, and don=92t know >> of anyone running FreeBSD on this SoC. >> >> Can you provide more details on what is failing, e.g. the boot log? >> >> Andrew --_000_BYAPR05MB6677AED54B7EA054E7B8C170B725ABYAPR05MB6677namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc= hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of= fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40">; <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"> <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:Menlo; panose-1:2 11 6 9 3 8 4 2 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; font-size:10.0pt; font-family:"Calibri",sans-serif;} span.EmailStyle19 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:windowtext;} p.p1, li.p1, div.p1 {mso-style-name:p1; margin:0cm; font-size:13.0pt; font-family:Menlo; color:black;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt; mso-ligatures:none;} @page WordSection1 {size:612.0pt 792.0pt; margin:72.0pt 72.0pt 72.0pt 72.0pt;} div.WordSection1 {page:WordSection1;} --></style> </head> <body lang=3D"EN-IN" link=3D"#0563C1" vlink=3D"#954F72" style=3D"word-wrap:= break-word"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US">Hello Andrew,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US">The addr2line for the address points to,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US"><o:p> </o:p></span></p> <p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q= uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">[sfaisal@svl-b= sdx-02 /b/sfaisal/stable_main/src]$ addr2line -e ../obj/arm64/juniper/kerne= ls/JNPR-ARM64-WIT/kernel/kernel.full -f 0xffff0000002244dc<o:p></o:p></span= ></p> <p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q= uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">sched_add<o:p>= </o:p></span></p> <p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q= uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">/b/sfaisal/sta= ble_main/src/sys/kern/sched_ule.c:1309<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US">Sched_setcpu is the exact function which calls TDQ_LOCK(</span><spa= n style=3D"font-size:11.0pt">__mtx_lock_spin_flags) </span><span style=3D"font-size:11.0pt;mso-fareast-language:EN-US">, and sc= hed_setcpu is an inline called from sched_add()<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US">Regards<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US">Faisal<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language= :EN-US"><o:p> </o:p></span></p> <div id=3D"mail-editor-reference-message-container"> <div> <div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm = 0cm 0cm"> <p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt"><b><span style=3D"fon= t-size:12.0pt;color:black"><br> </p> <p style=3D"font-family:Calibri;font-size:7pt;color:#000000;margin:15pt;fon= t-style:normal;font-weight:normal;text-decoration:none;" align=3D"Center"> Juniper Business Use Only<br> </p> From: </span></b><span style=3D"font-size:12.0pt;color:black">andrew@fubar.= geek.nz <andrew@fubar.geek.nz><br> <b>Date: </b>Thursday, 29 June 2023 at 20:17<br> <b>To: </b>Sha Faisal <sfaisal@juniper.net><br> <b>Cc: </b>freebsd-arm@FreeBSD.org <freebsd-arm@freebsd.org>, Steve K= iernan <stevek@juniper.net>, Mahesh Vardhamanaiah <maheshmv@junipe= r.net><br> <b>Subject: </b>Re: Kernel crash before login prompt<o:p></o:p></span> <p></p> </div> <div> <p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">[External Email. Be= cautious of content]<br> <br> <br> Can you find out which function 0xffff0000002244dc is from (i.e. the<br> caller of __mtx_lock_spin_flags)? It looks like it is in sched_pickcpu,<br> but it also looks like sched_pickcpu should return before calling<br> __mtx_lock_spin_flags based on the boot log.<br> <br> Andrew<br> <br> On 2023-06-29 14:26, Sha Faisal wrote:<br> > Below are the values you asked for<br> ><br> > Just after dpcpu_init() in initarm()<br> ><br> > --------In initarm and pcpup->pc_dynamic =3D 0x2886480<br> ><br> > --------In initarm and dpcpu_off[0] =3D 0x2886480<br> ><br> > --------In initarm and DPCPU_START =3D 0xffff000000681b80<br> ><br> > --------In initarm and DPCPU_STOP =3D 0xffff000000687300<br> ><br> > --------In initarm and DPCPU_BYTES =3D 0x5780<br> ><br> > --------In initarm and DPCPU_SIZE =3D 0x6000<br> ><br> > --------In initarm and PAGE_SIZE =3D 0x1000<br> ><br> > --------In initarm and pcpu =3D 0xffff00000095bd80<br> ><br> > --------In initarm and pcpu_entry_timerstate is 0xffff00000068ac80----= <br> ><br> ><br> > --------In initarm and __start_set_pcpu is 0----<br> ><br> > --------In initarm and pcpu_entry_tdq is 0xffff00000068ac80----<br> ><br> > ---<<BOOT>>---<br> ><br> > Below are after it crashes on db prompt:<br> ><br> > db> p __start_set_pcpu<br> > ffff000000681b80<br> > db> p __stop_set_pcpu<br> > ffff000000687300<br> > db> show dpcpu_off<br> > dpcpu_off[ 0] =3D 0x2886480 (+ DPCPU_START =3D 0xffff000002f08000)<br> ><br> > db> p pcpu_entry_tdq<br> ><br> > ffff000000681c80<br> ><br> > Regards<br> ><br> > Faisal<br> ><br> > Juniper Business Use Only<br> > From: owner-freebsd-arm@freebsd.org <owner-freebsd-arm@freebs= d.org><br> > on behalf of Andrew Turner <andrew@fubar.geek.nz><br> > Date: Thursday, 29 June 2023 at 15:27<br> > To: Sha Faisal <sfaisal@juniper.net><br> > Cc: freebsd-arm@FreeBSD.org <freebsd-arm@FreeBSD.org>, Steve Kie= rnan<br> > <stevek@juniper.net>, Mahesh Vardhamanaiah <maheshmv@juniper.= net><br> > Subject: Re: Kernel crash before login prompt<br> ><br> > [External Email. Be cautious of content]<br> ><br> > It looks like that exception is due to DPCPU_ID_PTR calculating an<br> > invalid pointer.<br> ><br> > Can you print the contents of the dpcpu_off array, and the addresses<b= r> > of __start_set_pcpu and pcpu_entry_tdq.<br> ><br> > Andrew<br> ><br> >> On 28 Jun 2023, at 19:45, Sha Faisal <sfaisal@juniper.net> w= rote:<br> >><br> >> Hello Andrew,<br> >><br> >> Attached is the boot log, also at the end I have pasted the = remote<br> >> gdb backtrace<br> >><br> >> Regards<br> >><br> >> faisal<br> >><br> >> Juniper Business Use Only<br> >><br> >> From: Andrew Turner <andrew@fubar.geek.nz><br> >> Date: Wednesday, 28 June 2023 at 20:47<br> >> To: Sha Faisal <sfaisal@juniper.net><br> >> Cc: freebsd-arm@FreeBSD.org <freebsd-arm@FreeBSD.org>, Steve= Kiernan<br> >> <stevek@juniper.net>, Mahesh Vardhamanaiah <maheshmv@juni= per.net><br> >> Subject: Re: Kernel crash before login prompt<br> >><br> >> [External Email. Be cautious of content]<br> >><br> >>> On 28 Jun 2023, at 13:18, Sha Faisal <sfaisal@juniper.net&g= t; wrote:<br> >>><br> >>> Hello FreeBSD team,<br> >>><br> >>> We have upgraded our FreeBSD kernel from 12 to main branch and= <br> >>> with latest main branch one of our boxes with<br> >>><br> >>> Broadcom iProc Northstar2 BCM5871X SoC (Quad-core ARM Cortex-A= 57<br> >>> 64-bit 2GHz ARMv8-A CPU architecture) is crashing at<br> >>> mtx_spin_lock_flags(), with data_abort exception.<br> >>><br> >>> Sometimes the exception is raised at DPCPCU_PTR() functions.<b= r> >>><br> >>> Has anyone tried to boot freeBSD14 on the above mentioned BCM = SoC?<br> >>><br> >>><br> >>> Please share your inputs and comment<br> >><br> >> I don=92t know of any issues with either of these, and don=92t kno= w<br> >> of anyone running FreeBSD on this SoC.<br> >><br> >> Can you provide more details on what is failing, e.g. the boot log= ?<br> >><br> >> Andrew<o:p></o:p></span></p> </div> </div> </div> </div> </body> </html> --_000_BYAPR05MB6677AED54B7EA054E7B8C170B725ABYAPR05MB6677namp_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi? <BYAPR05MB6677AED54B7EA054E7B8C170B725A>