Date: Tue, 4 Jul 2023 16:32:59 +0000 From: Sha Faisal <sfaisal@juniper.net> To: "andrew@fubar.geek.nz" <andrew@fubar.geek.nz>, Mahesh Vardhamanaiah <maheshmv@juniper.net> Cc: "freebsd-arm@FreeBSD.org" <freebsd-arm@freebsd.org>, Steve Kiernan <stevek@juniper.net>, Amit Verma <amitv@juniper.net> Subject: Re: Kernel crash before login prompt Message-ID: <BYAPR05MB667729839AB0CF24294B14F5B72EA@BYAPR05MB6677.namprd05.prod.outlook.com> In-Reply-To: <BYAPR05MB6677D1BF2C336E91D1DAFEABB72AA@BYAPR05MB6677.namprd05.prod.outlook.com> References: <BYAPR05MB66779D6947C5BE371C8EA3B3B724A@BYAPR05MB6677.namprd05.prod.outlook.com> <BYAPR05MB6677A86EDDAEDBE5E0DD6266B724A@BYAPR05MB6677.namprd05.prod.outlook.com> <E03D720A-B18B-45C6-9BD6-7CA6FDA83560@fubar.geek.nz> <BYAPR05MB6677E63414F88873FF17174BB724A@BYAPR05MB6677.namprd05.prod.outlook.com> <6537067F-670A-4B52-87FA-FF2DBAD68120@fubar.geek.nz> <BYAPR05MB667764EF552E113B08FD1222B725A@BYAPR05MB6677.namprd05.prod.outlook.com> <d12ce5cd719f9e1c2f4c5f53b89e0265@fubar.geek.nz> <BYAPR05MB6677AED54B7EA054E7B8C170B725A@BYAPR05MB6677.namprd05.prod.outlook.com> <1327534fa4a1da584a3a38b194999595@fubar.geek.nz> <BYAPR05MB6677FCD23054EF4D9CA0E4EAB725A@BYAPR05MB6677.namprd05.prod.outlook.com> <BYAPR05MB6677FCA278A0C5FED1B9B423B725A@BYAPR05MB6677.namprd05.prod.outlook.com> <PH0PR05MB7782189DE6FCAF371DC918E7D52AA@PH0PR05MB7782.namprd05.prod.outlook.com> <2bf93ae22548705e689e0bdd47f9d7db@fubar.geek.nz> <BYAPR05MB6677D1BF2C336E91D1DAFEABB72AA@BYAPR05MB6677.namprd05.prod.outlook.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--_000_BYAPR05MB667729839AB0CF24294B14F5B72EABYAPR05MB6677namp_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hello Andrew,
Can we have a short discussions on the below issue.
We are not able to debug further.
Please let me know your time zone and convenient time.
Regards
Faisal
Juniper Business Use Only
From: Sha Faisal <sfaisal@juniper.net>
Date: Friday, 30 June 2023 at 16:45
To: andrew@fubar.geek.nz <andrew@fubar.geek.nz>, Mahesh Vardhamanaiah <mahe=
shmv@juniper.net>
Cc: freebsd-arm@FreeBSD.org <freebsd-arm@freebsd.org>, Steve Kiernan <steve=
k@juniper.net>, Amit Verma <amitv@juniper.net>
Subject: Re: Kernel crash before login prompt
Hello Andrew,
Below is the disas of sched_pickcpu()
/b/sfaisal/stable_main/src/sys/kern/sched_ule.c:
1325 {
0xffff000000223724 <+0>: sub sp, sp, #0x90
0xffff000000223728 <+4>: stp x29, x30, [sp, #48]
0xffff00000022372c <+8>: stp x28, x27, [sp, #64]
0xffff000000223730 <+12>: stp x26, x25, [sp, #80]
0xffff000000223734 <+16>: stp x24, x23, [sp, #96]
0xffff000000223738 <+20>: stp x22, x21, [sp, #112]
0xffff00000022373c <+24>: stp x20, x19, [sp, #128]
0xffff000000223740 <+28>: add x29, sp, #0x30
1326 struct cpu_group *cg, *ccg;
1327 struct td_sched *ts;
1328 struct tdq *tdq;
1329 cpuset_t *mask;
1330 int cpu, pri, r, self, intr;
1331
1332 self =3D PCPU_GET(cpuid);
0xffff000000223744 <+32>: ldr w19, [x18, #60]
0xffff000000223748 <+36>: mov w21, w1
0xffff00000022374c <+40>: mov x20, x0
Also, below is the show allpcpu log
db> show allpcpu
Current CPU: 0
cpuid =3D 0
dynamic pcpu =3D 0x2886100
curthread =3D 0xffff0000008166a0: pid 0 tid 100000 critnest 1 "swapper"
curpcb =3D 0xffff00000068bb40
fpcurthread =3D none
idlethread =3D 0xffff000041b82000: tid 100003 "idle: cpu0"
spin locks held:
cpuid =3D 1
dynamic pcpu =3D 0x41393300
curthread =3D none
curpcb =3D 0
fpcurthread =3D none
idlethread =3D 0xffff000041b81880: tid 100004 "idle: cpu1"
spin locks held:
cpuid =3D 2
dynamic pcpu =3D 0x4139e300
curthread =3D none
curpcb =3D 0
fpcurthread =3D none
idlethread =3D 0xffff000041b81260: tid 100005 "idle: cpu2"
spin locks held:
cpuid =3D 3
dynamic pcpu =3D 0x413a9300
curthread =3D none
curpcb =3D 0
fpcurthread =3D none
idlethread =3D 0xffff000041b80c40: tid 100006 "idle: cpu3"
spin locks held:
Regards
Faisal
From: andrew@fubar.geek.nz <andrew@fubar.geek.nz>
Date: Friday, 30 June 2023 at 16:26
To: Mahesh Vardhamanaiah <maheshmv@juniper.net>
Cc: Sha Faisal <sfaisal@juniper.net>, freebsd-arm@FreeBSD.org <freebsd-arm@=
freebsd.org>, Steve Kiernan <stevek@juniper.net>, Amit Verma <amitv@juniper=
.net>
Subject: Re: Kernel crash before login prompt
[External Email. Be cautious of content]
On 2023-06-30 10:48, Mahesh Vardhamanaiah wrote:
> Hi Andrew,
>
> We are observing that self is getting -65536 at line 1329 in
> sched_pickcpu() before panic.
What is the disassembly around reading the cpuid? It looks like it is
getting 0xffff0000 when it should be zero.
>
> From the code flow
>
> sched_add [1](struct thread [2] *td [3], int flags [4])->
> sched_pickcpu()->PCPU_GET
>
> as below
>
> static int
>
> 1321 [5] sched_pickcpu [6](struct thread [2] *td [3], int flags [4])
>
> 1322 [7] {
>
> 1323 [8] struct cpu_group [9] *cg [10], *ccg;
>
> 1324 [11] struct td_sched [12] *ts [13];
>
> 1325 [14] struct tdq [15] *tdq [15];
>
> 1326 [16] cpuset_t [17] *mask [18];
>
> 1327 [19] int cpu [20], pri, r [21], self [22], intr;
>
> 1328 [23]
>
> 1329 [24] self [22] =3D PCPU_GET [25](cpuid [26]);
>
> PCPU_GET is defined differently in Freebsd14 compared to Freebsd12 but
> I am unable to dump the pcpup in db> prompt is it proper ?
>
> Is it expected to be a global ?
You can print all pcpu structs with "show allpcpu".
Andrew
--_000_BYAPR05MB667729839AB0CF24294B14F5B72EABYAPR05MB6677namp_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc=
hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of=
fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40">;
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"=
>
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0cm;
font-size:13.0pt;
font-family:Menlo;
color:black;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0cm;
font-size:13.0pt;
font-family:Menlo;
color:black;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang=3D"EN-IN" link=3D"#0563C1" vlink=3D"#954F72" style=3D"word-wrap:=
break-word">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US">Hello Andrew,
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US">Can we have a short discussions on the below issue.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US">We are not able to debug further.
<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US">Please let me know your time zone and convenient time.<o:p></o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US">Regards<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US">Faisal<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"><o:p> </o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"><o:p> </o:p></span></p>
<div id=3D"mail-editor-reference-message-container">
<div>
<div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm =
0cm 0cm">
<p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt"><b><span style=3D"fon=
t-size:12.0pt;color:black"><br>
</p>
<p style=3D"font-family:Calibri;font-size:7pt;color:#000000;margin:15pt;fon=
t-style:normal;font-weight:normal;text-decoration:none;" align=3D"Center">
Juniper Business Use Only<br>
</p>
From: </span></b><span style=3D"font-size:12.0pt;color:black">Sha Faisal &l=
t;sfaisal@juniper.net><br>
<b>Date: </b>Friday, 30 June 2023 at 16:45<br>
<b>To: </b>andrew@fubar.geek.nz <andrew@fubar.geek.nz>, Mahesh Vardha=
manaiah <maheshmv@juniper.net><br>
<b>Cc: </b>freebsd-arm@FreeBSD.org <freebsd-arm@freebsd.org>, Steve K=
iernan <stevek@juniper.net>, Amit Verma <amitv@juniper.net><br>
<b>Subject: </b>Re: Kernel crash before login prompt<o:p></o:p></span>
<p></p>
</div>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US">Hello Andrew,
</span><o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US">Below is the disas of sched_pickcpu()</span><o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"> </span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">/b/sfaisal/sta=
ble_main/src/sys/kern/sched_ule.c:</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">1325 &nb=
sp; {</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff000000223724 <+0>: sub sp, sp, #0x90=
</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff000000223728 <+4>: stp x29, x30, [sp=
, #48]</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff00000022372c <+8>: stp x28, x27, [sp=
, #64]</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff000000223730 <+12>: stp x26, x25, [sp=
, #80]</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff000000223734 <+16>: stp x24, x23, [sp=
, #96]</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff000000223738 <+20>: stp x22, x21, [sp=
, #112]</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff00000022373c <+24>: stp x20, x19, [sp=
, #128]</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff000000223740 <+28>: add x29, sp, #0x3=
0</span><o:p></o:p></p>
<p class=3D"p2"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> </span><=
o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">1326 &nb=
sp; struct cpu_group *cg, *ccg;</span><o:p></o:=
p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">1327 &nb=
sp; struct td_sched *ts;</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">1328 &nb=
sp; struct tdq *tdq;</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">1329 &nb=
sp; cpuset_t *mask;</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">1330 &nb=
sp; int cpu, pri, r, self, intr;</span><o:p></o=
:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">1331</span><o:=
p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US">1332 &nb=
sp; self =3D PCPU_GET(cpuid);</span><o:p></o:p>=
</p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff000000223744 <+32>: ldr w19, [x18, #6=
0]</span><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff000000223748 <+36>: mov w21, w1</span=
><o:p></o:p></p>
<p class=3D"p1"><span style=3D"font-size:11.0pt;font-family:"Calibri&q=
uot;,sans-serif;color:windowtext;mso-fareast-language:EN-US"> 0=
xffff00000022374c <+40>: mov x20, x0</span=
><o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"> </span><o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"> </span><o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"> </span><o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US">Also, below is the show allpcpu log</span><o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"> </span><o:p></o:p></p>
<p>db> show allpcpu<br>
Current CPU: 0<o:p></o:p></p>
<p>cpuid =3D 0<br>
dynamic pcpu =3D 0x2886100<br>
curthread =3D 0xffff0000008166a0: pid 0 tid 100000 critne=
st 1 "swapper"<br>
curpcb =3D 0xffff00000068bb40<br>
fpcurthread =3D none<br>
idlethread =3D 0xffff000041b82000: tid 100003 "idle: cpu0&=
quot;<br>
spin locks held:<o:p></o:p></p>
<p>cpuid =3D 1<br>
dynamic pcpu =3D 0x41393300<br>
curthread =3D none<br>
curpcb =3D 0<br>
fpcurthread =3D none<br>
idlethread =3D 0xffff000041b81880: tid 100004 "idle: cpu1&=
quot;<br>
spin locks held:<o:p></o:p></p>
<p>cpuid =3D 2<br>
dynamic pcpu =3D 0x4139e300<br>
curthread =3D none<br>
curpcb =3D 0<br>
fpcurthread =3D none<br>
idlethread =3D 0xffff000041b81260: tid 100005 "idle: cpu2&=
quot;<br>
spin locks held:<o:p></o:p></p>
<p>cpuid =3D 3<br>
dynamic pcpu =3D 0x413a9300<br>
curthread =3D none<br>
curpcb =3D 0<br>
fpcurthread =3D none<br>
idlethread =3D 0xffff000041b80c40: tid 100006 "idle: cpu3&=
quot;<br>
spin locks held:<o:p></o:p></p>
<p> <o:p></o:p></p>
<p>Regards<o:p></o:p></p>
<p>Faisal<o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"> </span><o:p></o:p></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt;mso-fareast-language=
:EN-US"> </span><o:p></o:p></p>
<div id=3D"mail-editor-reference-message-container">
<div>
<div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm =
0cm 0cm">
<p class=3D"MsoNormal" style=3D"margin-bottom:12.0pt"><b><span style=3D"fon=
t-size:12.0pt;color:black">From:
</span></b><span style=3D"font-size:12.0pt;color:black">andrew@fubar.geek.n=
z <andrew@fubar.geek.nz><br>
<b>Date: </b>Friday, 30 June 2023 at 16:26<br>
<b>To: </b>Mahesh Vardhamanaiah <maheshmv@juniper.net><br>
<b>Cc: </b>Sha Faisal <sfaisal@juniper.net>, freebsd-arm@FreeBSD.org =
<freebsd-arm@freebsd.org>, Steve Kiernan <stevek@juniper.net>, =
Amit Verma <amitv@juniper.net><br>
<b>Subject: </b>Re: Kernel crash before login prompt</span><o:p></o:p></p>
</div>
<div>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">[External Email. Be=
cautious of content]<br>
<br>
<br>
On 2023-06-30 10:48, Mahesh Vardhamanaiah wrote:<br>
> Hi Andrew,<br>
><br>
> We are observing that self is getting -65536 at line 1329 in<br>
> sched_pickcpu() before panic.<br>
<br>
What is the disassembly around reading the cpuid? It looks like it is<br>
getting 0xffff0000 when it should be zero.<br>
<br>
><br>
> From the code flow<br>
><br>
> sched_add [1](struct thread [2] *td [3], int flags [4])-><br>
> sched_pickcpu()->PCPU_GET<br>
><br>
> as below<br>
><br>
> static int<br>
><br>
> 1321 [5] sched_pickcpu [6](struct thread [2] *td [3], int flags =
[4])<br>
><br>
> 1322 [7] {<br>
><br>
> 1323 [8] struct =
cpu_group [9] *cg [10], *ccg;<br>
><br>
> 1324 [11] struct=
td_sched [12] *ts [13];<br>
><br>
> 1325 [14] struct=
tdq [15] *tdq [15];<br>
><br>
> 1326 [16] cpuset=
_t [17] *mask [18];<br>
><br>
> 1327 [19] int cp=
u [20], pri, r [21], self [22], intr;<br>
><br>
> 1328 [23]<br>
><br>
> 1329 [24] self [=
22] =3D PCPU_GET [25](cpuid [26]);<br>
><br>
> PCPU_GET is defined differently in Freebsd14 compared to Freebsd12 but=
<br>
> I am unable to dump the pcpup in db> prompt is it proper ?<br=
>
><br>
> Is it expected to be a global ?<br>
<br>
You can print all pcpu structs with "show allpcpu".<br>
<br>
Andrew</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>
--_000_BYAPR05MB667729839AB0CF24294B14F5B72EABYAPR05MB6677namp_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi? <BYAPR05MB667729839AB0CF24294B14F5B72EA>