Date: Sun, 17 Feb 2019 12:10:14 -0800 From: Mark Millard <marklmi@yahoo.com> To: Justin Hibbits <chmeeedalf@gmail.com> Cc: FreeBSD PowerPC ML <freebsd-ppc@freebsd.org>, Dennis Clarke <dclarke@blastwave.org> Subject: Re: Cleaned -up evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value [signficant new evidence] Message-ID: <924A8F35-88F6-4281-85D6-AF25161DAC8A@yahoo.com> In-Reply-To: <E7826D8D-4BA3-4A73-B006-9E7FBD1376B1@yahoo.com> References: <E7826D8D-4BA3-4A73-B006-9E7FBD1376B1@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[Reporting the how many times the DELAY(100) in
powermac_smp_start_cpu happens proved very interesting:
pc_awake for CPU 3 for what turns into a hang does
temporarily become non-zero as seen on cpu 0 (stopping
the wait loop) but becomes zero again as seen on CPU 0,
at least for some hung-up boots.]
On 2019-Feb-16, at 16:29, Mark Millard <marklmi at yahoo.com> wrote:
> I eliminated a bunch of reports that gave information
> that was the same for hang-ups and successful boots.
> So this is starting over with less material to go
> through.
>=20
> For an unsuccessful boot:
> (Values like 0x10, 0x25, 0x51 are use as recorded labels for points in =
the code.)
>=20
> Adding CPU 0, hwref=3Dcd38, awake=3D1
> Waking up CPU 3 (dev=3Dc480)
> After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x25
> After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D0, =
*(unsigned long*)0xc0000000000000f0=3D0x25
> Waking up CPU 2 (dev=3Dc768)
>=20
> So neither pc_awake nor *0xc0000000000000f0 have machdep_ap_bootstrap
> based values as seen by CPU 0 (from trying to start CPU 3).
>=20
> For a successful boot:
>=20
> Adding CPU 0, hwref=3Dcd38, awake=3D1
> Waking up CPU 3 (dev=3Dc480)
> After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x25
> After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D1, =
*(unsigned long*)0xc0000000000000f0=3D0x51
> Adding CPU 3, hwref=3Dc480, awake=3D1
> Waking up CPU 2 (dev=3Dc768)
> After reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x51
> After attempted wait for awake CPU 2, hwref=3Dc768, awake=3D1, =
*(unsigned long*)0xc0000000000000f0=3D0x51
> Adding CPU 2, hwref=3Dc768, awake=3D1
> Waking up CPU 1 (dev=3Dca50)
> After reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x51
> After attempted wait for awake CPU 1, hwref=3Dca50, awake=3D1, =
*(unsigned long*)0xc0000000000000f0=3D0x51
> Adding CPU 1, hwref=3Dca50, awake=3D1
> SMP: AP CPU #3 launched
> SMP: AP CPU #1 launched
> SMP: AP CPU #2 launched
> Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
> GEOM: new disk ada1
>=20
> So here waiting on cpu 0 does see pc_awake become 1 and does see
> 0xc0000000000000f0 updated by machdep_ap_bootstrap for CPU , for
> example.
>=20
> Note the awake=3D0 (i.e.., pc_awake) examples for each cpu even when
> there is already a 0x51 label for CPU 2 and CPU 3. (The code is
> shown later.)
>=20
> This is based on the labeling updates to:
> ("labels" stored at 0xc0000000000000f0)
>=20
> cpudep_ap_early_bootstrap (before-return labeled with value : 0x10)
> moea64_cpu_bootstrap_native (labeled with values: 0x21, 0x22, 0x23, =
0x24, 0x25 at various points)
> pmap_cpu_bootstrap (before-return labeled with value : 0x20)
> cpudep_ap_bootstrap (before-return labeled with value : 0x30)
> cpudep_ap_setup (before-return labeled with value : 0x40)
> machdep_ap_bootstrap (lableled with values: 0x5F, 0x51, and =
0x50 at various points)
>=20
> powermac_smp_start_cpu (reporting values after the resets for a =
CPU and after
> the pa_awake loop)
> . . .
>=20
=46rom a successful boot:
(note the "delay 100 count =3D 0," exmaples)
Adding CPU 0, hwref=3Dcd38, awake=3D1
Waking up CPU 3 (dev=3Dc480)
After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x25
After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D1, delay 100 =
count =3D 0, *(unsigned long*)0xc0000000000000f0=3D0x51
Adding CPU 3, hwref=3Dc480, awake=3D1
Waking up CPU 2 (dev=3Dc768)
After reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x51
After attempted wait for awake CPU 2, hwref=3Dc768, awake=3D1, delay 100 =
count =3D 0, *(unsigned long*)0xc0000000000000f0=3D0x51
Adding CPU 2, hwref=3Dc768, awake=3D1
Waking up CPU 1 (dev=3Dca50)
After reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x51
After attempted wait for awake CPU 1, hwref=3Dca50, awake=3D1, delay 100 =
count =3D 0, *(unsigned long*)0xc0000000000000f0=3D0x51
Adding CPU 1, hwref=3Dca50, awake=3D1
SMP: AP CPU #1 launched
SMP: AP CPU #2 launched
SMP: AP CPU #3 launched
Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
GEOM: new disk ada1
Examples of boot hung ups look something like:
(note the "delay 100 count =3D 2711," exmaples)
Adding CPU 0, hwref=3Dcd38, awake=3D1
Waking up CPU 3 (dev=3Dc480)
After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x25
After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D0, delay 100 =
count =3D 2711, *(unsigned long*)0xc0000000000000f0=3D0x25
Waking up CPU 2 (dev=3Dc768)
(The above is from a picture of the screen.)
2711 implies that pc_awake was for a time non-zero. The
code in question is:
if (bootverbose) // HACK!!!
printf("After reset 4&0 for CPU %d, hwref=3D%jx, =
awake=3D%x, *(unsigned long*)0xc0000000000000f0=3D0x%jx\n",
pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
pc->pc_awake,(uintmax_t)*(unsigned =
long*)0xc0000000000000f0);
timeout =3D 10000;
while (!pc->pc_awake && timeout--)
DELAY(100);
if (bootverbose) // HACK!!!
printf("After attempted wait for awake CPU %d, =
hwref=3D%jx, awake=3D%x, delay 100 count =3D %jx, *(unsigned =
long*)0xc0000000000000f0=3D0x%jx\n",
pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
pc->pc_awake, (uintmax_t)(10000-timeout), =
(uintmax_t)*(unsigned long*)0xc0000000000000f0);
(If the loop stops for timeout=3D=3D0 the calculation of the count will =
be
high by 1.)
Side note:
I've also eliminated the experimental, extra isync() additions.
The tested code is below.
Index: /usr/src/sys/powerpc/aim/moea64_native.c
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- /usr/src/sys/powerpc/aim/moea64_native.c (revision 344018)
+++ /usr/src/sys/powerpc/aim/moea64_native.c (working copy)
@@ -384,6 +384,9 @@
=20
mtmsr(mfmsr() & ~PSL_DR & ~PSL_IR);
=20
+ *(unsigned long*)0xc0000000000000f0 =3D 0x21; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
/*
* Install kernel SLB entries
*/
@@ -393,6 +396,9 @@
__asm __volatile ("slbmfee %0,%1; slbie %0;" : =
"=3Dr"(seg0) :
"r"(0));
=20
+ *(unsigned long*)0xc0000000000000f0 =3D 0x22; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
for (i =3D 0; i < n_slbs; i++) {
if (!(slb[i].slbe & SLBE_VALID))
continue;
@@ -400,6 +406,9 @@
__asm __volatile ("slbmte %0, %1" ::=20
"r"(slb[i].slbv), "r"(slb[i].slbe));=20
}
+
+ *(unsigned long*)0xc0000000000000f0 =3D 0x23; // HACK!!!
+ powerpc_sync(); // HACK!!!
#else
for (i =3D 0; i < 16; i++)
mtsrin(i << ADDR_SR_SHFT, =
kernel_pmap->pm_sr[i]);
@@ -412,7 +421,14 @@
__asm __volatile ("ptesync; mtsdr1 %0; isync"
:: "r"(((uintptr_t)moea64_pteg_table & ~DMAP_BASE_ADDRESS)
| (uintptr_t)(flsl(moea64_pteg_mask >> 11))));
+
+ *(unsigned long*)0xc0000000000000f0 =3D 0x24; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
tlbia();
+
+ *(unsigned long*)0xc0000000000000f0 =3D 0x25; // HACK!!!
+ powerpc_sync(); // HACK!!!
}
=20
static void
Index: /usr/src/sys/powerpc/aim/mp_cpudep.c
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- /usr/src/sys/powerpc/aim/mp_cpudep.c (revision 344018)
+++ /usr/src/sys/powerpc/aim/mp_cpudep.c (working copy)
@@ -105,6 +105,9 @@
=20
__asm __volatile("mtsprg 0, %0" :: "r"(ap_pcpu));
powerpc_sync();
+
+ *(unsigned long*)0xc0000000000000f0 =3D 0x10; // HACK!!!
+ powerpc_sync(); // HACK!!!
}
=20
uintptr_t
@@ -124,6 +127,9 @@
pcpup->pc_curpcb =3D pcpup->pc_curthread->td_pcb;
sp =3D pcpup->pc_curpcb->pcb_sp;
=20
+ *(unsigned long*)0xc0000000000000f0 =3D 0x30; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
return (sp);
}
=20
@@ -416,5 +422,8 @@
"suboptimal.\n");
break;
}
+
+ *(unsigned long*)0xc0000000000000f0 =3D 0x40; // HACK!!!
+ powerpc_sync(); // HACK!!!
}
=20
Index: /usr/src/sys/powerpc/include/vmparam.h
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- /usr/src/sys/powerpc/include/vmparam.h (revision 344018)
+++ /usr/src/sys/powerpc/include/vmparam.h (working copy)
@@ -109,9 +109,11 @@
#ifndef LOCORE
#define VM_MIN_KERNEL_ADDRESS 0xe000000000000000UL
#define VM_MAX_KERNEL_ADDRESS 0xe0000007ffffffffUL
+//#define VM_MAX_KERNEL_ADDRESS 0xe0000001c7ffffffUL
#else
#define VM_MIN_KERNEL_ADDRESS 0xe000000000000000
#define VM_MAX_KERNEL_ADDRESS 0xe0000007ffffffff
+//#define VM_MAX_KERNEL_ADDRESS 0xe0000001c7ffffff
#endif
#define VM_MAX_SAFE_KERNEL_ADDRESS VM_MAX_KERNEL_ADDRESS
#endif
Index: /usr/src/sys/powerpc/powermac/platform_powermac.c
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- /usr/src/sys/powerpc/powermac/platform_powermac.c (revision =
344018)
+++ /usr/src/sys/powerpc/powermac/platform_powermac.c (working copy)
@@ -382,10 +382,20 @@
(void)(*rstvec);
powerpc_sync();
=20
+ if (bootverbose) // HACK!!!
+ printf("After reset 4&0 for CPU %d, hwref=3D%jx, =
awake=3D%x, *(unsigned long*)0xc0000000000000f0=3D0x%jx\n",
+ pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
+ pc->pc_awake,(uintmax_t)*(unsigned =
long*)0xc0000000000000f0);
+
timeout =3D 10000;
while (!pc->pc_awake && timeout--)
DELAY(100);
=20
+ if (bootverbose) // HACK!!!
+ printf("After attempted wait for awake CPU %d, =
hwref=3D%jx, awake=3D%x, delay 100 count =3D %jx, *(unsigned =
long*)0xc0000000000000f0=3D0x%jx\n",
+ pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
+ pc->pc_awake, (uintmax_t)(10000-timeout), =
(uintmax_t)*(unsigned long*)0xc0000000000000f0);
+
return ((pc->pc_awake) ? 0 : EBUSY);
#else
/* No SMP support */
Index: /usr/src/sys/powerpc/powerpc/machdep.c
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- /usr/src/sys/powerpc/powerpc/machdep.c (revision 344018)
+++ /usr/src/sys/powerpc/powerpc/machdep.c (working copy)
@@ -141,8 +141,6 @@
extern vm_paddr_t kernload;
#endif
=20
-extern void *ap_pcpu;
-
struct pcpu __pcpu[MAXCPU];
static char init_kenv[2048];
=20
Index: /usr/src/sys/powerpc/powerpc/mp_machdep.c
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- /usr/src/sys/powerpc/powerpc/mp_machdep.c (revision 344018)
+++ /usr/src/sys/powerpc/powerpc/mp_machdep.c (working copy)
@@ -73,10 +73,15 @@
void
machdep_ap_bootstrap(void)
{
+ *(unsigned long*)0xc0000000000000f0 =3D 0x5F; // HACK!!!
+ powerpc_sync(); // HACK!!!
=20
PCPU_SET(awake, 1);
__asm __volatile("msync; isync");
=20
+ *(unsigned long*)0xc0000000000000f0 =3D 0x51; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
while (ap_letgo =3D=3D 0)
__asm __volatile("or 31,31,31");
__asm __volatile("or 6,6,6");
@@ -113,6 +118,9 @@
/* Start per-CPU event timers. */
cpu_initclocks_ap();
=20
+ *(unsigned long*)0xc0000000000000f0 =3D 0x50; // HACK!!!
+ powerpc_sync(); // HACK!!!
+
/* Announce ourselves awake, and enter the scheduler */
sched_throw(NULL);
}
@@ -256,6 +264,7 @@
}
=20
ap_awake =3D 1;
+ powerpc_sync(); // HACK!!!
=20
/* Provide our current DEC and TB values for APs */
ap_timebase =3D mftb() + 10;
Index: /usr/src/sys/powerpc/powerpc/pmap_dispatch.c
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- /usr/src/sys/powerpc/powerpc/pmap_dispatch.c (revision =
344018)
+++ /usr/src/sys/powerpc/powerpc/pmap_dispatch.c (working copy)
@@ -445,6 +445,9 @@
*/
=20
return (MMU_CPU_BOOTSTRAP(mmu_obj, ap));
+
+ *(unsigned long*)0xc0000000000000f0 =3D 0x20; // HACK!!!
+ powerpc_sync(); // HACK!!!
}
=20
void *
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?924A8F35-88F6-4281-85D6-AF25161DAC8A>