Date: Sun, 17 Feb 2019 19:11:20 -0800 From: Mark Millard <marklmi@yahoo.com> To: Justin Hibbits <chmeeedalf@gmail.com> Cc: FreeBSD PowerPC ML <freebsd-ppc@freebsd.org>, Dennis Clarke <dclarke@blastwave.org> Subject: Re: Cleaned -up evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value [Wrong radix invaldates prior "new" information] Message-ID: <F6C97774-F440-48AC-8FE1-3150D6C327B1@yahoo.com> In-Reply-To: <924A8F35-88F6-4281-85D6-AF25161DAC8A@yahoo.com> References: <E7826D8D-4BA3-4A73-B006-9E7FBD1376B1@yahoo.com> <924A8F35-88F6-4281-85D6-AF25161DAC8A@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[I used %jx instead of the %jd that I had intended and:
0x2711 =3D=3D 10001. SO IGNORE MY PRIOR CONCLUSION. In fact
the hang-up has the loop run until timeout=3D=3D0 .
No evidence of pc_awake being temporarily non-zero.]
On 2019-Feb-17, at 12:10, Mark Millard <marklmi at yahoo.com> wrote:
> [Reporting the how many times the DELAY(100) in
> powermac_smp_start_cpu happens proved very interesting:
> pc_awake for CPU 3 for what turns into a hang does
> temporarily become non-zero as seen on cpu 0 (stopping
> the wait loop) but becomes zero again as seen on CPU 0,
> at least for some hung-up boots.]
>=20
> On 2019-Feb-16, at 16:29, Mark Millard <marklmi at yahoo.com> wrote:
>=20
>> I eliminated a bunch of reports that gave information
>> that was the same for hang-ups and successful boots.
>> So this is starting over with less material to go
>> through.
>>=20
>> For an unsuccessful boot:
>> (Values like 0x10, 0x25, 0x51 are use as recorded labels for points =
in the code.)
>>=20
>> Adding CPU 0, hwref=3Dcd38, awake=3D1
>> Waking up CPU 3 (dev=3Dc480)
>> After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x25
>> After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D0, =
*(unsigned long*)0xc0000000000000f0=3D0x25
>> Waking up CPU 2 (dev=3Dc768)
>>=20
>> So neither pc_awake nor *0xc0000000000000f0 have machdep_ap_bootstrap
>> based values as seen by CPU 0 (from trying to start CPU 3).
>>=20
>> For a successful boot:
>>=20
>> Adding CPU 0, hwref=3Dcd38, awake=3D1
>> Waking up CPU 3 (dev=3Dc480)
>> After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x25
>> After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D1, =
*(unsigned long*)0xc0000000000000f0=3D0x51
>> Adding CPU 3, hwref=3Dc480, awake=3D1
>> Waking up CPU 2 (dev=3Dc768)
>> After reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x51
>> After attempted wait for awake CPU 2, hwref=3Dc768, awake=3D1, =
*(unsigned long*)0xc0000000000000f0=3D0x51
>> Adding CPU 2, hwref=3Dc768, awake=3D1
>> Waking up CPU 1 (dev=3Dca50)
>> After reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x51
>> After attempted wait for awake CPU 1, hwref=3Dca50, awake=3D1, =
*(unsigned long*)0xc0000000000000f0=3D0x51
>> Adding CPU 1, hwref=3Dca50, awake=3D1
>> SMP: AP CPU #3 launched
>> SMP: AP CPU #1 launched
>> SMP: AP CPU #2 launched
>> Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
>> GEOM: new disk ada1
>>=20
>> So here waiting on cpu 0 does see pc_awake become 1 and does see
>> 0xc0000000000000f0 updated by machdep_ap_bootstrap for CPU , for
>> example.
>>=20
>> Note the awake=3D0 (i.e.., pc_awake) examples for each cpu even when
>> there is already a 0x51 label for CPU 2 and CPU 3. (The code is
>> shown later.)
>>=20
>> This is based on the labeling updates to:
>> ("labels" stored at 0xc0000000000000f0)
>>=20
>> cpudep_ap_early_bootstrap (before-return labeled with value : 0x10)
>> moea64_cpu_bootstrap_native (labeled with values: 0x21, 0x22, 0x23, =
0x24, 0x25 at various points)
>> pmap_cpu_bootstrap (before-return labeled with value : 0x20)
>> cpudep_ap_bootstrap (before-return labeled with value : 0x30)
>> cpudep_ap_setup (before-return labeled with value : 0x40)
>> machdep_ap_bootstrap (lableled with values: 0x5F, 0x51, and =
0x50 at various points)
>>=20
>> powermac_smp_start_cpu (reporting values after the resets for a =
CPU and after
>> the pa_awake loop)
>> . . .
>>=20
>=20
> =46rom a successful boot:
> (note the "delay 100 count =3D 0," exmaples)
>=20
> Adding CPU 0, hwref=3Dcd38, awake=3D1
> Waking up CPU 3 (dev=3Dc480)
> After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x25
> After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D1, delay =
100 count =3D 0, *(unsigned long*)0xc0000000000000f0=3D0x51
> Adding CPU 3, hwref=3Dc480, awake=3D1
> Waking up CPU 2 (dev=3Dc768)
> After reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x51
> After attempted wait for awake CPU 2, hwref=3Dc768, awake=3D1, delay =
100 count =3D 0, *(unsigned long*)0xc0000000000000f0=3D0x51
> Adding CPU 2, hwref=3Dc768, awake=3D1
> Waking up CPU 1 (dev=3Dca50)
> After reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x51
> After attempted wait for awake CPU 1, hwref=3Dca50, awake=3D1, delay =
100 count =3D 0, *(unsigned long*)0xc0000000000000f0=3D0x51
> Adding CPU 1, hwref=3Dca50, awake=3D1
> SMP: AP CPU #1 launched
> SMP: AP CPU #2 launched
> SMP: AP CPU #3 launched
> Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
> GEOM: new disk ada1
>=20
>=20
> Examples of boot hung ups look something like:
> (note the "delay 100 count =3D 2711," exmaples)
>=20
> Adding CPU 0, hwref=3Dcd38, awake=3D1
> Waking up CPU 3 (dev=3Dc480)
> After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned =
long*)0xc0000000000000f0=3D0x25
> After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D0, delay =
100 count =3D 2711, *(unsigned long*)0xc0000000000000f0=3D0x25
> Waking up CPU 2 (dev=3Dc768)
>=20
> (The above is from a picture of the screen.)
The following is wrong because 2711 was hexadecimal,
instead of the decimal I had intended:
> 2711 implies that pc_awake was for a time non-zero. The
> code in question is:
0x2711 =3D=3D 10001, indicating that the loop ran to its
conclusion via timeout=3D=3D0 .
> if (bootverbose) // HACK!!!
> printf("After reset 4&0 for CPU %d, hwref=3D%jx, =
awake=3D%x, *(unsigned long*)0xc0000000000000f0=3D0x%jx\n",
> pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
> pc->pc_awake,(uintmax_t)*(unsigned =
long*)0xc0000000000000f0);
>=20
> timeout =3D 10000;
> while (!pc->pc_awake && timeout--)
> DELAY(100);
>=20
> if (bootverbose) // HACK!!!
> printf("After attempted wait for awake CPU %d, =
hwref=3D%jx, awake=3D%x, delay 100 count =3D %jx, *(unsigned =
long*)0xc0000000000000f0=3D0x%jx\n",
> pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
> pc->pc_awake, (uintmax_t)(10000-timeout), =
(uintmax_t)*(unsigned long*)0xc0000000000000f0);
Above: I used %jx for the count instead of the intended %jd.
> (If the loop stops for timeout=3D=3D0 the calculation of the count =
will be
> high by 1.)
>=20
>=20
>=20
> Side note:
>=20
> I've also eliminated the experimental, extra isync() additions.
> The tested code is below.
>=20
> Index: /usr/src/sys/powerpc/aim/moea64_native.c
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- /usr/src/sys/powerpc/aim/moea64_native.c (revision 344018)
> +++ /usr/src/sys/powerpc/aim/moea64_native.c (working copy)
> @@ -384,6 +384,9 @@
>=20
> mtmsr(mfmsr() & ~PSL_DR & ~PSL_IR);
>=20
> + *(unsigned long*)0xc0000000000000f0 =3D 0x21; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> /*
> * Install kernel SLB entries
> */
> @@ -393,6 +396,9 @@
> __asm __volatile ("slbmfee %0,%1; slbie %0;" : =
"=3Dr"(seg0) :
> "r"(0));
>=20
> + *(unsigned long*)0xc0000000000000f0 =3D 0x22; // =
HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> for (i =3D 0; i < n_slbs; i++) {
> if (!(slb[i].slbe & SLBE_VALID))
> continue;
> @@ -400,6 +406,9 @@
> __asm __volatile ("slbmte %0, %1" ::=20
> "r"(slb[i].slbv), "r"(slb[i].slbe));=20
> }
> +
> + *(unsigned long*)0xc0000000000000f0 =3D 0x23; // =
HACK!!!
> + powerpc_sync(); // HACK!!!
> #else
> for (i =3D 0; i < 16; i++)
> mtsrin(i << ADDR_SR_SHFT, =
kernel_pmap->pm_sr[i]);
> @@ -412,7 +421,14 @@
> __asm __volatile ("ptesync; mtsdr1 %0; isync"
> :: "r"(((uintptr_t)moea64_pteg_table & ~DMAP_BASE_ADDRESS)
> | (uintptr_t)(flsl(moea64_pteg_mask >> 11))));
> +
> + *(unsigned long*)0xc0000000000000f0 =3D 0x24; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> tlbia();
> +
> + *(unsigned long*)0xc0000000000000f0 =3D 0x25; // HACK!!!
> + powerpc_sync(); // HACK!!!
> }
>=20
> static void
> Index: /usr/src/sys/powerpc/aim/mp_cpudep.c
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- /usr/src/sys/powerpc/aim/mp_cpudep.c (revision 344018)
> +++ /usr/src/sys/powerpc/aim/mp_cpudep.c (working copy)
> @@ -105,6 +105,9 @@
>=20
> __asm __volatile("mtsprg 0, %0" :: "r"(ap_pcpu));
> powerpc_sync();
> +
> + *(unsigned long*)0xc0000000000000f0 =3D 0x10; // HACK!!!
> + powerpc_sync(); // HACK!!!
> }
>=20
> uintptr_t
> @@ -124,6 +127,9 @@
> pcpup->pc_curpcb =3D pcpup->pc_curthread->td_pcb;
> sp =3D pcpup->pc_curpcb->pcb_sp;
>=20
> + *(unsigned long*)0xc0000000000000f0 =3D 0x30; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> return (sp);
> }
>=20
> @@ -416,5 +422,8 @@
> "suboptimal.\n");
> break;
> }
> +
> + *(unsigned long*)0xc0000000000000f0 =3D 0x40; // HACK!!!
> + powerpc_sync(); // HACK!!!
> }
>=20
> Index: /usr/src/sys/powerpc/include/vmparam.h
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- /usr/src/sys/powerpc/include/vmparam.h (revision 344018)
> +++ /usr/src/sys/powerpc/include/vmparam.h (working copy)
> @@ -109,9 +109,11 @@
> #ifndef LOCORE
> #define VM_MIN_KERNEL_ADDRESS 0xe000000000000000UL
> #define VM_MAX_KERNEL_ADDRESS 0xe0000007ffffffffUL
> +//#define VM_MAX_KERNEL_ADDRESS 0xe0000001c7ffffffUL
> #else
> #define VM_MIN_KERNEL_ADDRESS 0xe000000000000000
> #define VM_MAX_KERNEL_ADDRESS 0xe0000007ffffffff
> +//#define VM_MAX_KERNEL_ADDRESS 0xe0000001c7ffffff
> #endif
> #define VM_MAX_SAFE_KERNEL_ADDRESS VM_MAX_KERNEL_ADDRESS
> #endif
> Index: /usr/src/sys/powerpc/powermac/platform_powermac.c
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- /usr/src/sys/powerpc/powermac/platform_powermac.c (revision =
344018)
> +++ /usr/src/sys/powerpc/powermac/platform_powermac.c (working copy)
> @@ -382,10 +382,20 @@
> (void)(*rstvec);
> powerpc_sync();
>=20
> + if (bootverbose) // HACK!!!
> + printf("After reset 4&0 for CPU %d, hwref=3D%jx, =
awake=3D%x, *(unsigned long*)0xc0000000000000f0=3D0x%jx\n",
> + pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
> + pc->pc_awake,(uintmax_t)*(unsigned =
long*)0xc0000000000000f0);
> +
> timeout =3D 10000;
> while (!pc->pc_awake && timeout--)
> DELAY(100);
>=20
> + if (bootverbose) // HACK!!!
> + printf("After attempted wait for awake CPU %d, =
hwref=3D%jx, awake=3D%x, delay 100 count =3D %jx, *(unsigned =
long*)0xc0000000000000f0=3D0x%jx\n",
> + pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
> + pc->pc_awake, (uintmax_t)(10000-timeout), =
(uintmax_t)*(unsigned long*)0xc0000000000000f0);
> +
> return ((pc->pc_awake) ? 0 : EBUSY);
> #else
> /* No SMP support */
Again (above): I used %jx for the count instead of the intended
%jd .
> Index: /usr/src/sys/powerpc/powerpc/machdep.c
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- /usr/src/sys/powerpc/powerpc/machdep.c (revision 344018)
> +++ /usr/src/sys/powerpc/powerpc/machdep.c (working copy)
> @@ -141,8 +141,6 @@
> extern vm_paddr_t kernload;
> #endif
>=20
> -extern void *ap_pcpu;
> -
> struct pcpu __pcpu[MAXCPU];
> static char init_kenv[2048];
>=20
> Index: /usr/src/sys/powerpc/powerpc/mp_machdep.c
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- /usr/src/sys/powerpc/powerpc/mp_machdep.c (revision 344018)
> +++ /usr/src/sys/powerpc/powerpc/mp_machdep.c (working copy)
> @@ -73,10 +73,15 @@
> void
> machdep_ap_bootstrap(void)
> {
> + *(unsigned long*)0xc0000000000000f0 =3D 0x5F; // HACK!!!
> + powerpc_sync(); // HACK!!!
>=20
> PCPU_SET(awake, 1);
> __asm __volatile("msync; isync");
>=20
> + *(unsigned long*)0xc0000000000000f0 =3D 0x51; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> while (ap_letgo =3D=3D 0)
> __asm __volatile("or 31,31,31");
> __asm __volatile("or 6,6,6");
> @@ -113,6 +118,9 @@
> /* Start per-CPU event timers. */
> cpu_initclocks_ap();
>=20
> + *(unsigned long*)0xc0000000000000f0 =3D 0x50; // HACK!!!
> + powerpc_sync(); // HACK!!!
> +
> /* Announce ourselves awake, and enter the scheduler */
> sched_throw(NULL);
> }
> @@ -256,6 +264,7 @@
> }
>=20
> ap_awake =3D 1;
> + powerpc_sync(); // HACK!!!
>=20
> /* Provide our current DEC and TB values for APs */
> ap_timebase =3D mftb() + 10;
> Index: /usr/src/sys/powerpc/powerpc/pmap_dispatch.c
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- /usr/src/sys/powerpc/powerpc/pmap_dispatch.c (revision =
344018)
> +++ /usr/src/sys/powerpc/powerpc/pmap_dispatch.c (working copy)
> @@ -445,6 +445,9 @@
> */
>=20
> return (MMU_CPU_BOOTSTRAP(mmu_obj, ap));
> +
> + *(unsigned long*)0xc0000000000000f0 =3D 0x20; // HACK!!!
> + powerpc_sync(); // HACK!!!
> }
>=20
> void *
>=20
>=20
>=20
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F6C97774-F440-48AC-8FE1-3150D6C327B1>