Date: Fri, 15 Feb 2019 11:51:26 -0800 From: Mark Millard <marklmi@yahoo.com> To: FreeBSD PowerPC ML <freebsd-ppc@freebsd.org> Subject: Re: Some evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value Message-ID: <1F3411CF-3D28-43C0-BEF1-4672B5CC1543@yahoo.com> In-Reply-To: <9FBCA729-CE80-44CD-8873-431853E55231@yahoo.com> References: <11680D15-D43D-4115-AF4F-5F6E4E0022C9@yahoo.com> <9FBCA729-CE80-44CD-8873-431853E55231@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[I compare/contrast the output with being based on the old
VM_MAX_KERNEL_ADDRESS value.]
On 2019-Feb-13, at 22:09, Mark Millard <marklmi at yahoo.com> wrote:
> [I provide additional bootverbose output for both a successful
> boot and a failing one.]
>=20
> On 2019-Feb-13, at 19:35, Mark Millard <marklmi at yahoo.com> wrote:
>=20
>> I added some bootverbose messages to:
>>=20
>> static int
>> powermac_smp_start_cpu(platform_t plat, struct pcpu *pc)
>> (in /usr/src/sys/powerpc/powermac/platform_powermac.c )
>>=20
>> and:
>>=20
>> void
>> machdep_ap_bootstrap(void)
>> ( /usr/src/sys/powerpc/powerpc/mp_machdep.c )
>>=20
>> and:
>>=20
>> static void
>> cpu_mp_unleash(void *dummy)
>> ( /usr/src/sys/powerpc/powerpc/mp_machdep.c )
>>=20
>>=20
>>=20
>> The result was (typed from a image and
>> showing normal messages too):
>>=20
>> Adding CPU 0, hwref=3Dcd38, awkae=3D1
>> Waling up CPU 3 (dev=3Dc480)
>> Before reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0
>> After reset 4&0 for CPU3, hwref=3Dc480, awake=3D0
>> After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D0
>> cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU3, =
hwref=3Dx480, awake=3D0
>> cpu_mp_unleash adding pc_cpuid to stopped_cpus: CPU 3 (dev=3Dc480)
>> Waking up CPU 2 (dev=3Dc768)
>> Before reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0
>>=20
>> There is no more text.
>>=20
>> Part of the information is what messages were not displayed.
>>=20
>> Some notes:
>>=20
>> Only CPU 0 gots a: smp_cpus++ (resulting in 1).
>>=20
>> There is no solid evidence for if machdep_ap_bootstrap was
>> ever used for CPU 3 or CPU 2.
>>=20
>> The wait loop after the platform_smp_start_cpu call in
>> cpu_mp_unleash for CPU 3 is not used. The code in
>> machdep_ap_bootstrap:
>>=20
>> PCPU_SET(awake, 1);
>> __asm __volatile("msync; isync");
>>=20
>> did not have a visible effect on the pc->pc_awake value in
>> platform_smp_start_cpu and cpu_mp_unleash. I have no
>> unique evidence that it was executed at all.
>>=20
>> My messages added to machdep_ap_bootstrap were not displayed.
>>=20
>> platform_smp_start_cpu did not return for CPU 2.
>> cpu_mp_unleash makes no more progress. In fact the
>> 2 resets do not complete overall for CPU 2 (but did
>> for CPU 3, even if it is unclear what code was
>> executed):
>>=20
>> *rstvec =3D 4;
>> powerpc_sync();
>> (void)(*rstvec);
>> powerpc_sync();
>> DELAY(1);
>> *rstvec =3D 0;
>> powerpc_sync();
>> (void)(*rstvec);
>> powerpc_sync();
>>=20
>> I'll see about getting more information about the resets used.
>>=20
>>=20
>>=20
>> For reference for the added messages:
>>=20
>> # svnlite diff /usr/src/sys/powerpc/powermac/platform_powermac.c =
/usr/src/sys/powerpc/powerpc/mp_machdep.c
>> Index: /usr/src/sys/powerpc/powermac/platform_powermac.c
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> --- /usr/src/sys/powerpc/powermac/platform_powermac.c =
(revision 344018)
>> +++ /usr/src/sys/powerpc/powermac/platform_powermac.c (working =
copy)
>> @@ -371,6 +371,11 @@
>> rstvec_virtbase =3D pmap_mapdev(0x80000000, PAGE_SIZE);
>>=20
>> rstvec =3D rstvec_virtbase + reset;
>> +=20
>> + if (bootverbose) // HACK!!!
>> + printf("Before reset 4&0 for CPU %d, hwref=3D%jx, =
awake=3D%x\n",
>> + pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
>> + pc->pc_awake);
>>=20
>> *rstvec =3D 4;
>> powerpc_sync();
>> @@ -382,10 +387,20 @@
>> (void)(*rstvec);
>> powerpc_sync();
>>=20
>> + if (bootverbose) // HACK!!!
>> + printf("After reset 4&0 for CPU %d, hwref=3D%jx, =
awake=3D%x\n",
>> + pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
>> + pc->pc_awake);
>> +
>> timeout =3D 10000;
>> while (!pc->pc_awake && timeout--)
>> DELAY(100);
>>=20
>> + if (bootverbose) // HACK!!!
>> + printf("After attempted wait for awake CPU %d, =
hwref=3D%jx, awake=3D%x\n",
>> + pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
>> + pc->pc_awake);
>> +
>> return ((pc->pc_awake) ? 0 : EBUSY);
>> #else
>> /* No SMP support */
>> Index: /usr/src/sys/powerpc/powerpc/mp_machdep.c
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> --- /usr/src/sys/powerpc/powerpc/mp_machdep.c (revision =
344018)
>> +++ /usr/src/sys/powerpc/powerpc/mp_machdep.c (working copy)
>> @@ -97,6 +97,9 @@
>> /* Initialize decrementer */
>> decr_ap_init();
>>=20
>> + if (bootverbose) // HACK!!!
>> + printf("machdep_ap_bootstrap before ap_boot_mtx =
lock: AP CPU #%d launched\n", PCPU_GET(cpuid));
>> +
>> /* Serialize console output and AP count increment */
>> mtx_lock_spin(&ap_boot_mtx);
>> ap_awake++;
>> @@ -109,6 +112,8 @@
>>=20
>> while(smp_started =3D=3D 0)
>> ;
>> + if (bootverbose) // HACK!!!
>> + printf("machdep_ap_bootstrap after smp_started!=3D0: =
AP CPU #%d launched\n", PCPU_GET(cpuid));
>>=20
>> /* Start per-CPU event timers. */
>> cpu_initclocks_ap();
>> @@ -238,10 +243,19 @@
>>=20
>> ret =3D platform_smp_start_cpu(pc);
>> if (ret =3D=3D 0) {
>> + if (bootverbose) // HACK!!!
>> + printf("cpu_mp_unleash =
attempting to wait for pc_awake: CPU %d, hwref=3D%jx, awake=3D%x\n",
>> + pc->pc_cpuid, =
(uintmax_t)pc->pc_hwref,
>> + pc->pc_awake);
>> +
>> timeout =3D 2000; /* wait 2sec for =
the AP */
>> while (!pc->pc_awake && --timeout > 0)
>> DELAY(1000);
>> }
>> + if (bootverbose) // HACK!!!
>> + printf("cpu_mp_unleash after =
platform_smp_start_cpu and waiting: CPU %d, hwref=3D%jx, awake=3D%x\n",
>> + pc->pc_cpuid, =
(uintmax_t)pc->pc_hwref,
>> + pc->pc_awake);
>> } else {
>> pc->pc_awake =3D 1;
>> }
>> @@ -252,7 +266,13 @@
>> pc->pc_awake);
>> smp_cpus++;
>> } else
>> + { // HACK!!!
>> + if (bootverbose) // HACK!!!
>> + printf("cpu_mp_unleash adding =
pc_cpuid to stopped_cpus: CPU %d (dev=3D%x)\n",
>> + pc->pc_cpuid, =
(int)pc->pc_hwref);
>> CPU_SET(pc->pc_cpuid, &stopped_cpus);
>> + } // HACK!!!
>> +
>> }
>>=20
>> ap_awake =3D 1;
>>=20
>=20
> I added some more bootverbose code.
>=20
> A boot that completed showed:
>=20
> Adding CPU 0, hwref=3Dcd38, awake=3D1
> Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
> Waking up CPU 3 (dev=3Dc480)
> powermac_smp_start_cpu 's OF_getprop for CPU 3, hwref=3Dc480, awake=3D0:=
res=3D4, reset=3D8c
> powermac_smp_start_cpu for CPU 3, hwref=3Dc480, awake=3D0: =
rstvec_virtbase=3D0xe000000087fd2000
> powermac_smp_start_cpu for CPU 3, hwref=3Dc480, awake=3D0: =
rstvec=3D0xe000000087fd208c
> Before reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0
> After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0
> After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D1
> cpu_mp_unleash attempting to wait for pc_awake: CPU 3, hwref=3Dc480, =
awake=3D1
> cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 3, =
hwref=3Dc480, awake=3D1
> Adding CPU 3, hwref=3Dc480, awake=3D1
> Waking up CPU 2 (dev=3Dc768)
> powermac_smp_start_cpu 's OF_getprop for CPU 2, hwref=3Dc768, awake=3D0:=
res=3D4, reset=3D8b
> powermac_smp_start_cpu for CPU 2, hwref=3Dc768, awake=3D0: =
rstvec=3D0xe000000087fd208b
> Before reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0
> After reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0
> After attempted wait for awake CPU 2, hwref=3Dc768, awake=3D1
> cpu_mp_unleash attempting to wait for pc_awake: CPU 2, hwref=3Dc768, =
awake=3D1
> cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 2, =
hwref=3Dc768, awake=3D1
> Adding CPU 2, hwref=3Dc768, awake=3D1
> Waking up CPU 1 (dev=3Dca50)
> powermac_smp_start_cpu 's OF_getprop for CPU 1, hwref=3Dca50, awake=3D0:=
res=3D4, reset=3D8a
> powermac_smp_start_cpu for CPU 1, hwref=3Dca50, awake=3D0: =
rstvec=3D0xe000000087fd208a
> Before reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0
> After reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0
> After attempted wait for awake CPU 1, hwref=3Dca50, awake=3D1
> cpu_mp_unleash attempting to wait for pc_awake: CPU 1, hwref=3Dca50, =
awake=3D1
> cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 1, =
hwref=3Dca50, awake=3D1
> Adding CPU 1, hwref=3Dca50, awake=3D1
> machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #3 launched
> machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #2 launched
> machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #1 launched
> SMP: AP CPU #3 launched
> SMP: AP CPU #2 launched
> SMP: AP CPU #1 launched
> machdep_ap_bootstrap after smp_started!=3D0: AP CPU #2 launched
> machdep_ap_bootstrap after smp_started!=3D0: AP CPU #3 launched
> machdep_ap_bootstrap after smp_started!=3D0: AP CPU #1 launched
The contrast in the output is just the rstvec_virtbase figure (and, so, =
the
rstvec figures too):
. . .
powermac_smp_start_cpu for CPU 3, hwref=3Dc480, awake=3D0: =
rstvec_virtbase=3D0xe0000000013ff000
powermac_smp_start_cpu for CPU 3, hwref=3Dc480, awake=3D0: =
rstvec=3D0xe0000000013ff08c
. . .
powermac_smp_start_cpu for CPU 2, hwref=3Dc768, awake=3D0: =
rstvec=3D0xe0000000013ff08b
. . .
powermac_smp_start_cpu for CPU 1, hwref=3Dca50, awake=3D0: =
rstvec=3D0xe0000000013ff08a
. . .
So:
Old: 0xe0000000013ff0??
New: 0xe000000087fd20??
So far I've never had the hang up with the older VM_MAX_KERNEL_ADDRESS =
value
and its rstvec_virtbase figure.
> A failing boot is very similar but for awake values (and consequences)
> and omitting much (the hang):
>=20
> Adding CPU 0, hwref=3Dcd38, awake=3D1
> Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
> Waking up CPU 3 (dev=3Dc480)
> powermac_smp_start_cpu 's OF_getprop for CPU 3, hwref=3Dc480, awake=3D0:=
res=3D4, reset=3D8c
> powermac_smp_start_cpu for CPU 3, hwref=3Dc480, awake=3D0: =
rstvec_virtbase=3D0xe000000087fd2000
> powermac_smp_start_cpu for CPU 3, hwref=3Dc480, awake=3D0: =
rstvec=3D0xe000000087fd208c
> Before reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0
> After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0
> After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D0
> cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 3, =
hwref=3Dc480, awake=3D0
> cpu_mp_unleash adding pc_cpuid to stopped_cpus: CPU 3 (dev=3Dc480)
> Waking up CPU 2 (dev=3Dc768)
> powermac_smp_start_cpu 's OF_getprop for CPU 2, hwref=3Dc768, awake=3D0:=
res=3D4, reset=3D8b
> powermac_smp_start_cpu for CPU 2, hwref=3Dc768, awake=3D0: =
rstvec=3D0xe000000087fd208b
> Before reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1F3411CF-3D28-43C0-BEF1-4672B5CC1543>