Date: Sat, 16 Feb 2019 12:07:18 -0800 From: Mark Millard <marklmi@yahoo.com> To: Justin Hibbits <chmeeedalf@gmail.com> Cc: Mark Millard via freebsd-ppc <freebsd-ppc@freebsd.org> Subject: Re: Some evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value [found more staging info] Message-ID: <518C5B96-75C4-4C24-BDEE-68A542242CA3@yahoo.com> In-Reply-To: <C35575A8-6316-40CB-B6E7-B2FB7F438EA4@yahoo.com> References: <11680D15-D43D-4115-AF4F-5F6E4E0022C9@yahoo.com> <9FBCA729-CE80-44CD-8873-431853E55231@yahoo.com> <1F3411CF-3D28-43C0-BEF1-4672B5CC1543@yahoo.com> <20190215151710.35545a26@ralga.knownspace> <6445CE54-26AA-4E21-B17E-921D72D4081A@yahoo.com> <20190215160942.1b282f71@ralga.knownspace> <744610C7-90EB-42A0-8B08-AFA0F12E5994@yahoo.com> <20190215180421.61afcae3@ralga.knownspace> <C35575A8-6316-40CB-B6E7-B2FB7F438EA4@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[I needed to allow more time after the 2 resets before
having CPU 0 look at the memory. It was reporting
older values instead of my added writes. The odd
non-zero value was from before the activity of interest.]
I start with the new result found, then give supporting
material.
I've now seen hangs with:
*(unsigned long*)0xc0000000000000f0)=3D0x10
for CPU 3. So the following completed:
void
cpudep_ap_early_bootstrap(void)
{
#ifndef __powerpc64__
register_t reg;
#endif
switch (mfpvr() >> 16) {
case IBM970:
case IBM970FX:
case IBM970MP:
/* Restore HID4 and HID5, which are necessary for the =
MMU */
#ifdef __powerpc64__
mtspr(SPR_HID4, bsp_state[2]); powerpc_sync(); isync();
mtspr(SPR_HID5, bsp_state[3]); powerpc_sync(); isync();
#else
__asm __volatile("ld %0, 16(%2); sync; isync; \
mtspr %1, %0; sync; isync;"
: "=3Dr"(reg) : "K"(SPR_HID4), "b"(bsp_state));
__asm __volatile("ld %0, 24(%2); sync; isync; \
mtspr %1, %0; sync; isync;"
: "=3Dr"(reg) : "K"(SPR_HID5), "b"(bsp_state));
#endif
powerpc_sync();
break;
case IBMPOWER8:
case IBMPOWER8E:
case IBMPOWER9:
#ifdef __powerpc64__
if (mfmsr() & PSL_HV) {
isync();
/*
* Direct interrupts to SRR instead of HSRR and
* reset LPCR otherwise
*/
mtspr(SPR_LPID, 0);
isync();
=20
mtspr(SPR_LPCR, lpcr);
isync();
}
#endif
break;
}
__asm __volatile("mtsprg 0, %0" :: "r"(ap_pcpu));
powerpc_sync();
*(unsigned long*)0xc0000000000000f0 =3D 0x10; // HACK!!!
powerpc_sync(); // HACK!!!
}
but the following (and later) did not complete:
void
pmap_cpu_bootstrap(int ap)
{
/* =20
* No KTR here because our console probably doesn't work yet
*/
=20
return (MMU_CPU_BOOTSTRAP(mmu_obj, ap));
=20
*(unsigned long*)0xc0000000000000f0 =3D 0x20; // HACK!!!
powerpc_sync(); // HACK!!!
}
Background for reference relative to showing
the intended values. . .
I now have:
*rstvec =3D 4;
powerpc_sync();
(void)(*rstvec);
powerpc_sync();
DELAY(1);
*rstvec =3D 0;
powerpc_sync();
(void)(*rstvec);
powerpc_sync();
if (bootverbose) // HACK!!!
printf("After reset 4&0 for CPU %d, hwref=3D%jx, =
awake=3D%x, *(unsigned long*)0xc0000000000000e0=3D0x%jx, *(unsigned =
long*)0xc0000000000000f0=3D0x%jx\n",
pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
pc->pc_awake,(uintmax_t)*(unsigned =
long*)0xc0000000000000e0,(uintmax_t)*(unsigned =
long*)0xc0000000000000f0);
timeout =3D 10000;
while (!pc->pc_awake && timeout--)
DELAY(100);
if (bootverbose) // HACK!!!
printf("After attempted wait for awake CPU %d, =
hwref=3D%jx, awake=3D%x, *(unsigned long*)0xc0000000000000e0=3D0x%jx, =
*(unsigned long*)0xc0000000000000f0=3D0x%jx\n",
pc->pc_cpuid, (uintmax_t)pc->pc_hwref,
pc->pc_awake,(uintmax_t)*(unsigned =
long*)0xc0000000000000e0,(uintmax_t)*(unsigned =
long*)0xc0000000000000f0);
The 2nd printf shows the expected values but the first
above shows old memory values.
The memory at 0xc0000000000000e0 hold the get_pcpu() result.
The memory at 0xc0000000000000f0 hold the 0x?? values that I'd reported =
earlier
although I've added a 0x5E but really use the value at =
0xc0000000000000e0 to
check since 0x5e would be replaced by 0x51:
struct pcpu* thepcpu =3D get_pcpu(); // HACK!!!
*(struct pcpu**)0xc0000000000000e0 =3D thepcpu; // HACK!!!
if ( thepcpu=3D=3D&__pcpu[1] // HACK!!!
|| thepcpu=3D=3D&__pcpu[2]
|| thepcpu=3D=3D&__pcpu[3]
)
*(unsigned long*)0xc0000000000000f0 =3D 0x5F; // HACK!!!
else
*(unsigned long*)0xc0000000000000f0 =3D 0x5E; // HACK!!!
powerpc_sync(); // HACK!!!
PCPU_SET(awake, 1);
__asm __volatile("msync; isync");
*(unsigned long*)0xc0000000000000f0 =3D 0x51; // HACK!!!
powerpc_sync(); // HACK!!!
while (ap_letgo =3D=3D 0)
__asm __volatile("or 31,31,31");
__asm __volatile("or 6,6,6");
For reference, a successful boot now looks like:
Adding CPU 0, hwref=3Dcd38, awake=3D1
Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]...
Waking up CPU 3 (dev=3Dc480)
powermac_smp_start_cpu 's OF_getprop for CPU 3, hwref=3Dc480, awake=3D0: =
res=3D4, reset=3D8c
powermac_smp_start_cpu for CPU 3, hwref=3Dc480, awake=3D0: =
rstvec_virtbase=3D0xe000000087fd2000
powermac_smp_start_cpu for CPU 3, hwref=3Dc480, awake=3D0: =
rstvec=3D0xe000000087fd208c
Before reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0
After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned =
long*)0xc0000000000000e0=3D0x0, *(unsigned long*)0xc0000000000000f0=3D0x0
After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D1, =
*(unsigned long*)0xc0000000000000e0=3D0xc0000000016c6100, *(unsigned =
long*)0xc0000000000000f0=3D0x51
cpu_mp_unleash attempting to wait for pc_awake: CPU 3, hwref=3Dc480, =
awake=3D1
cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 3, =
hwref=3Dc480, awake=3D1
Adding CPU 3, hwref=3Dc480, awake=3D1
Waking up CPU 2 (dev=3Dc768)
powermac_smp_start_cpu 's OF_getprop for CPU 2, hwref=3Dc768, awake=3D0: =
res=3D4, reset=3D8b
powermac_smp_start_cpu for CPU 2, hwref=3Dc768, awake=3D0: =
rstvec=3D0xe000000087fd208b
Before reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0
After reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0, *(unsigned =
long*)0xc0000000000000e0=3D0xc0000000016c6100, *(unsigned =
long*)0xc0000000000000f0=3D0x51
After attempted wait for awake CPU 2, hwref=3Dc768, awake=3D1, =
*(unsigned long*)0xc0000000000000e0=3D0xc0000000016c5100, *(unsigned =
long*)0xc0000000000000f0=3D0x51
cpu_mp_unleash attempting to wait for pc_awake: CPU 2, hwref=3Dc768, =
awake=3D1
cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 2, =
hwref=3Dc768, awake=3D1
Adding CPU 2, hwref=3Dc768, awake=3D1
Waking up CPU 1 (dev=3Dca50)
powermac_smp_start_cpu 's OF_getprop for CPU 1, hwref=3Dca50, awake=3D0: =
res=3D4, reset=3D8a
powermac_smp_start_cpu for CPU 1, hwref=3Dca50, awake=3D0: =
rstvec=3D0xe000000087fd208a
Before reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0
After reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0, *(unsigned =
long*)0xc0000000000000e0=3D0xc0000000016c5100, *(unsigned =
long*)0xc0000000000000f0=3D0x51
After attempted wait for awake CPU 1, hwref=3Dca50, awake=3D1, =
*(unsigned long*)0xc0000000000000e0=3D0xc0000000016c4100, *(unsigned =
long*)0xc0000000000000f0=3D0x51
cpu_mp_unleash attempting to wait for pc_awake: CPU 1, hwref=3Dca50, =
awake=3D1
cpu_mp_unleash after platform_smp_start_cpu and waiting: CPU 1, =
hwref=3Dca50, awake=3D1
Adding CPU 1, hwref=3Dca50, awake=3D1
machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #2 launched
machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #1 launched
machdep_ap_bootstrap before ap_boot_mtx lock: AP CPU #3 launched
SMP: AP CPU #2 launched
SMP: AP CPU #1 launched
SMP: AP CPU #3 launched
machdep_ap_bootstrap after smp_started!=3D0: AP CPU #2 launched
machdep_ap_bootstrap after smp_started!=3D0: AP CPU #1 launched
machdep_ap_bootstrap after smp_started!=3D0: AP CPU #3 launched
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?518C5B96-75C4-4C24-BDEE-68A542242CA3>