From owner-freebsd-ppc@freebsd.org Sun Feb 17 00:30:08 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E971814D8C5D for ; Sun, 17 Feb 2019 00:30:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-35.consmr.mail.ne1.yahoo.com (sonic317-35.consmr.mail.ne1.yahoo.com [66.163.184.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A825E742D9 for ; Sun, 17 Feb 2019 00:30:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: AzkdWtIVM1n4o9UohMlAU.w0Wt6REjqILh5EuXC9d5Jf_g8rZWTwdFdrX.CakmN 5TsXcvJZxQ._mTImHBU9ja5vj7Q6f.BHZ.6joygQFZKDdzmfhCE.c6Au4MU2MtW_egvj7hC_AJX5 ueZ.kWbtRaBCPf557vdLFFmbNoTfxcPO7KZyW4IYFFE2vVeR1DFbWbyuyL8_m0kbh7g43zps4G0v UmomvnXkGsydejLiaSTbdJv4sQv10HTTeYzLT21G2VeFcCZ6L7Y_FHs_jpNJh5KllIv.iTWng7ej Sy85nBZdBQ8KhDb5PJ6jDl1ZFagJBb88IEm7K045b.EpJ8K5rhK58ipwnq2wm6IvTtLRwwjlZTZ2 ZwXeUukyaSVub_TNZahDcZUm8D.KyAuZ63Ed40gZQS1m_aBYMLyjVsPpi322K.SolC3asx0zmGjS 0z6ukqNR9wZufTQldPsmE0I.fH0mZwI8vOKl1y7wAVE.E5Uppk5.B9rnfZVARyrFAqjF9xVWfGI1 6Tl25ajsXnhZDONl91lVGKIW47zncRBXhGppEut7gNqN_g7r.bGqJ_ddoFHAyeNHgovayMFzRnxI Y1Mio7XwsjdlcKAT2fZRDEGhd5EMu6TnN19epK1Y2oGgfYpAJRXePZYL_iID9jV7l_k8Emf4wry1 VtBBRYsK36g7jOoGUkQQaz1Kahzkl1SpxvMiXS6vm.Y2qkxEMQDDcxlMJI8Wn6aOO8qX_4mLXyAc n6BukdTwtB.RCwEy8aUxkmHj_Yvtn_masLXuNXXbFq2GlV7U_Htqz2UZkU_CBCrVUwBFQUA080F4 7lX0q8geVNEwk95OAWlTJanZeTyguKujXurpEmPeoDlgYSORBaTgfjt2vYCFR9SV90_0Ijpu2PEu yT_fyxySA42vHC__TMhdWGT.KisAq3tT3jG.lvVeDWgLKfvt.bmqqa7Sfu9k3NN5x_a36FFDB5Vi isszMSBGE91TSfhpuc8qscWJLeyqLizSjFqJnebcxXsRMYGfQOid4HM.l23C3TwJIw11SHQfrEVn FiDFfARTULIlO3Uih4Mld6A7fCisxvVxvA7MI5WT2nrGQymOgrzkKX2F9Zq8- Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.ne1.yahoo.com with HTTP; Sun, 17 Feb 2019 00:29:59 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp415.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 88dc9ec016cf75fec7969a98b9017f50; Sun, 17 Feb 2019 00:29:54 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Cleaned -up evidence about the PowerMac G5 multiprocessor boot hang ups with the modern VM_MAX_KERNEL_ADDRESS value Message-Id: Date: Sat, 16 Feb 2019 16:29:52 -0800 Cc: FreeBSD PowerPC ML To: Justin Hibbits X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: A825E742D9 X-Spamd-Bar: + X-Spamd-Result: default: False [1.94 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.23)[0.232,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.21)[ip: (4.15), ipnet: 66.163.184.0/21(1.10), asn: 36646(0.88), country: US(-0.07)]; NEURAL_SPAM_MEDIUM(0.87)[0.869,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.14)[0.140,0]; RCVD_IN_DNSWL_NONE(0.00)[46.184.163.66.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Feb 2019 00:30:08 -0000 I eliminated a bunch of reports that gave information that was the same for hang-ups and successful boots. So this is starting over with less material to go through. For an unsuccessful boot: (Values like 0x10, 0x25, 0x51 are use as recorded labels for points in = the code.) Adding CPU 0, hwref=3Dcd38, awake=3D1 Waking up CPU 3 (dev=3Dc480) After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned = long*)0xc0000000000000f0=3D0x25 After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D0, = *(unsigned long*)0xc0000000000000f0=3D0x25 Waking up CPU 2 (dev=3Dc768) So neither pc_awake nor *0xc0000000000000f0 have machdep_ap_bootstrap based values as seen by CPU 0 (from trying to start CPU 3). For a successful boot: Adding CPU 0, hwref=3Dcd38, awake=3D1 Waking up CPU 3 (dev=3Dc480) After reset 4&0 for CPU 3, hwref=3Dc480, awake=3D0, *(unsigned = long*)0xc0000000000000f0=3D0x25 After attempted wait for awake CPU 3, hwref=3Dc480, awake=3D1, = *(unsigned long*)0xc0000000000000f0=3D0x51 Adding CPU 3, hwref=3Dc480, awake=3D1 Waking up CPU 2 (dev=3Dc768) After reset 4&0 for CPU 2, hwref=3Dc768, awake=3D0, *(unsigned = long*)0xc0000000000000f0=3D0x51 After attempted wait for awake CPU 2, hwref=3Dc768, awake=3D1, = *(unsigned long*)0xc0000000000000f0=3D0x51 Adding CPU 2, hwref=3Dc768, awake=3D1 Waking up CPU 1 (dev=3Dca50) After reset 4&0 for CPU 1, hwref=3Dca50, awake=3D0, *(unsigned = long*)0xc0000000000000f0=3D0x51 After attempted wait for awake CPU 1, hwref=3Dca50, awake=3D1, = *(unsigned long*)0xc0000000000000f0=3D0x51 Adding CPU 1, hwref=3Dca50, awake=3D1 SMP: AP CPU #3 launched SMP: AP CPU #1 launched SMP: AP CPU #2 launched Trying to mount root from ufs:/dev/ufs/FBSDG5L2rootfs [rw,noatime]... GEOM: new disk ada1 So here waiting on cpu 0 does see pc_awake become 1 and does see 0xc0000000000000f0 updated by machdep_ap_bootstrap for CPU , for example. Note the awake=3D0 (i.e.., pc_awake) examples for each cpu even when there is already a 0x51 label for CPU 2 and CPU 3. (The code is shown later.) This is based on the labeling updates to: ("labels" stored at 0xc0000000000000f0) cpudep_ap_early_bootstrap (before-return labeled with value : 0x10) moea64_cpu_bootstrap_native (labeled with values: 0x21, 0x22, 0x23, = 0x24, 0x25 at various points) pmap_cpu_bootstrap (before-return labeled with value : 0x20) cpudep_ap_bootstrap (before-return labeled with value : 0x30) cpudep_ap_setup (before-return labeled with value : 0x40) machdep_ap_bootstrap (lableled with values: 0x5F, 0x51, and 0x50 = at various points) powermac_smp_start_cpu (reporting values after the resets for a CPU = and after the pa_awake loop) I happen to have experimented with additional isyncs in this variant of the code, not that they made much of a difference. The specifics are: # svnlite diff /usr/src/sys/powerpc/aim/mp_cpudep.c = /usr/src/sys/powerpc/powerpc/pmap_dispatch.c = /usr/src/sys/powerpc/powerpc/mp_machdep.c = /usr/src/sys/powerpc/powermac/platform_powermac.c = /usr/src/sys/powerpc/aim/moea64_native.c | more Index: /usr/src/sys/powerpc/aim/mp_cpudep.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/powerpc/aim/mp_cpudep.c (revision 344018) +++ /usr/src/sys/powerpc/aim/mp_cpudep.c (working copy) @@ -105,6 +105,10 @@ =20 __asm __volatile("mtsprg 0, %0" :: "r"(ap_pcpu)); powerpc_sync(); + + *(unsigned long*)0xc0000000000000f0 =3D 0x10; // HACK!!! + isync(); // HACK!!! + powerpc_sync(); // HACK!!! } =20 uintptr_t @@ -124,6 +128,10 @@ pcpup->pc_curpcb =3D pcpup->pc_curthread->td_pcb; sp =3D pcpup->pc_curpcb->pcb_sp; =20 + *(unsigned long*)0xc0000000000000f0 =3D 0x30; // HACK!!! + isync(); // HACK!!! + powerpc_sync(); // HACK!!! + return (sp); } =20 @@ -416,5 +424,9 @@ "suboptimal.\n"); break; } + + *(unsigned long*)0xc0000000000000f0 =3D 0x40; // HACK!!! + isync(); // HACK!!! + powerpc_sync(); // HACK!!! } =20 Index: /usr/src/sys/powerpc/aim/moea64_native.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/powerpc/aim/moea64_native.c (revision 344018) +++ /usr/src/sys/powerpc/aim/moea64_native.c (working copy) @@ -384,6 +384,9 @@ =20 mtmsr(mfmsr() & ~PSL_DR & ~PSL_IR); =20 + *(unsigned long*)0xc0000000000000f0 =3D 0x21; // HACK!!! + powerpc_sync(); // HACK!!! + /* * Install kernel SLB entries */ @@ -393,6 +396,9 @@ __asm __volatile ("slbmfee %0,%1; slbie %0;" : = "=3Dr"(seg0) : "r"(0)); =20 + *(unsigned long*)0xc0000000000000f0 =3D 0x22; // HACK!!! + powerpc_sync(); // HACK!!! + for (i =3D 0; i < n_slbs; i++) { if (!(slb[i].slbe & SLBE_VALID)) continue; @@ -400,6 +406,9 @@ __asm __volatile ("slbmte %0, %1" ::=20 "r"(slb[i].slbv), "r"(slb[i].slbe));=20 } + + *(unsigned long*)0xc0000000000000f0 =3D 0x23; // HACK!!! + powerpc_sync(); // HACK!!! #else for (i =3D 0; i < 16; i++) mtsrin(i << ADDR_SR_SHFT, = kernel_pmap->pm_sr[i]); @@ -412,7 +421,14 @@ __asm __volatile ("ptesync; mtsdr1 %0; isync" :: "r"(((uintptr_t)moea64_pteg_table & ~DMAP_BASE_ADDRESS) | (uintptr_t)(flsl(moea64_pteg_mask >> 11)))); + + *(unsigned long*)0xc0000000000000f0 =3D 0x24; // HACK!!! + powerpc_sync(); // HACK!!! + tlbia(); + + *(unsigned long*)0xc0000000000000f0 =3D 0x25; // HACK!!! + powerpc_sync(); // HACK!!! } Index: /usr/src/sys/powerpc/powerpc/pmap_dispatch.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/powerpc/powerpc/pmap_dispatch.c (revision = 344018) +++ /usr/src/sys/powerpc/powerpc/pmap_dispatch.c (working copy) @@ -445,6 +445,10 @@ */ =20 return (MMU_CPU_BOOTSTRAP(mmu_obj, ap)); + + *(unsigned long*)0xc0000000000000f0 =3D 0x20; // HACK!!! + isync(); // HACK!!! + powerpc_sync(); // HACK!!! } =20 void * Index: /usr/src/sys/powerpc/powerpc/mp_machdep.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/powerpc/powerpc/mp_machdep.c (revision 344018) +++ /usr/src/sys/powerpc/powerpc/mp_machdep.c (working copy) @@ -73,12 +73,20 @@ void machdep_ap_bootstrap(void) { + *(unsigned long*)0xc0000000000000f0 =3D 0x5F; // HACK!!! + isync(); // HACK!!! + powerpc_sync(); // HACK!!! =20 PCPU_SET(awake, 1); __asm __volatile("msync; isync"); =20 + *(unsigned long*)0xc0000000000000f0 =3D 0x51; // HACK!!! + isync(); // HACK!!! + powerpc_sync(); // HACK!!! + while (ap_letgo =3D=3D 0) __asm __volatile("or 31,31,31"); + isync(); // HACK!!! __asm __volatile("or 6,6,6"); =20 /* @@ -109,10 +117,15 @@ =20 while(smp_started =3D=3D 0) ; + isync(); // HACK!!! =20 /* Start per-CPU event timers. */ cpu_initclocks_ap(); =20 + *(unsigned long*)0xc0000000000000f0 =3D 0x50; // HACK!!! + isync(); // HACK!!! + powerpc_sync(); // HACK!!! + /* Announce ourselves awake, and enter the scheduler */ sched_throw(NULL); } @@ -241,11 +254,13 @@ timeout =3D 2000; /* wait 2sec for the = AP */ while (!pc->pc_awake && --timeout > 0) DELAY(1000); + isync(); // HACK!!! } } else { pc->pc_awake =3D 1; } if (pc->pc_awake) { + isync(); // HACK!!! if (bootverbose) printf("Adding CPU %d, hwref=3D%jx, = awake=3D%x\n", pc->pc_cpuid, = (uintmax_t)pc->pc_hwref, @@ -256,6 +271,8 @@ } =20 ap_awake =3D 1; + isync(); // HACK!!! + powerpc_sync(); // HACK!!! =20 /* Provide our current DEC and TB values for APs */ ap_timebase =3D mftb() + 10; @@ -268,6 +285,7 @@ =20 while (ap_awake < smp_cpus) ; + isync(); // HACK!!! =20 if (smp_cpus !=3D cpus || cpus !=3D mp_ncpus) { printf("SMP: %d CPUs found; %d CPUs usable; %d CPUs = woken\n", Index: /usr/src/sys/powerpc/powermac/platform_powermac.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /usr/src/sys/powerpc/powermac/platform_powermac.c (revision = 344018) +++ /usr/src/sys/powerpc/powermac/platform_powermac.c (working copy) @@ -382,10 +382,22 @@ (void)(*rstvec); powerpc_sync(); =20 + isync(); // HACK!!! + if (bootverbose) // HACK!!! + printf("After reset 4&0 for CPU %d, hwref=3D%jx, = awake=3D%x, *(unsigned long*)0xc0000000000000f0=3D0x%jx\n", + pc->pc_cpuid, (uintmax_t)pc->pc_hwref, + pc->pc_awake,(uintmax_t)*(unsigned = long*)0xc0000000000000f0); + timeout =3D 10000; while (!pc->pc_awake && timeout--) DELAY(100); + isync(); // HACK!!! =20 + if (bootverbose) // HACK!!! + printf("After attempted wait for awake CPU %d, = hwref=3D%jx, awake=3D%x, *(unsigned long*)0xc0000000000000f0=3D0x%jx\n", + pc->pc_cpuid, (uintmax_t)pc->pc_hwref, + pc->pc_awake,(uintmax_t)*(unsigned = long*)0xc0000000000000f0); + return ((pc->pc_awake) ? 0 : EBUSY); #else /* No SMP support */ =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)