From owner-freebsd-ppc@freebsd.org Wed Apr 17 00:21:07 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9EFBE1581D66 for ; Wed, 17 Apr 2019 00:21:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-22.consmr.mail.gq1.yahoo.com (sonic302-22.consmr.mail.gq1.yahoo.com [98.137.68.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2B5B187E3D for ; Wed, 17 Apr 2019 00:21:05 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 5lzxx.gVM1nbhgPZb21_zBGR1KKA.a4OYW6rfGKhF6tbLyoerey.ynNooQtSCOo CMCfif88fbBsAfjHi.d53ZrYAMuCJbEfU_bNlt_tpdAYGvnL4y2eiv6hxKhXgGd0iYHBarcCc6FF 3jtvMIYiIHBVfBHUs65Bk16LSxTdAvOBSVCupSSKakrRzT8WVIigYhJ2LhSAu9MUqQL0i0CgfnAB Wu4EtqVuWwuFo2Nkvl3KCGF4cApcmCiYQWB5EMwf5svybtgjMPQ_bj_Gw6Y06kIIsiPKoVxwkYcB GZVbMMuTnS1j5Xg2JmAiYsYqrALxu_uAFfrKgDci5fJiGaHQofgkxgk69iI_zKfHKxwsRDk6ffDn IWkpbccOaKFgLkv4cO_sf84XCBKmcsYiLE9TixLiF3GnIQbjbxoAhVB7Km98BYdDaWw91P1dOOgQ KXGjlmeyn9SWaDgGqx5t0o86_DSwL23k_8.PWNv7bPJ3aiaTDQNmf_k9TMXzfLj2zS16HrATLKnU X2q6fg5LMI1hD9NR1xNKFGLAimm5Nuqg03HW2USKnamOmkwcLAhlYeI.3K_Sckt.SMu0iFPm42dQ G.AOonWwRw1JstiAdqBMlyIhgg0ky.8NXJAfkFYrJUxgrcoifJYQF15xApAIxc7KTrghX3Ola1s8 E8JSVQcphTduVDrI2zDghhNtVMjmrZNhqnNTumFWmHpIkp1PkYf8gGb5Qd8YPmZZTAp0PeOWNW7z xDhIlG2.Mo.Vf9NVWx8CIWv5cG5ATt31xivr60tWy3jl_fUQJrnzvwiDkE9FLV6_JjY.ysYyDkdb jumWC7bFSfQ2hJJh3ATc4RMnSt61b.SYp9avbvWk7oAMqp0ga.6XO6zxqw4tJvkMfvoAO932GiWE LW2oRU7RIyBwBnTTnHt.Xu7nC1XCClBIbDDayGas_qcX1BRtRJuQly46h34MHi3FmPI9CaNp.qy9 VC4yxXa8gCpGFg5G8f0HWpYYYhWSc1wEerVGh0x_AQep0TQoG0M48lNGxhifld5FveNTcbctR3Xa VRzGJkJuhB4YFbzcWUGdw9W1JKQ3TMYJbrWZ766KGtEX8cBj7y708oHbeuzwblNaYj4yLlZ490Z7 .7YvsAnkQII_E6mBmD.5kvkM- Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.gq1.yahoo.com with HTTP; Wed, 17 Apr 2019 00:20:57 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp427.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 52c00388fd3233c48737238e1b07c9c0; Wed, 17 Apr 2019 00:20:56 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: head -r344018 powerpc64 variant on Powermac G5 (2 sockets, 2 cores each): [*buffer arena] shows up more . . .? [mpc85xx_smp_timebase_sync problem too] From: Mark Millard In-Reply-To: <8141B858-08B0-4B88-8439-747023A98822@yahoo.com> Date: Tue, 16 Apr 2019 17:20:56 -0700 Cc: freebsd-ppc Content-Transfer-Encoding: quoted-printable Message-Id: <306BC23A-3261-41BE-AB0B-4264E864DAA5@yahoo.com> References: <20190306151914.44ea831c@titan.knownspace> <8668AAF7-9E6A-4278-9D1B-2ECDBD3804AA@yahoo.com> <99AD89F8-0F90-48BE-A060-DA12FD7129E6@yahoo.com> <20190306223611.75c8a87e@titan.knownspace> <984188AE-93A9-4321-99EC-522C2E0CE9A9@yahoo.com> <20190416164210.26827a6d@titan.knownspace> <8141B858-08B0-4B88-8439-747023A98822@yahoo.com> To: Justin Hibbits X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 2B5B187E3D X-Spamd-Bar: +++ X-Spamd-Result: default: False [3.67 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; NEURAL_SPAM_SHORT(0.79)[0.788,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.89)[ip: (7.82), ipnet: 98.137.64.0/21(0.93), asn: 36647(0.75), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.68)[0.685,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.82)[0.820,0]; RCVD_IN_DNSWL_NONE(0.00)[148.68.137.98.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2019 00:21:07 -0000 [Looks a little odder in the details than I initially thought: the 0 in platform_smp_timebase_sync(timebase, 0) does not mean the bsp instead of some other ap. I've still not found a reasonable way to not have the sleep-gets-stuck issue for usefdt mode.] On 2019-Apr-16, at 15:36, Mark Millard wrote: > [Unfortunately cpufreq_drv_set leads to additional > platform_smp_timebase_sync(timebase, 0) use.] >=20 > On 2019-Apr-16, at 14:42, Justin Hibbits = wrote: >=20 >> On Tue, 16 Apr 2019 14:26:39 -0700 >> Mark Millard wrote: >>=20 >>> [Looks to me like the use and content of mpc85xx_smp_timebase_sync >>> have the same type of problems I noted for the proposed powermac >>> patch.] >>>=20 >> ... >>>>=20 >>>> As mentioned, I had only compiled it. Your examination of the code >>>> path demonstrates that the patch is insufficient, and would hang at >>>> unleash anyway. The sleep/wake logic probably needs to be updated >>>> anyway. It was written for a G4 powerbook primarily for the >>>> PMU-based cpufreq driver, so some bits might need to be moved >>>> around. Orthogonal to this issue, though. =20 >>>=20 >>> It appears to me that the overall sequence: >>>=20 >>> platform_smp_timebase_sync(0, 1); // called from >>> cpudep_ap_setup . . . >>> // The following are from in machdep_ap_bootstrap . . . >>> while (ap_letgo =3D=3D 0) >>> __asm __volatile("or 31,31,31"); >>> __asm __volatile("or 6,6,6"); >>> . . . >>> platform_smp_timebase_sync(ap_timebase, 1); >>>=20 >>> calls mpc85xx_smp_timebase_sync twice per ap and that the >>> 2nd call has tb_ready && mp_ncpus<=3Dcpu_done for each such >>> ap, leading to a lack of coordination of the activity that >>> 2nd time for each ap: >>>=20 >>> static void >>> mpc85xx_smp_timebase_sync(platform_t plat, u_long tb, int ap) >>> { >>> static volatile bool tb_ready; >>> static volatile int cpu_done; >>>=20 >>> if (ap) { >>> /* APs. Hold off until we get a stable timebase. */ >>> while (!tb_ready) >>> atomic_thread_fence_seq_cst(); >>> mttb(tb); >>> atomic_add_int(&cpu_done, 1); >>> while (cpu_done < mp_ncpus) >>> atomic_thread_fence_seq_cst(); >>> } else { >>> /* BSP */ >>> freeze_timebase(rcpm_dev, true); >>> tb_ready =3D true; >>> mttb(tb); >>> atomic_add_int(&cpu_done, 1); >>> while (cpu_done < mp_ncpus) >>> atomic_thread_fence_seq_cst(); >>> freeze_timebase(rcpm_dev, false); >>> } >>> } >>>=20 >>=20 >> Not for mpc85xx. This is call is only in the AIM cpudep_ap_setup, = and >> really shouldn't be there anyway. >=20 > Sorry for having looked in the wrong source. >=20 >> The original code to just set the >> timebase was there to set it to 0 just as a semi-sane value until the >> core got to a stable state. The original code was literally = "mttb(0)" >> for G4, and a check for hypervisor with mttb(0). Had that been >> sufficient, the platform_smp_timebase_sync() would not be a problem = to >> do the real sync as I had mentioned in my patch before. >>=20 >> The powermac patch that I had provided was derived from this >> well-working mpc85xx patch. However, I had neglected to check that = the >> sync wasn't used elsewhere as well. If the cpudep_ap_setup() use is >> removed, then this should work fine. >=20 > Unfortunately there is use from cpufreq_drv_set getting to cpu_sleep's > platform_smp_timebase_sync use as well: >=20 > /usr/src/sys/powerpc/powerpc/mp_machdep.c: = platform_smp_timebase_sync(ap_timebase, 1); > /usr/src/sys/powerpc/powerpc/mp_machdep.c: = platform_smp_timebase_sync(ap_timebase, 0); > /usr/src/sys/powerpc/aim/aim_machdep.c: = platform_smp_timebase_sync(timebase, 0); >=20 > that last is in cpu_sleep. Tracing towards what ultimately uses = cpu_sleep . . . >=20 > void > powermac_sleep(platform_t platform) > { >=20 > *(unsigned long *)0x80 =3D 0x100; > cpu_sleep(); > } >=20 > is used as platform_sleep. >=20 > pu_mu_set_speed calls platform_sleep and is called by pmufreq_set. > pmufreq_set is used as cpufreq_drv_set. >=20 > At least on the PowerMac11,2, this seems to be in use for > powerd if powerpd is started. >=20 > It also looks like such use does not try to make the other aps > track the change, just ap=3D=3D0 . >=20 > That may explain much of the variability across CPUs for usefdt mode! >=20 > It looks to me like the ap=3D=3D0 case in the = platform_smp_timebase_sync > use in cpu_sleep would not be well behaved under the patch, > even for that one ap: >=20 > static void > powermac_smp_timebase_sync(platform_t plat, u_long tb, int ap) > { > + static int cpus; > + static int unleash; > + > + if (ap) { > + atomic_add_int(&cpus, 1); > + while (!atomic_load_acq_int(&unleash)) > + ; > + } else { > + atomic_add_int(&cpus, 1); > + while (atomic_load_int(&cpus) !=3D mp_ncpus) > + ; > + atomic_store_rel_int(&unleash, 1); > + } >=20 > mttb(tb); > } >=20 > The else would increment cpus beyond mp_cpus and the loop > would be stuck. cpu_sleep getting to powermac_smp_timebase_sync seems to be based on cpu_reset_handler doing the longjmp in: GET_CPUINFO(%r5) ld %r3,(PC_RESTORE)(%r5) cmpldi %cr0,%r3,0 beq %cr0,2f nop li %r4,1 bl CNAME(longjmp) nop 2: #ifdef SMP bl CNAME(machdep_ap_bootstrap) /* And away! */ nop #endif The PC_RESTORE related test is tied to: PCPU_SET(restore, &resetjb); in cpu_sleep. The usage is: if (setjmp(resetjb) =3D=3D 0) { . . . timebase =3D mftb(); . . . while (1) mtmsr(msr); } platform_smp_timebase_sync(timebase, 0); ... So it seems there is a platform_smp_timebase_sync(timebase, 0) for every cpu that gets cpu_reset_handler invoked, even the cpu's that are not the bsp. I had assumed the ", 0" meant a bsp context previously. There is: static u_quad_t timebase =3D 0; So timebase is not preserved for each cpu if more than one ends up in cpu_sleep. I'm not sure the timebase value is synchronized across cpus. (It is not volatile either. I'm not sure that the code generation would always store and load the value from memory.) Overall it looks like the ", 0" is (ambiguously) intending to indicate that the platform_smp_timebase_sync activity should be unsynchronized with other cpus. May be a distinct, alternate argument value for that that is explicitly handled? So I'm still not to a stage of having a reasonable workaround for the "sleep gets stuck" problem for usefdt mode. (Just an inappropriate avoidance-hack.) https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D233863 has investigative patches for other usefdt tied issues, patches that could be useful starting points for official updates. But the hack that I've been using to avoid stuck-sleeps is not appropriate --and so is not submitted. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)