From owner-freebsd-ppc@freebsd.org Tue Apr 16 21:26:47 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDFB7157E081 for ; Tue, 16 Apr 2019 21:26:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-25.consmr.mail.gq1.yahoo.com (sonic303-25.consmr.mail.gq1.yahoo.com [98.137.64.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1BA15814A3 for ; Tue, 16 Apr 2019 21:26:44 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: slZkX6YVM1nTrv7E13d.loknuSsgevLvJIuVWWEkIoPjBB6e2edsmyBgkO0KAT8 y7ScIbnKBvSxb4QWx7HsSGYTmMJ3T0bwTbxcxfvI46kPWCZEumksH0Oifdx8dh2c3OjnzfPpbmLL _lBfg2NzmNoXSvxQZTMyrLQODsc7SDEzeK4wMkilfFKXeRbDspTR8YlRtuB4QymkEkNbVTSzyJeK Zk7iu5Kskii8eERabCB.PiV9h0UIAO9MZ3QjB3VHPCfjh1vlKZMqGHiGPqFVt5CbPhOfnSjHUqHJ xdSV6MWwSzPDJUmCqOwp5NIVDQiTA3h6bHH3ef4xiYNdFkxrPQadE4km20W9wVVY55YuL3X2cdq1 RSB6wYWzkgWPn916lVMZ6qkLbSOuRrp7lm6V6vayvUKZPOEzXDfza97REbcv4iFXgXbuet2U1mKo xi8gqAhy6PgZzlmRsDAdgnfMN06LUC46YuEn6ZfZ1vRl2YjLjw_NN9IrVf5IWIFZLolV1p2CHIew 2PK23U.gKnwxTM68vzQi51LA.N5bqOqxwAxyOPp_M7D1FOqqtX7QC9KDWPrG9R6y3x1k6BJnMJBp mE7pSwm8IFTlVh3Pep6JexK077Tl9mSWMp0zNdRRWWHHY57Fm_iwaZSASc4tkobQ4pofuWYo1w1A SH_NkBpGSobCgcHiDfKiU1Wb9B8QDrQfsS5j07yY1aw8yKXTdPJxQF0t2QplnpL9Jdas5FJftrbO NqKke24.08.8yjO39o8H_P1EGCugXrxZiBeHgUWIKfdH0E70cK048D.ieCRlga4dTaLCE0sheTk7 fWjLEpmBa9oWoLTUuKqRl6HQu._gv1wnQuO31pEWWowYJtNu0ZRr1Ew6qCn_qhcHrAGyVEWIsf37 xAwZW2fCL.jB6_7Ha2554wmPOzG.dMQAy.g8Oa8UiXT6MlQY8j_AQQwrKy_QorxeAHFbS9d.SsbD LNx8118dgLRWX3P7wmG4H8xt.k17IW6r2IEfORvgPbLCsDDB2sTUbrY5KGFa2YXcaPD.mSCETNRc tVQe_bNuaeB6w80fzVKySLvYFFjOgAtaNKiIxhQPHHqxF_nULnHCc.CJ52G2VvUyWEdcwpl595A4 - Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Tue, 16 Apr 2019 21:26:43 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp403.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 1135979b6d7e4fb92bf7daa05a32ff78; Tue, 16 Apr 2019 21:26:40 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: head -r344018 powerpc64 variant on Powermac G5 (2 sockets, 2 cores each): [*buffer arena] shows up more . . .? [mpc85xx_smp_timebase_sync problem too] From: Mark Millard In-Reply-To: <20190306223611.75c8a87e@titan.knownspace> Date: Tue, 16 Apr 2019 14:26:39 -0700 Cc: freebsd-ppc Content-Transfer-Encoding: 7bit Message-Id: <984188AE-93A9-4321-99EC-522C2E0CE9A9@yahoo.com> References: <20190306151914.44ea831c@titan.knownspace> <8668AAF7-9E6A-4278-9D1B-2ECDBD3804AA@yahoo.com> <99AD89F8-0F90-48BE-A060-DA12FD7129E6@yahoo.com> <20190306223611.75c8a87e@titan.knownspace> To: Justin Hibbits X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 1BA15814A3 X-Spamd-Bar: +++ X-Spamd-Result: default: False [3.45 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; NEURAL_SPAM_SHORT(0.87)[0.875,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.66)[ip: (6.66), ipnet: 98.137.64.0/21(0.93), asn: 36647(0.75), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.66)[0.657,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.77)[0.769,0]; RCVD_IN_DNSWL_NONE(0.00)[206.64.137.98.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Apr 2019 21:26:47 -0000 [Looks to me like the use and content of mpc85xx_smp_timebase_sync have the same type of problems I noted for the proposed powermac patch.] On 2019-Mar-6, at 20:36, Justin Hibbits wrote: > On Wed, 6 Mar 2019 18:35:42 -0800 > Mark Millard wrote: > >> [The patch is definitely wrong via a 3rd use of >> platform_smp_timebase_sync that I'd not noted before. Details at the >> end.] >> >> On 2019-Mar-6, at 16:39, Mark Millard wrote: >> >> >> >>> On 2019-Mar-6, at 13:19, Justin Hibbits >>> wrote: >>>> On Mon, 4 Mar 2019 19:43:09 -0800 >>>> Mark Millard via freebsd-ppc wrote: >>>> >>>>> [It is possible that the following is tied to my hack to >>>>> avoid threads ending up stuck-sleeping. But I ask about >>>>> an alternative that I see in the code.] >>>>> >>>>> Context: using the modern powerpc64 VM_MAX_KERNEL_ADDRESS >>>>> and using usefdt=1 on an old Powermac G5 (2 sockets, 2 cores >>>>> each). Hacks are in use to provide fairly reliable booting >>>>> and to avoid threads getting stuck sleeping. >>>>> >>>>> Before the modern VM_MAX_KERNEL_ADDRESS figure there were only >>>>> 2 or 3 bufspacedaemon-* threads as I remember. Now there are 8 >>>>> (plus bufdaemon and its worker), for example: >>>>> >>>>> root 23 0.0 0.0 0 288 - DL 15:48 0:00.39 >>>>> [bufdaemon/bufdaemon] root 23 0.0 0.0 0 288 - >>>>> DL 15:48 0:00.05 [bufdaemon/bufspaced] root 23 0.0 >>>>> 0.0 0 288 - DL 15:48 0:00.05 [bufdaemon/bufspaced] >>>>> root 23 0.0 0.0 0 288 - DL 15:48 0:00.05 >>>>> [bufdaemon/bufspaced] root 23 0.0 0.0 0 288 - >>>>> DL 15:48 0:00.05 [bufdaemon/bufspaced] root 23 0.0 >>>>> 0.0 0 288 - DL 15:48 0:00.05 [bufdaemon/bufspaced] >>>>> root 23 0.0 0.0 0 288 - DL 15:48 0:00.07 >>>>> [bufdaemon/bufspaced] root 23 0.0 0.0 0 288 - >>>>> DL 15:48 0:00.05 [bufdaemon/bufspaced] root 23 0.0 >>>>> 0.0 0 288 - DL 15:48 0:00.56 [bufdaemon// worker] >>>>> >>>>> I'm sometimes seeing processes showing [*buffer arena] that >>>>> seemed to wait for a fairly long time with that status, not >>>>> something I'd seen historically for those same types of >>>>> processes for a similar overall load (not much). During such >>>>> times trying to create processes to look around at what is >>>>> going on seems to also wait. (Probably with the same status?) >>>>> >>>> >>>> Hi Mark, >>>> >>>> Can you try the attached patch? It might be overkill in the >>>> synchronization, and I might be using the wrong barriers to be >>>> considered correct, but I think this should narrow the race down, >>>> and synchronize the timebases to within a very small margin. The >>>> real correct fix would be to suspend the timebase on all cores, >>>> which is feasible (there's a GPIO for the G4s, and i2c for G5s), >>>> but that's a non-trivial extra work. >>>> >>>> Be warned, I haven't tested it, I've only compiled it (I don't >>>> have a G5 to test with anymore). >>>> >>> >>> Sure, I'll try it when the G5 is again available: it is doing >>> a time consuming build. >>> >>> I do see one possible oddity: tracing another >>> platform_smp_timebase_sync use in the code . . . >>> >>> DEVMETHOD(cpufreq_drv_set, pmufreq_set) >>> >>> static int >>> pmufreq_set(device_t dev, const struct cf_setting *set) >>> { >>> . . . >>> error = pmu_set_speed(speed_sel); >>> . . . >>> } >>> >>> int >>> pmu_set_speed(int low_speed) >>> { >>> . . . >>> platform_sleep(); >>> . . . >>> } >>> >>> PLATFORMMETHOD(platform_sleep, powermac_sleep), >>> >>> void >>> powermac_sleep(platform_t platform) >>> { >>> >>> *(unsigned long *)0x80 = 0x100; >>> cpu_sleep(); >>> } >>> >>> void >>> cpu_sleep() >>> { >>> . . . >>> platform_smp_timebase_sync(timebase, 0); >>> . . . >>> } >>> >>> PLATFORMMETHOD(platform_smp_timebase_sync, >>> powermac_smp_timebase_sync), >>> >>> The issue: >>> >>> I do not see any matching platform_smp_timebase_sync(timebase, 1) >>> or other CPUs doing a powermac_smp_timebase_sync in this sequence. >>> >>> (If this makes testing the patch inappropriate, let me know.) >>> >> >> More important: There is also a use of: >> >> /* The following is needed for restoring from sleep. */ >> platform_smp_timebase_sync(0, 1); >> >> in cpudep_ap_setup . That in turn happens during cpu_reset_handler >> before machdep_ap_bootstrap is called (which does >> platform_smp_timebase_sync as well) : >> >> cpu_reset_handler: >> GET_TOCBASE(%r2) >> >> ld %r1,TOC_REF(tmpstk)(%r2) /* get new SP */ >> addi %r1,%r1,(TMPSTKSZ-48) >> >> bl CNAME(cpudep_ap_early_bootstrap) /* Set PCPU */ >> nop >> lis %r3,1@l >> bl CNAME(pmap_cpu_bootstrap) /* Turn on virtual >> memory */ nop >> bl CNAME(cpudep_ap_bootstrap) /* Set up PCPU and >> stack */ nop >> mr %r1,%r3 /* Use new stack */ >> bl CNAME(cpudep_ap_setup) >> nop >> GET_CPUINFO(%r5) >> ld %r3,(PC_RESTORE)(%r5) >> cmpldi %cr0,%r3,0 >> beq %cr0,2f >> nop >> li %r4,1 >> bl CNAME(longjmp) >> nop >> 2: >> #ifdef SMP >> bl CNAME(machdep_ap_bootstrap) /* And away! */ >> nop >> #endif >> >> Thus overall for ap's there is the sequence: >> >> platform_smp_timebase_sync(0, 1); >> . . . >> while (ap_letgo == 0) >> __asm __volatile("or 31,31,31"); >> __asm __volatile("or 6,6,6"); >> >> /* >> * Set timebase as soon as possible to meet an implicit >> rendezvous >> * from cpu_mp_unleash(), which sets ap_letgo and then >> immediately >> * sets timebase. >> * >> * Note that this is instrinsically racy and is only relevant >> on >> * platforms that do not support better mechanisms. >> */ >> platform_smp_timebase_sync(ap_timebase, 1); >> >> for each ap . So the (ap) case in powermac_smp_timebase_sync >> will wait with tb==0 (from cpudep_ap_setup) and the later calls >> from machdep_ap_bootstrap will not wait and will be after the >> unleash but not just local to powermac_smp_timebase_sync: >> >> static void >> powermac_smp_timebase_sync(platform_t plat, u_long tb, int ap) >> { >> static int cpus; >> static int unleash; >> >> if (ap) { >> atomic_add_int(&cpus, 1); >> while (!atomic_load_acq_int(&unleash)) >> ; >> } else { >> atomic_add_int(&cpus, 1); >> while (atomic_load_int(&cpus) != mp_ncpus) >> ; >> atomic_store_rel_int(&unleash, 1); >> } >> >> mttb(tb); >> } >> >> In the end cpus will have double counts of the ap cpus instead >> of matching mp_ncpus. >> >> cpufreq_drv_set activity is a seperate, additional issue from this. >> >> === >> Mark Millard >> marklmi at yahoo.com >> ( dsl-only.net went >> away in early 2018-Mar) >> > > As mentioned, I had only compiled it. Your examination of the code > path demonstrates that the patch is insufficient, and would hang at > unleash anyway. The sleep/wake logic probably needs to be updated > anyway. It was written for a G4 powerbook primarily for the PMU-based > cpufreq driver, so some bits might need to be moved around. Orthogonal > to this issue, though. It appears to me that the overall sequence: platform_smp_timebase_sync(0, 1); // called from cpudep_ap_setup . . . // The following are from in machdep_ap_bootstrap . . . while (ap_letgo == 0) __asm __volatile("or 31,31,31"); __asm __volatile("or 6,6,6"); . . . platform_smp_timebase_sync(ap_timebase, 1); calls mpc85xx_smp_timebase_sync twice per ap and that the 2nd call has tb_ready && mp_ncpus<=cpu_done for each such ap, leading to a lack of coordination of the activity that 2nd time for each ap: static void mpc85xx_smp_timebase_sync(platform_t plat, u_long tb, int ap) { static volatile bool tb_ready; static volatile int cpu_done; if (ap) { /* APs. Hold off until we get a stable timebase. */ while (!tb_ready) atomic_thread_fence_seq_cst(); mttb(tb); atomic_add_int(&cpu_done, 1); while (cpu_done < mp_ncpus) atomic_thread_fence_seq_cst(); } else { /* BSP */ freeze_timebase(rcpm_dev, true); tb_ready = true; mttb(tb); atomic_add_int(&cpu_done, 1); while (cpu_done < mp_ncpus) atomic_thread_fence_seq_cst(); freeze_timebase(rcpm_dev, false); } } === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)