From owner-freebsd-ppc@freebsd.org Thu Mar 7 02:35:51 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D89001508BED for ; Thu, 7 Mar 2019 02:35:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 768E787457 for ; Thu, 7 Mar 2019 02:35:48 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 5cwPnK8VM1lt9.6L8.6BDwdkK8aC6W65Vpp1AElh_l7Y_baNVbU1f0H96ZkWdQ7 PF5LSH5jx7JtilAoanUVPe9udcVMi0nOGj95EAGK7jbTeEJ7RJvTE2rHSxKyFOJ0YoZMkIIbnVf8 H4Mn8zpYgILGMjyMlOpQX5PUp_Yk7PPrHgJqmqqBOz_jkC2fdRp8Dyo0eH6r1qAwDk7sph_q2JTi RgbAHqtCLAUwtLMJwTl5PSFU4uH7GukR2HnB06ohMOOMH19V5bRNulkbfpwQv0zZqm7nF23iuLLW EdBRrB5SlpDoPhlpVabpAPml.RyuHI6dLoVcwYCft43.iCRfrGYHwN67iqVJbLTEBavbUskhr22_ FxBfWAs0uTx5DDQXRw_N5b7rfNFh90ADiHHlbvrXmo_XxwSPj_JBi5Oh9tYvnMwSQZOZtVI0B.nf c0ul4JQqtlUTYgmDZWYi898CkW2zfLFQydwLIUJiCWG.BwT_AKPzqXOgkOdEKEtvkKaewzyIbivx FY_S.3PmaNRZ.xF.OapWUWGmXelhJLQtXlSeh.SnN64rRo_IxwRmqyTkTUJ7Jx3lCNmT9qX_hrtA e7GMurI_qEQESGGXHY9UP1AUr4tKsiX0lx4PM4w8BM74___TlpEBC2fLPiqJa7AVmjqElw_AYoQz dAQEjJuGf3zITqJr2YnUEdwl3WzJUk89B_WK5pPowRx.X8LVCgbsByKHDt86j5iKNnML6zwHyW.P _Iqce8GCjjJquWMun9SZdz8SEDjcyhsOUFYhN9hfC4G7w.7Wz1Ishu5vUKBGdAHZSYf_Hqiy73.t 9I_lTGiA7SlhJlx3VdTflPSSpafsOgYcSEZa977j9lAbpi59.TGFJHzVyJW5lgKLPDUCrr1U4TsH SG6eySjiQKfCyWm.6O_f2VMb2IeGLrxQs8134tiWprNtcDBOvXUztJdHY_fLFPpDRnBHt7YeTiDD HgsCju9T52SpKUAc8710YyH0wLIQt5XknleUmZun51APcGgKMjLv51w2LQb4EkG5OP7osPoRqCbb H53hwi_kWkHawFv8IQT9EbCiy6cRD3po3th5k_Bsm5D5oNTiwdV8RBElD8RMAJw9JgIabwXxPyOa UstYfU9A- Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Thu, 7 Mar 2019 02:35:46 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.115]) ([67.170.167.181]) by smtp402.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 82ae82b154c65dbf4065e85c2c3af6d2; Thu, 07 Mar 2019 02:35:43 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.2 \(3445.102.3\)) Subject: Re: head -r344018 powerpc64 variant on Powermac G5 (2 sockets, 2 cores each): [*buffer arena] shows up more . . .? From: Mark Millard In-Reply-To: <8668AAF7-9E6A-4278-9D1B-2ECDBD3804AA@yahoo.com> Date: Wed, 6 Mar 2019 18:35:42 -0800 Cc: freebsd-ppc Content-Transfer-Encoding: 7bit Message-Id: <99AD89F8-0F90-48BE-A060-DA12FD7129E6@yahoo.com> References: <20190306151914.44ea831c@titan.knownspace> <8668AAF7-9E6A-4278-9D1B-2ECDBD3804AA@yahoo.com> To: Justin Hibbits X-Mailer: Apple Mail (2.3445.102.3) X-Rspamd-Queue-Id: 768E787457 X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.31 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.95)[0.953,0]; NEURAL_HAM_LONG(-0.23)[-0.227,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_MEDIUM(0.45)[0.445,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[148.69.137.98.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.65)[ip: (1.51), ipnet: 98.137.64.0/21(1.00), asn: 36647(0.80), country: US(-0.07)] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2019 02:35:51 -0000 [The patch is definitely wrong via a 3rd use of platform_smp_timebase_sync that I'd not noted before. Details at the end.] On 2019-Mar-6, at 16:39, Mark Millard wrote: > On 2019-Mar-6, at 13:19, Justin Hibbits wrote: > >> On Mon, 4 Mar 2019 19:43:09 -0800 >> Mark Millard via freebsd-ppc wrote: >> >>> [It is possible that the following is tied to my hack to >>> avoid threads ending up stuck-sleeping. But I ask about >>> an alternative that I see in the code.] >>> >>> Context: using the modern powerpc64 VM_MAX_KERNEL_ADDRESS >>> and using usefdt=1 on an old Powermac G5 (2 sockets, 2 cores >>> each). Hacks are in use to provide fairly reliable booting >>> and to avoid threads getting stuck sleeping. >>> >>> Before the modern VM_MAX_KERNEL_ADDRESS figure there were only >>> 2 or 3 bufspacedaemon-* threads as I remember. Now there are 8 >>> (plus bufdaemon and its worker), for example: >>> >>> root 23 0.0 0.0 0 288 - DL 15:48 0:00.39 >>> [bufdaemon/bufdaemon] root 23 0.0 0.0 0 288 - DL >>> 15:48 0:00.05 [bufdaemon/bufspaced] root 23 0.0 >>> 0.0 0 288 - DL 15:48 0:00.05 [bufdaemon/bufspaced] >>> root 23 0.0 0.0 0 288 - DL 15:48 0:00.05 >>> [bufdaemon/bufspaced] root 23 0.0 0.0 0 288 - DL >>> 15:48 0:00.05 [bufdaemon/bufspaced] root 23 0.0 >>> 0.0 0 288 - DL 15:48 0:00.05 [bufdaemon/bufspaced] >>> root 23 0.0 0.0 0 288 - DL 15:48 0:00.07 >>> [bufdaemon/bufspaced] root 23 0.0 0.0 0 288 - DL >>> 15:48 0:00.05 [bufdaemon/bufspaced] root 23 0.0 >>> 0.0 0 288 - DL 15:48 0:00.56 [bufdaemon// worker] >>> >>> I'm sometimes seeing processes showing [*buffer arena] that >>> seemed to wait for a fairly long time with that status, not >>> something I'd seen historically for those same types of >>> processes for a similar overall load (not much). During such >>> times trying to create processes to look around at what is >>> going on seems to also wait. (Probably with the same status?) >>> >> >> Hi Mark, >> >> Can you try the attached patch? It might be overkill in the >> synchronization, and I might be using the wrong barriers to be >> considered correct, but I think this should narrow the race down, and >> synchronize the timebases to within a very small margin. The real >> correct fix would be to suspend the timebase on all cores, which is >> feasible (there's a GPIO for the G4s, and i2c for G5s), but that's a >> non-trivial extra work. >> >> Be warned, I haven't tested it, I've only compiled it (I don't have a >> G5 to test with anymore). >> > > Sure, I'll try it when the G5 is again available: it is doing > a time consuming build. > > I do see one possible oddity: tracing another > platform_smp_timebase_sync use in the code . . . > > DEVMETHOD(cpufreq_drv_set, pmufreq_set) > > static int > pmufreq_set(device_t dev, const struct cf_setting *set) > { > . . . > error = pmu_set_speed(speed_sel); > . . . > } > > int > pmu_set_speed(int low_speed) > { > . . . > platform_sleep(); > . . . > } > > PLATFORMMETHOD(platform_sleep, powermac_sleep), > > void > powermac_sleep(platform_t platform) > { > > *(unsigned long *)0x80 = 0x100; > cpu_sleep(); > } > > void > cpu_sleep() > { > . . . > platform_smp_timebase_sync(timebase, 0); > . . . > } > > PLATFORMMETHOD(platform_smp_timebase_sync, powermac_smp_timebase_sync), > > The issue: > > I do not see any matching platform_smp_timebase_sync(timebase, 1) > or other CPUs doing a powermac_smp_timebase_sync in this sequence. > > (If this makes testing the patch inappropriate, let me know.) > More important: There is also a use of: /* The following is needed for restoring from sleep. */ platform_smp_timebase_sync(0, 1); in cpudep_ap_setup . That in turn happens during cpu_reset_handler before machdep_ap_bootstrap is called (which does platform_smp_timebase_sync as well) : cpu_reset_handler: GET_TOCBASE(%r2) ld %r1,TOC_REF(tmpstk)(%r2) /* get new SP */ addi %r1,%r1,(TMPSTKSZ-48) bl CNAME(cpudep_ap_early_bootstrap) /* Set PCPU */ nop lis %r3,1@l bl CNAME(pmap_cpu_bootstrap) /* Turn on virtual memory */ nop bl CNAME(cpudep_ap_bootstrap) /* Set up PCPU and stack */ nop mr %r1,%r3 /* Use new stack */ bl CNAME(cpudep_ap_setup) nop GET_CPUINFO(%r5) ld %r3,(PC_RESTORE)(%r5) cmpldi %cr0,%r3,0 beq %cr0,2f nop li %r4,1 bl CNAME(longjmp) nop 2: #ifdef SMP bl CNAME(machdep_ap_bootstrap) /* And away! */ nop #endif Thus overall for ap's there is the sequence: platform_smp_timebase_sync(0, 1); . . . while (ap_letgo == 0) __asm __volatile("or 31,31,31"); __asm __volatile("or 6,6,6"); /* * Set timebase as soon as possible to meet an implicit rendezvous * from cpu_mp_unleash(), which sets ap_letgo and then immediately * sets timebase. * * Note that this is instrinsically racy and is only relevant on * platforms that do not support better mechanisms. */ platform_smp_timebase_sync(ap_timebase, 1); for each ap . So the (ap) case in powermac_smp_timebase_sync will wait with tb==0 (from cpudep_ap_setup) and the later calls from machdep_ap_bootstrap will not wait and will be after the unleash but not just local to powermac_smp_timebase_sync: static void powermac_smp_timebase_sync(platform_t plat, u_long tb, int ap) { static int cpus; static int unleash; if (ap) { atomic_add_int(&cpus, 1); while (!atomic_load_acq_int(&unleash)) ; } else { atomic_add_int(&cpus, 1); while (atomic_load_int(&cpus) != mp_ncpus) ; atomic_store_rel_int(&unleash, 1); } mttb(tb); } In the end cpus will have double counts of the ap cpus instead of matching mp_ncpus. cpufreq_drv_set activity is a seperate, additional issue from this. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)