From owner-freebsd-ppc@freebsd.org Thu Mar 7 04:36:17 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87C0F150F1D5 for ; Thu, 7 Mar 2019 04:36:17 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: from mail-it1-x141.google.com (mail-it1-x141.google.com [IPv6:2607:f8b0:4864:20::141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2FBA58CB43 for ; Thu, 7 Mar 2019 04:36:16 +0000 (UTC) (envelope-from chmeeedalf@gmail.com) Received: by mail-it1-x141.google.com with SMTP id v2so13259301ith.3 for ; Wed, 06 Mar 2019 20:36:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=odaZ+eHeEuQU5nP4iQ1cxO7Md3T16x77bONXqDCdJx0=; b=E3zTAW6R082gzoeNwlAnHlkNuhNrdGznkBBYplqVF/0EzQ1cECI44brKqBARhcIfYe ex2ik8+6Zer/fIiP2Ai8Nxrx6cwyv3cXkUCfBWhARsj36OdxLJ5M68ngB6j2ngn0fPXS V1gKLaqaNpmW/YUjX453rjzzdvMV47A2s4u3zbc8aFIiu5q6kKQztpKZEWJ9uYLbP5I4 pEzjcGGDcfn564/XX6Of1R4RIlmxCknSgHk4T1aTR7/X0TLUshS7p/LhjgQZ744esTyk OxTwavH3vJIJGCjvIvVKyHWjsSVEKyEi6WfqgZznEwVU5XfCZ//Y1GTlLOt5sWpj6Qy4 Ad8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=odaZ+eHeEuQU5nP4iQ1cxO7Md3T16x77bONXqDCdJx0=; b=RetybH1rjHdScKbWMRTa7+KPva+VnLvc7tj81qvqhhiTXbyUlCcTE4Qp2RsFA6tP3P kzclrcpSKHOPpcq/MGOVQLsL/WnhU3o9uHPeMxp2th1vzl2UX3lcG6X4sDlFp+IQOehE Sra5uNQ1h0DS2bEsjLTLP8jXhul6bwRc/WnAgJX/FYt/Nzy89M4lUeLr9kAOvwQqWoGa s4939883zcWlLiPVyrFValdyORfDBBtGfZh5dDUnO+nSxHepPslpbv4beLXY8xdaJCGv eiuNaGXwdTVX7XKQcs/kivmTHCtx5ncyWr6IGMzQIzvCw+6VeKQabQJVxquMx3i4oI9u Ms/g== X-Gm-Message-State: APjAAAW72wBC0Tx2T48XUYE0I1CxAiny6BgUNpdun1kaIyfT5hI1G9/p jWS12FmrpFM1DsqskOzzExM= X-Google-Smtp-Source: APXvYqwpmUArYNx3EAdzkr0tmrRXNtbiKPasQcU/a+Qjra7uXiKo3prjENzBbzvifbtRx8uyIS1K+Q== X-Received: by 2002:a02:c786:: with SMTP id n6mr6619808jao.49.1551933375379; Wed, 06 Mar 2019 20:36:15 -0800 (PST) Received: from titan.knownspace (173-25-245-129.client.mchsi.com. [173.25.245.129]) by smtp.gmail.com with ESMTPSA id d184sm1889266itc.17.2019.03.06.20.36.14 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 06 Mar 2019 20:36:15 -0800 (PST) Date: Wed, 6 Mar 2019 22:36:11 -0600 From: Justin Hibbits To: Mark Millard Cc: freebsd-ppc Subject: Re: head -r344018 powerpc64 variant on Powermac G5 (2 sockets, 2 cores each): [*buffer arena] shows up more . . .? Message-ID: <20190306223611.75c8a87e@titan.knownspace> In-Reply-To: <99AD89F8-0F90-48BE-A060-DA12FD7129E6@yahoo.com> References: <20190306151914.44ea831c@titan.knownspace> <8668AAF7-9E6A-4278-9D1B-2ECDBD3804AA@yahoo.com> <99AD89F8-0F90-48BE-A060-DA12FD7129E6@yahoo.com> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; powerpc64-portbld-freebsd13.0) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 2FBA58CB43 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=E3zTAW6R; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of chmeeedalf@gmail.com designates 2607:f8b0:4864:20::141 as permitted sender) smtp.mailfrom=chmeeedalf@gmail.com X-Spamd-Result: default: False [-2.89 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[alt3.gmail-smtp-in.l.google.com,alt4.gmail-smtp-in.l.google.com,gmail-smtp-in.l.google.com,alt2.gmail-smtp-in.l.google.com,alt1.gmail-smtp-in.l.google.com]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.33)[-0.327,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-ppc@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.4.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.55)[ip: (2.07), ipnet: 2607:f8b0::/32(-2.71), asn: 15169(-2.04), country: US(-0.07)] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2019 04:36:17 -0000 On Wed, 6 Mar 2019 18:35:42 -0800 Mark Millard wrote: > [The patch is definitely wrong via a 3rd use of > platform_smp_timebase_sync that I'd not noted before. Details at the > end.] > > On 2019-Mar-6, at 16:39, Mark Millard wrote: > > > > > On 2019-Mar-6, at 13:19, Justin Hibbits > > wrote: > >> On Mon, 4 Mar 2019 19:43:09 -0800 > >> Mark Millard via freebsd-ppc wrote: > >> > >>> [It is possible that the following is tied to my hack to > >>> avoid threads ending up stuck-sleeping. But I ask about > >>> an alternative that I see in the code.] > >>> > >>> Context: using the modern powerpc64 VM_MAX_KERNEL_ADDRESS > >>> and using usefdt=1 on an old Powermac G5 (2 sockets, 2 cores > >>> each). Hacks are in use to provide fairly reliable booting > >>> and to avoid threads getting stuck sleeping. > >>> > >>> Before the modern VM_MAX_KERNEL_ADDRESS figure there were only > >>> 2 or 3 bufspacedaemon-* threads as I remember. Now there are 8 > >>> (plus bufdaemon and its worker), for example: > >>> > >>> root 23 0.0 0.0 0 288 - DL 15:48 0:00.39 > >>> [bufdaemon/bufdaemon] root 23 0.0 0.0 0 288 - > >>> DL 15:48 0:00.05 [bufdaemon/bufspaced] root 23 0.0 > >>> 0.0 0 288 - DL 15:48 0:00.05 [bufdaemon/bufspaced] > >>> root 23 0.0 0.0 0 288 - DL 15:48 0:00.05 > >>> [bufdaemon/bufspaced] root 23 0.0 0.0 0 288 - > >>> DL 15:48 0:00.05 [bufdaemon/bufspaced] root 23 0.0 > >>> 0.0 0 288 - DL 15:48 0:00.05 [bufdaemon/bufspaced] > >>> root 23 0.0 0.0 0 288 - DL 15:48 0:00.07 > >>> [bufdaemon/bufspaced] root 23 0.0 0.0 0 288 - > >>> DL 15:48 0:00.05 [bufdaemon/bufspaced] root 23 0.0 > >>> 0.0 0 288 - DL 15:48 0:00.56 [bufdaemon// worker] > >>> > >>> I'm sometimes seeing processes showing [*buffer arena] that > >>> seemed to wait for a fairly long time with that status, not > >>> something I'd seen historically for those same types of > >>> processes for a similar overall load (not much). During such > >>> times trying to create processes to look around at what is > >>> going on seems to also wait. (Probably with the same status?) > >>> > >> > >> Hi Mark, > >> > >> Can you try the attached patch? It might be overkill in the > >> synchronization, and I might be using the wrong barriers to be > >> considered correct, but I think this should narrow the race down, > >> and synchronize the timebases to within a very small margin. The > >> real correct fix would be to suspend the timebase on all cores, > >> which is feasible (there's a GPIO for the G4s, and i2c for G5s), > >> but that's a non-trivial extra work. > >> > >> Be warned, I haven't tested it, I've only compiled it (I don't > >> have a G5 to test with anymore). > >> > > > > Sure, I'll try it when the G5 is again available: it is doing > > a time consuming build. > > > > I do see one possible oddity: tracing another > > platform_smp_timebase_sync use in the code . . . > > > > DEVMETHOD(cpufreq_drv_set, pmufreq_set) > > > > static int > > pmufreq_set(device_t dev, const struct cf_setting *set) > > { > > . . . > > error = pmu_set_speed(speed_sel); > > . . . > > } > > > > int > > pmu_set_speed(int low_speed) > > { > > . . . > > platform_sleep(); > > . . . > > } > > > > PLATFORMMETHOD(platform_sleep, powermac_sleep), > > > > void > > powermac_sleep(platform_t platform) > > { > > > > *(unsigned long *)0x80 = 0x100; > > cpu_sleep(); > > } > > > > void > > cpu_sleep() > > { > > . . . > > platform_smp_timebase_sync(timebase, 0); > > . . . > > } > > > > PLATFORMMETHOD(platform_smp_timebase_sync, > > powermac_smp_timebase_sync), > > > > The issue: > > > > I do not see any matching platform_smp_timebase_sync(timebase, 1) > > or other CPUs doing a powermac_smp_timebase_sync in this sequence. > > > > (If this makes testing the patch inappropriate, let me know.) > > > > More important: There is also a use of: > > /* The following is needed for restoring from sleep. */ > platform_smp_timebase_sync(0, 1); > > in cpudep_ap_setup . That in turn happens during cpu_reset_handler > before machdep_ap_bootstrap is called (which does > platform_smp_timebase_sync as well) : > > cpu_reset_handler: > GET_TOCBASE(%r2) > > ld %r1,TOC_REF(tmpstk)(%r2) /* get new SP */ > addi %r1,%r1,(TMPSTKSZ-48) > > bl CNAME(cpudep_ap_early_bootstrap) /* Set PCPU */ > nop > lis %r3,1@l > bl CNAME(pmap_cpu_bootstrap) /* Turn on virtual > memory */ nop > bl CNAME(cpudep_ap_bootstrap) /* Set up PCPU and > stack */ nop > mr %r1,%r3 /* Use new stack */ > bl CNAME(cpudep_ap_setup) > nop > GET_CPUINFO(%r5) > ld %r3,(PC_RESTORE)(%r5) > cmpldi %cr0,%r3,0 > beq %cr0,2f > nop > li %r4,1 > bl CNAME(longjmp) > nop > 2: > #ifdef SMP > bl CNAME(machdep_ap_bootstrap) /* And away! */ > nop > #endif > > Thus overall for ap's there is the sequence: > > platform_smp_timebase_sync(0, 1); > . . . > while (ap_letgo == 0) > __asm __volatile("or 31,31,31"); > __asm __volatile("or 6,6,6"); > > /* > * Set timebase as soon as possible to meet an implicit > rendezvous > * from cpu_mp_unleash(), which sets ap_letgo and then > immediately > * sets timebase. > * > * Note that this is instrinsically racy and is only relevant > on > * platforms that do not support better mechanisms. > */ > platform_smp_timebase_sync(ap_timebase, 1); > > for each ap . So the (ap) case in powermac_smp_timebase_sync > will wait with tb==0 (from cpudep_ap_setup) and the later calls > from machdep_ap_bootstrap will not wait and will be after the > unleash but not just local to powermac_smp_timebase_sync: > > static void > powermac_smp_timebase_sync(platform_t plat, u_long tb, int ap) > { > static int cpus; > static int unleash; > > if (ap) { > atomic_add_int(&cpus, 1); > while (!atomic_load_acq_int(&unleash)) > ; > } else { > atomic_add_int(&cpus, 1); > while (atomic_load_int(&cpus) != mp_ncpus) > ; > atomic_store_rel_int(&unleash, 1); > } > > mttb(tb); > } > > In the end cpus will have double counts of the ap cpus instead > of matching mp_ncpus. > > cpufreq_drv_set activity is a seperate, additional issue from this. > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > As mentioned, I had only compiled it. Your examination of the code path demonstrates that the patch is insufficient, and would hang at unleash anyway. The sleep/wake logic probably needs to be updated anyway. It was written for a G4 powerbook primarily for the PMU-based cpufreq driver, so some bits might need to be moved around. Orthogonal to this issue, though. - Justin