From owner-freebsd-hackers@freebsd.org Sat Apr 6 08:25:22 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CA9EB157A43F for ; Sat, 6 Apr 2019 08:25:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-20.consmr.mail.gq1.yahoo.com (sonic302-20.consmr.mail.gq1.yahoo.com [98.137.68.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49C896F2EE for ; Sat, 6 Apr 2019 08:25:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 4cnKdn8VM1nYoyhy2wjSk5RYKMFVZTzfVtToQCcExGw7cscAKevnD4ij5g3CIU9 viVNlpClSvqK0OXvT2YrjznQdIvkDG7KaC3GknQ2OhUvQMfPwG9.Y9y4o_ZvFuJXalXuP66wv88s lBiFTjGGpMVQjPzDj3TL3FgGd0rP9R6yxRu.plvknkZsPSdn_nmzfz4_OBDD4xh5iT1mvb.xMqzy Mx2.zDXOvlTIKAbqNPYivl2hCq4tpH2oYq0etF3iDA3N9_WRnwRmX2_toCJSdfkSKnoUkKWAITRR Wo_jUDo6y5bnnNUgm1Ti5eshPNQgyqNP9wmNdn4mr8Re187njoQWywv_MMR12B_DagAdyitl5QRX 52eXpa4H65C9kUU4fMXSkLUN9VBaE0zTZsuw5IOm5sy16QKhtNZgTmwtxV.zD84VQiaJkqEhy8Gv P3cVySaEoF4fTwQNAhQzUDaw.tzlZS6eR0Yqm_Ha3__qbfbXvTCffJFvs8wyrw5QRSJWyeFZiFES JEIRw9Q05So3VWcrhPSk7xKbdZgIrgOHvwZs9HUGbejQ1bjOjE4Aw0VqRelhkldu17Ow3ECT09VV kctRcPrPFNnIC_Gl42j7RvBtuYc1gRYjDDBTEGEep0Xgm6PQiv5eynmq3k5TyVATnHdzeo8xv_Jl u0_a8np3rBonMGR5mYRso4Bs6xukNBdB2IlgPQOF7DCApD8a8J3synuuYgp3MIvNbS41uFpvc2Tu DSMVPR1GcyNtr1m6FezGO9ncw9gTl1owzzQfG5fncpZLVH397UukvV.x1FW2TzbRv38KWExnPS8y xUGzY18SvDUFqdwOdQs3GrzTSVI.E3cGzW2g375Wk_IPIBqJQDU5DU.88HikFMNjQriA8Fv_qNVF 2yZRtM7G8XceAyh1iFab_xEpPwnoLuUOUsUx6WaAF_x8FYwHvmSkT4iZsHs6lyXCBE.SlwS41uCx gJma5CzFClBXJY49fqO.1Uao3CJkm.D8OzlFCLsEWXM2JAsL_nsJCFEEAgbmFwVrJq42QoynW0Sx vYHpvudnhVw3c2FWpijRE4dlqtfUW.A2b91YtzTcfph7UC3nBRLYIpvOr1Yu1BLyEb3FNA_c4Xg- - Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.gq1.yahoo.com with HTTP; Sat, 6 Apr 2019 08:25:18 +0000 Received: from c-67-170-167-181.hsd1.or.comcast.net (EHLO [192.168.1.113]) ([67.170.167.181]) by smtp420.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 75821d711dc554505c099e91b68f42a6; Sat, 06 Apr 2019 08:25:15 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] From: Mark Millard In-Reply-To: <3592A7F1-9EAE-4A33-B51A-678BE104E18C@yahoo.com> Date: Sat, 6 Apr 2019 01:25:13 -0700 Cc: freebsd-hackers Hackers , FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: <7B096BB4-7DBA-4B95-AC1E-DDD9A7DF3B22@yahoo.com> References: <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405150236.A959@besplex.bde.org> <20190405201055.I2396@besplex.bde.org> <20190405113504.GA1923@kib.kiev.ua> <55604EF6-CB81-4300-8E9E-E3A94504D0B5@yahoo.com> <3592A7F1-9EAE-4A33-B51A-678BE104E18C@yahoo.com> To: Konstantin Belousov , Bruce Evans X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 49C896F2EE X-Spamd-Bar: + X-Spamd-Result: default: False [1.45 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(0.89)[0.891,0]; NEURAL_HAM_LONG(-0.52)[-0.521,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.86)[ip: (2.66), ipnet: 98.137.64.0/21(0.95), asn: 36647(0.76), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.73)[0.727,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[146.68.137.98.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 06 Apr 2019 08:25:22 -0000 [Another document was more explicit about 1/16 mode.] On 2019-Apr-6, at 00:54, Mark Millard wrote: > On 2019-Apr-5, at 14:31, Mark Millard wrote: >=20 > [I found a document covering PLLs for the 970MP's in the old > PowerMac G5 involved.] >=20 >> On 2019-Apr-5, at 04:35, Konstantin Belousov = wrote: >>>=20 >>> On Fri, Apr 05, 2019 at 09:13:50PM +1100, Bruce Evans wrote: >>>> On Fri, 5 Apr 2019, Mark Millard wrote: >>>>=20 >>>>> On 2019-Apr-4, at 21:38, Bruce Evans = wrote: >>>>>=20 >>>>>> On Thu, 4 Apr 2019, Mark Millard wrote: >>>>>>> ... >>>>>>> Unfortunately, all the multi-socket contexts that I sometimes = have >>>>>>> access to are old PowerMacs. And, currently, the only such = context >>>>>>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So = I've >>>>>>> not been able to set up other types of examples to see if = problems >>>>>>> repeat. >>>>>>>=20 >>>>>>> I do not have access to a single-socket powerpc64 for contrast = in >>>>>>> that direction. >>>>>>=20 >>>>>> Testing 1 socket is time-consuming enough. Do these old systems >>>>>> use the equivalent of an x86 TSC for the timecounter? With = multiple >>>>>> sockets, it isn't clear how even a hardware timer independent of = the >>>>>> CPUs can be distributed so as to appear to be monotonic on all = cors. >>>>>=20 >>>>> "The DEC frequency is based on the same implementation-dependent >>>>> frequency that drives the time base." The frequency may well be >>>>> fixed by the PowerMac G5 model implementation but is not fixed >>>>> by the powerpc64 architecture. >>>>>=20 >>>>> The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD = terms) >>>>> increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its = value >>>>> can be set (mttb instruction) and the boot sequence in FreeBSD = does >>>>> attempt to adjust as the FreeBSD CPU is brought-up/started. mftb = is >>>>> used to read the 64-bit value. FreeBSD masks it down to 32-bits to >>>>> contribute to the time-counter. >>>>>=20 >>>>> (Is that description sufficient for what you were after? I've = never >>>>> seen documentation of how the 33,333,333 MHz is produced.) >>>>=20 >>>> This seems to be equivalent to an x86 TSC. >>> It is equivalent from the interface PoV. I saw some references that >>> Powers do have per-core PLLs, the best I can find now is >>> = https://stackoverflow.com/questions/43844438/are-multiple-clock-domains-co= mmon-in-modern-processors >>=20 >> I finally found a document covering the 970MP in the old >> PowerMac G5 in question: >>=20 >> http://datasheets.chipdb.org/IBM/PowerPC/970/PowerPC-970MP.pdf >>=20 >> It has material about the PLL's, pinout, etc. : >>=20 >> "This section will help in configuring the PLL and determining SYSCLK = input >> frequency for PowerPC 970MP systems." Also: 100 MHz to 700 MHz = SYSCLK. >> "SYSCLK minimum frequency is for PLL mode. In PLL bypass mode (BYPASS = low), >> SYSCLK frequency may be as low as 10MHz." BUS_CFG[0:2], PLL_MULT, and >> PLL_RANGE[1:0] are involved for PLL mode. >>=20 >> Ratio from BUS_CFG: 3:1, 6:1, 12:1 normally use PLL_MULT=3D0 for a = PLL multiplier of 12. >> Ratio from BUS_CFG: 2:1, 4:1, 8:1 normally use PLL_MULT=3D1 for a PLL = multiplier of 8. >>=20 >> (It describes the PLL power supply filtering circuit as well.) >>=20 >> So far I've not seen anything that is directly for the rate that the = TBR >> and DEC registers change. >>=20 >>> For Intel there is no much information, the best guess is that TSC = is in >>> uncore and effectively shared by all cores. See also >>> = https://software.intel.com/en-us/articles/best-timing-function-for-measuri= ng-ipp-api-timing/ >>>=20 >>>>=20 >>>> x86 P-state-invariant TSCs apparently use a 100 MHz clock which = corresponds >>>> even more closely to this, except for historical reasons this clock = is >>>> scaled and interpolated to a clock resembling the CPU cycle count = at a >>>> nominal frequency. >>> For Intel designs, there is indeed a single-source 100MHz signal = which >>> is distributed over all consumers using clock fan-out buffers like = DB1900Z. >>>=20 >>>>=20 >>>>> As FreeBSD supports multi-socket, what are its criteria for a = sufficient >>>>> context for it to work with for supporting sbinuptime and the = like? Is >>>>> FreeBSD supposed to then make it appear that sbinputime and the = like are >>>>> weakly increasing, even as threads migrate between CPUs (cores, >>>>> hw-threads)? >>>>=20 >>>> CLOCK_MONOTONIC has to be monotonic, but it is only clear what this = means >>>> within a single thread: sequential clock_gettime() calls must occur = in >>>> program order and the results must be consistent with that order. = Across >>>> threads, I think it should mean that the results must be consistent = with >>>> any order established using any supported ordering methods. >>>>=20 >>>>> ... >>>>>>> One oddity is that the eventtimer's decrementer and timecounter >>>>>>> may change (nearly) together: both change at 33,333,333 Hz, as = if >>>>>>> they are tied to the same clock (at least on one socket). >>>>>>=20 >>>>>> I think this is from a normal hardware implementation. On all of >>>>>> my x86 systems with a TSC, the TSC frequency is an exact = fractional >>>>>> multiple of the i8254, the ACPI timer (if present) and the HPET = (if >>>>>> present). Only the RTC has an independent frequency. The = fraction is >>>>>> changed by changing the nominal TSC frequency in the BIOS, but is = not >>>>>> changed by temperature variations. This must be because most = clocks are >>>>>> derived from a common clock using a PLL. I use this to calibrate = all >>>>>> clocks (except the RTC) by calibrating only 1. >>>>>=20 >>>>> I'm not aware of the OpenFirmware having any control over the >>>>> TBR-change frequency behavior. I've no evidence about any = variability >>>>> based on temperature. >>>>=20 >>>> Temperature changes usually affect the actual frequency but not the >>>> nominal frequency. >>>>=20 >>=20 >=20 > I found more about the 970MP's TB/DEC rate: >=20 > "The TBEN input pin can be used as an enable for the internal > timebase/decrementer or as an external clock input." Details: >=20 > HID0 bit 19 =3D 0: update at 1/16th processor core frequency, > but only when TBEN is high. That 1/16th should be of "the full processor frequency, even when the processor itself is running at a lower frequency". The 970FX has 1/8th instead of 1/16th. "Since the mesh clock frequency can be lowered to 1/64th of the full-speed, the time base/decrementer may be increased/decreased by more than one at a time." The decrementer tests for wrap, not for reaching zero. > HID0 bit 19 =3D 1: update at rising edge of TBEN > (must not exceed 1/16th of the core processor's > maximum frequency). >=20 > So the 33,333,333 Hz TB&DEC update rate vs. 2500 MHz would mean > that rising edges of TBEN are in use (HID0 bit 19 =3D 1) in the > PowerMac G5 context in question. >=20 > I've no information about how closely matched the TBEN signals are > at the 2 sockets, beyond the nominal frequency. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)