Date: Sat, 6 Apr 2019 01:25:13 -0700 From: Mark Millard <marklmi@yahoo.com> To: Konstantin Belousov <kostikbel@gmail.com>, Bruce Evans <brde@optusnet.com.au> Cc: freebsd-hackers Hackers <freebsd-hackers@freebsd.org>, FreeBSD PowerPC ML <freebsd-ppc@freebsd.org> Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] Message-ID: <7B096BB4-7DBA-4B95-AC1E-DDD9A7DF3B22@yahoo.com> In-Reply-To: <3592A7F1-9EAE-4A33-B51A-678BE104E18C@yahoo.com> References: <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <E0785613-2B6E-4BB3-95CD-03DD96902CD8@fh-muenster.de> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <F22CCA2C-08BB-452E-B00C-A36CD4611540@yahoo.com> <20190405150236.A959@besplex.bde.org> <F3BBD355-5198-41A0-A461-8E5E12984BE7@yahoo.com> <20190405201055.I2396@besplex.bde.org> <20190405113504.GA1923@kib.kiev.ua> <55604EF6-CB81-4300-8E9E-E3A94504D0B5@yahoo.com> <3592A7F1-9EAE-4A33-B51A-678BE104E18C@yahoo.com>
index | next in thread | previous in thread | raw e-mail
[Another document was more explicit about 1/16 mode.] On 2019-Apr-6, at 00:54, Mark Millard <marklmi at yahoo.com> wrote: > On 2019-Apr-5, at 14:31, Mark Millard <marklmi at yahoo.com> wrote: > > [I found a document covering PLLs for the 970MP's in the old > PowerMac G5 involved.] > >> On 2019-Apr-5, at 04:35, Konstantin Belousov <kostikbel at gmail.com> wrote: >>> >>> On Fri, Apr 05, 2019 at 09:13:50PM +1100, Bruce Evans wrote: >>>> On Fri, 5 Apr 2019, Mark Millard wrote: >>>> >>>>> On 2019-Apr-4, at 21:38, Bruce Evans <brde at optusnet.com.au> wrote: >>>>> >>>>>> On Thu, 4 Apr 2019, Mark Millard wrote: >>>>>>> ... >>>>>>> Unfortunately, all the multi-socket contexts that I sometimes have >>>>>>> access to are old PowerMacs. And, currently, the only such context >>>>>>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've >>>>>>> not been able to set up other types of examples to see if problems >>>>>>> repeat. >>>>>>> >>>>>>> I do not have access to a single-socket powerpc64 for contrast in >>>>>>> that direction. >>>>>> >>>>>> Testing 1 socket is time-consuming enough. Do these old systems >>>>>> use the equivalent of an x86 TSC for the timecounter? With multiple >>>>>> sockets, it isn't clear how even a hardware timer independent of the >>>>>> CPUs can be distributed so as to appear to be monotonic on all cors. >>>>> >>>>> "The DEC frequency is based on the same implementation-dependent >>>>> frequency that drives the time base." The frequency may well be >>>>> fixed by the PowerMac G5 model implementation but is not fixed >>>>> by the powerpc64 architecture. >>>>> >>>>> The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD terms) >>>>> increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its value >>>>> can be set (mttb instruction) and the boot sequence in FreeBSD does >>>>> attempt to adjust as the FreeBSD CPU is brought-up/started. mftb is >>>>> used to read the 64-bit value. FreeBSD masks it down to 32-bits to >>>>> contribute to the time-counter. >>>>> >>>>> (Is that description sufficient for what you were after? I've never >>>>> seen documentation of how the 33,333,333 MHz is produced.) >>>> >>>> This seems to be equivalent to an x86 TSC. >>> It is equivalent from the interface PoV. I saw some references that >>> Powers do have per-core PLLs, the best I can find now is >>> https://stackoverflow.com/questions/43844438/are-multiple-clock-domains-common-in-modern-processors >> >> I finally found a document covering the 970MP in the old >> PowerMac G5 in question: >> >> http://datasheets.chipdb.org/IBM/PowerPC/970/PowerPC-970MP.pdf >> >> It has material about the PLL's, pinout, etc. : >> >> "This section will help in configuring the PLL and determining SYSCLK input >> frequency for PowerPC 970MP systems." Also: 100 MHz to 700 MHz SYSCLK. >> "SYSCLK minimum frequency is for PLL mode. In PLL bypass mode (BYPASS low), >> SYSCLK frequency may be as low as 10MHz." BUS_CFG[0:2], PLL_MULT, and >> PLL_RANGE[1:0] are involved for PLL mode. >> >> Ratio from BUS_CFG: 3:1, 6:1, 12:1 normally use PLL_MULT=0 for a PLL multiplier of 12. >> Ratio from BUS_CFG: 2:1, 4:1, 8:1 normally use PLL_MULT=1 for a PLL multiplier of 8. >> >> (It describes the PLL power supply filtering circuit as well.) >> >> So far I've not seen anything that is directly for the rate that the TBR >> and DEC registers change. >> >>> For Intel there is no much information, the best guess is that TSC is in >>> uncore and effectively shared by all cores. See also >>> https://software.intel.com/en-us/articles/best-timing-function-for-measuring-ipp-api-timing/ >>> >>>> >>>> x86 P-state-invariant TSCs apparently use a 100 MHz clock which corresponds >>>> even more closely to this, except for historical reasons this clock is >>>> scaled and interpolated to a clock resembling the CPU cycle count at a >>>> nominal frequency. >>> For Intel designs, there is indeed a single-source 100MHz signal which >>> is distributed over all consumers using clock fan-out buffers like DB1900Z. >>> >>>> >>>>> As FreeBSD supports multi-socket, what are its criteria for a sufficient >>>>> context for it to work with for supporting sbinuptime and the like? Is >>>>> FreeBSD supposed to then make it appear that sbinputime and the like are >>>>> weakly increasing, even as threads migrate between CPUs (cores, >>>>> hw-threads)? >>>> >>>> CLOCK_MONOTONIC has to be monotonic, but it is only clear what this means >>>> within a single thread: sequential clock_gettime() calls must occur in >>>> program order and the results must be consistent with that order. Across >>>> threads, I think it should mean that the results must be consistent with >>>> any order established using any supported ordering methods. >>>> >>>>> ... >>>>>>> One oddity is that the eventtimer's decrementer and timecounter >>>>>>> may change (nearly) together: both change at 33,333,333 Hz, as if >>>>>>> they are tied to the same clock (at least on one socket). >>>>>> >>>>>> I think this is from a normal hardware implementation. On all of >>>>>> my x86 systems with a TSC, the TSC frequency is an exact fractional >>>>>> multiple of the i8254, the ACPI timer (if present) and the HPET (if >>>>>> present). Only the RTC has an independent frequency. The fraction is >>>>>> changed by changing the nominal TSC frequency in the BIOS, but is not >>>>>> changed by temperature variations. This must be because most clocks are >>>>>> derived from a common clock using a PLL. I use this to calibrate all >>>>>> clocks (except the RTC) by calibrating only 1. >>>>> >>>>> I'm not aware of the OpenFirmware having any control over the >>>>> TBR-change frequency behavior. I've no evidence about any variability >>>>> based on temperature. >>>> >>>> Temperature changes usually affect the actual frequency but not the >>>> nominal frequency. >>>> >> > > I found more about the 970MP's TB/DEC rate: > > "The TBEN input pin can be used as an enable for the internal > timebase/decrementer or as an external clock input." Details: > > HID0 bit 19 = 0: update at 1/16th processor core frequency, > but only when TBEN is high. That 1/16th should be of "the full processor frequency, even when the processor itself is running at a lower frequency". The 970FX has 1/8th instead of 1/16th. "Since the mesh clock frequency can be lowered to 1/64th of the full-speed, the time base/decrementer may be increased/decreased by more than one at a time." The decrementer tests for wrap, not for reaching zero. > HID0 bit 19 = 1: update at rising edge of TBEN > (must not exceed 1/16th of the core processor's > maximum frequency). > > So the 33,333,333 Hz TB&DEC update rate vs. 2500 MHz would mean > that rising edges of TBEN are in use (HID0 bit 19 = 1) in the > PowerMac G5 context in question. > > I've no information about how closely matched the TBEN signals are > at the 2 sockets, beyond the nominal frequency. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7B096BB4-7DBA-4B95-AC1E-DDD9A7DF3B22>