From owner-freebsd-hackers@freebsd.org Fri Apr 5 11:35:17 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A17221573ACD; Fri, 5 Apr 2019 11:35:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C87618CFE7; Fri, 5 Apr 2019 11:35:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x35BZ6jL002604 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 5 Apr 2019 14:35:09 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x35BZ6jL002604 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x35BZ4Vo002581; Fri, 5 Apr 2019 14:35:04 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 5 Apr 2019 14:35:04 +0300 From: Konstantin Belousov To: Bruce Evans Cc: Mark Millard , freebsd-hackers Hackers , FreeBSD PowerPC ML Subject: Re: powerpc64 head -r344018 stuck sleeping problems: th->th_scale * tc_delta(th) overflows unsigned 64 bits sometimes [patched failed] Message-ID: <20190405113504.GA1923@kib.kiev.ua> References: <20190307222220.GK2492@kib.kiev.ua> <20190309144844.K1166@besplex.bde.org> <20190324110138.GR1923@kib.kiev.ua> <20190403070045.GW1923@kib.kiev.ua> <20190404011802.E2390@besplex.bde.org> <20190405150236.A959@besplex.bde.org> <20190405201055.I2396@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190405201055.I2396@besplex.bde.org> User-Agent: Mutt/1.11.4 (2019-03-13) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Apr 2019 11:35:18 -0000 On Fri, Apr 05, 2019 at 09:13:50PM +1100, Bruce Evans wrote: > On Fri, 5 Apr 2019, Mark Millard wrote: > > > On 2019-Apr-4, at 21:38, Bruce Evans wrote: > > > >> On Thu, 4 Apr 2019, Mark Millard wrote: > >>> ... > >>> Unfortunately, all the multi-socket contexts that I sometimes have > >>> access to are old PowerMacs. And, currently, the only such context > >>> is the G5 with 2 sockets, 2 cores per socket (powerpc64). So I've > >>> not been able to set up other types of examples to see if problems > >>> repeat. > >>> > >>> I do not have access to a single-socket powerpc64 for contrast in > >>> that direction. > >> > >> Testing 1 socket is time-consuming enough. Do these old systems > >> use the equivalent of an x86 TSC for the timecounter? With multiple > >> sockets, it isn't clear how even a hardware timer independent of the > >> CPUs can be distributed so as to appear to be monotonic on all cors. > > > > "The DEC frequency is based on the same implementation-dependent > > frequency that drives the time base." The frequency may well be > > fixed by the PowerMac G5 model implementation but is not fixed > > by the powerpc64 architecture. > > > > The Time Base Register (TBR) in a powerpc64 core (cpu in FreeBSD terms) > > increments at 33,333,333 Hz (nominal) and is 64 bits wide. Its value > > can be set (mttb instruction) and the boot sequence in FreeBSD does > > attempt to adjust as the FreeBSD CPU is brought-up/started. mftb is > > used to read the 64-bit value. FreeBSD masks it down to 32-bits to > > contribute to the time-counter. > > > > (Is that description sufficient for what you were after? I've never > > seen documentation of how the 33,333,333 MHz is produced.) > > This seems to be equivalent to an x86 TSC. It is equivalent from the interface PoV. I saw some references that Powers do have per-core PLLs, the best I can find now is https://stackoverflow.com/questions/43844438/are-multiple-clock-domains-common-in-modern-processors For Intel there is no much information, the best guess is that TSC is in uncore and effectively shared by all cores. See also https://software.intel.com/en-us/articles/best-timing-function-for-measuring-ipp-api-timing/ > > x86 P-state-invariant TSCs apparently use a 100 MHz clock which corresponds > even more closely to this, except for historical reasons this clock is > scaled and interpolated to a clock resembling the CPU cycle count at a > nominal frequency. For Intel designs, there is indeed a single-source 100MHz signal which is distributed over all consumers using clock fan-out buffers like DB1900Z. > > > As FreeBSD supports multi-socket, what are its criteria for a sufficient > > context for it to work with for supporting sbinuptime and the like? Is > > FreeBSD supposed to then make it appear that sbinputime and the like are > > weakly increasing, even as threads migrate between CPUs (cores, > > hw-threads)? > > CLOCK_MONOTONIC has to be monotonic, but it is only clear what this means > within a single thread: sequential clock_gettime() calls must occur in > program order and the results must be consistent with that order. Across > threads, I think it should mean that the results must be consistent with > any order established using any supported ordering methods. > > >... > >>> One oddity is that the eventtimer's decrementer and timecounter > >>> may change (nearly) together: both change at 33,333,333 Hz, as if > >>> they are tied to the same clock (at least on one socket). > >> > >> I think this is from a normal hardware implementation. On all of > >> my x86 systems with a TSC, the TSC frequency is an exact fractional > >> multiple of the i8254, the ACPI timer (if present) and the HPET (if > >> present). Only the RTC has an independent frequency. The fraction is > >> changed by changing the nominal TSC frequency in the BIOS, but is not > >> changed by temperature variations. This must be because most clocks are > >> derived from a common clock using a PLL. I use this to calibrate all > >> clocks (except the RTC) by calibrating only 1. > > > > I'm not aware of the OpenFirmware having any control over the > > TBR-change frequency behavior. I've no evidence about any variability > > based on temperature. > > Temperature changes usually affect the actual frequency but not the > nominal frequency. > > Bruce