From owner-svn-src-all@freebsd.org Thu Mar 24 16:25:00 2016 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 78C35ADCFDE; Thu, 24 Mar 2016 16:25:00 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EE84312C1; Thu, 24 Mar 2016 16:24:59 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u2OGOmmG065542 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Thu, 24 Mar 2016 18:24:48 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u2OGOmmG065542 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u2OGOlDq065541; Thu, 24 Mar 2016 18:24:47 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 24 Mar 2016 18:24:47 +0200 From: Konstantin Belousov To: Bruce Evans Cc: John Baldwin , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org, "'rstone@freebsd.org'" Subject: Re: svn commit: r297039 - head/sys/x86/x86 Message-ID: <20160324162447.GD1741@kib.kiev.ua> References: <201603181948.u2IJmndg063765@repo.freebsd.org> <1866602.Bp7VFd5f42@ralph.baldwin.cx> <20160323075842.GX1741@kib.kiev.ua> <2922763.uITxoCVqGR@ralph.baldwin.cx> <20160324090917.GC1741@kib.kiev.ua> <20160325010649.H898@besplex.bde.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160325010649.H898@besplex.bde.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Mar 2016 16:25:00 -0000 On Fri, Mar 25, 2016 at 01:57:32AM +1100, Bruce Evans wrote: > > In fact, if we can use TSC with the only requirement of being monotonic, > > I do not see why do we need TSC at all. We can return to pre-r278325 > > loop, but calibrate the number of loop iterations for known delay in > > 1us, once on boot. Do you agree with this ? > > As a comment in the previous version says, the old method is highly > bogus since it is sensitive to CPU clock speed. We do not care if we time-out in the 5usec or 50usec. We can even wait for 1 second. The only thing that must be avoided is infinite wait to prevent a situation when CPU spins with disabled interrupts infinitely. > > My systems allow speed variations of about 4000:800 = 5:1 for one CPU and > about 50:1 for different CPUs. So the old method gave a variation of up > to 50:1. This can be reduced to only 5:1 using the boot-time calibration. What do you mean by 'for different CPUs' ? I understand that modern ESS can give us CPU frequency between 800-4200MHz, which is what you mean by 'for one CPU'. We definitely do not care if 5usec timeout becomes 25usecs, since we practically never time-out there at all. As I understand the original report, the LAPIC becomes ready for next IPI command much faster than 1usec, but not that fast to report readiness on the next register read. The timeout is more practical for APs startup, where microcode must initialize and test core, which does take time, and sometimes other issues do prevent APs from starting. But this happens during early boot, when ESS and trottling cannot happen, so initial calibration is more or less accurate. > > Old versions of DELAY() had similar problems. E.g., in FreeBSD-7: > > X if (tsc_freq != 0 && !tsc_is_broken) { > X uint64_t start, end, now; > X > X sched_pin(); > X start = rdtsc(); > X end = start + (tsc_freq * n) / 1000000; > X do { > X cpu_spinwait(); > X now = rdtsc(); > X } while (now < end || (now > start && end < start)); > X sched_unpin(); > X return; > X } > > This only works if the TSC is invariant. tsc_freq_changed() updates > tsc_freq, but it is too hard to do this atomically, so the above is > broken when it is called while the frequency is being updated (and > this can be arranged by single stepping through the update code). > The bug might break at least the keyboard i/o used to do the single > stepping, depending on whether the DELAY()s in it are really needed > and on how much these DELAYS()s are lengthened or shortened. > > This loop doesn't need to use the TSC, but can use a simple calibrated > loop since the delay only needs to be right to within a few usec of > a factor of 2 or so, or any longer time (so that it doesn't matter if > the thread is preempted many quanta). The calibration just needs to > be updated whenever the CPU execution rate changes. I do not see why would we need to re-calibrate for ipi_wait(), the possible inaccuracy is not important. > > Newer CPUs give complications like the CPU execution rate being independent > of the (TSC) CPU frequency. An invariant TSC gives an invariant CPU > frequency, but the execution rate may change in lots of ways for lots > of reasons. > > The cputicker has similar problems with non-invariant CPUs. It > recalibrates, but there are races and other bugs in the recalibration. > These bugs are small compared with counting ticks at old rate(s) as > being at the current rate. Yes, but I do not believe any of the listed problems are relevant for the wait for LAPIC readiness.