Date: Sun, 18 Jul 2010 16:05:08 +0200 From: Marius Strobl <marius@alchemy.franken.de> To: Alexander Motin <mav@FreeBSD.org> Cc: freebsd-sparc64@FreeBSD.org Subject: Re: [RFC] Event timers on sparc64/sun4v Message-ID: <20100718140508.GX4706@alchemy.franken.de> In-Reply-To: <4C41D99B.10202@FreeBSD.org> References: <4C404018.6040405@FreeBSD.org> <20100716213503.GT4706@alchemy.franken.de> <4C40D6F5.6070208@FreeBSD.org> <20100717152459.GU4706@alchemy.franken.de> <4C41D99B.10202@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Jul 17, 2010 at 07:26:03PM +0300, Alexander Motin wrote: > Marius Strobl wrote: > > On Sat, Jul 17, 2010 at 01:02:29AM +0300, Alexander Motin wrote: > >> Marius Strobl wrote: > > If it is granularity, then it is caused not by the base frequency. Here > is typical x86 timer frequencies: > kern.timecounter.tc.i8254.frequency: 1193182 > kern.timecounter.tc.ACPI-fast.frequency: 3579545 > kern.timecounter.tc.HPET.frequency: 14318180 > kern.timecounter.tc.TSC.frequency: 2000085680 > , but TSC is not used on SMP. Others have frequencies not higher then > stick and still working fine. IMHO while counter is monotonic and it's > frequency is higher then frequency of context switches - it should not > be important. I would looked for some different reason. If you have an idea what else could be causing it I'm all ears :) Part of the problem is that there's so much stuff ticking inside the kernel - hard, prof soft and clocks, time counters, cycle counters and CPU tickers - it's hard to follow which source is used for what and how they are supposed to work and also might interact. > > > Using the stick counter on machines consisting of CPUs running at > > different speeds (well, actually all the combinations of using > > stick/tick for hardclock, timecounter, CPU ticker and cycle > > counter I tried as they didn't appear totally wrong) additionally > > has the problem of processes getting killed as they are diagnosed > > to have exceeded their maximum CPU limit, although with the in-tree > > code only the timecounter provided by the host-PCI-bridge should > > be used for this calculation as far as the MD initialization is > > concerned when the stick counter is used to drive hardclock. > > On my SB100 I've seen only tick timecounter registered. If there is some > other timecounter hardware in a system, why it is not registered? It > would be much easier to experiment, having more trusted spare parts. The timecounter I was talking about is implemented by (ab)using one of their performance counters of Schizo host-PCI-bridges in bus cycle counting mode. The host-SBus-bridges and the Psycho host-PCI-bridges also have a timecounter but the Hummingbird host-PCI-bridges found in Blade 100 don't have such counters. > > >>> Thus the more desireable variant for these machines > >>> probably is to provide the tick counter of the BSP as the only > >>> non-per-CPU timer and forward it to the APs via IPIs. > >> It would be possible if timer was programmable from any CPU. But as I > >> understand - it require thread to be binded, which handled by > >> infrastructure only for per-CPU timer. > > > > Wouldn't it be sufficient to bind curthread to the BSP in > > tick_et_start() in that case? For one-shot mode this probably > > is to much overhead (assuming a tickless kernel) but for > > periodic mode IMO this approach should be sound. > > tick_et_start() is called under spin lock and sometimes critical > section. You can't call CPU binding there. For per-CPU timers > reconfiguration there is special logic implemented in MI code using IPIs. Too bad. > > By the way I have some doubts about tick_get_timecount_mp() correctness. > It tries to bind thread to BSP, but what if it is called inside > interrupt handler, or under lock, or some else. I have doubt binding > will work in that case. I've no idea whether sched_bind() works under locks etc as there's no man page describing it, however as it requires curthread to be locked and thread_lock() itself uses a spin lock and locking(9) basically says that acquiring a spinlock with any other lock held is okay I assume that the whole thing is fine with any lock held. Also if there were such restrictions I'd expect there some KASSERTs etc to be in place in the functions invovled preventing incorrect use but tick_get_timecount_mp() doesn't trigger such. Apart from that I'm not really happy about that construct myself but I don't see an alternative to always bind to the same CPU when reading the tick counter in order to get reliable results and in US-IIIi-based machines there just isn't another piece of hardware besides the per-CPU stick and tick counters that could be used as a timecounter available. > > >>> This also > >>> leaves the stick counter of all >= US-III machines generally > >>> available for driving statclock, which likely is also desirable. > >> It would be nice, but I don't know how separate their interrupts. > > > > I think this should be possible in the soft interrupt dispatch. > > However, meanwhile it came to my mind that there was a problem > > with using the stick counter on US-IIIi machines (which also > > only can consist of CPUs running at the same frequency though). > > Is this hardware working at all? May be there is something wrong by > definition or it is misused? I haven't tried to verify whether the frequency actually is correct but it in US-III-based machines it's at least incrementing and generating interrupts and OpenSolaris also is using it (and also just getting its frequency via the OFW device tree) so it should be basically fine. Marius
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100718140508.GX4706>