From owner-freebsd-hackers@freebsd.org Tue Oct 27 09:05:46 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4988A8246 for ; Tue, 27 Oct 2015 09:05:46 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 58D9419F1; Tue, 27 Oct 2015 09:05:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id LAA16928; Tue, 27 Oct 2015 11:05:43 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Zr0CM-000GsT-U7; Tue, 27 Oct 2015 11:05:42 +0200 Subject: Re: instability of timekeeping To: Konstantin Belousov References: <56261398.60102@FreeBSD.org> <56261FE6.90302@FreeBSD.org> <56274FFC.2000608@FreeBSD.org> <20151021184850.GX2257@kib.kiev.ua> Cc: freebsd-hackers , Poul-Henning Kamp , Jung-uk Kim From: Andriy Gapon X-Enigmail-Draft-Status: N1110 Message-ID: <562F3E2F.2010100@FreeBSD.org> Date: Tue, 27 Oct 2015 11:04:47 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <20151021184850.GX2257@kib.kiev.ua> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Oct 2015 09:05:46 -0000 On 21/10/2015 21:48, Konstantin Belousov wrote: > Am I right that the tsc synchronization test passes on your machine ? Yes. > If yes, you probably cannot read 'slightly behind' timecounter after IPI > on other core. Right. But please note that my hypothesis does not require that the following is true: CPU_A reads TSC value X, CPU_A sends an IPI to CPU_B, CPU_B gets the IPI, CPU_B reads TSC value Y, Y < X. In my configuration there is one master CPU and 3 slaves. Also, it could be possible that the IPI is delayed for so long that there is an interaction between handling of 2 consecutive hardclock IPIs. > Might be, try to change CPUID instruction in the test to > MFENCE and see if the test still able to pass. > Does the symptom disappear if you switch the eventtimer to LAPIC ? And now another observation. I have C1E option enabled in BIOS. It means that if all cores enter C1 state, then the whole processor is magically placed into a deep C-state (C3, I think). LAPIC timer on this CPU model does not run in the deep C-state. So, I had to disable C1E option to test the LAPIC timer in a useful way. But before actually testing it I first tried to reproduce the problem. As you might have already guessed the problem is gone with that option disabled. Scratching my head to understand the implications of this observation. > What happens if you turn off usermode gettimeofday() > by setting kern.timercounter.fast_gettime to 0 ? -- Andriy Gapon