From owner-freebsd-hackers@freebsd.org Thu Oct 29 22:31:27 2015 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC015A216CB for ; Thu, 29 Oct 2015 22:31:27 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id F28351294; Thu, 29 Oct 2015 22:31:25 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id AAA04100; Fri, 30 Oct 2015 00:31:18 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Zrvj3-000KuV-Ov; Fri, 30 Oct 2015 00:31:17 +0200 Subject: Re: instability of timekeeping To: Konstantin Belousov , Alexander Motin References: <56261398.60102@FreeBSD.org> <56261FE6.90302@FreeBSD.org> <56274FFC.2000608@FreeBSD.org> <20151021184850.GX2257@kib.kiev.ua> <562F3E2F.2010100@FreeBSD.org> <20151027115810.GU2257@kib.kiev.ua> <562F8109.4050203@FreeBSD.org> <20151027140403.GB2257@kib.kiev.ua> <5630FC3B.2070908@FreeBSD.org> <5631FB66.4000007@FreeBSD.org> Cc: freebsd-hackers , Poul-Henning Kamp , Jung-uk Kim From: Andriy Gapon X-Enigmail-Draft-Status: N1110 Message-ID: <56329E11.1070102@FreeBSD.org> Date: Fri, 30 Oct 2015 00:30:41 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <5631FB66.4000007@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Oct 2015 22:31:27 -0000 On 29/10/2015 12:56, Andriy Gapon wrote: > Okay, more data: > https://people.freebsd.org/~avg/timekeeping-ktr.2.patch > https://people.freebsd.org/~avg/timekeeping.2.ktrdump.txt > > I think that the snippet (amended with some notes of mine) makes it painfully > obvious that the timer interrupt got very delayed when all CPUs entered the idle > state. > I do not see anything that could suggest a FreeBSD bug. > > There is another sad discovery. Turns out that my CPU model provides two ways > of doing C1E magic. The sane one: the north bridge logic in the CPU performs a > read of a configured LVL3 register so that C3 is entered. The insane one: the > CPU NB performs a write of a configured value to a configured SMI register, so > that the SMI is generated and an SMM handler does the job (probably reading from > LVL2 or LVL3). Looking at MSR C001_0055 I see that my BIOS has chosen the > insane approach[*], quite unfortunately. Bugs in the SMM code are not unheard > of, to put it mildly, so that could be an explanation for my problem. > > So, I guess I'll just disable C1E and end this investigation. Just in case anyone is still interested in this, I have found that a newer BIOS version is available for my motherboard and it includes a newer version of AGESA. So, I upgraded the BIOS and the new version does C1E in the sane way: $ cpucontrol -m 0xc0010055 /dev/cpuctl0 MSR 0xc0010055: 0x00000000 0x14c14015 C1eOnCmpHalt: C1E on chip multi-processing halt. - read from 0x4015 Also, after a few hours of testing I do not see any problems with the timekeeping. So, either the problem was indeed in the SMM code in the older BIOS or the older BIOS incorrectly configured the sleep break events. Unfortunately, the newer BIOS has a regression in a different area: it fails to access HDD LBAs greater than INT32_MAX (offsets beyond 1TB on my HDDs with 512-byte native sector sizes). But that's a different story. > [*] > $ cpucontrol -m 0xc0010055 /dev/cpuctl0 > > > MSR 0xc0010055: 0x00000000 0x083400b0 > > SmiOnCmpHalt: SMI on chip multi-processing halt. > - write 0x34 to port 0xb0 > -- Andriy Gapon