From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 18 14:16:27 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BEF66F2E for ; Thu, 18 Apr 2013 14:16:27 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-bk0-x232.google.com (mail-bk0-x232.google.com [IPv6:2a00:1450:4008:c01::232]) by mx1.freebsd.org (Postfix) with ESMTP id 553981A5 for ; Thu, 18 Apr 2013 14:16:27 +0000 (UTC) Received: by mail-bk0-f50.google.com with SMTP id jg1so1295976bkc.37 for ; Thu, 18 Apr 2013 07:16:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=8FtMKd3N1GKGimTPv+8BgVP68hc6RKn+yvYKHLLRHKQ=; b=QLuhkN28lgtvqgCruxyPxKqmM6bGGu6gem4aHaXFvPv0a/9UxzPcjz8/aGQOUsKVx8 4cxuuWqx+/IfC7s9+fq6e4r34LVpbXTszHaZGe4aD/5uDBaLCMNFa4wGnkjCDbLkBJ9s SG4MzxEfrn3A0CUaq40is6Fcb4PP0xep/2qX49FmgRrmKNbnqZIlJ8ZbbkHYMvIgtr3M nYOi84VXJi0AuafxYvnxPSlV39t22ZWBgqEGyA7iC2geXC6FU+db3wUJ4FcfUPYnEQgj NAAZQez7baTkzVzFeeDbnTYdZXuGGPpSpWLhyUrcfoOrIR27STHXnHXW0lae6dC5cYKL qPKQ== X-Received: by 10.204.170.202 with SMTP id e10mr4052928bkz.73.1366294586371; Thu, 18 Apr 2013 07:16:26 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPS id jm15sm3166060bkb.13.2013.04.18.07.16.23 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 18 Apr 2013 07:16:25 -0700 (PDT) Sender: Alexander Motin Message-ID: <51700036.3000306@FreeBSD.org> Date: Thu, 18 Apr 2013 17:16:22 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130413 Thunderbird/17.0.5 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Synchronizing TSC References: <516DCAF7.20400@FreeBSD.org> <516E4537.7050205@FreeBSD.org> <20130417085052.GZ2930@kib.kiev.ua> In-Reply-To: <20130417085052.GZ2930@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , Jim Harris X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Apr 2013 14:16:27 -0000 On 17.04.2013 11:50, Konstantin Belousov wrote: > On Wed, Apr 17, 2013 at 09:46:15AM +0300, Alexander Motin wrote: >> On 17.04.2013 03:25, Jim Harris wrote: >>> >>> On Tue, Apr 16, 2013 at 3:04 PM, Alexander Motin >> > wrote: >>> >>> Hi. >>> >>> Recently I've got 6-core/12-thread system on Sandy Bridge-E Core >>> i7-3930K CPU and was unpleasantly surprised to see that TSCs are not >>> synchronized there. While all 11 APs were synchronized, BSP was far >>> behind them. Since it is single-socket system, I don't know any good >>> reason for such behavior except some BIOS bug. But I've recalled >>> that somewhere was some discussions about possible TSC >>> synchronization. I've implemented patch below that allows to adjust >>> TSC values of BSPs to AP's one on boot using CPU MSRs, hoping that >>> they should not diverge after that: >>> http://people.freebsd.org/~__mav/tsc_adj2.patch >>> >>> >>> I don't know very much about all different TSC hardware to predict >>> when it is safe to enable the functionality, but at least on my >>> system being enabled via loader tunable it seems working well. >>> >>> Comments? >>> >>> >>> You may be remembering this thread on r238755 last year: >>> >>> http://lists.freebsd.org/pipermail/svn-src-head/2012-July/038992.html >>> >>> This was a bug fix in the TSC synchronization test code though, not >>> anything for trying to adjust out-of-sync TSCs. >> >> I remember that thread, but I think I've seen somebody told somewhere >> that it could be interesting to implement some MI mechanism. Never mind. >> >>> The Intel SDM (volume 3, section 17.13 of March 2013 revision) says >>> earlier models can only write to lower 32 bits of >>> IA32_TIME_STAMP_COUNTER, but these models also should not have invariant >>> TSC so they would never even get to your new routine. So your patch >>> seems OK for Intel CPUs, at least as a tunable that is disabled by default. >> >> Thanks. >> >>> My only concern would be why TSC on the BSP started out-of-sync on your >>> system. Theoretically, BIOS could adjust TSCs in SMM to try to hide SMI >>> code execution from the OS, which could then make them out-of-sync >>> again. Not sure if that's what's happening here, but might be worth a >>> test putting the TSC test code on a periodic timer to see if they ever >>> get out of sync again. >> >> I did one more interesting observation: on every reboot drift between >> BSP and APs is growing proportionally to the previous system power-on >> time. On first boot it is -3878361036 (just above one second), after >> reboot some minutes later it is -1123454492776 (about 6 minutes), after >> another reboot it is -1853033521804 (about 10 minutes). >> >> Unless my adjustment code would be active, I would guess that AP's TSC >> is running linearly while BSP's for some reason reset to zero on every >> reboot. But since I am synchronizing them on each boot, the only >> possibility for it I see is that there is some other timer(s) / >> counter(s) not affected by MSR writes that ticks linearly and reloading >> AP's TSC, but for some reason not reloading BSP's. > > For me it sounds as the BIOS bug, indeed. Could you verify the content > of IA32_TSC_ADJUST on all cores (I believe it is present on E5) ? > Also, using TSC_ADJUST to correct the skew seems to be preferrable, > according to the Intel docs. IA32_TSC_ADJUST register seems not present there. At least cpucontrol doesn't want to read it. In Intel docs I also see it mentioned only in context of future Haswell generation. And I don't see "Standard Extended Features" line in dmesg. > Why do you use cpuid in the assembly sequence ? As I understand, you > ensure that there is a serialization point, but why do you need it ? The idea was to minimize time distance between following MSR read and write. But may be it is not needed, I am not exactly sure about that magic. -- Alexander Motin