From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 17 06:53:08 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B7A9CFCB for ; Wed, 17 Apr 2013 06:53:08 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ea0-x230.google.com (mail-ea0-x230.google.com [IPv6:2a00:1450:4013:c01::230]) by mx1.freebsd.org (Postfix) with ESMTP id 54969EF3 for ; Wed, 17 Apr 2013 06:53:08 +0000 (UTC) Received: by mail-ea0-f176.google.com with SMTP id h10so548238eaj.7 for ; Tue, 16 Apr 2013 23:53:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=q5dKrG2KjVrvtji1z59W3IqMT5iUHs4/9/y2C0ViPC4=; b=tT4gKGxGhPiEm0GxOfsBiaotRyx+g5918UmqFoiV/rXtaaQBSDUx8HxVT+0wSRUvFa LAcVb9WCYpjErsV/2Wl2F+0KeN+M/WcZBMdry/FZtqkbyN93OtHGdWBym4sVRT0rh6GM zvUGsqn3dv8ctLRFnNXRMR8sZS0mwTHNDUWKCjvO6I+YL1n+g4X2q8S383KycOrUe7+2 zdXmpTu9WgrMA6bD7XRJOKA6BRDlT/GmwC9g4U+Mpf2NYLqGNnfVq/PDwRrUEZzDMKVl scTT5LlqLGnw7Xs2IJR4yzuRp4Xn55+Q314m6FlxjUmrtRGkzlFeqkId0im4lUNZmeDj g61g== X-Received: by 10.15.99.201 with SMTP id bl49mr14716983eeb.43.1366181178391; Tue, 16 Apr 2013 23:46:18 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPS id b5sm6703242eew.16.2013.04.16.23.46.16 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 16 Apr 2013 23:46:17 -0700 (PDT) Sender: Alexander Motin Message-ID: <516E4537.7050205@FreeBSD.org> Date: Wed, 17 Apr 2013 09:46:15 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130413 Thunderbird/17.0.5 MIME-Version: 1.0 To: Jim Harris Subject: Re: Synchronizing TSC References: <516DCAF7.20400@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2013 06:53:08 -0000 On 17.04.2013 03:25, Jim Harris wrote: > > On Tue, Apr 16, 2013 at 3:04 PM, Alexander Motin > wrote: > > Hi. > > Recently I've got 6-core/12-thread system on Sandy Bridge-E Core > i7-3930K CPU and was unpleasantly surprised to see that TSCs are not > synchronized there. While all 11 APs were synchronized, BSP was far > behind them. Since it is single-socket system, I don't know any good > reason for such behavior except some BIOS bug. But I've recalled > that somewhere was some discussions about possible TSC > synchronization. I've implemented patch below that allows to adjust > TSC values of BSPs to AP's one on boot using CPU MSRs, hoping that > they should not diverge after that: > http://people.freebsd.org/~__mav/tsc_adj2.patch > > > I don't know very much about all different TSC hardware to predict > when it is safe to enable the functionality, but at least on my > system being enabled via loader tunable it seems working well. > > Comments? > > > You may be remembering this thread on r238755 last year: > > http://lists.freebsd.org/pipermail/svn-src-head/2012-July/038992.html > > This was a bug fix in the TSC synchronization test code though, not > anything for trying to adjust out-of-sync TSCs. I remember that thread, but I think I've seen somebody told somewhere that it could be interesting to implement some MI mechanism. Never mind. > The Intel SDM (volume 3, section 17.13 of March 2013 revision) says > earlier models can only write to lower 32 bits of > IA32_TIME_STAMP_COUNTER, but these models also should not have invariant > TSC so they would never even get to your new routine. So your patch > seems OK for Intel CPUs, at least as a tunable that is disabled by default. Thanks. > My only concern would be why TSC on the BSP started out-of-sync on your > system. Theoretically, BIOS could adjust TSCs in SMM to try to hide SMI > code execution from the OS, which could then make them out-of-sync > again. Not sure if that's what's happening here, but might be worth a > test putting the TSC test code on a periodic timer to see if they ever > get out of sync again. I did one more interesting observation: on every reboot drift between BSP and APs is growing proportionally to the previous system power-on time. On first boot it is -3878361036 (just above one second), after reboot some minutes later it is -1123454492776 (about 6 minutes), after another reboot it is -1853033521804 (about 10 minutes). Unless my adjustment code would be active, I would guess that AP's TSC is running linearly while BSP's for some reason reset to zero on every reboot. But since I am synchronizing them on each boot, the only possibility for it I see is that there is some other timer(s) / counter(s) not affected by MSR writes that ticks linearly and reloading AP's TSC, but for some reason not reloading BSP's. -- Alexander Motin