From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 17 00:25:10 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6D97EA6E; Wed, 17 Apr 2013 00:25:10 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: from mail-ee0-f52.google.com (mail-ee0-f52.google.com [74.125.83.52]) by mx1.freebsd.org (Postfix) with ESMTP id DC0811FC6; Wed, 17 Apr 2013 00:25:09 +0000 (UTC) Received: by mail-ee0-f52.google.com with SMTP id b57so326984eek.25 for ; Tue, 16 Apr 2013 17:25:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=tklgdmI2e1nLMRrFm1D9XUtlzGHV2QbplP2dodQrXF0=; b=E5G+pN9m/fz9AWrEuma0YMzF0wzA7BqfVA1aIqrf3fbsUdpp38Kok6M+SsAFsS0xbg j3iUXjNA5iYyVib2s65zVYaMwS4P8l4Mm8mjHvC1oJg6iT5emxgQRex+8WbDLWhm9lns fvHjmnHEoxemSgtkurIpshIHwgaH/L8qsDfQeJXzDEDC1U5ofy7kPJ2nP87KwvMLyZEW CxuVkGuWivNzjr6YsJmm1Np/VyPqykXcihJbx7i+XSblh52TGJEdRw5UtQZYDdZg8kc8 9CSQNmLoqM9w/kyJzP5NzZaMqA2fnxxVfwkCK7IlhkxGBSglLTNoZ+b6qcj4gR6tOpO7 P3Nw== MIME-Version: 1.0 X-Received: by 10.14.39.5 with SMTP id c5mr12008234eeb.27.1366158302767; Tue, 16 Apr 2013 17:25:02 -0700 (PDT) Received: by 10.14.111.2 with HTTP; Tue, 16 Apr 2013 17:25:02 -0700 (PDT) In-Reply-To: <516DCAF7.20400@FreeBSD.org> References: <516DCAF7.20400@FreeBSD.org> Date: Tue, 16 Apr 2013 17:25:02 -0700 Message-ID: Subject: Re: Synchronizing TSC From: Jim Harris To: Alexander Motin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Apr 2013 00:25:10 -0000 On Tue, Apr 16, 2013 at 3:04 PM, Alexander Motin wrote: > Hi. > > Recently I've got 6-core/12-thread system on Sandy Bridge-E Core i7-3930K > CPU and was unpleasantly surprised to see that TSCs are not synchronized > there. While all 11 APs were synchronized, BSP was far behind them. Since > it is single-socket system, I don't know any good reason for such behavior > except some BIOS bug. But I've recalled that somewhere was some discussions > about possible TSC synchronization. I've implemented patch below that > allows to adjust TSC values of BSPs to AP's one on boot using CPU MSRs, > hoping that they should not diverge after that: > http://people.freebsd.org/~**mav/tsc_adj2.patch > > I don't know very much about all different TSC hardware to predict when it > is safe to enable the functionality, but at least on my system being > enabled via loader tunable it seems working well. > > Comments? > > You may be remembering this thread on r238755 last year: http://lists.freebsd.org/pipermail/svn-src-head/2012-July/038992.html This was a bug fix in the TSC synchronization test code though, not anything for trying to adjust out-of-sync TSCs. The Intel SDM (volume 3, section 17.13 of March 2013 revision) says earlier models can only write to lower 32 bits of IA32_TIME_STAMP_COUNTER, but these models also should not have invariant TSC so they would never even get to your new routine. So your patch seems OK for Intel CPUs, at least as a tunable that is disabled by default. My only concern would be why TSC on the BSP started out-of-sync on your system. Theoretically, BIOS could adjust TSCs in SMM to try to hide SMI code execution from the OS, which could then make them out-of-sync again. Not sure if that's what's happening here, but might be worth a test putting the TSC test code on a periodic timer to see if they ever get out of sync again. -Jim