From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 18 16:12:02 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 902C49C6; Thu, 18 Apr 2013 16:12:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 046FB9C4; Thu, 18 Apr 2013 16:12:01 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.6/8.14.6) with ESMTP id r3IGBv2f077829; Thu, 18 Apr 2013 19:11:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.2 kib.kiev.ua r3IGBv2f077829 Received: (from kostik@localhost) by tom.home (8.14.6/8.14.6/Submit) id r3IGBvjZ077828; Thu, 18 Apr 2013 19:11:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 18 Apr 2013 19:11:57 +0300 From: Konstantin Belousov To: Alexander Motin Subject: Re: Synchronizing TSC Message-ID: <20130418161157.GI85148@kib.kiev.ua> References: <516DCAF7.20400@FreeBSD.org> <516E4537.7050205@FreeBSD.org> <20130417085052.GZ2930@kib.kiev.ua> <51700036.3000306@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="54ZiyWcDhi/7bWb8" Content-Disposition: inline In-Reply-To: <51700036.3000306@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: "freebsd-hackers@freebsd.org" , Jim Harris X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Apr 2013 16:12:02 -0000 --54ZiyWcDhi/7bWb8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 18, 2013 at 05:16:22PM +0300, Alexander Motin wrote: > On 17.04.2013 11:50, Konstantin Belousov wrote: > > On Wed, Apr 17, 2013 at 09:46:15AM +0300, Alexander Motin wrote: > >> On 17.04.2013 03:25, Jim Harris wrote: > >>> > >>> On Tue, Apr 16, 2013 at 3:04 PM, Alexander Motin >>> > wrote: > >>> > >>> Hi. > >>> > >>> Recently I've got 6-core/12-thread system on Sandy Bridge-E Core > >>> i7-3930K CPU and was unpleasantly surprised to see that TSCs are= not > >>> synchronized there. While all 11 APs were synchronized, BSP was = far > >>> behind them. Since it is single-socket system, I don't know any = good > >>> reason for such behavior except some BIOS bug. But I've recalled > >>> that somewhere was some discussions about possible TSC > >>> synchronization. I've implemented patch below that allows to adj= ust > >>> TSC values of BSPs to AP's one on boot using CPU MSRs, hoping th= at > >>> they should not diverge after that: > >>> http://people.freebsd.org/~__mav/tsc_adj2.patch > >>> > >>> > >>> I don't know very much about all different TSC hardware to predi= ct > >>> when it is safe to enable the functionality, but at least on my > >>> system being enabled via loader tunable it seems working well. > >>> > >>> Comments? > >>> > >>> > >>> You may be remembering this thread on r238755 last year: > >>> > >>> http://lists.freebsd.org/pipermail/svn-src-head/2012-July/038992.html > >>> > >>> This was a bug fix in the TSC synchronization test code though, not > >>> anything for trying to adjust out-of-sync TSCs. > >> > >> I remember that thread, but I think I've seen somebody told somewhere > >> that it could be interesting to implement some MI mechanism. Never min= d. > >> > >>> The Intel SDM (volume 3, section 17.13 of March 2013 revision) says > >>> earlier models can only write to lower 32 bits of > >>> IA32_TIME_STAMP_COUNTER, but these models also should not have invari= ant > >>> TSC so they would never even get to your new routine. So your patch > >>> seems OK for Intel CPUs, at least as a tunable that is disabled by de= fault. > >> > >> Thanks. > >> > >>> My only concern would be why TSC on the BSP started out-of-sync on yo= ur > >>> system. Theoretically, BIOS could adjust TSCs in SMM to try to hide = SMI > >>> code execution from the OS, which could then make them out-of-sync > >>> again. Not sure if that's what's happening here, but might be worth a > >>> test putting the TSC test code on a periodic timer to see if they ever > >>> get out of sync again. > >> > >> I did one more interesting observation: on every reboot drift between > >> BSP and APs is growing proportionally to the previous system power-on > >> time. On first boot it is -3878361036 (just above one second), after > >> reboot some minutes later it is -1123454492776 (about 6 minutes), after > >> another reboot it is -1853033521804 (about 10 minutes). > >> > >> Unless my adjustment code would be active, I would guess that AP's TSC > >> is running linearly while BSP's for some reason reset to zero on every > >> reboot. But since I am synchronizing them on each boot, the only > >> possibility for it I see is that there is some other timer(s) / > >> counter(s) not affected by MSR writes that ticks linearly and reloading > >> AP's TSC, but for some reason not reloading BSP's. > > > > For me it sounds as the BIOS bug, indeed. Could you verify the content > > of IA32_TSC_ADJUST on all cores (I believe it is present on E5) ? > > Also, using TSC_ADJUST to correct the skew seems to be preferrable, > > according to the Intel docs. >=20 > IA32_TSC_ADJUST register seems not present there. At least cpucontrol=20 > doesn't want to read it. In Intel docs I also see it mentioned only in=20 > context of future Haswell generation. And I don't see "Standard Extended= =20 > Features" line in dmesg. >=20 > > Why do you use cpuid in the assembly sequence ? As I understand, you > > ensure that there is a serialization point, but why do you need it ? >=20 > The idea was to minimize time distance between following MSR read and=20 > write. But may be it is not needed, I am not exactly sure about that magi= c. Well, I do not believe that such trick is useful. Patch with removed cpuid and with resync disabled by default (as it is now, AFAIR) would be IMO fine for the commit. --54ZiyWcDhi/7bWb8 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQIcBAEBAgAGBQJRcBtNAAoJEJDCuSvBvK1B1uwP/jTHbObpjYzDIS2tFjFpTZNA PYJLx9FEaLVAqc0kz5XZqXNM7k7XpobjKQuzZmVaIRzdjQaXcuCkfgWf5ABS4y9d E6Hqp/Qdp2YKtJBLwMpvtgj9IXo29fYeBj4495jFnWZSH2tYcaNZSEioBAY6Bx1w Ac/h+E9k47Bl5iVMBaE+WA239D1G5M9bDHyEMpcrIijJrUBFjMYouR+vyTYC+gXj UtsIviC0ZrRdHS0woqWpGAMgLEJJJyW53PIo6nqZb3B+Tr4VAKu5FiDrZrnT+fdf nPZDnZUHC1ggYnO/Y4ohYQXthu1u9cmq3ip3idsdVCm7eQLofDbONrHOGDJUDvQP zXjFui2D67GIUNBlLyxocPbjnkLuwjo1WVml3cKbgABmTqhV+r+wG8W1pj+VFeX9 CH0oqd5PzyecptN6SnyL1LzpKjFagZ0ZKE/L6SPhjZBh5HCtVABzOlPVIMLtMopd lklwT44jWnkdLSH39fhyb34iI29rdtgHdgo0nyi7ClWDJgCXAcVMpE37PuCWPkvd j7yUDHg2EcWdlQTD6lyZAsA8EFHTUk/NHCpVtXa2Urzx912kYx+j6+ZM3JcQ4xEp UGf833xUaTG8P516eS0+gOI9PRb7EhV9GmE7juxa/tzhht8Jym7qLX1EWXtbBY10 z/zn1VvmksC3/nsZeFH4 =Y3OR -----END PGP SIGNATURE----- --54ZiyWcDhi/7bWb8--