From owner-freebsd-emulation@FreeBSD.ORG  Wed Nov 28 16:16:17 2012
Return-Path: <owner-freebsd-emulation@FreeBSD.ORG>
Delivered-To: freebsd-emulation@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
 by hub.freebsd.org (Postfix) with ESMTP id E7EA3980;
 Wed, 28 Nov 2012 16:16:17 +0000 (UTC) (envelope-from avg@FreeBSD.org)
Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140])
 by mx1.freebsd.org (Postfix) with ESMTP id 77C858FC15;
 Wed, 28 Nov 2012 16:16:16 +0000 (UTC)
Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua
 [212.40.38.101])
 by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id SAA28276;
 Wed, 28 Nov 2012 18:16:14 +0200 (EET) (envelope-from avg@FreeBSD.org)
Message-ID: <50B638CD.4020208@FreeBSD.org>
Date: Wed, 28 Nov 2012 18:16:13 +0200
From: Andriy Gapon <avg@FreeBSD.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:17.0) Gecko/17.0 Thunderbird/17.0
MIME-Version: 1.0
To: Alex Chistyakov <alexclear@gmail.com>
Subject: Re: VirtualBox 4.2.4 on FreeBSD 9.1-PRERELEASE problem: VMs behave
 very different when pinned to different cores
References: <CA+kq2xvh3j5CM7UzRVfXCeLhHwpTY+_M7dCJx0c27NtV8EVJwg@mail.gmail.com>
 <CAE-m3X1UPsy+wbqm_02JpXMr-UO3m7N6z_ZwY2HNo4GL0YUi1w@mail.gmail.com>
 <CA+kq2xva61m_bHdzBZM2TYL5z7XiohvkxsYWtOyoBwQkpyvp0A@mail.gmail.com>
 <50AFAD05.1050604@FreeBSD.org>
 <CA+kq2xv+U4ZnfK=1js4PRaNpTNdW-y-G50GV4+MVP0LugBf1pQ@mail.gmail.com>
 <50B25C17.20208@FreeBSD.org>
 <CA+kq2xvjDa1BeuzPUuH99bgriEA-GJH36AZGiqScKSo4QZmHDg@mail.gmail.com>
 <50B62CB3.70101@FreeBSD.org>
 <CA+kq2xvGhWBVEPomBnRkfa0em8woi6qBzSFX9hiUSNH-+d8-3g@mail.gmail.com>
In-Reply-To: <CA+kq2xvGhWBVEPomBnRkfa0em8woi6qBzSFX9hiUSNH-+d8-3g@mail.gmail.com>
X-Enigmail-Version: 1.4.6
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Cc: "freebsd-emulation@freebsd.org" <freebsd-emulation@FreeBSD.org>,
 Alexander Motin <mav@FreeBSD.org>
X-BeenThere: freebsd-emulation@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Development of Emulators of other operating systems
 <freebsd-emulation.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-emulation>, 
 <mailto:freebsd-emulation-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-emulation>
List-Post: <mailto:freebsd-emulation@freebsd.org>
List-Help: <mailto:freebsd-emulation-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-emulation>, 
 <mailto:freebsd-emulation-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 28 Nov 2012 16:16:18 -0000

on 28/11/2012 17:34 Alex Chistyakov said the following:
> On Wed, Nov 28, 2012 at 7:24 PM, Andriy Gapon <avg@freebsd.org> wrote:
>> on 26/11/2012 09:10 Alex Chistyakov said the following:
>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3200.18-MHz K8-class CPU)
>>>   Origin = "GenuineIntel"  Id = 0x206d7  Family = 0x6  Model = 0x2d
>>> Stepping = 7
>>> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>>> Features2=0x1fbee3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
>>>   AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
>>>   AMD Features2=0x1<LAHF>
>>>   TSC: P-state invariant, performance statistics
>>>
>>
>> Is this a multi-socket system?
> 
> No, this is a single-socket desktop grade baseboard, Intel DX79TO.
> 

I'd say that there must be some kind of hardware or firmware (BIOS) problem to
create such a discrepancy in TSC readings.

>> It would be very strange that a modern CPU like this would have such a skew
>> between TSC on different cores.
>>
>> On my Core i5-3570 I see that the _observed_ skew is no more than 100 ticks (after
>> many days of uptime).  It could be zero, in fact, given the inaccuracy of
>> inter-core measurements.

You could try the following (totally untested) patch as an experiment.  Just to
see if indeed there is some connection between the TSC issue and the main issue.

--- a/sys/x86/x86/tsc.c
+++ b/sys/x86/x86/tsc.c
@@ -398,6 +398,13 @@ comp_smp_tsc(void *arg)
 		}
 }

+static void
+reset_tsc(void *arg)
+{
+
+	wrmsr(MSR_TSC, (uintptr_t)arg);
+}
+
 static int
 test_smp_tsc(void)
 {
@@ -406,6 +413,9 @@ test_smp_tsc(void)

 	if (!smp_tsc && !tsc_is_invariant)
 		return (-100);
+
+	smp_rendezvous(NULL, reset_tsc, NULL, (void*)0);
+
 	size = (mp_maxid + 1) * 3;
 	data = malloc(sizeof(*data) * size * N, M_TEMP, M_WAITOK);
 	for (i = 0, tsc = data; i < N; i++, tsc += size)

-- 
Andriy Gapon