From owner-freebsd-emulation@FreeBSD.ORG Mon Nov 26 02:12:07 2012 Return-Path: Delivered-To: freebsd-emulation@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 7F5EA2D8; Mon, 26 Nov 2012 02:12:07 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id EAD128FC08; Mon, 26 Nov 2012 02:12:06 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id l1so2872757vba.13 for ; Sun, 25 Nov 2012 18:12:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=pEsQfYihpLsTzG7I1wo46d4WZTcu4TyhVMcAwzgvu+I=; b=DUZX2DtxUscxK1x/JQdGfkNez74MQuMrSg2pFfWdeP07AKhkd1/UOqEHntKnUvHxUi soJ2csocqW7zJHPX3UJbnd/PBrlYD+ZilF8AKLJ9lZWhSMe6qyWV1JKeewLQu5NLbFpe kPAPIaADnloBgWLAmIMUhQJZ12D5iqVWiL5lxOAuOeahtjCksS6Tn8/xAOEtphMsEmz6 kwlthOZd1F1O2mX7XzFSiW5pfKCQwKWcDSosx8BHng4diP3m65UBPNWSDCcuygu61PHf o7WN23eCmr3XoM70q3qZ87Ayx4n9p72ne07WSn96FQvBSJH3nBSot5NkymjKpBQaGKsh cUmA== Received: by 10.52.75.70 with SMTP id a6mr14758145vdw.6.1353895926046; Sun, 25 Nov 2012 18:12:06 -0800 (PST) Received: from mavbook.mavhome.dp.ua (cpe-67-244-107-195.nyc.res.rr.com. [67.244.107.195]) by mx.google.com with ESMTPS id cd19sm490926vdb.20.2012.11.25.18.12.04 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 25 Nov 2012 18:12:05 -0800 (PST) Sender: Alexander Motin Message-ID: <50B2CFF3.9060407@FreeBSD.org> Date: Mon, 26 Nov 2012 04:12:03 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:13.0) Gecko/20120628 Thunderbird/13.0.1 MIME-Version: 1.0 To: Andriy Gapon Subject: Re: VirtualBox 4.2.4 on FreeBSD 9.1-PRERELEASE problem: VMs behave very different when pinned to different cores References: <50AFAD05.1050604@FreeBSD.org> <50B25C17.20208@FreeBSD.org> <50B29D53.5080802@FreeBSD.org> In-Reply-To: <50B29D53.5080802@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-emulation@freebsd.org" X-BeenThere: freebsd-emulation@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Development of Emulators of other operating systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 Nov 2012 02:12:07 -0000 On 26.11.2012 00:36, Andriy Gapon wrote: > on 25/11/2012 19:57 Andriy Gapon said the following: >> on 24/11/2012 00:17 Alex Chistyakov said the following: >>> I collected two samples and put them here: http://1888.spb.ru/samples.zip >>> sched-cpu0.ktr is for a VM running on CPU #0 and sched-cpu1.ktr is for >>> a VM running on CPU #1 >>> They seem to be very different. >> >> It looks like you didn't stop ktr tracing before running ktrdump or something >> like that. schedgraph can not grok the files because it believes that the >> timestamps are incorrect. >> >> # - While the workload is continuing (i.e. before it finishes), disable >> # KTR tracing by setting 'sysctl debug.ktr.mask=0'. This is necessary >> # to avoid a race condition while running ktrdump, i.e. the KTR ring buffer >> # will cycle a bit while ktrdump runs, and this confuses schedgraph because >> # the timestamps appear to go backwards at some point. > > Hmm, looks like this assessment is not correct. > I now think that the root cause of schedgraph issue might be a too wild > difference in what TSC counters produce on different (logical/physical) CPUs. > > E.g.: > 131059 1 33232414877586 ... > 131058 1 33232414876546 ... > 131057 3 33232416064514 ... > 131056 3 33232416064198 ... > > Or even: > 131038 0 33232862369416 ... > 131037 3 33232409671570 ... > 131036 0 33232862367256 ... > 131035 3 33232409670982 ... > > That's 455111586 ticks! I've tried to calculate those offsets manually to make schedgraph happy. It seems that while running on one CPU VirtualBox consumes all the CPU time but behaves different on another CPU. I am not sure how it is related to scheduler or specific CPUs. My only guess that there is some unexpected synchronization happens that makes VirtualBox to do some extra spinning. -- Alexander Motin