From owner-freebsd-emulation@FreeBSD.ORG Sun Nov 25 17:57:49 2012 Return-Path: Delivered-To: freebsd-emulation@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9DEE84A4; Sun, 25 Nov 2012 17:57:49 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 641448FC0C; Sun, 25 Nov 2012 17:57:48 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA20424; Sun, 25 Nov 2012 19:57:45 +0200 (EET) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1TcgSf-000L6K-8Q; Sun, 25 Nov 2012 19:57:45 +0200 Message-ID: <50B25C17.20208@FreeBSD.org> Date: Sun, 25 Nov 2012 19:57:43 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Alex Chistyakov Subject: Re: VirtualBox 4.2.4 on FreeBSD 9.1-PRERELEASE problem: VMs behave very different when pinned to different cores References: <50AFAD05.1050604@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.4.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-emulation@freebsd.org" , Alexander Motin X-BeenThere: freebsd-emulation@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Development of Emulators of other operating systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Nov 2012 17:57:49 -0000 on 24/11/2012 00:17 Alex Chistyakov said the following: > On Fri, Nov 23, 2012 at 9:06 PM, Andriy Gapon wrote: >> >> I've cc-ed Alexander who is deeply familiar with both the scheduler and the timer >> code. >> I think that it would be nice to get ktr(4) information suitable for use with >> schedgraph (please google for these keywords). > > I collected two samples and put them here: http://1888.spb.ru/samples.zip > sched-cpu0.ktr is for a VM running on CPU #0 and sched-cpu1.ktr is for > a VM running on CPU #1 > They seem to be very different. It looks like you didn't stop ktr tracing before running ktrdump or something like that. schedgraph can not grok the files because it believes that the timestamps are incorrect. # - While the workload is continuing (i.e. before it finishes), disable # KTR tracing by setting 'sysctl debug.ktr.mask=0'. This is necessary # to avoid a race condition while running ktrdump, i.e. the KTR ring buffer # will cycle a bit while ktrdump runs, and this confuses schedgraph because # the timestamps appear to go backwards at some point. >> Also, version of your kernel, > > kern.version: FreeBSD 9.1-PRERELEASE #4: Fri Nov 23 22:38:47 MSK 2012 > Sources were grabbed on Nov, 16. > >> output of sysctls kern.eventtimer and kern.sched. > > kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) > i8254(100) RTC(0) > kern.eventtimer.et.LAPIC.flags: 7 > kern.eventtimer.et.LAPIC.frequency: 50002806 > kern.eventtimer.et.LAPIC.quality: 600 > kern.eventtimer.et.RTC.flags: 17 > kern.eventtimer.et.RTC.frequency: 32768 > kern.eventtimer.et.RTC.quality: 0 > kern.eventtimer.et.i8254.flags: 1 > kern.eventtimer.et.i8254.frequency: 1193182 > kern.eventtimer.et.i8254.quality: 100 > kern.eventtimer.et.HPET.flags: 7 > kern.eventtimer.et.HPET.frequency: 14318180 > kern.eventtimer.et.HPET.quality: 550 > kern.eventtimer.et.HPET1.flags: 3 > kern.eventtimer.et.HPET1.frequency: 14318180 > kern.eventtimer.et.HPET1.quality: 440 > kern.eventtimer.et.HPET2.flags: 3 > kern.eventtimer.et.HPET2.frequency: 14318180 > kern.eventtimer.et.HPET2.quality: 440 > kern.eventtimer.periodic: 0 > kern.eventtimer.timer: LAPIC > kern.eventtimer.activetick: 1 > kern.eventtimer.idletick: 0 > kern.eventtimer.singlemul: 2 > kern.sched.cpusetsize: 8 > kern.sched.preemption: 1 > kern.sched.topology_spec: > kern.sched.steal_thresh: 2 > kern.sched.steal_idle: 1 > kern.sched.balance_interval: 127 > kern.sched.balance: 1 > kern.sched.affinity: 1 > kern.sched.idlespinthresh: 16 > kern.sched.idlespins: 10000 > kern.sched.static_boost: 152 > kern.sched.preempt_thresh: 80 > kern.sched.interact: 30 > kern.sched.slice: 12 > kern.sched.quantum: 94488 > kern.sched.name: ULE > > I tried kern.eventtimer.periodic=1 and > kern.timecounter.hardware=ACPI-fast but that did not help. Could you please also provide the CPU identification block from dmesg? >> BTW, do you use the default ULE scheduler? > > Yep. > I tried SCHED_4BSD and the situation became much better but not ideal. > %si was around 3-7% on the guest and I had to boot with noacpi and > disable the tickless kernel on the guest to lower it. > At least I was able to run a VM on CPU #0 and all cores became equal. Interesting results. >> Also, is your kernel DTrace enabled? Could you please run the following script for some seconds and report its output for both of the scenarios: profile:::profile-4001 { @stacks[pid, tid, execname, stack()] = count(); } END { trunc(@stacks, 40); printa(@stacks); } -- Andriy Gapon