Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 25 Nov 2012 19:57:43 +0200
From:      Andriy Gapon <avg@FreeBSD.org>
To:        Alex Chistyakov <alexclear@gmail.com>
Cc:        "freebsd-emulation@freebsd.org" <freebsd-emulation@FreeBSD.org>, Alexander Motin <mav@FreeBSD.org>
Subject:   Re: VirtualBox 4.2.4 on FreeBSD 9.1-PRERELEASE problem: VMs behave very different when pinned to different cores
Message-ID:  <50B25C17.20208@FreeBSD.org>
In-Reply-To: <CA%2Bkq2xv%2BU4ZnfK=1js4PRaNpTNdW-y-G50GV4%2BMVP0LugBf1pQ@mail.gmail.com>
References:  <CA%2Bkq2xvh3j5CM7UzRVfXCeLhHwpTY%2B_M7dCJx0c27NtV8EVJwg@mail.gmail.com> <CAE-m3X1UPsy%2Bwbqm_02JpXMr-UO3m7N6z_ZwY2HNo4GL0YUi1w@mail.gmail.com> <CA%2Bkq2xva61m_bHdzBZM2TYL5z7XiohvkxsYWtOyoBwQkpyvp0A@mail.gmail.com> <50AFAD05.1050604@FreeBSD.org> <CA%2Bkq2xv%2BU4ZnfK=1js4PRaNpTNdW-y-G50GV4%2BMVP0LugBf1pQ@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
on 24/11/2012 00:17 Alex Chistyakov said the following:
> On Fri, Nov 23, 2012 at 9:06 PM, Andriy Gapon <avg@freebsd.org> wrote:
>>
>> I've cc-ed Alexander who is deeply familiar with both the scheduler and the timer
>> code.
>> I think that it would be nice to get ktr(4) information suitable for use with
>> schedgraph (please google for these keywords).
> 
> I collected two samples and put them here: http://1888.spb.ru/samples.zip
> sched-cpu0.ktr is for a VM running on CPU #0 and sched-cpu1.ktr is for
> a VM running on CPU #1
> They seem to be very different.

It looks like you didn't stop ktr tracing before running ktrdump or something
like that.  schedgraph can not grok the files because it believes that the
timestamps are incorrect.

# - While the workload is continuing (i.e. before it finishes), disable
#   KTR tracing by setting 'sysctl debug.ktr.mask=0'.  This is necessary
#   to avoid a race condition while running ktrdump, i.e. the KTR ring buffer
#   will cycle a bit while ktrdump runs, and this confuses schedgraph because
#   the timestamps appear to go backwards at some point.
>>  Also, version of your kernel,
> 
> kern.version: FreeBSD 9.1-PRERELEASE #4: Fri Nov 23 22:38:47 MSK 2012
> Sources were grabbed on Nov, 16.
> 
>> output of sysctls kern.eventtimer and kern.sched.
> 
> kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440)
> i8254(100) RTC(0)
> kern.eventtimer.et.LAPIC.flags: 7
> kern.eventtimer.et.LAPIC.frequency: 50002806
> kern.eventtimer.et.LAPIC.quality: 600
> kern.eventtimer.et.RTC.flags: 17
> kern.eventtimer.et.RTC.frequency: 32768
> kern.eventtimer.et.RTC.quality: 0
> kern.eventtimer.et.i8254.flags: 1
> kern.eventtimer.et.i8254.frequency: 1193182
> kern.eventtimer.et.i8254.quality: 100
> kern.eventtimer.et.HPET.flags: 7
> kern.eventtimer.et.HPET.frequency: 14318180
> kern.eventtimer.et.HPET.quality: 550
> kern.eventtimer.et.HPET1.flags: 3
> kern.eventtimer.et.HPET1.frequency: 14318180
> kern.eventtimer.et.HPET1.quality: 440
> kern.eventtimer.et.HPET2.flags: 3
> kern.eventtimer.et.HPET2.frequency: 14318180
> kern.eventtimer.et.HPET2.quality: 440
> kern.eventtimer.periodic: 0
> kern.eventtimer.timer: LAPIC
> kern.eventtimer.activetick: 1
> kern.eventtimer.idletick: 0
> kern.eventtimer.singlemul: 2
> kern.sched.cpusetsize: 8
> kern.sched.preemption: 1
> kern.sched.topology_spec: <groups>
> kern.sched.steal_thresh: 2
> kern.sched.steal_idle: 1
> kern.sched.balance_interval: 127
> kern.sched.balance: 1
> kern.sched.affinity: 1
> kern.sched.idlespinthresh: 16
> kern.sched.idlespins: 10000
> kern.sched.static_boost: 152
> kern.sched.preempt_thresh: 80
> kern.sched.interact: 30
> kern.sched.slice: 12
> kern.sched.quantum: 94488
> kern.sched.name: ULE
> 
> I tried kern.eventtimer.periodic=1 and
> kern.timecounter.hardware=ACPI-fast but that did not help.

Could you please also provide the CPU identification block from dmesg?

>> BTW, do you use the default ULE scheduler?
> 
> Yep.
> I tried SCHED_4BSD and the situation became much better but not ideal.
> %si was around 3-7% on the guest and I had to boot with noacpi and
> disable the tickless kernel on the guest to lower it.
> At least I was able to run a VM on CPU #0 and all cores became equal.

Interesting results.

>> Also, is your kernel DTrace enabled?

Could you please run the following script for some seconds and report its output
for both of the scenarios:

profile:::profile-4001
{
        @stacks[pid, tid, execname, stack()] = count();
}

END
{
        trunc(@stacks, 40);
        printa(@stacks);
}

-- 
Andriy Gapon



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50B25C17.20208>