Date: Mon, 26 Nov 2012 10:10:54 +0300 From: Alex Chistyakov <alexclear@gmail.com> To: Andriy Gapon <avg@freebsd.org> Cc: "freebsd-emulation@freebsd.org" <freebsd-emulation@freebsd.org>, Alexander Motin <mav@freebsd.org> Subject: Re: VirtualBox 4.2.4 on FreeBSD 9.1-PRERELEASE problem: VMs behave very different when pinned to different cores Message-ID: <CA%2Bkq2xvjDa1BeuzPUuH99bgriEA-GJH36AZGiqScKSo4QZmHDg@mail.gmail.com> In-Reply-To: <50B25C17.20208@FreeBSD.org> References: <CA%2Bkq2xvh3j5CM7UzRVfXCeLhHwpTY%2B_M7dCJx0c27NtV8EVJwg@mail.gmail.com> <CAE-m3X1UPsy%2Bwbqm_02JpXMr-UO3m7N6z_ZwY2HNo4GL0YUi1w@mail.gmail.com> <CA%2Bkq2xva61m_bHdzBZM2TYL5z7XiohvkxsYWtOyoBwQkpyvp0A@mail.gmail.com> <50AFAD05.1050604@FreeBSD.org> <CA%2Bkq2xv%2BU4ZnfK=1js4PRaNpTNdW-y-G50GV4%2BMVP0LugBf1pQ@mail.gmail.com> <50B25C17.20208@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Nov 25, 2012 at 9:57 PM, Andriy Gapon <avg@freebsd.org> wrote: > on 24/11/2012 00:17 Alex Chistyakov said the following: >> On Fri, Nov 23, 2012 at 9:06 PM, Andriy Gapon <avg@freebsd.org> wrote: >>> >>> I've cc-ed Alexander who is deeply familiar with both the scheduler and the timer >>> code. >>> I think that it would be nice to get ktr(4) information suitable for use with >>> schedgraph (please google for these keywords). >> >> I collected two samples and put them here: http://1888.spb.ru/samples.zip >> sched-cpu0.ktr is for a VM running on CPU #0 and sched-cpu1.ktr is for >> a VM running on CPU #1 >> They seem to be very different. > > It looks like you didn't stop ktr tracing before running ktrdump or something > like that. schedgraph can not grok the files because it believes that the > timestamps are incorrect. > > # - While the workload is continuing (i.e. before it finishes), disable > # KTR tracing by setting 'sysctl debug.ktr.mask=0'. This is necessary > # to avoid a race condition while running ktrdump, i.e. the KTR ring buffer > # will cycle a bit while ktrdump runs, and this confuses schedgraph because > # the timestamps appear to go backwards at some point. >>> Also, version of your kernel, >> >> kern.version: FreeBSD 9.1-PRERELEASE #4: Fri Nov 23 22:38:47 MSK 2012 >> Sources were grabbed on Nov, 16. >> >>> output of sysctls kern.eventtimer and kern.sched. >> >> kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) >> i8254(100) RTC(0) >> kern.eventtimer.et.LAPIC.flags: 7 >> kern.eventtimer.et.LAPIC.frequency: 50002806 >> kern.eventtimer.et.LAPIC.quality: 600 >> kern.eventtimer.et.RTC.flags: 17 >> kern.eventtimer.et.RTC.frequency: 32768 >> kern.eventtimer.et.RTC.quality: 0 >> kern.eventtimer.et.i8254.flags: 1 >> kern.eventtimer.et.i8254.frequency: 1193182 >> kern.eventtimer.et.i8254.quality: 100 >> kern.eventtimer.et.HPET.flags: 7 >> kern.eventtimer.et.HPET.frequency: 14318180 >> kern.eventtimer.et.HPET.quality: 550 >> kern.eventtimer.et.HPET1.flags: 3 >> kern.eventtimer.et.HPET1.frequency: 14318180 >> kern.eventtimer.et.HPET1.quality: 440 >> kern.eventtimer.et.HPET2.flags: 3 >> kern.eventtimer.et.HPET2.frequency: 14318180 >> kern.eventtimer.et.HPET2.quality: 440 >> kern.eventtimer.periodic: 0 >> kern.eventtimer.timer: LAPIC >> kern.eventtimer.activetick: 1 >> kern.eventtimer.idletick: 0 >> kern.eventtimer.singlemul: 2 >> kern.sched.cpusetsize: 8 >> kern.sched.preemption: 1 >> kern.sched.topology_spec: <groups> >> kern.sched.steal_thresh: 2 >> kern.sched.steal_idle: 1 >> kern.sched.balance_interval: 127 >> kern.sched.balance: 1 >> kern.sched.affinity: 1 >> kern.sched.idlespinthresh: 16 >> kern.sched.idlespins: 10000 >> kern.sched.static_boost: 152 >> kern.sched.preempt_thresh: 80 >> kern.sched.interact: 30 >> kern.sched.slice: 12 >> kern.sched.quantum: 94488 >> kern.sched.name: ULE >> >> I tried kern.eventtimer.periodic=1 and >> kern.timecounter.hardware=ACPI-fast but that did not help. > > Could you please also provide the CPU identification block from dmesg? CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3200.18-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x1fbee3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x1<LAHF> TSC: P-state invariant, performance statistics > >>> BTW, do you use the default ULE scheduler? >> >> Yep. >> I tried SCHED_4BSD and the situation became much better but not ideal. >> %si was around 3-7% on the guest and I had to boot with noacpi and >> disable the tickless kernel on the guest to lower it. >> At least I was able to run a VM on CPU #0 and all cores became equal. > > Interesting results. > >>> Also, is your kernel DTrace enabled? > > Could you please run the following script for some seconds and report its output > for both of the scenarios: > > profile:::profile-4001 > { > @stacks[pid, tid, execname, stack()] = count(); > } > > END > { > trunc(@stacks, 40); > printa(@stacks); > } A DTrace result for "bad" core is at https://gist.github.com/4145782 and a result for "good" core is at https://gist.github.com/4146946 Thank you! -- SY, Alex
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2Bkq2xvjDa1BeuzPUuH99bgriEA-GJH36AZGiqScKSo4QZmHDg>