Date: Thu, 2 Apr 2015 17:58:46 +0000 From: Andrew Daugherity <adaugherity@tamu.edu> To: "freebsd-xen@freebsd.org" <freebsd-xen@freebsd.org> Subject: Re: Poor performance with FreeBSD 10.1 under Xen 4.2 Message-ID: <5CC2ABAB-AE46-4C9F-A610-D4EAD735ECA0@tamu.edu> In-Reply-To: <57429F3F-8CC9-4C4F-86DF-3E63C5853B01@tamu.edu> References: <115BE54D-078A-4C45-8904-861DAB316C03@tamu.edu> <5516A998.10206@citrix.com> <57429F3F-8CC9-4C4F-86DF-3E63C5853B01@tamu.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mar 30, 2015, at 6:52 PM, Andrew Daugherity <adaugherity@tamu.edu> wrote=
:
> On Mar 28, 2015, at 8:16 AM, Roger Pau Monn=E9 <roger.pau@citrix.com> wro=
te:
>> I'm Ccing feld because IIRC he found something similar on one of his
>> boxes, that also had VTx but no EPT (just like yours). Would it be
>> possible for you to try the same set of tests on a different hardware?
>=20
> I think you're on to something. I copied this FreeBSD 10.1 VM to a syste=
m running the same version of Xen (and same SLES in the Dom0), but with an =
Opteron 2360SE CPU (which has both SVM and NPT), and it is *much* faster (a=
nd feels more responsive too):
> [snip]
>> Also, if even FreeBSD 10.1 compiled without XENHVM shows this issue it
>> means there's something in the generic code that doesn't work well when
>> running virtualized on this specific hardware, but I'm afraid figuring
>> it out is not trivial. One place to start would be asking on
>> freebsd-hackers and freebsd-virt.
>=20
> I suppose this performance delta with presence of EPT/NPT vs. lack thereo=
f means it's time to take it to those lists? My next step will be to test =
10.1 under KVM on the Xeon to confirm whether it's a Xen issue or strictly =
EPT.
It seems I spoke too soon. I booted into the "default" (non-Xen) Linux ker=
nel on the Xeon E5420 box and launched the same FreeBSD 10.1 VM under KVM, =
and performance is much, much better:
Forked, executed and destroyed 5000 processes in 12.307276 seconds.
[ Always remains at this time and does not become slower. ]
sssd configure: 24.713u 13.530s 0:36.89 103.6% 5337+233k 434+298io 69pf+0w
sssd build: 288.788u 235.076s 4:29.13 194.6% 3640+202k 233+28io 87pf+0w
It may not be the Xen platform drivers at fault (since my NOHVM kernel didn=
't help), but something is causing FreeBSD 10.x to perform badly under Xen =
on a CPU w/o EPT, whereas KVM does not have this issue.
Full KVM dmesg follows at the end for comparison; I've highlighted some dif=
ferences here. I wonder if any of the CPU/ACPI/timer differences are relev=
ant?
--- dmesg.xen 2015-03-27 18:06:12.000000000 -0500
+++ dmesg.kvm 2015-04-02 12:39:51.000000000 -0500
@@ -5,54 +5,55 @@
FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21 UTC 2015
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
-XEN: Hypervisor version 4.2 detected.
-CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (2493.90-MHz K8-class=
CPU)
- Origin =3D "GenuineIntel" Id =3D 0x10676 Family =3D 0x6 Model =3D 0x1=
7 Stepping =3D 6
- Features=3D0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,P=
GE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
- Features2=3D0x81282201<SSE3,SSSE3,CX16,SSE4.1,x2APIC,TSCDLT,HV>
+CPU: QEMU Virtual CPU version 1.4.2 (2493.82-MHz K8-class CPU)
+ Origin =3D "GenuineIntel" Id =3D 0x623 Family =3D 0x6 Model =3D 0x2 =
Stepping =3D 3
+ Features=3D0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MC=
A,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
+ Features2=3D0x80002001<SSE3,CX16,HV>
AMD Features=3D0x20100800<SYSCALL,NX,LM>
AMD Features2=3D0x1<LAHF>
real memory =3D 1073741824 (1024 MB)
-avail memory =3D 1010737152 (963 MB)
+avail memory =3D 1010745344 (963 MB)
Event timer "LAPIC" quality 400
-ACPI APIC Table: <Xen HVM>
+ACPI APIC Table: <BOCHS BXPCAPIC>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
-FreeBSD/SMP: 1 package(s) x 2 core(s)
+FreeBSD/SMP: 2 package(s) x 1 core(s)
cpu0 (BSP): APIC ID: 0
- cpu1 (AP): APIC ID: 2
-ioapic0: Changing APIC ID to 1
-MADT: Forcing active-low polarity and level trigger for SCI
-ioapic0 <Version 1.1> irqs 0-47 on motherboard
+ cpu1 (AP): APIC ID: 1
+ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
random: <Software, Yarrow> initialized
-xen_et0: <Xen PV Clock> on motherboard
-Event timer "XENTIMER" frequency 1000000000 Hz quality 950
-Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
-acpi0: <Xen> on motherboard
+acpi0: <BOCHS BXPCRSDT> on motherboard
acpi0: Power Button (fixed)
-acpi0: Sleep Button (fixed)
-acpi0: reservation of 0, a0000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
-attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
-Timecounter "i8254" frequency 1193182 Hz quality 0
-Event timer "i8254" frequency 1193182 Hz quality 100
-atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
+atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
+hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
+Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
-acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
+acpi_timer0: <24-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
[snip]
+orm0: <ISA Option ROM> at iomem 0xef000-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=3D0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
+attimer0: <AT timer> at port 0x40 on isa0
+Timecounter "i8254" frequency 1193182 Hz quality 0
+Event timer "i8254" frequency 1193182 Hz quality 100=20
[more snipped]
Any idea how to track this down? I suppose I should file a bug to keep tra=
ck of this, but maybe wait until we narrow down the cause?
Thanks,
Andrew
dmesg.kvm:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21 UTC 2015
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: QEMU Virtual CPU version 1.4.2 (2493.82-MHz K8-class CPU)
Origin =3D "GenuineIntel" Id =3D 0x623 Family =3D 0x6 Model =3D 0x2 S=
tepping =3D 3
Features=3D0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA=
,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
Features2=3D0x80002001<SSE3,CX16,HV>
AMD Features=3D0x20100800<SYSCALL,NX,LM>
AMD Features2=3D0x1<LAHF>
real memory =3D 1073741824 (1024 MB)
avail memory =3D 1010745344 (963 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <BOCHS BXPCAPIC>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 2 package(s) x 1 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
random: <Software, Yarrow> initialized
acpi0: <BOCHS BXPCRSDT> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x72-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 100000000 Hz quality 950
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,=
0x376,0xc060-0xc06f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
uhci0: <Intel 82371SB (PIIX3) USB controller> port 0xc000-0xc01f irq 11 at =
device 1.2 on pci0
usbus0: controller did not stop
usbus0 on uhci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xfc000000-0xfdffffff,0xfebf0000-0xfe=
bf0fff at device 2.0 on pci0
vgapci0: Boot video device
virtio_pci0: <VirtIO PCI Network adapter> port 0xc020-0xc03f mem 0xfebf1000=
-0xfebf1fff irq 11 at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
virtio_pci0: host features: 0x719fffe3 <EventIdx,RingIndirect,NotifyOnEmpty=
,SetMacAddress,RxModeExtra,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,TxUF=
O,TxTSOECN,TxTSOv6,TxTSOv4,RxUFO,RxECN,RxTSOv6,RxTSOv4,TxAllGSO,MacAddress,=
RxChecksum,TxChecksum>
virtio_pci0: negotiated features: 0x308fbbe3 <EventIdx,RingIndirect,SetMacA=
ddress,VLanFilter,RxMode,ControlVq,Status,MrgRxBuf,TxTSOECN,TxTSOv6,TxTSOv4=
,RxECN,RxTSOv6,RxTSOv4,TxAllGSO,MacAddress,RxChecksum,TxChecksum>
vtnet0: Ethernet address: 52:54:00:7d:be:ff
virtio_pci1: <VirtIO PCI Balloon adapter> port 0xc040-0xc05f irq 10 at devi=
ce 5.0 on pci0
vtballoon0: <VirtIO Balloon Adapter> on virtio_pci1
virtio_pci1: host features: 0x71000002 <EventIdx,RingIndirect,NotifyOnEmpty=
,StatsVq>
virtio_pci1: negotiated features: 0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
orm0: <ISA Option ROM> at iomem 0xef000-0xeffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=3D0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
attimer0: <AT timer> at port 0x40 on isa0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
fdc0: No FDOUT register!
ppc0: cannot reserve I/O port range
Timecounters tick every 10.000 msec
usbus0: 12Mbps Full Speed USB v1.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ada0 at ata0 bus 0 scbus0 target 0 lun 0
ada0: <QEMU HARDDISK 1.4.2> ATA-7 device
ada0: Serial Number QM00001
ada0: 16.700MB/s transfers (WDMA2, PIO 8192bytes)
ada0: 6144MB (12582912 512 byte sectors: 16H 63S/T 12483C)
ada0: Previously was known as ad0
random: unblocking device.
SMP: AP CPU #1 Launched!
Root mount waiting for: usbus0
uhub0: 2 ports with 2 removable, self powered
Trying to mount root from ufs:/dev/ada0p2 [rw]...=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5CC2ABAB-AE46-4C9F-A610-D4EAD735ECA0>