From owner-freebsd-xen@FreeBSD.ORG Thu Apr 2 19:49:17 2015 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6EFA21BA for ; Thu, 2 Apr 2015 19:49:17 +0000 (UTC) Received: from os-mail-1.tamu.edu (os-mail-1.tamu.edu [165.91.23.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp-relay.tamu.edu", Issuer "InCommon RSA Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3A96B9F1 for ; Thu, 2 Apr 2015 19:49:16 +0000 (UTC) X-TAMU-Auth-ID: None X-TAMU-SenderIP: 165.91.22.240 Received: from exchange.tamu.edu (exch-2p-snat-pool.tamu.edu [165.91.22.240]) by os-mail-1.itio.tamu.edu (8.14.7/8.14.7) with ESMTP id t32HxEkK022072 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT) for ; Thu, 2 Apr 2015 12:59:22 -0500 Received: from MB03.ads.tamu.edu ([169.254.3.105]) by caex02.ads.tamu.edu ([128.194.147.21]) with mapi id 14.03.0195.001; Thu, 2 Apr 2015 12:58:46 -0500 From: Andrew Daugherity To: "freebsd-xen@freebsd.org" Subject: Re: Poor performance with FreeBSD 10.1 under Xen 4.2 Thread-Topic: Poor performance with FreeBSD 10.1 under Xen 4.2 Thread-Index: AQHQaOOfinlni718o0SDQUrW328qGJ0yNL4AgAPWZYCABFQ6gA== Date: Thu, 2 Apr 2015 17:58:46 +0000 Message-ID: <5CC2ABAB-AE46-4C9F-A610-D4EAD735ECA0@tamu.edu> References: <115BE54D-078A-4C45-8904-861DAB316C03@tamu.edu> <5516A998.10206@citrix.com> <57429F3F-8CC9-4C4F-86DF-3E63C5853B01@tamu.edu> In-Reply-To: <57429F3F-8CC9-4C4F-86DF-3E63C5853B01@tamu.edu> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [128.194.89.247] Content-Type: text/plain; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.13.68, 1.0.33, 0.0.0000 definitions=2015-04-02_06:2015-04-02,2015-04-02,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 kscore.is_bulkscore=1.00272401493129e-10 kscore.compositescore=0 circleOfTrustscore=0 compositescore=0.999586866343301 suspectscore=0 recipient_domain_to_sender_totalscore=0 phishscore=0 bulkscore=0 kscore.is_spamscore=0 rbsscore=0.999586866343301 recipient_to_sender_totalscore=0 recipient_domain_to_sender_domain_totalscore=0 spamscore=0 recipient_to_sender_domain_totalscore=0 urlsuspectscore=0.999586866343301 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1504020169 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2015 19:49:17 -0000 On Mar 30, 2015, at 6:52 PM, Andrew Daugherity wrote= : > On Mar 28, 2015, at 8:16 AM, Roger Pau Monn=E9 wro= te: >> I'm Ccing feld because IIRC he found something similar on one of his >> boxes, that also had VTx but no EPT (just like yours). Would it be >> possible for you to try the same set of tests on a different hardware? >=20 > I think you're on to something. I copied this FreeBSD 10.1 VM to a syste= m running the same version of Xen (and same SLES in the Dom0), but with an = Opteron 2360SE CPU (which has both SVM and NPT), and it is *much* faster (a= nd feels more responsive too): > [snip] >> Also, if even FreeBSD 10.1 compiled without XENHVM shows this issue it >> means there's something in the generic code that doesn't work well when >> running virtualized on this specific hardware, but I'm afraid figuring >> it out is not trivial. One place to start would be asking on >> freebsd-hackers and freebsd-virt. >=20 > I suppose this performance delta with presence of EPT/NPT vs. lack thereo= f means it's time to take it to those lists? My next step will be to test = 10.1 under KVM on the Xeon to confirm whether it's a Xen issue or strictly = EPT. It seems I spoke too soon. I booted into the "default" (non-Xen) Linux ker= nel on the Xeon E5420 box and launched the same FreeBSD 10.1 VM under KVM, = and performance is much, much better: Forked, executed and destroyed 5000 processes in 12.307276 seconds. [ Always remains at this time and does not become slower. ] sssd configure: 24.713u 13.530s 0:36.89 103.6% 5337+233k 434+298io 69pf+0w sssd build: 288.788u 235.076s 4:29.13 194.6% 3640+202k 233+28io 87pf+0w It may not be the Xen platform drivers at fault (since my NOHVM kernel didn= 't help), but something is causing FreeBSD 10.x to perform badly under Xen = on a CPU w/o EPT, whereas KVM does not have this issue. Full KVM dmesg follows at the end for comparison; I've highlighted some dif= ferences here. I wonder if any of the CPU/ACPI/timer differences are relev= ant? --- dmesg.xen 2015-03-27 18:06:12.000000000 -0500 +++ dmesg.kvm 2015-04-02 12:39:51.000000000 -0500 @@ -5,54 +5,55 @@ FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 -XEN: Hypervisor version 4.2 detected. -CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (2493.90-MHz K8-class= CPU) - Origin =3D "GenuineIntel" Id =3D 0x10676 Family =3D 0x6 Model =3D 0x1= 7 Stepping =3D 6 - Features=3D0x1783fbff - Features2=3D0x81282201 +CPU: QEMU Virtual CPU version 1.4.2 (2493.82-MHz K8-class CPU) + Origin =3D "GenuineIntel" Id =3D 0x623 Family =3D 0x6 Model =3D 0x2 = Stepping =3D 3 + Features=3D0x783fbfd + Features2=3D0x80002001 AMD Features=3D0x20100800 AMD Features2=3D0x1 real memory =3D 1073741824 (1024 MB) -avail memory =3D 1010737152 (963 MB) +avail memory =3D 1010745344 (963 MB) Event timer "LAPIC" quality 400 -ACPI APIC Table: +ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs -FreeBSD/SMP: 1 package(s) x 2 core(s) +FreeBSD/SMP: 2 package(s) x 1 core(s) cpu0 (BSP): APIC ID: 0 - cpu1 (AP): APIC ID: 2 -ioapic0: Changing APIC ID to 1 -MADT: Forcing active-low polarity and level trigger for SCI -ioapic0 irqs 0-47 on motherboard + cpu1 (AP): APIC ID: 1 +ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 random: initialized -xen_et0: on motherboard -Event timer "XENTIMER" frequency 1000000000 Hz quality 950 -Timecounter "XENTIMER" frequency 1000000000 Hz quality 950 -acpi0: on motherboard +acpi0: on motherboard acpi0: Power Button (fixed) -acpi0: Sleep Button (fixed) -acpi0: reservation of 0, a0000 (3) failed cpu0: on acpi0 cpu1: on acpi0 -attimer0: port 0x40-0x43 irq 0 on acpi0 -Timecounter "i8254" frequency 1193182 Hz quality 0 -Event timer "i8254" frequency 1193182 Hz quality 100 -atrtc0: port 0x70-0x71 irq 8 on acpi0 +atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 +hpet0: iomem 0xfed00000-0xfed003ff on acpi0 +Timecounter "HPET" frequency 100000000 Hz quality 950 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 -acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0 +acpi_timer0: <24-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0 [snip] +orm0: at iomem 0xef000-0xeffff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 +attimer0: at port 0x40 on isa0 +Timecounter "i8254" frequency 1193182 Hz quality 0 +Event timer "i8254" frequency 1193182 Hz quality 100=20 [more snipped] Any idea how to track this down? I suppose I should file a bug to keep tra= ck of this, but maybe wait until we narrow down the cause? Thanks, Andrew dmesg.kvm: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.1-RELEASE-p6 #0: Tue Feb 24 19:00:21 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 CPU: QEMU Virtual CPU version 1.4.2 (2493.82-MHz K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x623 Family =3D 0x6 Model =3D 0x2 S= tepping =3D 3 Features=3D0x783fbfd Features2=3D0x80002001 AMD Features=3D0x20100800 AMD Features2=3D0x1 real memory =3D 1073741824 (1024 MB) avail memory =3D 1010745344 (963 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 2 package(s) x 1 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 random: initialized acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 100000000 Hz quality 950 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 isab0: at device 1.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,= 0x376,0xc060-0xc06f at device 1.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 uhci0: port 0xc000-0xc01f irq 11 at = device 1.2 on pci0 usbus0: controller did not stop usbus0 on uhci0 pci0: at device 1.3 (no driver attached) vgapci0: mem 0xfc000000-0xfdffffff,0xfebf0000-0xfe= bf0fff at device 2.0 on pci0 vgapci0: Boot video device virtio_pci0: port 0xc020-0xc03f mem 0xfebf1000= -0xfebf1fff irq 11 at device 3.0 on pci0 vtnet0: on virtio_pci0 virtio_pci0: host features: 0x719fffe3 virtio_pci0: negotiated features: 0x308fbbe3 vtnet0: Ethernet address: 52:54:00:7d:be:ff virtio_pci1: port 0xc040-0xc05f irq 10 at devi= ce 5.0 on pci0 vtballoon0: on virtio_pci1 virtio_pci1: host features: 0x71000002 virtio_pci1: negotiated features: 0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 fdc0: port 0x3f2-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (9600,n,8,1) orm0: at iomem 0xef000-0xeffff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x100> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 attimer0: at port 0x40 on isa0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 fdc0: No FDOUT register! ppc0: cannot reserve I/O port range Timecounters tick every 10.000 msec usbus0: 12Mbps Full Speed USB v1.0 ugen0.1: at usbus0 uhub0: on usbus0 ada0 at ata0 bus 0 scbus0 target 0 lun 0 ada0: ATA-7 device ada0: Serial Number QM00001 ada0: 16.700MB/s transfers (WDMA2, PIO 8192bytes) ada0: 6144MB (12582912 512 byte sectors: 16H 63S/T 12483C) ada0: Previously was known as ad0 random: unblocking device. SMP: AP CPU #1 Launched! Root mount waiting for: usbus0 uhub0: 2 ports with 2 removable, self powered Trying to mount root from ufs:/dev/ada0p2 [rw]...=