From owner-freebsd-xen@freebsd.org Sat Oct 12 16:43:08 2019 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E937213BFB2 for ; Sat, 12 Oct 2019 16:43:08 +0000 (UTC) (envelope-from royger@gmail.com) Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46r9cD11Z7z4J8w for ; Sat, 12 Oct 2019 16:43:07 +0000 (UTC) (envelope-from royger@gmail.com) Received: by mail-ot1-f41.google.com with SMTP id y39so10550518ota.7 for ; Sat, 12 Oct 2019 09:43:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=QjQSZMEqJBAK2hjLDLqjorvjDNyIqfYBfCVItUEwckI=; b=ARPhV+DKFrao03t+SMKvqU6Pp8+u2YKnAABSXehX1K322XWXNYWvQefYcnGuGdAXwM xv+amJIR6vkal+kknETcVgs5AoMIRvzpVjpGSsDDg98mUq9CfOnjwn8DGYHAp5i1glK3 i4fJgoqZn6+JveuCh5Avii4/r1qbBYYkeY/GvOqXF5s55HFd5eHnIjR7nhZX6kjUmIjA Jub4xMQvxHjHRNflk5tj+xCv4c30GOy4zya3T8aAIBQ4hHFK4uGc4NkuQhxVIzKiSG5J EkSrdMPiyq9h0zlDhnTz5ymwV+7IAgHmgZ5PwOsOHKFHxfdtb1arpFa3vxIWU/LSGtPX gYqQ== X-Gm-Message-State: APjAAAWA+naPoD9scPhA6tffi5i95kQXcd+8q8Dyg18DP4iHOpaqXuSG oJ0CaDNyc6xrtRHnB5WhhWNF2M0ovF2Ern7nESw= X-Google-Smtp-Source: APXvYqwD626pvkEInfaUhf2i79+qY17wq/XHJZoxkHbW/kZ8kwJ6OxmxQm7ZdNxxGHQ9JNgvio5wuK9prj+4PkppvRA= X-Received: by 2002:a9d:6b82:: with SMTP id b2mr16826674otq.56.1570898586231; Sat, 12 Oct 2019 09:43:06 -0700 (PDT) MIME-Version: 1.0 References: <20191012095900.GA14005@Air-de-Roger> <201910121531.x9CFVaGT014403@lothlorien.nfbcal.org> In-Reply-To: <201910121531.x9CFVaGT014403@lothlorien.nfbcal.org> From: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Date: Sat, 12 Oct 2019 18:42:53 +0200 Message-ID: Subject: Re: i/o is very slow on FreeBSD dom0 with Xen-4.12 and Freebsd-12 To: Brian Buhrow Cc: Roger Pau Monne , freebsd-xen@freebsd.org X-Rspamd-Queue-Id: 46r9cD11Z7z4J8w X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of royger@gmail.com designates 209.85.210.41 as permitted sender) smtp.mailfrom=royger@gmail.com X-Spamd-Result: default: False [-3.19 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-xen@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[41.210.85.209.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_GOOD(0.00)[41.210.85.209.rep.mailspike.net : 127.0.0.18]; IP_SCORE(-1.19)[ip: (-0.52), ipnet: 209.85.128.0/17(-3.25), asn: 15169(-2.12), country: US(-0.05)]; FORGED_SENDER(0.30)[royger@freebsd.org,royger@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[royger@freebsd.org,royger@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Oct 2019 16:43:09 -0000 Replying from my phone, sorry for the format. El ds., 12 d=E2=80=99oct. 2019, 17:32, Brian Buhrow va = escriure: > Hello. Here are the logs with remapping enabled, plus the i, m > and z > xen console commands. > Here's hoping you can provide a fix soon! Or, perhaps, a clue on how to > fix. > -thanks > -Brian > > > > boot_serial=3D"YES" # -h: Use serial console > comconsole_speed=3D"115200" # Set the current serial console speed > comconsole_port=3D"0x2f8" # Set the current serial console port > #console=3D"vidconsole" # A comma separated list of console(s) > console=3D"comconsole" # A comma separated list of console(s) > geom_mirror_load=3D"YES" # RAID1 disk driver (see gmirror(8)) > ahci_load=3D"YES" # ahci driver > ipmi_load=3D"YES" # Ripmi driver > if_tap_load=3D"YES" # Load bridge driver. > if_vlan_load=3D"YES" # Load vlan driver. > # Turn on Xen (BB 10/08/2019) > xen_kernel=3D"/boot/xen" > xen_cmdline=3D"dom0_mem=3D4096m dom0_max_vcpus=3D4 dom0=3Dpvh,verbose > console=3Dcom2,vga iommu=3Dverbose,debug sync_console=3Dtrue com2=3D11520= 0,8n1 > guest_loglvl=3Dall loglvl=3Dall" > > > > (XEN) Xen version 4.12.1 (buhrow@) (FreeBSD clang version 6.0.1 > (tags/RELEASE_601/final 335540) (based on LLVM 6.0.1)) debug=3Dn Thu Oct= 3 > 11:34:20 PDT 2019 > (XEN) Latest ChangeSet: > (XEN) Console output is synchronous. > (XEN) Bootloader: FreeBSD Loader > (XEN) Command line: dom0_mem=3D4096m dom0_max_vcpus=3D4 dom0=3Dpvh,verbos= e > console=3Dcom2,vga iommu=3Dverbose,debug sync_console=3Dtrue com2=3D11520= 0,8n1 > guest_loglvl=3Dall loglvl=3Dall > (XEN) Xen image load base address: 0 > (XEN) Video information: > (XEN) VGA is text mode 80x25, font 8x16 > (XEN) VBE/DDC methods: none; EDID transfer time: 0 seconds > (XEN) EDID info not retrieved because no DDC retrieval method detected > (XEN) Disc information: > (XEN) Found 4 MBR signatures > (XEN) Found 4 EDD information structures > (XEN) Xen-e820 RAM map: > (XEN) 0000000000000000 - 0000000000098400 (usable) > (XEN) 0000000000098400 - 00000000000a0000 (reserved) > (XEN) 00000000000e0000 - 0000000000100000 (reserved) > (XEN) 0000000000100000 - 00000000bf780000 (usable) > (XEN) 00000000bf78e000 - 00000000bf790000 type 9 > (XEN) 00000000bf790000 - 00000000bf79e000 (ACPI data) > (XEN) 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS) > (XEN) 00000000bf7d0000 - 00000000bf7e0000 (reserved) > (XEN) 00000000bf7ec000 - 00000000c0000000 (reserved) > (XEN) 00000000e0000000 - 00000000f0000000 (reserved) > (XEN) 00000000fee00000 - 00000000fee01000 (reserved) > (XEN) 00000000ffc00000 - 0000000100000000 (reserved) > (XEN) 0000000100000000 - 0000001840000000 (usable) > (XEN) New Xen image base address: 0xbf000000 > (XEN) ACPI: RSDP 000FACE0, 0024 (r2 ACPIAM) > (XEN) ACPI: XSDT BF790100, 008C (r1 SMCI 20120803 MSFT 9= 7) > (XEN) ACPI: FACP BF790290, 00F4 (r3 080312 FACP1521 20120803 MSFT 9= 7) > (XEN) ACPI: DSDT BF7906A0, 6580 (r1 10600 10600000 0 INTL 2005111= 7) > (XEN) ACPI: FACS BF79E000, 0040 > (XEN) ACPI: APIC BF790390, 011E (r1 080312 APIC1521 20120803 MSFT 9= 7) > (XEN) ACPI: MCFG BF7904B0, 003C (r1 080312 OEMMCFG 20120803 MSFT 9= 7) > (XEN) ACPI: SLIT BF7904F0, 0030 (r1 080312 OEMSLIT 20120803 MSFT 9= 7) > (XEN) ACPI: OEMB BF79E040, 0086 (r1 080312 OEMB1521 20120803 MSFT 9= 7) > (XEN) ACPI: SRAT BF79A6A0, 01D0 (r1 080312 OEMSRAT 1 INTL = 1) > (XEN) ACPI: HPET BF79A870, 0038 (r1 080312 OEMHPET 20120803 MSFT 9= 7) > (XEN) ACPI: DMAR BF79E0D0, 0130 (r1 AMI OEMDMAR 1 MSFT 9= 7) > (XEN) ACPI: SSDT BF7A1710, 0363 (r1 DpgPmm CpuPm 12 INTL 2005111= 7) > (XEN) ACPI: EINJ BF79A8B0, 0130 (r1 AMIER AMI_EINJ 20120803 MSFT 9= 7) > (XEN) ACPI: BERT BF79AA40, 0030 (r1 AMIER AMI_BERT 20120803 MSFT 9= 7) > (XEN) ACPI: ERST BF79AA70, 01B0 (r1 AMIER AMI_ERST 20120803 MSFT 9= 7) > (XEN) ACPI: HEST BF79AC20, 00A8 (r1 AMIER ABC_HEST 20120803 MSFT 9= 7) > (XEN) System RAM: 98295MB (100654176kB) > (XEN) SRAT: PXM 0 -> APIC 00 -> Node 0 > (XEN) SRAT: PXM 0 -> APIC 02 -> Node 0 > (XEN) SRAT: PXM 0 -> APIC 04 -> Node 0 > (XEN) SRAT: PXM 0 -> APIC 06 -> Node 0 > (XEN) SRAT: PXM 0 -> APIC 01 -> Node 0 > (XEN) SRAT: PXM 0 -> APIC 03 -> Node 0 > (XEN) SRAT: PXM 0 -> APIC 05 -> Node 0 > (XEN) SRAT: PXM 0 -> APIC 07 -> Node 0 > (XEN) SRAT: PXM 1 -> APIC 10 -> Node 1 > (XEN) SRAT: PXM 1 -> APIC 12 -> Node 1 > (XEN) SRAT: PXM 1 -> APIC 14 -> Node 1 > (XEN) SRAT: PXM 1 -> APIC 16 -> Node 1 > (XEN) SRAT: PXM 1 -> APIC 11 -> Node 1 > (XEN) SRAT: PXM 1 -> APIC 13 -> Node 1 > (XEN) SRAT: PXM 1 -> APIC 15 -> Node 1 > (XEN) SRAT: PXM 1 -> APIC 17 -> Node 1 > (XEN) SRAT: Node 0 PXM 0 0-a0000 > (XEN) SRAT: Node 0 PXM 0 100000-c0000000 > (XEN) SRAT: Node 0 PXM 0 100000000-c40000000 > (XEN) SRAT: Node 1 PXM 1 c40000000-1840000000 > (XEN) NUMA: Allocated memnodemap from 183e1b7000 - 183e1d0000 > (XEN) NUMA: Using 8 for the hash shift. > (XEN) Domain heap initialised DMA width 32 bits > (XEN) CPU Vendor: Intel, Family 6 (0x6), Model 26 (0x1a), Stepping 5 (raw > 000106a5) > (XEN) found SMP MP-table at 000ff780 > (XEN) DMI present. > (XEN) Using APIC driver default > (XEN) ACPI: PM-Timer IO Port: 0x808 (32 bits) > (XEN) ACPI: SLEEP INFO: pm1x_cnt[1:804,1:0], pm1x_evt[1:800,1:0] > (XEN) ACPI: wakeup_vec[bf79e00c], vec_size[20] > (XEN) ACPI: Local APIC address 0xfee00000 > (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x10] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x12] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x14] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x06] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x16] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x01] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x11] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x03] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x13] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x05] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x15] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x07] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x17] enabled) > (XEN) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x90] disabled) > (XEN) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x91] disabled) > (XEN) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x92] disabled) > (XEN) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x93] disabled) > (XEN) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x94] disabled) > (XEN) ACPI: LAPIC (acpi_id[0x16] lapic_id[0x95] disabled) > (XEN) ACPI: LAPIC (acpi_id[0x17] lapic_id[0x96] disabled) > (XEN) ACPI: LAPIC (acpi_id[0x18] lapic_id[0x97] disabled) > (XEN) ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) > (XEN) Overriding APIC driver with bigsmp > (XEN) ACPI: IOAPIC (id[0x08] address[0xfec00000] gsi_base[0]) > (XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23 > (XEN) ACPI: IOAPIC (id[0x09] address[0xfec8a000] gsi_base[24]) > (XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec8a000, GSI 24-47 > (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) > (XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) > (XEN) ACPI: IRQ0 used by override. > (XEN) ACPI: IRQ2 used by override. > (XEN) ACPI: IRQ9 used by override. > (XEN) Enabling APIC mode: Phys. Using 2 I/O APICs > (XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000 > (XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff > (XEN) PCI: MCFG area at e0000000 reserved in E820 > (XEN) PCI: Using MCFG for segment 0000 bus 00-ff > (XEN) [VT-D]Host address width 40 > (XEN) [VT-D]found ACPI_DMAR_DRHD: > (XEN) [VT-D] dmaru->address =3D fbffe000 > (XEN) [VT-D]drhd->address =3D fbffe000 iommu->reg =3D ffff82c00021d000 > (XEN) [VT-D]cap =3D c90780106f0462 ecap =3D f020fe > (XEN) [VT-D] IOAPIC: 0000:f0:1f.7 > (XEN) [VT-D] IOAPIC: 0000:00:13.0 > (XEN) [VT-D] flags: INCLUDE_ALL > (XEN) [VT-D]found ACPI_DMAR_RMRR: > (XEN) [VT-D] endpoint: 0000:00:1d.0 > (XEN) [VT-D] endpoint: 0000:00:1d.1 > (XEN) [VT-D] endpoint: 0000:00:1d.2 > (XEN) [VT-D] endpoint: 0000:00:1d.7 > (XEN) [VT-D] endpoint: 0000:00:1a.0 > (XEN) [VT-D] endpoint: 0000:00:1a.1 > (XEN) [VT-D] endpoint: 0000:00:1a.2 > (XEN) [VT-D] endpoint: 0000:00:1a.7 > (XEN) [VT-D]found ACPI_DMAR_RMRR: > (XEN) [VT-D] endpoint: 0000:00:1d.0 > (XEN) [VT-D] endpoint: 0000:00:1d.1 > (XEN) [VT-D] endpoint: 0000:00:1d.2 > (XEN) [VT-D] endpoint: 0000:00:1d.7 > (XEN) [VT-D] endpoint: 0000:00:1a.0 > (XEN) [VT-D] endpoint: 0000:00:1a.1 > (XEN) [VT-D] endpoint: 0000:00:1a.2 > (XEN) [VT-D] endpoint: 0000:00:1a.7 > (XEN) [VT-D]found ACPI_DMAR_ATSR: > (XEN) [VT-D] atsru->all_ports: 0 > (XEN) [VT-D] bridge: 0000:00:01.0 start=3D0 sec=3D1 sub=3D1 > (XEN) [VT-D] bridge: 0000:00:03.0 start=3D0 sec=3D2 sub=3D2 > (XEN) [VT-D] bridge: 0000:00:05.0 start=3D0 sec=3D3 sub=3D3 > (XEN) [VT-D] bridge: 0000:00:07.0 start=3D0 sec=3D4 sub=3D4 > (XEN) [VT-D] bridge: 0000:00:09.0 start=3D0 sec=3D5 sub=3D5 > (XEN) Failed to get Error Log Address Range. > (XEN) HEST: Table parsing has been initialized > (XEN) Using ACPI (MADT) for SMP configuration information > (XEN) SMP: Allowing 24 CPUs (8 hotplug CPUs) > (XEN) IRQ limits: 48 GSI, 3040 MSI/MSI-X > (XEN) CPU0: Intel machine check reporting enabled > (XEN) Speculative mitigation facilities: > (XEN) Hardware features: > (XEN) Compiled-in support: SHADOW_PAGING > (XEN) Xen settings: BTI-Thunk N/A, SPEC_CTRL: No, Other: > (XEN) L1TF: believed vulnerable, maxphysaddr L1D 44, CPUID 40, Safe > address 10000000000 > (XEN) Support for HVM VMs: RSB EAGER_FPU > (XEN) Support for PV VMs: RSB EAGER_FPU > (XEN) XPTI (64-bit PV only): Dom0 enabled, DomU enabled (without PCID) > (XEN) PV L1TF shadowing: Dom0 disabled, DomU enabled > (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2) > (XEN) Initializing Credit2 scheduler > (XEN) load_precision_shift: 18 > (XEN) load_window_shift: 30 > (XEN) underload_balance_tolerance: 0 > (XEN) overload_balance_tolerance: -3 > (XEN) runqueues arrangement: socket > (XEN) cap enforcement granularity: 10ms > (XEN) load tracking window length 1073741824 ns > (XEN) Adding cpu 0 to runqueue 0 > (XEN) First cpu on runqueue, activating > (XEN) Platform timer is 14.318MHz HPET > (XEN) Detected 2266.768 MHz processor. > (XEN) Initing memory sharing. > (XEN) alt table ffff82d0804309b8 -> ffff82d0804318d0 > (XEN) Intel VT-d iommu 0 supported page sizes: 4kB. > (XEN) Intel VT-d Snoop Control enabled. > (XEN) Intel VT-d Dom0 DMA Passthrough not enabled. > (XEN) Intel VT-d Queued Invalidation enabled. > (XEN) Intel VT-d Interrupt Remapping enabled. > (XEN) Intel VT-d Posted Interrupt not enabled. > (XEN) Intel VT-d Shared EPT tables not enabled. > (XEN) I/O virtualisation enabled > (XEN) - Dom0 mode: Relaxed > (XEN) Interrupt remapping enabled > (XEN) nr_sockets: 3 > (XEN) ENABLING IO-APIC IRQs > (XEN) -> Using new ACK method > Can you try to add ioapic_ack=3Dold to the xen command line and report if that makes a difference? Roger.