Date: Thu, 13 Jan 2022 15:57:04 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= <roger.pau@citrix.com> To: Andriy Gapon <avg@FreeBSD.org> Cc: <xen@freebsd.org> Subject: Re: VCPUOP_send_nmi returns -38 Message-ID: <YeA9wDPWQ%2BAJH1aN@Air-de-Roger> In-Reply-To: <787faaf6-c639-0f6a-1a16-c4fb7a5506e5@FreeBSD.org> References: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> <a27fb9ea-1078-fbbd-3dd9-9ce60de08c22@FreeBSD.org> <Yd1bR6mSAjL6VMYD@Air-de-Roger> <787faaf6-c639-0f6a-1a16-c4fb7a5506e5@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--s7dTevK8xyWDIcXS Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit On Tue, Jan 11, 2022 at 12:28:01PM +0200, Andriy Gapon wrote: > On 11/01/2022 12:26, Roger Pau Monné wrote: > > On Tue, Jan 11, 2022 at 12:07:28PM +0200, Andriy Gapon wrote: > > > On 11/01/2022 11:50, Andriy Gapon wrote: > > > > > > > > Recently I got a report of crashes related to using procstat -k on one > > > > of our systems. The system runs FreeBSD 12.2 on an AWS Xen-based > > > > instance (can get more specifics about it later). > > > > > > The instance type is t2.large. > > > Here are all lines from verbose boot that mention Xen: > > > > > > XEN: Hypervisor version 4.2 detected. > > > Disabling MSI-X interrupt migration due to Xen hypervisor bug. > > > XEN: disabling emulated disks > > > XEN: disabling emulated nics > > > Hypervisor: Origin = "XenVMMXenVMM" > > > x2APIC available but disabled due to running under XEN > > > > Thanks. I've asked someone at AWS for comments. I guess the only way > > to workaround this is to switch back to using the emulated APIC. I > > will try to prepare a patch later. > > Thank you very much! I've got two patches for you to try, the first one is 'mni-fix.patch' and should fix the NMI sending issue. I'm also attaching a second patch, that should allow your guest to use x2APIC. FreeBSD has previously prevented Xen guests from using x2APIC because of a bug in old Xen versions, but I think this should be solved now, so if FreeBSD can successfully boot on your VM with the patch applied I think we are safe to enable x2APIC. That one is called 'x2apic.patch'. Let me know how the testing goes. Thanks, Roger. --s7dTevK8xyWDIcXS Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="mni-fix.patch" >From 8a4a13210d7e55b65c164ed5d0a9d9e4e6f9b593 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= <royger@FreeBSD.org> Date: Thu, 13 Jan 2022 14:48:14 +0100 Subject: [PATCH] x86/xen: workaround for VCPUOP_send_nmi failing It has been reported that on some AWS instances VCPUOP_send_nmi returns -38 (ENOSYS). Add a fallback to use the native NMI sending procedure when that happens, so that the NMI is not lost. Reported by: avg MFC after: 1 week --- sys/x86/xen/xen_apic.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c index b553e5248716..701598e44f4e 100644 --- a/sys/x86/xen/xen_apic.c +++ b/sys/x86/xen/xen_apic.c @@ -107,6 +107,12 @@ static struct xen_ipi_handler xen_ipis[] = }; #endif +/* + * Save old handler as a fallback. It has been reported some AWS system don't + * support VCPUOP_send_nmi and return ENOSYS. + */ +void (*native_ipi_vectored)(u_int, int); + /*------------------------------- Per-CPU Data -------------------------------*/ #ifdef SMP DPCPU_DEFINE(xen_intr_handle_t, ipi_handle[nitems(xen_ipis)]); @@ -273,10 +279,11 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest) } #define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field) -static void +static int send_nmi(int dest) { unsigned int cpu; + int rc = 0; /* * NMIs are not routed over event channels, and instead delivered as on @@ -286,24 +293,30 @@ send_nmi(int dest) */ switch(dest) { case APIC_IPI_DEST_SELF: - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); break; case APIC_IPI_DEST_ALL: CPU_FOREACH(cpu) - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + rc |= HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(cpu, vcpu_id), NULL); break; case APIC_IPI_DEST_OTHERS: CPU_FOREACH(cpu) if (cpu != PCPU_GET(cpuid)) - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + rc |= HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(cpu, vcpu_id), NULL); break; default: - HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(apic_cpuid(dest), vcpu_id), NULL); break; } + + /* + * NB: for the loops above it's expected that all calls to + * VCPUOP_send_nmi will either fail or succeed. + */ + return rc; } #undef PCPU_ID_GET @@ -314,7 +327,8 @@ xen_pv_lapic_ipi_vectored(u_int vector, int dest) int ipi_idx, to_cpu, self; if (vector >= IPI_NMI_FIRST) { - send_nmi(dest); + if (send_nmi(dest)) + native_ipi_vectored(vector, dest); return; } @@ -580,8 +594,8 @@ xen_setup_cpus(void) xen_cpu_ipi_init(i); /* Set the xen pv ipi ops to replace the native ones */ - if (xen_hvm_domain()) - apic_ops.ipi_vectored = xen_pv_lapic_ipi_vectored; + native_ipi_vectored = apic_ops.ipi_vectored; + apic_ops.ipi_vectored = xen_pv_lapic_ipi_vectored; } /* Switch to using PV IPIs as soon as the vcpu_id is set. */ -- 2.34.1 --s7dTevK8xyWDIcXS Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="x2apic.patch" >From d2c5cbe40836f9784307ca072f030365269586f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= <royger@FreeBSD.org> Date: Thu, 13 Jan 2022 14:54:21 +0100 Subject: [PATCH] x86/madt: allow Xen guest to use x2APIC mode The old bogus Xen versions that would deliver a GPF when writing to the LAPIC MSR are likely retired, so it's safe to enable x2APIC unconditionally now if available. --- sys/x86/acpica/madt.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/sys/x86/acpica/madt.c b/sys/x86/acpica/madt.c index 8e343a6619f3..880857a8cfad 100644 --- a/sys/x86/acpica/madt.c +++ b/sys/x86/acpica/madt.c @@ -168,9 +168,6 @@ madt_x2apic_disable_reason(void) return ("inside VMWare without intr redirection"); } - if (vm_guest == VM_GUEST_XEN) - return ("due to running under XEN"); - if (vm_guest == VM_GUEST_NO && CPUID_TO_FAMILY(cpu_id) == 0x6 && CPUID_TO_MODEL(cpu_id) == 0x2a) { -- 2.34.1 --s7dTevK8xyWDIcXS--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YeA9wDPWQ%2BAJH1aN>