Date: Thu, 13 Jan 2022 15:57:04 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= <roger.pau@citrix.com> To: Andriy Gapon <avg@FreeBSD.org> Cc: <xen@freebsd.org> Subject: Re: VCPUOP_send_nmi returns -38 Message-ID: <YeA9wDPWQ%2BAJH1aN@Air-de-Roger> In-Reply-To: <787faaf6-c639-0f6a-1a16-c4fb7a5506e5@FreeBSD.org> References: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> <a27fb9ea-1078-fbbd-3dd9-9ce60de08c22@FreeBSD.org> <Yd1bR6mSAjL6VMYD@Air-de-Roger> <787faaf6-c639-0f6a-1a16-c4fb7a5506e5@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--s7dTevK8xyWDIcXS
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
On Tue, Jan 11, 2022 at 12:28:01PM +0200, Andriy Gapon wrote:
> On 11/01/2022 12:26, Roger Pau Monné wrote:
> > On Tue, Jan 11, 2022 at 12:07:28PM +0200, Andriy Gapon wrote:
> > > On 11/01/2022 11:50, Andriy Gapon wrote:
> > > >
> > > > Recently I got a report of crashes related to using procstat -k on one
> > > > of our systems. The system runs FreeBSD 12.2 on an AWS Xen-based
> > > > instance (can get more specifics about it later).
> > >
> > > The instance type is t2.large.
> > > Here are all lines from verbose boot that mention Xen:
> > >
> > > XEN: Hypervisor version 4.2 detected.
> > > Disabling MSI-X interrupt migration due to Xen hypervisor bug.
> > > XEN: disabling emulated disks
> > > XEN: disabling emulated nics
> > > Hypervisor: Origin = "XenVMMXenVMM"
> > > x2APIC available but disabled due to running under XEN
> >
> > Thanks. I've asked someone at AWS for comments. I guess the only way
> > to workaround this is to switch back to using the emulated APIC. I
> > will try to prepare a patch later.
>
> Thank you very much!
I've got two patches for you to try, the first one is 'mni-fix.patch'
and should fix the NMI sending issue.
I'm also attaching a second patch, that should allow your guest to use
x2APIC. FreeBSD has previously prevented Xen guests from using x2APIC
because of a bug in old Xen versions, but I think this should be
solved now, so if FreeBSD can successfully boot on your VM with the
patch applied I think we are safe to enable x2APIC. That one is called
'x2apic.patch'.
Let me know how the testing goes.
Thanks, Roger.
--s7dTevK8xyWDIcXS
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename="mni-fix.patch"
>From 8a4a13210d7e55b65c164ed5d0a9d9e4e6f9b593 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= <royger@FreeBSD.org>
Date: Thu, 13 Jan 2022 14:48:14 +0100
Subject: [PATCH] x86/xen: workaround for VCPUOP_send_nmi failing
It has been reported that on some AWS instances VCPUOP_send_nmi
returns -38 (ENOSYS). Add a fallback to use the native NMI sending
procedure when that happens, so that the NMI is not lost.
Reported by: avg
MFC after: 1 week
---
sys/x86/xen/xen_apic.c | 30 ++++++++++++++++++++++--------
1 file changed, 22 insertions(+), 8 deletions(-)
diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c
index b553e5248716..701598e44f4e 100644
--- a/sys/x86/xen/xen_apic.c
+++ b/sys/x86/xen/xen_apic.c
@@ -107,6 +107,12 @@ static struct xen_ipi_handler xen_ipis[] =
};
#endif
+/*
+ * Save old handler as a fallback. It has been reported some AWS system don't
+ * support VCPUOP_send_nmi and return ENOSYS.
+ */
+void (*native_ipi_vectored)(u_int, int);
+
/*------------------------------- Per-CPU Data -------------------------------*/
#ifdef SMP
DPCPU_DEFINE(xen_intr_handle_t, ipi_handle[nitems(xen_ipis)]);
@@ -273,10 +279,11 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest)
}
#define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field)
-static void
+static int
send_nmi(int dest)
{
unsigned int cpu;
+ int rc = 0;
/*
* NMIs are not routed over event channels, and instead delivered as on
@@ -286,24 +293,30 @@ send_nmi(int dest)
*/
switch(dest) {
case APIC_IPI_DEST_SELF:
- HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL);
+ rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL);
break;
case APIC_IPI_DEST_ALL:
CPU_FOREACH(cpu)
- HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
+ rc |= HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
PCPU_ID_GET(cpu, vcpu_id), NULL);
break;
case APIC_IPI_DEST_OTHERS:
CPU_FOREACH(cpu)
if (cpu != PCPU_GET(cpuid))
- HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
+ rc |= HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
PCPU_ID_GET(cpu, vcpu_id), NULL);
break;
default:
- HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
+ rc = HYPERVISOR_vcpu_op(VCPUOP_send_nmi,
PCPU_ID_GET(apic_cpuid(dest), vcpu_id), NULL);
break;
}
+
+ /*
+ * NB: for the loops above it's expected that all calls to
+ * VCPUOP_send_nmi will either fail or succeed.
+ */
+ return rc;
}
#undef PCPU_ID_GET
@@ -314,7 +327,8 @@ xen_pv_lapic_ipi_vectored(u_int vector, int dest)
int ipi_idx, to_cpu, self;
if (vector >= IPI_NMI_FIRST) {
- send_nmi(dest);
+ if (send_nmi(dest))
+ native_ipi_vectored(vector, dest);
return;
}
@@ -580,8 +594,8 @@ xen_setup_cpus(void)
xen_cpu_ipi_init(i);
/* Set the xen pv ipi ops to replace the native ones */
- if (xen_hvm_domain())
- apic_ops.ipi_vectored = xen_pv_lapic_ipi_vectored;
+ native_ipi_vectored = apic_ops.ipi_vectored;
+ apic_ops.ipi_vectored = xen_pv_lapic_ipi_vectored;
}
/* Switch to using PV IPIs as soon as the vcpu_id is set. */
--
2.34.1
--s7dTevK8xyWDIcXS
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename="x2apic.patch"
>From d2c5cbe40836f9784307ca072f030365269586f7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= <royger@FreeBSD.org>
Date: Thu, 13 Jan 2022 14:54:21 +0100
Subject: [PATCH] x86/madt: allow Xen guest to use x2APIC mode
The old bogus Xen versions that would deliver a GPF when writing to
the LAPIC MSR are likely retired, so it's safe to enable x2APIC
unconditionally now if available.
---
sys/x86/acpica/madt.c | 3 ---
1 file changed, 3 deletions(-)
diff --git a/sys/x86/acpica/madt.c b/sys/x86/acpica/madt.c
index 8e343a6619f3..880857a8cfad 100644
--- a/sys/x86/acpica/madt.c
+++ b/sys/x86/acpica/madt.c
@@ -168,9 +168,6 @@ madt_x2apic_disable_reason(void)
return ("inside VMWare without intr redirection");
}
- if (vm_guest == VM_GUEST_XEN)
- return ("due to running under XEN");
-
if (vm_guest == VM_GUEST_NO &&
CPUID_TO_FAMILY(cpu_id) == 0x6 &&
CPUID_TO_MODEL(cpu_id) == 0x2a) {
--
2.34.1
--s7dTevK8xyWDIcXS--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YeA9wDPWQ%2BAJH1aN>