From owner-freebsd-xen@freebsd.org Fri Nov 8 09:58:03 2019 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8A87A1AD421 for ; Fri, 8 Nov 2019 09:58:03 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa2.hc3370-68.iphmx.com (esa2.hc3370-68.iphmx.com [216.71.145.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa2.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 478bLM1jsmz480d; Fri, 8 Nov 2019 09:58:02 +0000 (UTC) (envelope-from roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1573207083; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=eWNaOUNbSLq1KV0tLVKLg/k29eTMRE2PS+T7kW82Kfw=; b=HmJtRgK4P9q/oljcF2kY4EHW9udXHYC7PYsNpajB4auDHWg5LwSlWPvk Lywk3mD0vN20MYIQuFemLMw1e2C9vA4snILqWS+VmWG8BOgQmJgovanml E1ATWEs/t+xPP/C6U/xBOOSvHjOLOoonJ5l1+tVEx/L6pS6vaaaRqRYrx 4=; Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: 0Sv6LbfvmYX7pHomwrE6A7Gk+4+eq0JTe15Q+TdKe6l5oOrPNrdXn3m67NOSkCBqBhI1WF//fK 2ZhbgiPdAybizDchmgtdlS7Iu86F3tVLxPRc0txfhLc2yfhDBgqRzdwgwgu0G4OvqmNadDfvbV MUsdPiwrRhO9ykw+T50USQ0rr0s2GIuPHGHWBZEMeylTPM5tAZmic5QrAUo4I6ULpz0NmSfSwl 8hVdMC0JM1BQLvT9Zjx2grE0IzVGzo4cJCBSSWUpnUkbPXrNPf17BEPEYRt4SLZcIiD0f8IGbb jOI= X-SBRS: 2.7 X-MesageID: 8052046 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.68,281,1569297600"; d="scan'208";a="8052046" Date: Fri, 8 Nov 2019 10:57:54 +0100 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Andriy Gapon CC: Subject: Re: Xen (HVM) and NMI Message-ID: <20191108095754.GV14005@Air-de-Roger> References: <62c12d1e-658c-93fc-fad8-3d97d45b4dd1@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-ClientProxiedBy: AMSPEX02CAS01.citrite.net (10.69.22.112) To AMSPEX02CL03.citrite.net (10.69.22.127) X-Rspamd-Queue-Id: 478bLM1jsmz480d X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-5.99 / 15.00]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Nov 2019 09:58:03 -0000 On Fri, Nov 08, 2019 at 10:19:01AM +0200, Andriy Gapon wrote: > On 08/11/2019 10:03, Andriy Gapon wrote: > > On 07/11/2019 20:08, Andriy Gapon wrote: > >> For CPUs that do get interrupted I see stack traces like: > >> cpustop_handler+0x28 ipi_nmi_handler+0x44 xen_cpustophard_handler+0x9 > >> intr_event_handle+0x8b intr_execute_handlers+0x58 xen_intr_handle_upcall+0x15a > >> xen_intr_upcall_u+0x96 ... > >> So, it looks like the NMI is delivered by the same mechanism as normal > >> interrupts. If a processor has interrupts disabled then the NMI would not get > >> delivered? Yes, sorry, I certainly didn't code this correctly regarding NMI handling. > >> Is there anything we could do to improve this? > > > > I found this in Linux code: > > HYPERVISOR_vcpu_op(VCPUOP_send_nmi, xen_vcpu_nr(cpu), NULL); > > It's in xen_send_IPI_one(). > > I wonder if that's that or if there is more to this than meets the eye. Yes, something along this lines should work, we could even use the native NMI signaling using the local APIC, but the hypercall shortcut should be faster in most cases. > I also found this in an old post. > Ian Campbell wrote: > > You need to register a callback with CALLBACKOP_register > > CALLBACKTYPE_nmi. You also need to write the code in entry.S to receive > > that callback. IIRC you also need to arrange that returning from an NMI > > is always done with HYPERVISOR_iret and not optimised to a direct iret > > as it can be otherwise. This is to allow the hypervisor to implement NMI > > masking correctly. That's AFAIK for PV guests which use a completely different mechanism in order to receive interrupts. None of this is needed for FreeBSD because there's no classic PV support. Can you try the patch below? Roger. ---8<--- diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c index 7d254ef3f734..7a6964d60cdb 100644 --- a/sys/x86/xen/xen_apic.c +++ b/sys/x86/xen/xen_apic.c @@ -72,7 +72,6 @@ static driver_filter_t xen_invlcache; static driver_filter_t xen_ipi_bitmap_handler; static driver_filter_t xen_cpustop_handler; static driver_filter_t xen_cpususpend_handler; -static driver_filter_t xen_cpustophard_handler; #endif /*---------------------------------- Macros ----------------------------------*/ @@ -96,7 +95,6 @@ static struct xen_ipi_handler xen_ipis[] = [IPI_TO_IDX(IPI_BITMAP_VECTOR)] = { xen_ipi_bitmap_handler, "b" }, [IPI_TO_IDX(IPI_STOP)] = { xen_cpustop_handler, "st" }, [IPI_TO_IDX(IPI_SUSPEND)] = { xen_cpususpend_handler, "sp" }, - [IPI_TO_IDX(IPI_STOP_HARD)] = { xen_cpustophard_handler, "sth" }, }; #endif @@ -259,12 +257,52 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest) XEN_APIC_UNSUPPORTED; } +#define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field) +static void +send_nmi(int dest) +{ + unsigned int cpu; + + /* + * NMIs are not routed over event channels, and instead delivered as on + * native using the exception vector (#2). Triggering them can be done + * using the local APIC, or an hypercall as a shortcut like it's done + * below. + */ + switch(dest) { + case APIC_IPI_DEST_SELF: + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); + break; + case APIC_IPI_DEST_ALL: + CPU_FOREACH(cpu) + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + PCPU_ID_GET(cpu, vcpu_id), NULL); + break; + case APIC_IPI_DEST_OTHERS: + CPU_FOREACH(cpu) + if (cpu != PCPU_GET(cpuid)) + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + PCPU_ID_GET(cpu, vcpu_id), NULL); + break; + default: + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_ID_GET(dest, vcpu_id), + NULL); + break; + } +} +#undef PCPU_ID_GET + static void xen_pv_lapic_ipi_vectored(u_int vector, int dest) { xen_intr_handle_t *ipi_handle; int ipi_idx, to_cpu, self; + if (vector >= IPI_NMI_FIRST) { + send_nmi(dest); + return; + } + ipi_idx = IPI_TO_IDX(vector); if (ipi_idx >= nitems(xen_ipis)) panic("IPI out of range"); @@ -522,14 +560,6 @@ xen_cpususpend_handler(void *arg) return (FILTER_HANDLED); } -static int -xen_cpustophard_handler(void *arg) -{ - - ipi_nmi_handler(); - return (FILTER_HANDLED); -} - /*----------------------------- XEN PV IPI setup -----------------------------*/ /* * Those functions are provided outside of the Xen PV APIC implementation