From owner-freebsd-xen@freebsd.org Fri Nov 8 14:01:00 2019 Return-Path: Delivered-To: freebsd-xen@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1AA8D1B3386 for ; Fri, 8 Nov 2019 14:01:00 +0000 (UTC) (envelope-from roger.pau@citrix.com) Received: from esa4.hc3370-68.iphmx.com (esa4.hc3370-68.iphmx.com [216.71.155.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "esa4.hc3370-68.iphmx.com", Issuer "HydrantID SSL ICA G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 478hkg5SK3z4P05; Fri, 8 Nov 2019 14:00:59 +0000 (UTC) (envelope-from roger.pau@citrix.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1573221659; h=date:from:to:cc:subject:message-id:references: mime-version:content-transfer-encoding:in-reply-to; bh=3UeKj+UIKN5+9af1AK2WbLLDuX/65FLNlUDYXTLjKRU=; b=QdpxKKVlUvlof5JwRmuK+jvsKBLS/MMSqw04+5n/nvcloAUT271Qnnxq KFMBFHO2ucw22kFKKSEX94NwgM0JleqOwSwieqDs6N2wBDuvdafW0a+bJ J2FqIwLj6NKfrRHPzgzQCOaxqpWL2+WP+I5DXjPThDQwZ/+RIu2gpIHOj U=; Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa4.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa4.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa4.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: OK8jDI/oaOgxpkylYPOrL9Roi8UAfYokLrTa3Wyz7XLZjKjMoHg3qxahEaMtDKlUUo6Z4tqLHu snFYlFriQTAuVDtJNlNHU057OSzohq38tYBfYs/iJQR6JIa+VgkUivXoUdCazbgQTHDB/AAFKX 5xunCr4/EqeTKTvYFNXSZWD6pz0www7UdOpGN8JHVOk7TxLkibBJDc7SVMD9GTF2CNQSWJuGZC Np8eU3f0BzjaGKFbsc8rAIdaFogf/Wmx6F3y4W9SVOPbBYthmdwma+MKqtLyW//8GJxz+0/6+x alo= X-SBRS: 2.7 X-MesageID: 8573630 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.68,281,1569297600"; d="scan'208";a="8573630" Date: Fri, 8 Nov 2019 15:00:49 +0100 From: Roger Pau =?iso-8859-1?Q?Monn=E9?= To: Andriy Gapon CC: Subject: Re: Xen (HVM) and NMI Message-ID: <20191108140049.GW14005@Air-de-Roger> References: <62c12d1e-658c-93fc-fad8-3d97d45b4dd1@FreeBSD.org> <20191108095754.GV14005@Air-de-Roger> <760f4fe2-8a5b-87a7-f84f-f250bb4eb31c@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <760f4fe2-8a5b-87a7-f84f-f250bb4eb31c@FreeBSD.org> User-Agent: Mutt/1.12.2 (2019-09-21) X-ClientProxiedBy: AMSPEX02CAS02.citrite.net (10.69.22.113) To AMSPEX02CL03.citrite.net (10.69.22.127) X-Rspamd-Queue-Id: 478hkg5SK3z4P05 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-5.99 / 15.00]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Nov 2019 14:01:00 -0000 On Fri, Nov 08, 2019 at 03:15:40PM +0200, Andriy Gapon wrote: > On 08/11/2019 11:57, Roger Pau Monné wrote: > > On Fri, Nov 08, 2019 at 10:19:01AM +0200, Andriy Gapon wrote: > >>> I found this in Linux code: > >>> HYPERVISOR_vcpu_op(VCPUOP_send_nmi, xen_vcpu_nr(cpu), NULL); > >>> It's in xen_send_IPI_one(). > >>> I wonder if that's that or if there is more to this than meets the eye. > > > > Yes, something along this lines should work, we could even use the > > native NMI signaling using the local APIC, but the hypercall shortcut > > should be faster in most cases. > > > >> I also found this in an old post. > >> Ian Campbell wrote: > >>> You need to register a callback with CALLBACKOP_register > >>> CALLBACKTYPE_nmi. You also need to write the code in entry.S to receive > >>> that callback. IIRC you also need to arrange that returning from an NMI > >>> is always done with HYPERVISOR_iret and not optimised to a direct iret > >>> as it can be otherwise. This is to allow the hypervisor to implement NMI > >>> masking correctly. > > > > That's AFAIK for PV guests which use a completely different mechanism > > in order to receive interrupts. None of this is needed for FreeBSD > > because there's no classic PV support. > > Perhaps HYPERVISOR_iret is still needed even for us? No, that's just for classic PV. > This is said with zero knowledge of the topic. > > > Can you try the patch below? > > I tried it and, unfortunately, it was not a success. And I don't have much > diagnostic. My bad, that's what happens when you send untested patches, the previous patch was missing an apic_cpuid and was hitting a page-fault in the kernel. Patch below should work fine. Roger. ---8<--- diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c index 7d254ef3f734..8bf2158dbec2 100644 --- a/sys/x86/xen/xen_apic.c +++ b/sys/x86/xen/xen_apic.c @@ -72,7 +72,6 @@ static driver_filter_t xen_invlcache; static driver_filter_t xen_ipi_bitmap_handler; static driver_filter_t xen_cpustop_handler; static driver_filter_t xen_cpususpend_handler; -static driver_filter_t xen_cpustophard_handler; #endif /*---------------------------------- Macros ----------------------------------*/ @@ -96,7 +95,6 @@ static struct xen_ipi_handler xen_ipis[] = [IPI_TO_IDX(IPI_BITMAP_VECTOR)] = { xen_ipi_bitmap_handler, "b" }, [IPI_TO_IDX(IPI_STOP)] = { xen_cpustop_handler, "st" }, [IPI_TO_IDX(IPI_SUSPEND)] = { xen_cpususpend_handler, "sp" }, - [IPI_TO_IDX(IPI_STOP_HARD)] = { xen_cpustophard_handler, "sth" }, }; #endif @@ -259,12 +257,52 @@ xen_pv_lapic_ipi_raw(register_t icrlo, u_int dest) XEN_APIC_UNSUPPORTED; } +#define PCPU_ID_GET(id, field) (pcpu_find(id)->pc_##field) +static void +send_nmi(int dest) +{ + unsigned int cpu; + + /* + * NMIs are not routed over event channels, and instead delivered as on + * native using the exception vector (#2). Triggering them can be done + * using the local APIC, or an hypercall as a shortcut like it's done + * below. + */ + switch(dest) { + case APIC_IPI_DEST_SELF: + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, PCPU_GET(vcpu_id), NULL); + break; + case APIC_IPI_DEST_ALL: + CPU_FOREACH(cpu) + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + PCPU_ID_GET(cpu, vcpu_id), NULL); + break; + case APIC_IPI_DEST_OTHERS: + CPU_FOREACH(cpu) + if (cpu != PCPU_GET(cpuid)) + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + PCPU_ID_GET(cpu, vcpu_id), NULL); + break; + default: + HYPERVISOR_vcpu_op(VCPUOP_send_nmi, + PCPU_ID_GET(apic_cpuid(dest), vcpu_id), NULL); + break; + } +} +#undef PCPU_ID_GET + static void xen_pv_lapic_ipi_vectored(u_int vector, int dest) { xen_intr_handle_t *ipi_handle; int ipi_idx, to_cpu, self; + if (vector >= IPI_NMI_FIRST) { + send_nmi(dest); + return; + } + ipi_idx = IPI_TO_IDX(vector); if (ipi_idx >= nitems(xen_ipis)) panic("IPI out of range"); @@ -522,14 +560,6 @@ xen_cpususpend_handler(void *arg) return (FILTER_HANDLED); } -static int -xen_cpustophard_handler(void *arg) -{ - - ipi_nmi_handler(); - return (FILTER_HANDLED); -} - /*----------------------------- XEN PV IPI setup -----------------------------*/ /* * Those functions are provided outside of the Xen PV APIC implementation