From owner-freebsd-xen@FreeBSD.ORG Mon Jan 13 11:06:57 2014 Return-Path: Delivered-To: freebsd-xen@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F35B76F9 for ; Mon, 13 Jan 2014 11:06:56 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D3C231166 for ; Mon, 13 Jan 2014 11:06:56 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id s0DB6urU096069 for ; Mon, 13 Jan 2014 11:06:56 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id s0DB6uei096067 for freebsd-xen@FreeBSD.org; Mon, 13 Jan 2014 11:06:56 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 13 Jan 2014 11:06:56 GMT Message-Id: <201401131106.s0DB6uei096067@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-xen@FreeBSD.org Subject: Current problem reports assigned to freebsd-xen@FreeBSD.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Jan 2014 11:06:57 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/183139 xen [xen] [patch] ifconfig options on xn0 lost after xen v o kern/180788 xen [xen] [panic] XEN PV kernel 9.2-BETA1 panics on boot o kern/180403 xen [xen] Problems with GENERIC and XENHVM kernels with Xe o kern/180402 xen [xen] XEN kernel does not load in XenClient 4.5.5 o kern/179814 xen [xen] mountroot fails with error=19 under Xen on 9-STA o kern/176471 xen [xen] xn driver crash on detach o kern/176053 xen [xen] [patch] i386: Correct wrong usage of vsnprintf() o kern/175954 xen [xen] XENHVM xn network driver extreme packet loss dur o kern/175822 xen [xen] FreeBSD 9.1 does not work with Xen 4.0 o kern/175757 xen [xen] [patch] xen pvhvm looses keyboard input from VNC o kern/171873 xen [xen] xn network device floods warning in dmesg o kern/171118 xen [xen] FreeBSD XENHVM guest doesn't shutdown cleanly o kern/166174 xen [xen] Problems ROOT MOUNT ERROR o kern/165418 xen [xen] Problems mounting root filesystem from XENHVM o kern/164630 xen [xen] XEN HVM kernel: run_interrupt_driven_hooks: stil o kern/164450 xen [xen] Failed to install FreeeBSD 9.0-RELEASE from CD i o kern/162677 xen [xen] FreeBSD not compatible with "Current Stable Xen" o kern/161318 xen [xen] sysinstall crashes with floating point exception o kern/155468 xen [xen] Xen PV i386 multi-kernel CPU system is not worki o kern/155353 xen [xen] [patch] put "nudging TOD" message under boot_ver o kern/154833 xen [xen]: xen 4.0 - DomU freebsd8.2RC3 i386, XEN kernel. o kern/154473 xen [xen] xen 4.0 - DomU freebsd8.1 i386, XEN kernel. Not o kern/154472 xen [xen] xen 4.0 - DomU freebsd8.1 i386 xen kernel reboot o kern/154428 xen [xen] xn0 network interface and PF - Massive performan o kern/153674 xen [xen] i386/XEN idle thread shows wrong percentages o kern/153672 xen [xen] [panic] i386/XEN panics under heavy fork load o kern/153620 xen [xen] Xen guest system clock drifts in AWS EC2 (FreeBS o kern/153477 xen [xen] XEN pmap code abuses vm page queue lock o kern/153150 xen [xen] xen/ec2: disable checksum offloading on interfac o kern/152228 xen [xen] [panic] Xen/PV panic with machdep.idle_mwait=1 o kern/144629 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143398 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143340 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor f kern/143069 xen [xen] [panic] Xen Kernel Panic - Memory modified after f kern/135667 xen ufs filesystem corruption on XEN DomU system f kern/135421 xen [xen] FreeBSD Xen PVM DomU network failure - netfronc. f kern/135178 xen [xen] Xen domU outgoing data transfer stall when TSO i p kern/135069 xen [xen] FreeBSD-current/Xen SMP doesn't function at all f i386/124516 xen [xen] FreeBSD-CURRENT Xen Kernel Segfaults when config o kern/118734 xen [xen] FreeBSD 6.3-RC1 and FreeBSD 7.0-BETA 4 fail to b 40 problems total. From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:00:09 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4EAFEE74; Tue, 14 Jan 2014 15:00:09 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BFFBC1DC1; Tue, 14 Jan 2014 15:00:07 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92694647" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 14:59:47 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 09:59:46 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T0-0006J6-BW; Tue, 14 Jan 2014 14:59:46 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 02/20] xen: add macro to detect if running as Dom0 Date: Tue, 14 Jan 2014 15:59:24 +0100 Message-ID: <1389711582-66908-3-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:00:09 -0000 --- sys/xen/xen-os.h | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/sys/xen/xen-os.h b/sys/xen/xen-os.h index c7474d8..e8a5a99 100644 --- a/sys/xen/xen-os.h +++ b/sys/xen/xen-os.h @@ -82,6 +82,13 @@ xen_hvm_domain(void) return (xen_domain_type == XEN_HVM_DOMAIN); } +static inline int +xen_initial_domain(void) +{ + return (xen_domain() && HYPERVISOR_start_info && + HYPERVISOR_start_info->flags & SIF_INITDOMAIN); +} + #ifndef xen_mb #define xen_mb() mb() #endif -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:00:11 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ED09FEEF; Tue, 14 Jan 2014 15:00:10 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 681AA1DC2; Tue, 14 Jan 2014 15:00:09 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92694666" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 14:59:48 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 09:59:47 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T1-0006J6-TZ; Tue, 14 Jan 2014 14:59:47 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 05/20] xen: rework xen timer so it can be used early in boot process Date: Tue, 14 Jan 2014 15:59:27 +0100 Message-ID: <1389711582-66908-6-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:00:11 -0000 This should not introduce any functional change, and makes the functions suitable to be called before we have actually mapped the vcpu_info struct on a per-cpu basis. --- sys/dev/xen/timer/timer.c | 29 ++++++++++++++++++++--------- 1 files changed, 20 insertions(+), 9 deletions(-) diff --git a/sys/dev/xen/timer/timer.c b/sys/dev/xen/timer/timer.c index 354085b..b2f6bcd 100644 --- a/sys/dev/xen/timer/timer.c +++ b/sys/dev/xen/timer/timer.c @@ -230,22 +230,22 @@ xen_fetch_vcpu_tinfo(struct vcpu_time_info *dst, struct vcpu_time_info *src) /** * \brief Get the current time, in nanoseconds, since the hypervisor booted. * + * \param vcpu vcpu_info structure to fetch the time from. + * * \note This function returns the current CPU's idea of this value, unless * it happens to be less than another CPU's previously determined value. */ static uint64_t -xen_fetch_vcpu_time(void) +xen_fetch_vcpu_time(struct vcpu_info *vcpu) { struct vcpu_time_info dst; struct vcpu_time_info *src; uint32_t pre_version; uint64_t now; volatile uint64_t last; - struct vcpu_info *vcpu = DPCPU_GET(vcpu_info); src = &vcpu->time; - critical_enter(); do { pre_version = xen_fetch_vcpu_tinfo(&dst, src); barrier(); @@ -266,16 +266,19 @@ xen_fetch_vcpu_time(void) } } while (!atomic_cmpset_64(&xen_timer_last_time, last, now)); - critical_exit(); - return (now); } static uint32_t xentimer_get_timecount(struct timecounter *tc) { + uint32_t xen_time; - return ((uint32_t)xen_fetch_vcpu_time() & UINT_MAX); + critical_enter(); + xen_time = (uint32_t)xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)) & UINT_MAX; + critical_exit(); + + return (xen_time); } /** @@ -305,7 +308,12 @@ xen_fetch_wallclock(struct timespec *ts) static void xen_fetch_uptime(struct timespec *ts) { - uint64_t uptime = xen_fetch_vcpu_time(); + uint64_t uptime; + + critical_enter(); + uptime = xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)); + critical_exit(); + ts->tv_sec = uptime / NSEC_IN_SEC; ts->tv_nsec = uptime % NSEC_IN_SEC; } @@ -354,7 +362,7 @@ xentimer_intr(void *arg) struct xentimer_softc *sc = (struct xentimer_softc *)arg; struct xentimer_pcpu_data *pcpu = DPCPU_PTR(xentimer_pcpu); - pcpu->last_processed = xen_fetch_vcpu_time(); + pcpu->last_processed = xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)); if (pcpu->timer != 0 && sc->et.et_active) sc->et.et_event_cb(&sc->et, sc->et.et_arg); @@ -415,7 +423,10 @@ xentimer_et_start(struct eventtimer *et, do { if (++i == 60) panic("can't schedule timer"); - next_time = xen_fetch_vcpu_time() + first_in_ns; + critical_enter(); + next_time = xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)) + + first_in_ns; + critical_exit(); error = xentimer_vcpu_start_timer(cpu, next_time); } while (error == -ETIME); -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:00:07 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5C6D5DED; Tue, 14 Jan 2014 15:00:07 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D1C391DBE; Tue, 14 Jan 2014 15:00:05 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92694623" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 14:59:45 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 09:59:45 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35Sz-0006J6-7G; Tue, 14 Jan 2014 14:59:45 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 00/20] FreeBSD PVH DomU support Date: Tue, 14 Jan 2014 15:59:22 +0100 Message-ID: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:00:07 -0000 This series is a split of the previous patch "Xen x86 DomU PVH support", with the aim to make the review of the code easier. The series can also be found on my git repo: git://xenbits.xen.org/people/royger/freebsd.git pvh_v10 or http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/pvh_v10 PVH mode is basically a PV guest inside an HVM container, and shares a great amount of code with PVHVM. The main difference is the way the guest is started, PVH uses the PV start sequence, jumping directly into the kernel entry point in long mode and with page tables set. The main work of this patch consists in setting the environment as similar as possible to what native FreeBSD expects, and then adding hooks to the PV ops when necessary. This new version of the series (v10) addresses the comments from the previous posted version (v9). Major changes between v9 and v10: * Add a identify routine to xenpv instead of attaching it manually from the Xen nexus. * Remove bus routines from xenpci (devices are now attached to xenpv instead). * Add __printflike modifier to xc_printf. From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:00:09 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3136FE4F; Tue, 14 Jan 2014 15:00:09 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5C0EC1DC0; Tue, 14 Jan 2014 15:00:07 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92694644" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 14:59:46 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 09:59:45 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35Sz-0006J6-OP; Tue, 14 Jan 2014 14:59:45 +0000 From: Roger Pau Monne To: , , , , , , Subject: =?UTF-8?q?=5BPATCH=20v10=2001/20=5D=20xen=3A=20add=20PV/PVH=20kernel=20entry=20point?= Date: Tue, 14 Jan 2014 15:59:23 +0100 Message-ID: <1389711582-66908-2-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:00:09 -0000 Add the PV/PVH entry point and the low level functions for PVH initialization. --- sys/amd64/amd64/locore.S | 1 + sys/amd64/amd64/xen-locore.S | 83 +++++++++++++++++++++++++++++ sys/amd64/include/asmacros.h | 26 +++++++++ sys/conf/files.amd64 | 2 + sys/i386/xen/xen_machdep.c | 2 + sys/x86/xen/hvm.c | 1 + sys/x86/xen/pv.c | 119 ++++++++++++++++++++++++++++++++++++++++++ sys/xen/xen-os.h | 4 ++ 8 files changed, 238 insertions(+), 0 deletions(-) create mode 100644 sys/amd64/amd64/xen-locore.S create mode 100644 sys/x86/xen/pv.c diff --git a/sys/amd64/amd64/locore.S b/sys/amd64/amd64/locore.S index 55cda3a..4acef97 100644 --- a/sys/amd64/amd64/locore.S +++ b/sys/amd64/amd64/locore.S @@ -84,5 +84,6 @@ NON_GPROF_ENTRY(btext) .bss ALIGN_DATA /* just to be sure */ + .globl bootstack .space 0x1000 /* space for bootstack - temporary stack */ bootstack: diff --git a/sys/amd64/amd64/xen-locore.S b/sys/amd64/amd64/xen-locore.S new file mode 100644 index 0000000..84287c4 --- /dev/null +++ b/sys/amd64/amd64/xen-locore.S @@ -0,0 +1,83 @@ +/*- + * Copyright (c) 2003 Peter Wemm + * Copyright (c) 2013 Roger Pau Monne + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#include +#include +#include +#include + +#include +#define __ASSEMBLY__ +#include + +#include "assym.s" + +.section __xen_guest + ELFNOTE(Xen, XEN_ELFNOTE_GUEST_OS, .asciz, "FreeBSD") + ELFNOTE(Xen, XEN_ELFNOTE_GUEST_VERSION, .asciz, "HEAD") + ELFNOTE(Xen, XEN_ELFNOTE_XEN_VERSION, .asciz, "xen-3.0") + ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE, .quad, KERNBASE) + ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET, .quad, KERNBASE) /* Xen honours elf->p_paddr; compensate for this */ + ELFNOTE(Xen, XEN_ELFNOTE_ENTRY, .quad, xen_start) + ELFNOTE(Xen, XEN_ELFNOTE_HYPERCALL_PAGE, .quad, hypercall_page) + ELFNOTE(Xen, XEN_ELFNOTE_HV_START_LOW, .quad, HYPERVISOR_VIRT_START) + ELFNOTE(Xen, XEN_ELFNOTE_FEATURES, .asciz, "writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel|hvm_callback_vector") + ELFNOTE(Xen, XEN_ELFNOTE_PAE_MODE, .asciz, "yes") + ELFNOTE(Xen, XEN_ELFNOTE_L1_MFN_VALID, .long, PG_V, PG_V) + ELFNOTE(Xen, XEN_ELFNOTE_LOADER, .asciz, "generic") + ELFNOTE(Xen, XEN_ELFNOTE_SUSPEND_CANCEL, .long, 0) + ELFNOTE(Xen, XEN_ELFNOTE_BSD_SYMTAB, .asciz, "yes") + + .text +.p2align PAGE_SHIFT, 0x90 /* Hypercall_page needs to be PAGE aligned */ + +NON_GPROF_ENTRY(hypercall_page) + .skip 0x1000, 0x90 /* Fill with "nop"s */ + +NON_GPROF_ENTRY(xen_start) + /* Don't trust what the loader gives for rflags. */ + pushq $PSL_KERNEL + popfq + + /* Parameters for the xen init function */ + movq %rsi, %rdi /* shared_info (arg 1) */ + movq %rsp, %rsi /* xenstack (arg 2) */ + + /* Use our own stack */ + movq $bootstack,%rsp + xorl %ebp, %ebp + + /* u_int64_t hammer_time_xen(start_info_t *si, u_int64_t xenstack); */ + call hammer_time_xen + movq %rax, %rsp /* set up kstack for mi_startup() */ + call mi_startup /* autoconfiguration, mountroot etc */ + + /* NOTREACHED */ +0: hlt + jmp 0b diff --git a/sys/amd64/include/asmacros.h b/sys/amd64/include/asmacros.h index 1fb592a..ce8dce4 100644 --- a/sys/amd64/include/asmacros.h +++ b/sys/amd64/include/asmacros.h @@ -201,4 +201,30 @@ #endif /* LOCORE */ +#ifdef __STDC__ +#define ELFNOTE(name, type, desctype, descdata...) \ +.pushsection .note.name ; \ + .align 4 ; \ + .long 2f - 1f /* namesz */ ; \ + .long 4f - 3f /* descsz */ ; \ + .long type ; \ +1:.asciz #name ; \ +2:.align 4 ; \ +3:desctype descdata ; \ +4:.align 4 ; \ +.popsection +#else /* !__STDC__, i.e. -traditional */ +#define ELFNOTE(name, type, desctype, descdata) \ +.pushsection .note.name ; \ + .align 4 ; \ + .long 2f - 1f /* namesz */ ; \ + .long 4f - 3f /* descsz */ ; \ + .long type ; \ +1:.asciz "name" ; \ +2:.align 4 ; \ +3:desctype descdata ; \ +4:.align 4 ; \ +.popsection +#endif /* __STDC__ */ + #endif /* !_MACHINE_ASMACROS_H_ */ diff --git a/sys/conf/files.amd64 b/sys/conf/files.amd64 index d1bdcd9..16029d8 100644 --- a/sys/conf/files.amd64 +++ b/sys/conf/files.amd64 @@ -119,6 +119,7 @@ amd64/amd64/in_cksum.c optional inet | inet6 amd64/amd64/initcpu.c standard amd64/amd64/io.c optional io amd64/amd64/locore.S standard no-obj +amd64/amd64/xen-locore.S optional xenhvm amd64/amd64/machdep.c standard amd64/amd64/mem.c optional mem amd64/amd64/minidump_machdep.c standard @@ -566,3 +567,4 @@ x86/x86/nexus.c standard x86/x86/tsc.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm +x86/xen/pv.c optional xenhvm diff --git a/sys/i386/xen/xen_machdep.c b/sys/i386/xen/xen_machdep.c index 7049be6..fd575ee 100644 --- a/sys/i386/xen/xen_machdep.c +++ b/sys/i386/xen/xen_machdep.c @@ -89,6 +89,7 @@ IDTVEC(div), IDTVEC(dbg), IDTVEC(nmi), IDTVEC(bpt), IDTVEC(ofl), int xendebug_flags; start_info_t *xen_start_info; +start_info_t *HYPERVISOR_start_info; shared_info_t *HYPERVISOR_shared_info; xen_pfn_t *xen_machine_phys = machine_to_phys_mapping; xen_pfn_t *xen_phys_machine; @@ -927,6 +928,7 @@ initvalues(start_info_t *startinfo) HYPERVISOR_vm_assist(VMASST_CMD_enable, VMASST_TYPE_4gb_segments_notify); #endif xen_start_info = startinfo; + HYPERVISOR_start_info = startinfo; xen_phys_machine = (xen_pfn_t *)startinfo->mfn_list; IdlePTD = (pd_entry_t *)((uint8_t *)startinfo->pt_base + PAGE_SIZE); diff --git a/sys/x86/xen/hvm.c b/sys/x86/xen/hvm.c index 72811dc..b397721 100644 --- a/sys/x86/xen/hvm.c +++ b/sys/x86/xen/hvm.c @@ -159,6 +159,7 @@ DPCPU_DEFINE(xen_intr_handle_t, ipi_handle[nitems(xen_ipis)]); /** Hypercall table accessed via HYPERVISOR_*_op() methods. */ char *hypercall_stubs; shared_info_t *HYPERVISOR_shared_info; +start_info_t *HYPERVISOR_start_info; #ifdef SMP /*---------------------------- XEN PV IPI Handlers ---------------------------*/ diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c new file mode 100644 index 0000000..5571ecf --- /dev/null +++ b/sys/x86/xen/pv.c @@ -0,0 +1,119 @@ +/* + * Copyright (c) 2004 Christian Limpach. + * Copyright (c) 2004-2006,2008 Kip Macy + * Copyright (c) 2013 Roger Pau Monné + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +/* Native initial function */ +extern u_int64_t hammer_time(u_int64_t, u_int64_t); +/* Xen initial function */ +extern u_int64_t hammer_time_xen(start_info_t *, u_int64_t); + +/* + * First function called by the Xen PVH boot sequence. + * + * Set some Xen global variables and prepare the environment so it is + * as similar as possible to what native FreeBSD init function expects. + */ +u_int64_t +hammer_time_xen(start_info_t *si, u_int64_t xenstack) +{ + u_int64_t physfree; + u_int64_t *PT4 = (u_int64_t *)xenstack; + u_int64_t *PT3 = (u_int64_t *)(xenstack + PAGE_SIZE); + u_int64_t *PT2 = (u_int64_t *)(xenstack + 2 * PAGE_SIZE); + int i; + + if ((si == NULL) || (xenstack == 0)) { + HYPERVISOR_shutdown(SHUTDOWN_crash); + } + + /* We use 3 pages of xen stack for the boot pagetables */ + physfree = xenstack + 3 * PAGE_SIZE - KERNBASE; + + /* Setup Xen global variables */ + HYPERVISOR_start_info = si; + HYPERVISOR_shared_info = + (shared_info_t *)(si->shared_info + KERNBASE); + + /* + * Setup some misc global variables for Xen devices + * + * XXX: devices that need this specific variables should + * be rewritten to fetch this info by themselves from the + * start_info page. + */ + xen_store = (struct xenstore_domain_interface *) + (ptoa(si->store_mfn) + KERNBASE); + + xen_domain_type = XEN_PV_DOMAIN; + vm_guest = VM_GUEST_XEN; + + /* + * Use the stack Xen gives us to build the page tables + * as native FreeBSD expects to find them (created + * by the boot trampoline). + */ + for (i = 0; i < 512; i++) { + /* Each slot of the level 4 pages points to the same level 3 page */ + PT4[i] = ((u_int64_t)&PT3[0]) - KERNBASE; + PT4[i] |= PG_V | PG_RW | PG_U; + + /* Each slot of the level 3 pages points to the same level 2 page */ + PT3[i] = ((u_int64_t)&PT2[0]) - KERNBASE; + PT3[i] |= PG_V | PG_RW | PG_U; + + /* The level 2 page slots are mapped with 2MB pages for 1GB. */ + PT2[i] = i * (2 * 1024 * 1024); + PT2[i] |= PG_V | PG_RW | PG_PS | PG_U; + } + load_cr3(((u_int64_t)&PT4[0]) - KERNBASE); + + /* Now we can jump into the native init function */ + return (hammer_time(0, physfree)); +} diff --git a/sys/xen/xen-os.h b/sys/xen/xen-os.h index 87644e9..c7474d8 100644 --- a/sys/xen/xen-os.h +++ b/sys/xen/xen-os.h @@ -51,6 +51,10 @@ void force_evtchn_callback(void); extern shared_info_t *HYPERVISOR_shared_info; +extern start_info_t *HYPERVISOR_start_info; + +/* XXX: we need to get rid of this and use HYPERVISOR_start_info directly */ +extern struct xenstore_domain_interface *xen_store; enum xen_domain_type { XEN_NATIVE, /* running on bare hardware */ -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:00:11 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7E6CDEF7; Tue, 14 Jan 2014 15:00:11 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AF9831DC4; Tue, 14 Jan 2014 15:00:09 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92694676" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 14:59:49 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 09:59:48 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T2-0006J6-Dt; Tue, 14 Jan 2014 14:59:48 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 06/20] xen: implement an early timer for Xen PVH Date: Tue, 14 Jan 2014 15:59:28 +0100 Message-ID: <1389711582-66908-7-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:00:11 -0000 When running as a PVH guest, there's no emulated i8254, so we need to use the Xen PV timer as the early source for DELAY. This change allows for different implementations of the early DELAY function and implements a Xen variant for it. --- sys/amd64/amd64/machdep.c | 6 ++- sys/amd64/include/clock.h | 5 ++ sys/amd64/include/sysarch.h | 2 + sys/conf/files.amd64 | 1 + sys/conf/files.i386 | 1 + sys/dev/xen/timer/timer.c | 33 +++++++++++++ sys/i386/include/clock.h | 5 ++ sys/x86/isa/clock.c | 53 +-------------------- sys/x86/x86/delay.c | 112 +++++++++++++++++++++++++++++++++++++++++++ sys/x86/xen/pv.c | 3 + 10 files changed, 167 insertions(+), 54 deletions(-) create mode 100644 sys/x86/x86/delay.c diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index 343f9b8..b8d6dc2 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -172,6 +172,8 @@ static caddr_t native_parse_preload_data(u_int64_t); /* Default init_ops implementation. */ struct init_ops init_ops = { .parse_preload_data = native_parse_preload_data, + .early_delay_init = i8254_init, + .early_delay = i8254_delay, }; /* @@ -1822,10 +1824,10 @@ hammer_time(u_int64_t modulep, u_int64_t physfree) lidt(&r_idt); /* - * Initialize the i8254 before the console so that console + * Initialize the early delay before the console so that console * initialization can use DELAY(). */ - i8254_init(); + init_ops.early_delay_init(); /* * Initialize the console before we print anything out. diff --git a/sys/amd64/include/clock.h b/sys/amd64/include/clock.h index d7f7d82..ac8818f 100644 --- a/sys/amd64/include/clock.h +++ b/sys/amd64/include/clock.h @@ -25,6 +25,11 @@ extern int smp_tsc; #endif void i8254_init(void); +void i8254_delay(int); +#ifdef XENHVM +void xen_delay_init(void); +void xen_delay(int); +#endif /* * Driver to clock driver interface. diff --git a/sys/amd64/include/sysarch.h b/sys/amd64/include/sysarch.h index 58ac8cd..60fa635 100644 --- a/sys/amd64/include/sysarch.h +++ b/sys/amd64/include/sysarch.h @@ -13,6 +13,8 @@ */ struct init_ops { caddr_t (*parse_preload_data)(u_int64_t); + void (*early_delay_init)(void); + void (*early_delay)(int); }; extern struct init_ops init_ops; diff --git a/sys/conf/files.amd64 b/sys/conf/files.amd64 index 16029d8..109a796 100644 --- a/sys/conf/files.amd64 +++ b/sys/conf/files.amd64 @@ -565,6 +565,7 @@ x86/x86/mptable_pci.c optional mptable pci x86/x86/msi.c optional pci x86/x86/nexus.c standard x86/x86/tsc.c standard +x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm x86/xen/pv.c optional xenhvm diff --git a/sys/conf/files.i386 b/sys/conf/files.i386 index eb8697c..790296d 100644 --- a/sys/conf/files.i386 +++ b/sys/conf/files.i386 @@ -600,5 +600,6 @@ x86/x86/mptable_pci.c optional apic native pci x86/x86/msi.c optional apic pci x86/x86/nexus.c standard x86/x86/tsc.c standard +x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm diff --git a/sys/dev/xen/timer/timer.c b/sys/dev/xen/timer/timer.c index b2f6bcd..96372ab 100644 --- a/sys/dev/xen/timer/timer.c +++ b/sys/dev/xen/timer/timer.c @@ -59,6 +59,9 @@ __FBSDID("$FreeBSD$"); #include #include +/* For the declaration of clock_lock */ +#include + #include "clock_if.h" static devclass_t xentimer_devclass; @@ -584,6 +587,36 @@ xentimer_suspend(device_t dev) return (0); } +/* + * Xen delay early init + */ +void xen_delay_init(void) +{ + /* Init the clock lock */ + mtx_init(&clock_lock, "clk", NULL, MTX_SPIN | MTX_NOPROFILE); +} +/* + * Xen PV DELAY function + * + * When running on PVH mode we don't have an emulated i8524, so + * make use of the Xen time info in order to code a simple DELAY + * function that can be used during early boot. + */ +void xen_delay(int n) +{ + uint64_t end_ns; + uint64_t current; + + end_ns = xen_fetch_vcpu_time(&HYPERVISOR_shared_info->vcpu_info[0]); + end_ns += n * NSEC_IN_USEC; + + for (;;) { + current = xen_fetch_vcpu_time(&HYPERVISOR_shared_info->vcpu_info[0]); + if (current >= end_ns) + break; + } +} + static device_method_t xentimer_methods[] = { DEVMETHOD(device_identify, xentimer_identify), DEVMETHOD(device_probe, xentimer_probe), diff --git a/sys/i386/include/clock.h b/sys/i386/include/clock.h index d980ec7..b831445 100644 --- a/sys/i386/include/clock.h +++ b/sys/i386/include/clock.h @@ -22,6 +22,11 @@ extern int tsc_is_invariant; extern int tsc_perf_stat; void i8254_init(void); +void i8254_delay(int); +#ifdef XENHVM +void xen_delay_init(void); +void xen_delay(int); +#endif /* * Driver to clock driver interface. diff --git a/sys/x86/isa/clock.c b/sys/x86/isa/clock.c index a12e175..a5aed1c 100644 --- a/sys/x86/isa/clock.c +++ b/sys/x86/isa/clock.c @@ -247,61 +247,13 @@ getit(void) return ((high << 8) | low); } -#ifndef DELAYDEBUG -static u_int -get_tsc(__unused struct timecounter *tc) -{ - - return (rdtsc32()); -} - -static __inline int -delay_tc(int n) -{ - struct timecounter *tc; - timecounter_get_t *func; - uint64_t end, freq, now; - u_int last, mask, u; - - tc = timecounter; - freq = atomic_load_acq_64(&tsc_freq); - if (tsc_is_invariant && freq != 0) { - func = get_tsc; - mask = ~0u; - } else { - if (tc->tc_quality <= 0) - return (0); - func = tc->tc_get_timecount; - mask = tc->tc_counter_mask; - freq = tc->tc_frequency; - } - now = 0; - end = freq * n / 1000000; - if (func == get_tsc) - sched_pin(); - last = func(tc) & mask; - do { - cpu_spinwait(); - u = func(tc) & mask; - if (u < last) - now += mask - last + u + 1; - else - now += u - last; - last = u; - } while (now < end); - if (func == get_tsc) - sched_unpin(); - return (1); -} -#endif - /* * Wait "n" microseconds. * Relies on timer 1 counting down from (i8254_freq / hz) * Note: timer had better have been programmed before this is first used! */ void -DELAY(int n) +i8254_delay(int n) { int delta, prev_tick, tick, ticks_left; #ifdef DELAYDEBUG @@ -317,9 +269,6 @@ DELAY(int n) } if (state == 1) printf("DELAY(%d)...", n); -#else - if (delay_tc(n)) - return; #endif /* * Read the counter first, so that the rest of the setup overhead is diff --git a/sys/x86/x86/delay.c b/sys/x86/x86/delay.c new file mode 100644 index 0000000..d13c727 --- /dev/null +++ b/sys/x86/x86/delay.c @@ -0,0 +1,112 @@ +/*- + * Copyright (c) 1990 The Regents of the University of California. + * Copyright (c) 2010 Alexander Motin + * All rights reserved. + * + * This code is derived from software contributed to Berkeley by + * William Jolitz and Don Ahn. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 4. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * from: @(#)clock.c 7.2 (Berkeley) 5/12/91 + */ + +#include +__FBSDID("$FreeBSD$"); + +/* Generic x86 routines to handle delay */ + +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +static u_int +get_tsc(__unused struct timecounter *tc) +{ + + return (rdtsc32()); +} + +static int +delay_tc(int n) +{ + struct timecounter *tc; + timecounter_get_t *func; + uint64_t end, freq, now; + u_int last, mask, u; + + tc = timecounter; + freq = atomic_load_acq_64(&tsc_freq); + if (tsc_is_invariant && freq != 0) { + func = get_tsc; + mask = ~0u; + } else { + if (tc->tc_quality <= 0) + return (0); + func = tc->tc_get_timecount; + mask = tc->tc_counter_mask; + freq = tc->tc_frequency; + } + now = 0; + end = freq * n / 1000000; + if (func == get_tsc) + sched_pin(); + last = func(tc) & mask; + do { + cpu_spinwait(); + u = func(tc) & mask; + if (u < last) + now += mask - last + u + 1; + else + now += u - last; + last = u; + } while (now < end); + if (func == get_tsc) + sched_unpin(); + return (1); +} + +#ifndef XEN +void +DELAY(int n) +{ + + if (delay_tc(n)) + return; + +#ifdef __amd64__ + init_ops.early_delay(n); +#else + i8254_delay(n); +#endif +} +#endif diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c index 908b50b..0ec4b54 100644 --- a/sys/x86/xen/pv.c +++ b/sys/x86/xen/pv.c @@ -47,6 +47,7 @@ __FBSDID("$FreeBSD$"); #include #include +#include #include #include @@ -65,6 +66,8 @@ static void xen_pv_set_init_ops(void); /* Xen init_ops implementation. */ struct init_ops xen_init_ops = { .parse_preload_data = xen_pv_parse_preload_data, + .early_delay_init = xen_delay_init, + .early_delay = xen_delay, }; static struct -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:00:11 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 50BA3EF3; Tue, 14 Jan 2014 15:00:11 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8D0B61DC3; Tue, 14 Jan 2014 15:00:09 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92694686" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 14:59:49 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.80) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 09:59:48 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T2-0006J6-V2; Tue, 14 Jan 2014 14:59:49 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 07/20] xen: implement hook to fetch e820 memory map Date: Tue, 14 Jan 2014 15:59:29 +0100 Message-ID: <1389711582-66908-8-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:00:11 -0000 --- sys/amd64/amd64/machdep.c | 50 ++++++++++++++++++++++++++---------------- sys/amd64/include/pc/bios.h | 2 + sys/amd64/include/sysarch.h | 1 + sys/x86/xen/pv.c | 25 +++++++++++++++++++++ 4 files changed, 59 insertions(+), 19 deletions(-) diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index b8d6dc2..64df89a 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -169,11 +169,15 @@ SYSINIT(cpu, SI_SUB_CPU, SI_ORDER_FIRST, cpu_startup, NULL); /* Preload data parse function */ static caddr_t native_parse_preload_data(u_int64_t); +/* Native function to fetch and parse the e820 map */ +static void native_parse_memmap(caddr_t, vm_paddr_t *, int *); + /* Default init_ops implementation. */ struct init_ops init_ops = { .parse_preload_data = native_parse_preload_data, .early_delay_init = i8254_init, .early_delay = i8254_delay, + .parse_memmap = native_parse_memmap, }; /* @@ -1403,21 +1407,12 @@ add_physmap_entry(uint64_t base, uint64_t length, vm_paddr_t *physmap, return (1); } -static void -add_smap_entries(struct bios_smap *smapbase, vm_paddr_t *physmap, - int *physmap_idx) +void +bios_add_smap_entries(struct bios_smap *smapbase, u_int32_t smapsize, + vm_paddr_t *physmap, int *physmap_idx) { struct bios_smap *smap, *smapend; - u_int32_t smapsize; - /* - * Memory map from INT 15:E820. - * - * subr_module.c says: - * "Consumer may safely assume that size value precedes data." - * ie: an int32_t immediately precedes smap. - */ - smapsize = *((u_int32_t *)smapbase - 1); smapend = (struct bios_smap *)((uintptr_t)smapbase + smapsize); for (smap = smapbase; smap < smapend; smap++) { @@ -1434,6 +1429,29 @@ add_smap_entries(struct bios_smap *smapbase, vm_paddr_t *physmap, } } +static void +native_parse_memmap(caddr_t kmdp, vm_paddr_t *physmap, int *physmap_idx) +{ + struct bios_smap *smap; + u_int32_t size; + + /* + * Memory map from INT 15:E820. + * + * subr_module.c says: + * "Consumer may safely assume that size value precedes data." + * ie: an int32_t immediately precedes smap. + */ + + smap = (struct bios_smap *)preload_search_info(kmdp, + MODINFO_METADATA | MODINFOMD_SMAP); + if (smap == NULL) + panic("No BIOS smap info from loader!"); + size = *((u_int32_t *)smap - 1); + + bios_add_smap_entries(smap, size, physmap, physmap_idx); +} + /* * Populate the (physmap) array with base/bound pairs describing the * available physical memory in the system, then test this memory and @@ -1451,19 +1469,13 @@ getmemsize(caddr_t kmdp, u_int64_t first) vm_paddr_t pa, physmap[PHYSMAP_SIZE]; u_long physmem_start, physmem_tunable, memtest; pt_entry_t *pte; - struct bios_smap *smapbase; quad_t dcons_addr, dcons_size; bzero(physmap, sizeof(physmap)); basemem = 0; physmap_idx = 0; - smapbase = (struct bios_smap *)preload_search_info(kmdp, - MODINFO_METADATA | MODINFOMD_SMAP); - if (smapbase == NULL) - panic("No BIOS smap info from loader!"); - - add_smap_entries(smapbase, physmap, &physmap_idx); + init_ops.parse_memmap(kmdp, physmap, &physmap_idx); /* * Find the 'base memory' segment for SMP diff --git a/sys/amd64/include/pc/bios.h b/sys/amd64/include/pc/bios.h index e7d568e..95ef703 100644 --- a/sys/amd64/include/pc/bios.h +++ b/sys/amd64/include/pc/bios.h @@ -106,6 +106,8 @@ struct bios_oem { int bios_oem_strings(struct bios_oem *oem, u_char *buffer, size_t maxlen); uint32_t bios_sigsearch(uint32_t start, u_char *sig, int siglen, int paralen, int sigofs); +void bios_add_smap_entries(struct bios_smap *smapbase, u_int32_t smapsize, + vm_paddr_t *physmap, int *physmap_idx); #endif #endif /* _MACHINE_PC_BIOS_H_ */ diff --git a/sys/amd64/include/sysarch.h b/sys/amd64/include/sysarch.h index 60fa635..084223e 100644 --- a/sys/amd64/include/sysarch.h +++ b/sys/amd64/include/sysarch.h @@ -15,6 +15,7 @@ struct init_ops { caddr_t (*parse_preload_data)(u_int64_t); void (*early_delay_init)(void); void (*early_delay)(int); + void (*parse_memmap)(caddr_t, vm_paddr_t *, int *); }; extern struct init_ops init_ops; diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c index 0ec4b54..d11bc1a 100644 --- a/sys/x86/xen/pv.c +++ b/sys/x86/xen/pv.c @@ -48,6 +48,7 @@ __FBSDID("$FreeBSD$"); #include #include +#include #include #include @@ -57,8 +58,11 @@ extern u_int64_t hammer_time(u_int64_t, u_int64_t); /* Xen initial function */ extern u_int64_t hammer_time_xen(start_info_t *, u_int64_t); +#define MAX_E820_ENTRIES 128 + /*--------------------------- Forward Declarations ---------------------------*/ static caddr_t xen_pv_parse_preload_data(u_int64_t); +static void xen_pv_parse_memmap(caddr_t, vm_paddr_t *, int *); static void xen_pv_set_init_ops(void); @@ -68,6 +72,7 @@ struct init_ops xen_init_ops = { .parse_preload_data = xen_pv_parse_preload_data, .early_delay_init = xen_delay_init, .early_delay = xen_delay, + .parse_memmap = xen_pv_parse_memmap, }; static struct @@ -88,6 +93,8 @@ static struct {NULL, 0} }; +static struct bios_smap xen_smap[MAX_E820_ENTRIES]; + /*-------------------------------- Xen PV init -------------------------------*/ /* * First function called by the Xen PVH boot sequence. @@ -201,6 +208,24 @@ xen_pv_parse_preload_data(u_int64_t modulep) } static void +xen_pv_parse_memmap(caddr_t kmdp, vm_paddr_t *physmap, int *physmap_idx) +{ + struct xen_memory_map memmap; + u_int32_t size; + int rc; + + /* Fetch the E820 map from Xen */ + memmap.nr_entries = MAX_E820_ENTRIES; + set_xen_guest_handle(memmap.buffer, xen_smap); + rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap); + if (rc) + panic("unable to fetch Xen E820 memory map"); + size = memmap.nr_entries * sizeof(xen_smap[0]); + + bios_add_smap_entries(xen_smap, size, physmap, physmap_idx); +} + +static void xen_pv_set_init_ops(void) { /* Init ops for Xen PV */ -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:00:13 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C6957FF7; Tue, 14 Jan 2014 15:00:13 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5B8E51DC6; Tue, 14 Jan 2014 15:00:11 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92694698" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 14:59:50 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 09:59:49 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T3-0006J6-Fw; Tue, 14 Jan 2014 14:59:49 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 08/20] xen: use the same hypercall mechanism for XEN and XENHVM Date: Tue, 14 Jan 2014 15:59:30 +0100 Message-ID: <1389711582-66908-9-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:00:13 -0000 --- sys/amd64/include/xen/hypercall.h | 7 ------- sys/i386/i386/locore.s | 9 +++++++++ sys/i386/include/xen/hypercall.h | 8 -------- sys/x86/xen/hvm.c | 24 ++++++++++-------------- 4 files changed, 19 insertions(+), 29 deletions(-) diff --git a/sys/amd64/include/xen/hypercall.h b/sys/amd64/include/xen/hypercall.h index a1b2a5c..499fb4d 100644 --- a/sys/amd64/include/xen/hypercall.h +++ b/sys/amd64/include/xen/hypercall.h @@ -51,15 +51,8 @@ #define CONFIG_XEN_COMPAT 0x030002 #define __must_check -#ifdef XEN #define HYPERCALL_STR(name) \ "call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)" -#else -#define HYPERCALL_STR(name) \ - "mov $("STR(__HYPERVISOR_##name)" * 32),%%eax; "\ - "add hypercall_stubs(%%rip),%%rax; " \ - "call *%%rax" -#endif #define _hypercall0(type, name) \ ({ \ diff --git a/sys/i386/i386/locore.s b/sys/i386/i386/locore.s index 68cb430..bd136b1 100644 --- a/sys/i386/i386/locore.s +++ b/sys/i386/i386/locore.s @@ -898,3 +898,12 @@ done_pde: #endif ret + +#ifdef XENHVM +/* Xen Hypercall page */ + .text +.p2align PAGE_SHIFT, 0x90 /* Hypercall_page needs to be PAGE aligned */ + +NON_GPROF_ENTRY(hypercall_page) + .skip 0x1000, 0x90 /* Fill with "nop"s */ +#endif diff --git a/sys/i386/include/xen/hypercall.h b/sys/i386/include/xen/hypercall.h index edc13f4..16b5ee2 100644 --- a/sys/i386/include/xen/hypercall.h +++ b/sys/i386/include/xen/hypercall.h @@ -39,16 +39,8 @@ #define ENOXENSYS 38 #define CONFIG_XEN_COMPAT 0x030002 - -#if defined(XEN) #define HYPERCALL_STR(name) \ "call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)" -#else -#define HYPERCALL_STR(name) \ - "mov hypercall_stubs,%%eax; " \ - "add $("STR(__HYPERVISOR_##name)" * 32),%%eax; " \ - "call *%%eax" -#endif #define _hypercall0(type, name) \ ({ \ diff --git a/sys/x86/xen/hvm.c b/sys/x86/xen/hvm.c index b397721..9a0411e 100644 --- a/sys/x86/xen/hvm.c +++ b/sys/x86/xen/hvm.c @@ -157,7 +157,7 @@ DPCPU_DEFINE(xen_intr_handle_t, ipi_handle[nitems(xen_ipis)]); /*------------------ Hypervisor Access Shared Memory Regions -----------------*/ /** Hypercall table accessed via HYPERVISOR_*_op() methods. */ -char *hypercall_stubs; +extern char *hypercall_page; shared_info_t *HYPERVISOR_shared_info; start_info_t *HYPERVISOR_start_info; @@ -559,7 +559,7 @@ xen_hvm_cpuid_base(void) * Allocate and fill in the hypcall page. */ static int -xen_hvm_init_hypercall_stubs(void) +xen_hvm_init_hypercall_stubs(enum xen_hvm_init_type init_type) { uint32_t base, regs[4]; int i; @@ -568,7 +568,7 @@ xen_hvm_init_hypercall_stubs(void) if (base == 0) return (ENXIO); - if (hypercall_stubs == NULL) { + if (init_type == XEN_HVM_INIT_COLD) { do_cpuid(base + 1, regs); printf("XEN: Hypervisor version %d.%d detected.\n", regs[0] >> 16, regs[0] & 0xffff); @@ -578,18 +578,9 @@ xen_hvm_init_hypercall_stubs(void) * Find the hypercall pages. */ do_cpuid(base + 2, regs); - - if (hypercall_stubs == NULL) { - size_t call_region_size; - - call_region_size = regs[0] * PAGE_SIZE; - hypercall_stubs = malloc(call_region_size, M_XENHVM, M_NOWAIT); - if (hypercall_stubs == NULL) - panic("Unable to allocate Xen hypercall region"); - } for (i = 0; i < regs[0]; i++) - wrmsr(regs[1], vtophys(hypercall_stubs + i * PAGE_SIZE) + i); + wrmsr(regs[1], vtophys(&hypercall_page + i * PAGE_SIZE) + i); return (0); } @@ -692,7 +683,12 @@ xen_hvm_init(enum xen_hvm_init_type init_type) if (init_type == XEN_HVM_INIT_CANCELLED_SUSPEND) return; - error = xen_hvm_init_hypercall_stubs(); + if (xen_pv_domain()) { + /* hypercall page is already set in the PV case */ + error = 0; + } else { + error = xen_hvm_init_hypercall_stubs(init_type); + } switch (init_type) { case XEN_HVM_INIT_COLD: -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:01:51 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 78930797; Tue, 14 Jan 2014 15:01:51 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A4DAD1E75; Tue, 14 Jan 2014 15:01:49 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="90575047" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 14 Jan 2014 14:59:47 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 09:59:46 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T0-0006J6-SE; Tue, 14 Jan 2014 14:59:46 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 03/20] xen: add and enable Xen console for PVH guests Date: Tue, 14 Jan 2014 15:59:25 +0100 Message-ID: <1389711582-66908-4-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:01:51 -0000 This adds and enables the console used on XEN kernels. --- sys/conf/files | 4 +- sys/dev/xen/console/console.c | 37 +++++++++++++++++++++++++++++------ sys/dev/xen/console/xencons_ring.c | 15 +++++++++---- sys/i386/include/xen/xen-os.h | 1 - sys/i386/xen/xen_machdep.c | 17 ---------------- sys/x86/xen/pv.c | 4 +++ sys/xen/xen-os.h | 4 +++ 7 files changed, 50 insertions(+), 32 deletions(-) diff --git a/sys/conf/files b/sys/conf/files index 33fc75d..bddf021 100644 --- a/sys/conf/files +++ b/sys/conf/files @@ -2493,8 +2493,8 @@ dev/xe/if_xe_pccard.c optional xe pccard dev/xen/balloon/balloon.c optional xen | xenhvm dev/xen/blkfront/blkfront.c optional xen | xenhvm dev/xen/blkback/blkback.c optional xen | xenhvm -dev/xen/console/console.c optional xen -dev/xen/console/xencons_ring.c optional xen +dev/xen/console/console.c optional xen | xenhvm +dev/xen/console/xencons_ring.c optional xen | xenhvm dev/xen/control/control.c optional xen | xenhvm dev/xen/netback/netback.c optional xen | xenhvm dev/xen/netfront/netfront.c optional xen | xenhvm diff --git a/sys/dev/xen/console/console.c b/sys/dev/xen/console/console.c index 23eaee2..899dffc 100644 --- a/sys/dev/xen/console/console.c +++ b/sys/dev/xen/console/console.c @@ -69,11 +69,14 @@ struct mtx cn_mtx; static char wbuf[WBUF_SIZE]; static char rbuf[RBUF_SIZE]; static int rc, rp; -static unsigned int cnsl_evt_reg; +unsigned int cnsl_evt_reg; static unsigned int wc, wp; /* write_cons, write_prod */ xen_intr_handle_t xen_intr_handle; device_t xencons_dev; +/* Virtual address of the shared console page */ +char *console_page; + #ifdef KDB static int xc_altbrk; #endif @@ -110,9 +113,26 @@ static struct ttydevsw xc_ttydevsw = { .tsw_outwakeup = xcoutwakeup, }; +/*----------------------------- Debug function -------------------------------*/ +#define XC_PRINTF_BUFSIZE 1024 +void +xc_printf(const char *fmt, ...) +{ + static char buf[XC_PRINTF_BUFSIZE]; + __va_list ap; + + va_start(ap, fmt); + vsnprintf(buf, sizeof(buf), fmt, ap); + va_end(ap); + HYPERVISOR_console_write(buf, strlen(buf)); +} + static void xc_cnprobe(struct consdev *cp) { + if (!xen_pv_domain()) + return; + cp->cn_pri = CN_REMOTE; sprintf(cp->cn_name, "%s0", driver_name); } @@ -175,7 +195,7 @@ static void xc_cnputc(struct consdev *dev, int c) { - if (xen_start_info->flags & SIF_INITDOMAIN) + if (xen_initial_domain()) xc_cnputc_dom0(dev, c); else xc_cnputc_domu(dev, c); @@ -206,8 +226,7 @@ xcons_putc(int c) xcons_force_flush(); #endif } - if (cnsl_evt_reg) - __xencons_tx_flush(); + __xencons_tx_flush(); /* inform start path that we're pretty full */ return ((wp - wc) >= WBUF_SIZE - 100) ? TRUE : FALSE; @@ -217,6 +236,10 @@ static void xc_identify(driver_t *driver, device_t parent) { device_t child; + + if (!xen_pv_domain()) + return; + child = BUS_ADD_CHILD(parent, 0, driver_name, 0); device_set_driver(child, driver); device_set_desc(child, "Xen Console"); @@ -245,7 +268,7 @@ xc_attach(device_t dev) cnsl_evt_reg = 1; callout_reset(&xc_callout, XC_POLLTIME, xc_timeout, xccons); - if (xen_start_info->flags & SIF_INITDOMAIN) { + if (xen_initial_domain()) { error = xen_intr_bind_virq(dev, VIRQ_CONSOLE, 0, NULL, xencons_priv_interrupt, NULL, INTR_TYPE_TTY, &xen_intr_handle); @@ -309,7 +332,7 @@ __xencons_tx_flush(void) sz = wp - wc; if (sz > (WBUF_SIZE - WBUF_MASK(wc))) sz = WBUF_SIZE - WBUF_MASK(wc); - if (xen_start_info->flags & SIF_INITDOMAIN) { + if (xen_initial_domain()) { HYPERVISOR_console_io(CONSOLEIO_write, sz, &wbuf[WBUF_MASK(wc)]); wc += sz; } else { @@ -424,7 +447,7 @@ xcons_force_flush(void) { int sz; - if (xen_start_info->flags & SIF_INITDOMAIN) + if (xen_initial_domain()) return; /* Spin until console data is flushed through to the domain controller. */ diff --git a/sys/dev/xen/console/xencons_ring.c b/sys/dev/xen/console/xencons_ring.c index 3701551..d826363 100644 --- a/sys/dev/xen/console/xencons_ring.c +++ b/sys/dev/xen/console/xencons_ring.c @@ -32,9 +32,9 @@ __FBSDID("$FreeBSD$"); #define console_evtchn console.domU.evtchn xen_intr_handle_t console_handle; -extern char *console_page; extern struct mtx cn_mtx; extern device_t xencons_dev; +extern int cnsl_evt_reg; static inline struct xencons_interface * xencons_interface(void) @@ -60,6 +60,8 @@ xencons_ring_send(const char *data, unsigned len) struct xencons_interface *intf; XENCONS_RING_IDX cons, prod; int sent; + struct evtchn_send send = { .port = + HYPERVISOR_start_info->console_evtchn }; intf = xencons_interface(); cons = intf->out_cons; @@ -76,7 +78,10 @@ xencons_ring_send(const char *data, unsigned len) wmb(); intf->out_prod = prod; - xen_intr_signal(console_handle); + if (cnsl_evt_reg) + xen_intr_signal(console_handle); + else + HYPERVISOR_event_channel_op(EVTCHNOP_send, &send); return sent; @@ -125,11 +130,11 @@ xencons_ring_init(void) { int err; - if (!xen_start_info->console_evtchn) + if (!HYPERVISOR_start_info->console_evtchn) return 0; err = xen_intr_bind_local_port(xencons_dev, - xen_start_info->console_evtchn, NULL, xencons_handle_input, NULL, + HYPERVISOR_start_info->console_evtchn, NULL, xencons_handle_input, NULL, INTR_TYPE_MISC | INTR_MPSAFE, &console_handle); if (err) { return err; @@ -145,7 +150,7 @@ void xencons_suspend(void) { - if (!xen_start_info->console_evtchn) + if (!HYPERVISOR_start_info->console_evtchn) return; xen_intr_unbind(&console_handle); diff --git a/sys/i386/include/xen/xen-os.h b/sys/i386/include/xen/xen-os.h index a8fba61..3d1ef04 100644 --- a/sys/i386/include/xen/xen-os.h +++ b/sys/i386/include/xen/xen-os.h @@ -45,7 +45,6 @@ static inline void rep_nop(void) #define cpu_relax() rep_nop() #ifndef XENHVM -void xc_printf(const char *fmt, ...); #ifdef SMP extern int gdtset; diff --git a/sys/i386/xen/xen_machdep.c b/sys/i386/xen/xen_machdep.c index fd575ee..09c01f1 100644 --- a/sys/i386/xen/xen_machdep.c +++ b/sys/i386/xen/xen_machdep.c @@ -186,21 +186,6 @@ xen_boothowto(char *envp) return howto; } -#define XC_PRINTF_BUFSIZE 1024 -void -xc_printf(const char *fmt, ...) -{ - __va_list ap; - int retval; - static char buf[XC_PRINTF_BUFSIZE]; - - va_start(ap, fmt); - retval = vsnprintf(buf, XC_PRINTF_BUFSIZE - 1, fmt, ap); - va_end(ap); - buf[retval] = 0; - (void)HYPERVISOR_console_write(buf, retval); -} - #define XPQUEUE_SIZE 128 @@ -745,8 +730,6 @@ void initvalues(start_info_t *startinfo); struct xenstore_domain_interface; extern struct xenstore_domain_interface *xen_store; -char *console_page; - void * bootmem_alloc(unsigned int size) { diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c index 5571ecf..db3b7a3 100644 --- a/sys/x86/xen/pv.c +++ b/sys/x86/xen/pv.c @@ -70,9 +70,12 @@ hammer_time_xen(start_info_t *si, u_int64_t xenstack) int i; if ((si == NULL) || (xenstack == 0)) { + xc_printf("ERROR: invalid start_info or xen stack, halting\n"); HYPERVISOR_shutdown(SHUTDOWN_crash); } + xc_printf("FreeBSD PVH running on %s\n", si->magic); + /* We use 3 pages of xen stack for the boot pagetables */ physfree = xenstack + 3 * PAGE_SIZE - KERNBASE; @@ -90,6 +93,7 @@ hammer_time_xen(start_info_t *si, u_int64_t xenstack) */ xen_store = (struct xenstore_domain_interface *) (ptoa(si->store_mfn) + KERNBASE); + console_page = (char *)(ptoa(si->console.domU.mfn) + KERNBASE); xen_domain_type = XEN_PV_DOMAIN; vm_guest = VM_GUEST_XEN; diff --git a/sys/xen/xen-os.h b/sys/xen/xen-os.h index e8a5a99..b1aa0a9 100644 --- a/sys/xen/xen-os.h +++ b/sys/xen/xen-os.h @@ -55,6 +55,7 @@ extern start_info_t *HYPERVISOR_start_info; /* XXX: we need to get rid of this and use HYPERVISOR_start_info directly */ extern struct xenstore_domain_interface *xen_store; +extern char *console_page; enum xen_domain_type { XEN_NATIVE, /* running on bare hardware */ @@ -89,6 +90,9 @@ xen_initial_domain(void) HYPERVISOR_start_info->flags & SIF_INITDOMAIN); } +/* Debug function, prints directly to hypervisor console */ +void xc_printf(const char *, ...) __printflike(1, 2); + #ifndef xen_mb #define xen_mb() mb() #endif -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:01:57 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D16DB81D; Tue, 14 Jan 2014 15:01:57 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 763451E7B; Tue, 14 Jan 2014 15:01:56 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="90575051" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 14 Jan 2014 14:59:48 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 09:59:47 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T1-0006J6-Ck; Tue, 14 Jan 2014 14:59:47 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 04/20] amd64: introduce hook for custom preload metadata parsers Date: Tue, 14 Jan 2014 15:59:26 +0100 Message-ID: <1389711582-66908-5-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:01:57 -0000 --- sys/amd64/amd64/machdep.c | 41 ++++++++++++++++------ sys/amd64/include/sysarch.h | 12 ++++++ sys/x86/xen/pv.c | 82 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 124 insertions(+), 11 deletions(-) diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index f0d4ea8..343f9b8 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -126,6 +126,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #ifdef PERFMON #include #endif @@ -165,6 +166,14 @@ static int set_fpcontext(struct thread *td, const mcontext_t *mcp, char *xfpustate, size_t xfpustate_len); SYSINIT(cpu, SI_SUB_CPU, SI_ORDER_FIRST, cpu_startup, NULL); +/* Preload data parse function */ +static caddr_t native_parse_preload_data(u_int64_t); + +/* Default init_ops implementation. */ +struct init_ops init_ops = { + .parse_preload_data = native_parse_preload_data, +}; + /* * The file "conf/ldscript.amd64" defines the symbol "kernphys". Its value is * the physical address at which the kernel is loaded. @@ -1685,6 +1694,26 @@ do_next: msgbufp = (struct msgbuf *)PHYS_TO_DMAP(phys_avail[pa_indx]); } +static caddr_t +native_parse_preload_data(u_int64_t modulep) +{ + caddr_t kmdp; + + preload_metadata = (caddr_t)(uintptr_t)(modulep + KERNBASE); + preload_bootstrap_relocate(KERNBASE); + kmdp = preload_search_by_type("elf kernel"); + if (kmdp == NULL) + kmdp = preload_search_by_type("elf64 kernel"); + boothowto = MD_FETCH(kmdp, MODINFOMD_HOWTO, int); + kern_envp = MD_FETCH(kmdp, MODINFOMD_ENVP, char *) + KERNBASE; +#ifdef DDB + ksym_start = MD_FETCH(kmdp, MODINFOMD_SSYM, uintptr_t); + ksym_end = MD_FETCH(kmdp, MODINFOMD_ESYM, uintptr_t); +#endif + + return (kmdp); +} + u_int64_t hammer_time(u_int64_t modulep, u_int64_t physfree) { @@ -1709,17 +1738,7 @@ hammer_time(u_int64_t modulep, u_int64_t physfree) */ proc_linkup0(&proc0, &thread0); - preload_metadata = (caddr_t)(uintptr_t)(modulep + KERNBASE); - preload_bootstrap_relocate(KERNBASE); - kmdp = preload_search_by_type("elf kernel"); - if (kmdp == NULL) - kmdp = preload_search_by_type("elf64 kernel"); - boothowto = MD_FETCH(kmdp, MODINFOMD_HOWTO, int); - kern_envp = MD_FETCH(kmdp, MODINFOMD_ENVP, char *) + KERNBASE; -#ifdef DDB - ksym_start = MD_FETCH(kmdp, MODINFOMD_SSYM, uintptr_t); - ksym_end = MD_FETCH(kmdp, MODINFOMD_ESYM, uintptr_t); -#endif + kmdp = init_ops.parse_preload_data(modulep); /* Init basic tunables, hz etc */ init_param1(); diff --git a/sys/amd64/include/sysarch.h b/sys/amd64/include/sysarch.h index cd380d4..58ac8cd 100644 --- a/sys/amd64/include/sysarch.h +++ b/sys/amd64/include/sysarch.h @@ -4,3 +4,15 @@ /* $FreeBSD$ */ #include + +/* + * Struct containing pointers to init functions whose + * implementation is run time selectable. Selection can be made, + * for example, based on detection of a BIOS variant or + * hypervisor environment. + */ +struct init_ops { + caddr_t (*parse_preload_data)(u_int64_t); +}; + +extern struct init_ops init_ops; diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c index db3b7a3..908b50b 100644 --- a/sys/x86/xen/pv.c +++ b/sys/x86/xen/pv.c @@ -46,6 +46,8 @@ __FBSDID("$FreeBSD$"); #include #include +#include + #include #include @@ -54,6 +56,36 @@ extern u_int64_t hammer_time(u_int64_t, u_int64_t); /* Xen initial function */ extern u_int64_t hammer_time_xen(start_info_t *, u_int64_t); +/*--------------------------- Forward Declarations ---------------------------*/ +static caddr_t xen_pv_parse_preload_data(u_int64_t); + +static void xen_pv_set_init_ops(void); + +/*-------------------------------- Global Data -------------------------------*/ +/* Xen init_ops implementation. */ +struct init_ops xen_init_ops = { + .parse_preload_data = xen_pv_parse_preload_data, +}; + +static struct +{ + const char *ev; + int mask; +} howto_names[] = { + {"boot_askname", RB_ASKNAME}, + {"boot_single", RB_SINGLE}, + {"boot_nosync", RB_NOSYNC}, + {"boot_halt", RB_ASKNAME}, + {"boot_serial", RB_SERIAL}, + {"boot_cdrom", RB_CDROM}, + {"boot_gdb", RB_GDB}, + {"boot_gdb_pause", RB_RESERVED1}, + {"boot_verbose", RB_VERBOSE}, + {"boot_multicons", RB_MULTIPLE}, + {NULL, 0} +}; + +/*-------------------------------- Xen PV init -------------------------------*/ /* * First function called by the Xen PVH boot sequence. * @@ -118,6 +150,56 @@ hammer_time_xen(start_info_t *si, u_int64_t xenstack) } load_cr3(((u_int64_t)&PT4[0]) - KERNBASE); + /* Set the hooks for early functions that diverge from bare metal */ + xen_pv_set_init_ops(); + /* Now we can jump into the native init function */ return (hammer_time(0, physfree)); } + +/*-------------------------------- PV specific -------------------------------*/ +/* + * Functions to convert the "extra" parameters passed by Xen + * into FreeBSD boot options (from the i386 Xen port). + */ +static char * +xen_setbootenv(char *cmd_line) +{ + char *cmd_line_next; + + /* Skip leading spaces */ + for (; *cmd_line == ' '; cmd_line++); + + for (cmd_line_next = cmd_line; strsep(&cmd_line_next, ",") != NULL;); + return (cmd_line); +} + +static int +xen_boothowto(char *envp) +{ + int i, howto = 0; + + /* get equivalents from the environment */ + for (i = 0; howto_names[i].ev != NULL; i++) + if (getenv(howto_names[i].ev) != NULL) + howto |= howto_names[i].mask; + return (howto); +} + +static caddr_t +xen_pv_parse_preload_data(u_int64_t modulep) +{ + /* Parse the extra boot information given by Xen */ + if (HYPERVISOR_start_info->cmd_line) + kern_envp = xen_setbootenv(HYPERVISOR_start_info->cmd_line); + boothowto |= xen_boothowto(kern_envp); + + return (NULL); +} + +static void +xen_pv_set_init_ops(void) +{ + /* Init ops for Xen PV */ + init_ops = xen_init_ops; +} -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:01:49 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 33EF86EC; Tue, 14 Jan 2014 15:01:49 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id AD0D41E71; Tue, 14 Jan 2014 15:01:47 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="90575064" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 14 Jan 2014 14:59:50 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.80) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 09:59:49 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T4-0006J6-0S; Tue, 14 Jan 2014 14:59:50 +0000 From: Roger Pau Monne To: , , , , , , Subject: =?UTF-8?q?=5BPATCH=20v10=2009/20=5D=20xen=3A=20add=20a=20apic=5Fenumerator=20for=20PVH?= Date: Tue, 14 Jan 2014 15:59:31 +0100 Message-ID: <1389711582-66908-10-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:01:49 -0000 --- sys/conf/files.amd64 | 1 + sys/x86/xen/pvcpu_enum.c | 136 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+), 0 deletions(-) create mode 100644 sys/x86/xen/pvcpu_enum.c diff --git a/sys/conf/files.amd64 b/sys/conf/files.amd64 index 109a796..a3491da 100644 --- a/sys/conf/files.amd64 +++ b/sys/conf/files.amd64 @@ -569,3 +569,4 @@ x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm x86/xen/pv.c optional xenhvm +x86/xen/pvcpu_enum.c optional xenhvm diff --git a/sys/x86/xen/pvcpu_enum.c b/sys/x86/xen/pvcpu_enum.c new file mode 100644 index 0000000..0384886 --- /dev/null +++ b/sys/x86/xen/pvcpu_enum.c @@ -0,0 +1,136 @@ +/*- + * Copyright (c) 2003 John Baldwin + * Copyright (c) 2013 Roger Pau Monné + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the author nor the names of any co-contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include + +#include +#include + +#include + +static int xenpv_probe(void); +static int xenpv_probe_cpus(void); +static int xenpv_setup_local(void); +static int xenpv_setup_io(void); + +static struct apic_enumerator xenpv_enumerator = { + "Xen PV", + xenpv_probe, + xenpv_probe_cpus, + xenpv_setup_local, + xenpv_setup_io +}; + +/* + * This enumerator will only be registered on PVH + */ +static int +xenpv_probe(void) +{ + return (-100); +} + +/* + * Test each possible vCPU in order to find the number of vCPUs + */ +static int +xenpv_probe_cpus(void) +{ +#ifdef SMP + int i, ret; + + for (i = 0; i < MAXCPU; i++) { + ret = HYPERVISOR_vcpu_op(VCPUOP_is_up, i, NULL); + if (ret >= 0) + cpu_add((i * 2), (i == 0)); + } +#endif + return (0); +} + +/* + * Initialize the vCPU id of the BSP + */ +static int +xenpv_setup_local(void) +{ + PCPU_SET(vcpu_id, 0); + return (0); +} + +/* + * On PVH guests there's no IO APIC + */ +static int +xenpv_setup_io(void) +{ + return (0); +} + +static void +xenpv_register(void *dummy __unused) +{ + if (xen_pv_domain()) { + apic_register_enumerator(&xenpv_enumerator); + } +} +SYSINIT(xenpv_register, SI_SUB_TUNABLES - 1, SI_ORDER_FIRST, xenpv_register, NULL); + +/* + * Setup per-CPU vCPU IDs + */ +static void +xenpv_set_ids(void *dummy) +{ + struct pcpu *pc; + int i; + + CPU_FOREACH(i) { + pc = pcpu_find(i); + pc->pc_vcpu_id = i; + } +} +SYSINIT(xenpv_set_ids, SI_SUB_CPU, SI_ORDER_MIDDLE, xenpv_set_ids, NULL); -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:25:25 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A4BC3616; Tue, 14 Jan 2014 15:25:25 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 288061089; Tue, 14 Jan 2014 15:25:23 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92709101" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 15:25:21 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:25:21 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T6-0006J6-L4; Tue, 14 Jan 2014 14:59:52 +0000 From: Roger Pau Monne To: , , , , , , Subject: =?UTF-8?q?=5BPATCH=20v10=2014/20=5D=20xen=3A=20introduce=20xenpv=20bus=20and=20a=20dummy=20pvcpu=20device?= Date: Tue, 14 Jan 2014 15:59:36 +0100 Message-ID: <1389711582-66908-15-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:25:25 -0000 Since Xen PVH guests doesn't have ACPI, we need to create a dummy bus so top level Xen devices can attach to it (instead of attaching directly to the nexus) and a pvcpu device that will be used to fill the pcpu->pc_device field. --- sys/conf/files.amd64 | 1 + sys/conf/files.i386 | 1 + sys/x86/xen/xenpv.c | 128 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 130 insertions(+), 0 deletions(-) create mode 100644 sys/x86/xen/xenpv.c diff --git a/sys/conf/files.amd64 b/sys/conf/files.amd64 index a3491da..d7c98cc 100644 --- a/sys/conf/files.amd64 +++ b/sys/conf/files.amd64 @@ -570,3 +570,4 @@ x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm x86/xen/pv.c optional xenhvm x86/xen/pvcpu_enum.c optional xenhvm +x86/xen/xenpv.c optional xenhvm diff --git a/sys/conf/files.i386 b/sys/conf/files.i386 index 790296d..81142e3 100644 --- a/sys/conf/files.i386 +++ b/sys/conf/files.i386 @@ -603,3 +603,4 @@ x86/x86/tsc.c standard x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm +x86/xen/xenpv.c optional xen | xenhvm diff --git a/sys/x86/xen/xenpv.c b/sys/x86/xen/xenpv.c new file mode 100644 index 0000000..e1282cf --- /dev/null +++ b/sys/x86/xen/xenpv.c @@ -0,0 +1,128 @@ +/* + * Copyright (c) 2013 Roger Pau Monné + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include + +static devclass_t xenpv_devclass; + +static void +xenpv_identify(driver_t *driver, device_t parent) +{ + if (!xen_domain()) + return; + + /* Make sure there's only one xenpv device. */ + if (devclass_get_device(xenpv_devclass, 0)) + return; + + /* + * Use a high order number so xenpv is attached after + * xenpci on HVM guests. + */ + BUS_ADD_CHILD(parent, 200, "xenpv", 0); +} + +static int +xenpv_probe(device_t dev) +{ + + device_set_desc(dev, "Xen PV bus"); + device_quiet(dev); + return (BUS_PROBE_NOWILDCARD); +} + +static int +xenpv_attach(device_t dev) +{ + device_t child; + + if (xen_hvm_domain()) { + device_t xenpci; + devclass_t dc; + + /* Make sure xenpci has been attached */ + dc = devclass_find("xenpci"); + if (dc == NULL) + panic("unable to find xenpci devclass"); + + xenpci = devclass_get_device(dc, 0); + if (xenpci == NULL) + panic("unable to find xenpci device"); + + if (!device_is_attached(xenpci)) + panic("trying to attach xenpv before xenpci"); + } + + /* + * Let our child drivers identify any child devices that they + * can find. Once that is done attach any devices that we + * found. + */ + bus_generic_probe(dev); + bus_generic_attach(dev); + + if (!devclass_get_device(devclass_find("isa"), 0)) { + child = BUS_ADD_CHILD(dev, 0, "isa", 0); + if (child == NULL) + panic("xenpv_attach isa"); + device_probe_and_attach(child); + } + + return 0; +} + +static device_method_t xenpv_methods[] = { + /* Device interface */ + DEVMETHOD(device_identify, xenpv_identify), + DEVMETHOD(device_probe, xenpv_probe), + DEVMETHOD(device_attach, xenpv_attach), + DEVMETHOD(device_suspend, bus_generic_suspend), + DEVMETHOD(device_resume, bus_generic_resume), + + /* Bus interface */ + DEVMETHOD(bus_add_child, bus_generic_add_child), + + DEVMETHOD_END +}; + +static driver_t xenpv_driver = { + "xenpv", + xenpv_methods, + 1, /* no softc */ +}; + +DRIVER_MODULE(xenpv, nexus, xenpv_driver, xenpv_devclass, 0, 0); -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:25:37 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8AAA8842; Tue, 14 Jan 2014 15:25:37 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0ED37109A; Tue, 14 Jan 2014 15:25:35 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92709209" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 15:25:35 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:25:34 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T7-0006J6-6J; Tue, 14 Jan 2014 14:59:53 +0000 From: Roger Pau Monne To: , , , , , , Subject: =?UTF-8?q?=5BPATCH=20v10=2015/20=5D=20xen=3A=20create=20a=20PV=20CPU=20device=20for=20PVH=20guests?= Date: Tue, 14 Jan 2014 15:59:37 +0100 Message-ID: <1389711582-66908-16-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:25:37 -0000 Since there's no ACPI on PVH guests, we need to create a dummy CPU device in order to fill the pcpu->pc_device field. --- sys/conf/files | 1 + sys/dev/xen/pvcpu/pvcpu.c | 101 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 102 insertions(+), 0 deletions(-) create mode 100644 sys/dev/xen/pvcpu/pvcpu.c diff --git a/sys/conf/files b/sys/conf/files index bddf021..178d5e2 100644 --- a/sys/conf/files +++ b/sys/conf/files @@ -2500,6 +2500,7 @@ dev/xen/netback/netback.c optional xen | xenhvm dev/xen/netfront/netfront.c optional xen | xenhvm dev/xen/xenpci/xenpci.c optional xenpci dev/xen/timer/timer.c optional xen | xenhvm +dev/xen/pvcpu/pvcpu.c optional xen | xenhvm dev/xl/if_xl.c optional xl pci dev/xl/xlphy.c optional xl pci fs/deadfs/dead_vnops.c standard diff --git a/sys/dev/xen/pvcpu/pvcpu.c b/sys/dev/xen/pvcpu/pvcpu.c new file mode 100644 index 0000000..7f3697c --- /dev/null +++ b/sys/dev/xen/pvcpu/pvcpu.c @@ -0,0 +1,101 @@ +/* + * Copyright (c) 2013 Roger Pau Monné + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include + +/* + * Dummy Xen cpu device + * + * Since there's no ACPI on PVH guests, we need to create a dummy + * CPU device in order to fill the pcpu->pc_device field. + */ + +static void +xenpvcpu_identify(driver_t *driver, device_t parent) +{ + device_t child; + int i; + + /* Only attach to PV guests, HVM guests use the ACPI CPU devices */ + if (!xen_pv_domain()) + return; + + CPU_FOREACH(i) { + child = BUS_ADD_CHILD(parent, 0, "pvcpu", i); + if (child == NULL) + panic("xenpvcpu_identify add pvcpu"); + } +} + +static int +xenpvcpu_probe(device_t dev) +{ + + device_set_desc(dev, "Xen PV CPU"); + return (BUS_PROBE_NOWILDCARD); +} + +static int +xenpvcpu_attach(device_t dev) +{ + struct pcpu *pc; + int cpu; + + cpu = device_get_unit(dev); + pc = pcpu_find(cpu); + pc->pc_device = dev; + return (0); +} + +static device_method_t xenpvcpu_methods[] = { + DEVMETHOD(device_identify, xenpvcpu_identify), + DEVMETHOD(device_probe, xenpvcpu_probe), + DEVMETHOD(device_attach, xenpvcpu_attach), + + DEVMETHOD_END +}; + +static driver_t xenpvcpu_driver = { + "pvcpu", + xenpvcpu_methods, + 0, +}; + +devclass_t xenpvcpu_devclass; + +DRIVER_MODULE(xenpvcpu, xenpv, xenpvcpu_driver, xenpvcpu_devclass, 0, 0); +MODULE_DEPEND(xenpvcpu, xenpv, 1, 1, 1); -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:25:39 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A988A8B2; Tue, 14 Jan 2014 15:25:39 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5C449109C; Tue, 14 Jan 2014 15:25:36 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="90590385" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 14 Jan 2014 15:25:19 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:25:19 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T9-0006J6-Ts; Tue, 14 Jan 2014 14:59:55 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 20/20] isa: allow ISA bus to attach to xenpv device Date: Tue, 14 Jan 2014 15:59:42 +0100 Message-ID: <1389711582-66908-21-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:25:39 -0000 --- sys/x86/isa/isa.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/sys/x86/isa/isa.c b/sys/x86/isa/isa.c index 1a57137..9287ff2 100644 --- a/sys/x86/isa/isa.c +++ b/sys/x86/isa/isa.c @@ -241,3 +241,6 @@ isa_release_resource(device_t bus, device_t child, int type, int rid, * On this platform, isa can also attach to the legacy bus. */ DRIVER_MODULE(isa, legacy, isa_driver, isa_devclass, 0, 0); +#ifdef XENHVM +DRIVER_MODULE(isa, xenpv, isa_driver, isa_devclass, 0, 0); +#endif -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:25:44 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B7B839BF; Tue, 14 Jan 2014 15:25:44 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 3756510A6; Tue, 14 Jan 2014 15:25:41 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="90590647" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 14 Jan 2014 15:25:38 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:25:36 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T5-0006J6-It; Tue, 14 Jan 2014 14:59:51 +0000 From: Roger Pau Monne To: , , , , , , Subject: =?UTF-8?q?=5BPATCH=20v10=2012/20=5D=20xen=3A=20add=20a=20hook=20to=20perform=20AP=20startup?= Date: Tue, 14 Jan 2014 15:59:34 +0100 Message-ID: <1389711582-66908-13-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:25:44 -0000 AP startup on PVH follows the PV method, so we need to add a hook in order to diverge from bare metal. --- sys/amd64/amd64/mp_machdep.c | 14 +++--- sys/amd64/include/cpu.h | 1 + sys/amd64/include/smp.h | 1 + sys/x86/xen/hvm.c | 12 +++++- sys/x86/xen/pv.c | 85 ++++++++++++++++++++++++++++++++++++++++++ sys/xen/pv.h | 32 ++++++++++++++++ 6 files changed, 137 insertions(+), 8 deletions(-) create mode 100644 sys/xen/pv.h diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c index 4af4f8f..17e957d 100644 --- a/sys/amd64/amd64/mp_machdep.c +++ b/sys/amd64/amd64/mp_machdep.c @@ -90,7 +90,7 @@ extern struct pcpu __pcpu[]; /* AP uses this during bootstrap. Do not staticize. */ char *bootSTK; -static int bootAP; +int bootAP; /* Free these after use */ void *bootstacks[MAXCPU]; @@ -124,7 +124,8 @@ static u_long *ipi_hardclock_counts[MAXCPU]; /* Default cpu_ops implementation. */ struct cpu_ops cpu_ops = { - .ipi_vectored = lapic_ipi_vectored + .ipi_vectored = lapic_ipi_vectored, + .start_all_aps = native_start_all_aps, }; extern inthand_t IDTVEC(fast_syscall), IDTVEC(fast_syscall32); @@ -138,7 +139,7 @@ extern int pmap_pcid_enabled; static volatile cpuset_t ipi_nmi_pending; /* used to hold the AP's until we are ready to release them */ -static struct mtx ap_boot_mtx; +struct mtx ap_boot_mtx; /* Set to 1 once we're ready to let the APs out of the pen. */ static volatile int aps_ready = 0; @@ -165,7 +166,6 @@ static int cpu_cores; /* cores per package */ static void assign_cpu_ids(void); static void set_interrupt_apic_ids(void); -static int start_all_aps(void); static int start_ap(int apic_id); static void release_aps(void *dummy); @@ -569,7 +569,7 @@ cpu_mp_start(void) assign_cpu_ids(); /* Start each Application Processor */ - start_all_aps(); + cpu_ops.start_all_aps(); set_interrupt_apic_ids(); } @@ -908,8 +908,8 @@ assign_cpu_ids(void) /* * start each AP in our list */ -static int -start_all_aps(void) +int +native_start_all_aps(void) { vm_offset_t va = boot_address + KERNBASE; u_int64_t *pt4, *pt3, *pt2; diff --git a/sys/amd64/include/cpu.h b/sys/amd64/include/cpu.h index 3c5d5df..98dc3e0 100644 --- a/sys/amd64/include/cpu.h +++ b/sys/amd64/include/cpu.h @@ -64,6 +64,7 @@ struct cpu_ops { void (*cpu_init)(void); void (*cpu_resume)(void); void (*ipi_vectored)(u_int, int); + int (*start_all_aps)(void); }; extern struct cpu_ops cpu_ops; diff --git a/sys/amd64/include/smp.h b/sys/amd64/include/smp.h index d1b366b..15bc823 100644 --- a/sys/amd64/include/smp.h +++ b/sys/amd64/include/smp.h @@ -79,6 +79,7 @@ void smp_masked_invlpg_range(cpuset_t mask, struct pmap *pmap, vm_offset_t startva, vm_offset_t endva); void smp_invltlb(struct pmap *pmap); void smp_masked_invltlb(cpuset_t mask, struct pmap *pmap); +int native_start_all_aps(void); #endif /* !LOCORE */ #endif /* SMP */ diff --git a/sys/x86/xen/hvm.c b/sys/x86/xen/hvm.c index fb1ed79..49caacf 100644 --- a/sys/x86/xen/hvm.c +++ b/sys/x86/xen/hvm.c @@ -53,6 +53,9 @@ __FBSDID("$FreeBSD$"); #include #include #include +#ifdef __amd64__ +#include +#endif #include #include @@ -119,7 +122,10 @@ enum xen_domain_type xen_domain_type = XEN_NATIVE; struct cpu_ops xen_hvm_cpu_ops = { .ipi_vectored = lapic_ipi_vectored, .cpu_init = xen_hvm_cpu_init, - .cpu_resume = xen_hvm_cpu_resume + .cpu_resume = xen_hvm_cpu_resume, +#ifdef __amd64__ + .start_all_aps = native_start_all_aps, +#endif }; static MALLOC_DEFINE(M_XENHVM, "xen_hvm", "Xen HVM PV Support"); @@ -698,6 +704,10 @@ xen_hvm_init(enum xen_hvm_init_type init_type) setup_xen_features(); cpu_ops = xen_hvm_cpu_ops; vm_guest = VM_GUEST_XEN; +#ifdef __amd64__ + if (xen_pv_domain()) + cpu_ops.start_all_aps = xen_pv_start_all_aps; +#endif break; case XEN_HVM_INIT_RESUME: if (error != 0) diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c index d11bc1a..22fd6a6 100644 --- a/sys/x86/xen/pv.c +++ b/sys/x86/xen/pv.c @@ -34,8 +34,11 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include +#include +#include #include #include @@ -49,9 +52,13 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include +#include + +#include /* Native initial function */ extern u_int64_t hammer_time(u_int64_t, u_int64_t); @@ -65,6 +72,15 @@ static caddr_t xen_pv_parse_preload_data(u_int64_t); static void xen_pv_parse_memmap(caddr_t, vm_paddr_t *, int *); static void xen_pv_set_init_ops(void); +/*---------------------------- Extern Declarations ---------------------------*/ +/* Variables used by amd64 mp_machdep to start APs */ +extern struct mtx ap_boot_mtx; +extern void *bootstacks[]; +extern char *doublefault_stack; +extern char *nmi_stack; +extern void *dpcpu; +extern int bootAP; +extern char *bootSTK; /*-------------------------------- Global Data -------------------------------*/ /* Xen init_ops implementation. */ @@ -168,6 +184,75 @@ hammer_time_xen(start_info_t *si, u_int64_t xenstack) } /*-------------------------------- PV specific -------------------------------*/ + +static int +start_xen_ap(int cpu) +{ + struct vcpu_guest_context *ctxt; + int ms, cpus = mp_naps; + + ctxt = malloc(sizeof(*ctxt), M_TEMP, M_NOWAIT | M_ZERO); + if (ctxt == NULL) + panic("unable to allocate memory"); + + ctxt->flags = VGCF_IN_KERNEL; + ctxt->user_regs.rip = (unsigned long) init_secondary; + ctxt->user_regs.rsp = (unsigned long) bootSTK; + + /* Set the AP to use the same page tables */ + ctxt->ctrlreg[3] = KPML4phys; + + if (HYPERVISOR_vcpu_op(VCPUOP_initialise, cpu, ctxt)) + panic("unable to initialize AP#%d\n", cpu); + + free(ctxt, M_TEMP); + + /* Launch the vCPU */ + if (HYPERVISOR_vcpu_op(VCPUOP_up, cpu, NULL)) + panic("unable to start AP#%d\n", cpu); + + /* Wait up to 5 seconds for it to start. */ + for (ms = 0; ms < 5000; ms++) { + if (mp_naps > cpus) + return (1); /* return SUCCESS */ + DELAY(1000); + } + + return (0); +} + +int +xen_pv_start_all_aps(void) +{ + int cpu; + + mtx_init(&ap_boot_mtx, "ap boot", NULL, MTX_SPIN); + + for (cpu = 1; cpu < mp_ncpus; cpu++) { + + /* allocate and set up an idle stack data page */ + bootstacks[cpu] = (void *)kmem_malloc(kernel_arena, + KSTACK_PAGES * PAGE_SIZE, M_WAITOK | M_ZERO); + doublefault_stack = (char *)kmem_malloc(kernel_arena, + PAGE_SIZE, M_WAITOK | M_ZERO); + nmi_stack = (char *)kmem_malloc(kernel_arena, PAGE_SIZE, + M_WAITOK | M_ZERO); + dpcpu = (void *)kmem_malloc(kernel_arena, DPCPU_SIZE, + M_WAITOK | M_ZERO); + + bootSTK = (char *)bootstacks[cpu] + KSTACK_PAGES * PAGE_SIZE - 8; + bootAP = cpu; + + /* attempt to start the Application Processor */ + if (!start_xen_ap(cpu)) + panic("AP #%d failed to start!", cpu); + + CPU_SET(cpu, &all_cpus); /* record AP in CPU map */ + } + + return (mp_naps); +} + /* * Functions to convert the "extra" parameters passed by Xen * into FreeBSD boot options (from the i386 Xen port). diff --git a/sys/xen/pv.h b/sys/xen/pv.h new file mode 100644 index 0000000..45b7473 --- /dev/null +++ b/sys/xen/pv.h @@ -0,0 +1,32 @@ +/* + * Copyright (c) 2013 Roger Pau Monné + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#ifndef __XEN_PV_H__ +#define __XEN_PV_H__ + +int xen_pv_start_all_aps(void); + +#endif /* __XEN_PV_H__ */ -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:26:05 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 56AA3F0D; Tue, 14 Jan 2014 15:26:05 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BB74210C6; Tue, 14 Jan 2014 15:26:03 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="90590893" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 14 Jan 2014 15:26:01 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:26:00 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T8-0006J6-SQ; Tue, 14 Jan 2014 14:59:54 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 18/20] xen: xenstore changes to support PVH Date: Tue, 14 Jan 2014 15:59:40 +0100 Message-ID: <1389711582-66908-19-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:26:05 -0000 --- sys/xen/xenstore/xenstore.c | 18 +++++++++--------- 1 files changed, 9 insertions(+), 9 deletions(-) diff --git a/sys/xen/xenstore/xenstore.c b/sys/xen/xenstore/xenstore.c index b5cf413..7fa08cc 100644 --- a/sys/xen/xenstore/xenstore.c +++ b/sys/xen/xenstore/xenstore.c @@ -229,13 +229,11 @@ struct xs_softc { */ struct sx xenwatch_mutex; -#ifdef XENHVM /** * The HVM guest pseudo-physical frame number. This is Xen's mapping * of the true machine frame number into our "physical address space". */ unsigned long gpfn; -#endif /** * The event channel for communicating with the @@ -1147,13 +1145,15 @@ xs_attach(device_t dev) /* Initialize the interface to xenstore. */ struct proc *p; -#ifdef XENHVM - xs.evtchn = hvm_get_parameter(HVM_PARAM_STORE_EVTCHN); - xs.gpfn = hvm_get_parameter(HVM_PARAM_STORE_PFN); - xen_store = pmap_mapdev(xs.gpfn * PAGE_SIZE, PAGE_SIZE); -#else - xs.evtchn = xen_start_info->store_evtchn; -#endif + if (xen_hvm_domain()) { + xs.evtchn = hvm_get_parameter(HVM_PARAM_STORE_EVTCHN); + xs.gpfn = hvm_get_parameter(HVM_PARAM_STORE_PFN); + xen_store = pmap_mapdev(xs.gpfn * PAGE_SIZE, PAGE_SIZE); + } else if (xen_pv_domain()) { + xs.evtchn = HYPERVISOR_start_info->store_evtchn; + } else { + panic("Unknown domain type, cannot initialize xenstore\n"); + } TAILQ_INIT(&xs.reply_list); TAILQ_INIT(&xs.watch_events); -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:25:49 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A64D4A9A; Tue, 14 Jan 2014 15:25:49 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A935F10AB; Tue, 14 Jan 2014 15:25:46 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="90590737" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 14 Jan 2014 15:25:44 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:25:39 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T7-0006J6-Ph; Tue, 14 Jan 2014 14:59:53 +0000 From: Roger Pau Monne To: , , , , , , Subject: =?UTF-8?q?=5BPATCH=20v10=2016/20=5D=20xen=3A=20create=20a=20Xen=20nexus=20to=20use=20in=20PV/PVH?= Date: Tue, 14 Jan 2014 15:59:38 +0100 Message-ID: <1389711582-66908-17-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:25:49 -0000 Introduce a Xen specific nexus that is going to be in charge for attaching Xen specific devices. --- sys/conf/files.amd64 | 1 + sys/conf/files.i386 | 1 + sys/dev/xen/console/console.c | 2 +- sys/dev/xen/timer/timer.c | 4 +- sys/dev/xen/xenpci/xenpci.c | 64 ++--------------------------------- sys/x86/xen/xen_nexus.c | 76 +++++++++++++++++++++++++++++++++++++++++ sys/xen/xenstore/xenstore.c | 6 +--- 7 files changed, 85 insertions(+), 69 deletions(-) create mode 100644 sys/x86/xen/xen_nexus.c diff --git a/sys/conf/files.amd64 b/sys/conf/files.amd64 index d7c98cc..f378983 100644 --- a/sys/conf/files.amd64 +++ b/sys/conf/files.amd64 @@ -571,3 +571,4 @@ x86/xen/xen_intr.c optional xen | xenhvm x86/xen/pv.c optional xenhvm x86/xen/pvcpu_enum.c optional xenhvm x86/xen/xenpv.c optional xenhvm +x86/xen/xen_nexus.c optional xenhvm diff --git a/sys/conf/files.i386 b/sys/conf/files.i386 index 81142e3..02887a33 100644 --- a/sys/conf/files.i386 +++ b/sys/conf/files.i386 @@ -604,3 +604,4 @@ x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm x86/xen/xenpv.c optional xen | xenhvm +x86/xen/xen_nexus.c optional xen | xenhvm diff --git a/sys/dev/xen/console/console.c b/sys/dev/xen/console/console.c index 899dffc..91538fe 100644 --- a/sys/dev/xen/console/console.c +++ b/sys/dev/xen/console/console.c @@ -462,4 +462,4 @@ xcons_force_flush(void) } } -DRIVER_MODULE(xc, nexus, xc_driver, xc_devclass, 0, 0); +DRIVER_MODULE(xc, xenpv, xc_driver, xc_devclass, 0, 0); diff --git a/sys/dev/xen/timer/timer.c b/sys/dev/xen/timer/timer.c index 96372ab..f16f5a5 100644 --- a/sys/dev/xen/timer/timer.c +++ b/sys/dev/xen/timer/timer.c @@ -636,5 +636,5 @@ static driver_t xentimer_driver = { sizeof(struct xentimer_softc), }; -DRIVER_MODULE(xentimer, nexus, xentimer_driver, xentimer_devclass, 0, 0); -MODULE_DEPEND(xentimer, nexus, 1, 1, 1); +DRIVER_MODULE(xentimer, xenpv, xentimer_driver, xentimer_devclass, 0, 0); +MODULE_DEPEND(xentimer, xenpv, 1, 1, 1); diff --git a/sys/dev/xen/xenpci/xenpci.c b/sys/dev/xen/xenpci/xenpci.c index dd2ad92..e334051 100644 --- a/sys/dev/xen/xenpci/xenpci.c +++ b/sys/dev/xen/xenpci/xenpci.c @@ -51,8 +51,6 @@ __FBSDID("$FreeBSD$"); extern void xen_intr_handle_upcall(struct trapframe *trap_frame); -static device_t nexus; - /* * This is used to find our platform device instance. */ @@ -188,36 +186,6 @@ xenpci_alloc_space(size_t sz, vm_paddr_t *pa) } } -static struct resource * -xenpci_alloc_resource(device_t dev, device_t child, int type, int *rid, - u_long start, u_long end, u_long count, u_int flags) -{ - return (BUS_ALLOC_RESOURCE(nexus, child, type, rid, start, - end, count, flags)); -} - - -static int -xenpci_release_resource(device_t dev, device_t child, int type, int rid, - struct resource *r) -{ - return (BUS_RELEASE_RESOURCE(nexus, child, type, rid, r)); -} - -static int -xenpci_activate_resource(device_t dev, device_t child, int type, int rid, - struct resource *r) -{ - return (BUS_ACTIVATE_RESOURCE(nexus, child, type, rid, r)); -} - -static int -xenpci_deactivate_resource(device_t dev, device_t child, int type, - int rid, struct resource *r) -{ - return (BUS_DEACTIVATE_RESOURCE(nexus, child, type, rid, r)); -} - /* * Probe - just check device ID. */ @@ -229,7 +197,7 @@ xenpci_probe(device_t dev) return (ENXIO); device_set_desc(dev, "Xen Platform Device"); - return (bus_generic_probe(dev)); + return (BUS_PROBE_DEFAULT); } /* @@ -239,20 +207,8 @@ static int xenpci_attach(device_t dev) { struct xenpci_softc *scp = device_get_softc(dev); - devclass_t dc; int error; - /* - * Find and record nexus0. Since we are not really on the - * PCI bus, all resource operations are directed to nexus - * instead of through our parent. - */ - if ((dc = devclass_find("nexus")) == 0 - || (nexus = devclass_get_device(dc, 0)) == 0) { - device_printf(dev, "unable to find nexus."); - return (ENOENT); - } - error = xenpci_allocate_resources(dev); if (error) { device_printf(dev, "xenpci_allocate_resources failed(%d).\n", @@ -270,7 +226,7 @@ xenpci_attach(device_t dev) goto errexit; } - return (bus_generic_attach(dev)); + return (0); errexit: /* @@ -309,16 +265,10 @@ xenpci_detach(device_t dev) } static int -xenpci_suspend(device_t dev) -{ - return (bus_generic_suspend(dev)); -} - -static int xenpci_resume(device_t dev) { xen_hvm_set_callback(dev); - return (bus_generic_resume(dev)); + return (0); } static device_method_t xenpci_methods[] = { @@ -326,16 +276,8 @@ static device_method_t xenpci_methods[] = { DEVMETHOD(device_probe, xenpci_probe), DEVMETHOD(device_attach, xenpci_attach), DEVMETHOD(device_detach, xenpci_detach), - DEVMETHOD(device_suspend, xenpci_suspend), DEVMETHOD(device_resume, xenpci_resume), - /* Bus interface */ - DEVMETHOD(bus_add_child, bus_generic_add_child), - DEVMETHOD(bus_alloc_resource, xenpci_alloc_resource), - DEVMETHOD(bus_release_resource, xenpci_release_resource), - DEVMETHOD(bus_activate_resource, xenpci_activate_resource), - DEVMETHOD(bus_deactivate_resource, xenpci_deactivate_resource), - { 0, 0 } }; diff --git a/sys/x86/xen/xen_nexus.c b/sys/x86/xen/xen_nexus.c new file mode 100644 index 0000000..c5b7c43 --- /dev/null +++ b/sys/x86/xen/xen_nexus.c @@ -0,0 +1,76 @@ +/* + * Copyright (c) 2013 Roger Pau Monné + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include + +/* + * Xen nexus(4) driver. + */ +static int +nexus_xen_probe(device_t dev) +{ + if (!xen_pv_domain()) + return (ENXIO); + + return (BUS_PROBE_DEFAULT); +} + +static int +nexus_xen_attach(device_t dev) +{ + + nexus_init_resources(); + bus_generic_probe(dev); + bus_generic_attach(dev); + + return 0; +} + +static device_method_t nexus_xen_methods[] = { + /* Device interface */ + DEVMETHOD(device_probe, nexus_xen_probe), + DEVMETHOD(device_attach, nexus_xen_attach), + + { 0, 0 } +}; + +DEFINE_CLASS_1(nexus, nexus_xen_driver, nexus_xen_methods, 1, nexus_driver); +static devclass_t nexus_devclass; + +DRIVER_MODULE(nexus_xen, root, nexus_xen_driver, nexus_devclass, 0, 0); diff --git a/sys/xen/xenstore/xenstore.c b/sys/xen/xenstore/xenstore.c index d404862..b5cf413 100644 --- a/sys/xen/xenstore/xenstore.c +++ b/sys/xen/xenstore/xenstore.c @@ -1261,11 +1261,7 @@ static device_method_t xenstore_methods[] = { DEFINE_CLASS_0(xenstore, xenstore_driver, xenstore_methods, 0); static devclass_t xenstore_devclass; -#ifdef XENHVM -DRIVER_MODULE(xenstore, xenpci, xenstore_driver, xenstore_devclass, 0, 0); -#else -DRIVER_MODULE(xenstore, nexus, xenstore_driver, xenstore_devclass, 0, 0); -#endif +DRIVER_MODULE(xenstore, xenpv, xenstore_driver, xenstore_devclass, 0, 0); /*------------------------------- Sysctl Data --------------------------------*/ /* XXX Shouldn't the node be somewhere else? */ -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:25:30 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EC9D671B; Tue, 14 Jan 2014 15:25:30 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5BC4F108B; Tue, 14 Jan 2014 15:25:29 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92709156" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 15:25:28 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:25:28 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T9-0006J6-Cw; Tue, 14 Jan 2014 14:59:55 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 19/20] xen: changes to gnttab for PVH Date: Tue, 14 Jan 2014 15:59:41 +0100 Message-ID: <1389711582-66908-20-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:25:31 -0000 --- sys/xen/gnttab.c | 26 +++++++++++++++++++++----- 1 files changed, 21 insertions(+), 5 deletions(-) diff --git a/sys/xen/gnttab.c b/sys/xen/gnttab.c index 03c32b7..6949be5 100644 --- a/sys/xen/gnttab.c +++ b/sys/xen/gnttab.c @@ -25,6 +25,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -607,6 +608,7 @@ gnttab_resume(void) { int error; unsigned int max_nr_gframes, nr_gframes; + void *alloc_mem; nr_gframes = nr_grant_frames; max_nr_gframes = max_nr_grant_frames(); @@ -614,11 +616,25 @@ gnttab_resume(void) return (ENOSYS); if (!resume_frames) { - error = xenpci_alloc_space(PAGE_SIZE * max_nr_gframes, - &resume_frames); - if (error) { - printf("error mapping gnttab share frames\n"); - return (error); + if (xen_pv_domain()) { + /* + * This is a waste of physical memory, + * we should use ballooned pages instead, + * but it will do for now. + */ + alloc_mem = contigmalloc(max_nr_gframes * PAGE_SIZE, + M_DEVBUF, M_NOWAIT, 0, + ULONG_MAX, PAGE_SIZE, 0); + KASSERT((alloc_mem != NULL), + ("unable to alloc memory for gnttab")); + resume_frames = vtophys(alloc_mem); + } else { + error = xenpci_alloc_space(PAGE_SIZE * max_nr_gframes, + &resume_frames); + if (error) { + printf("error mapping gnttab share frames\n"); + return (error); + } } } -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:25:53 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 224F7BC1; Tue, 14 Jan 2014 15:25:53 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9E53910B4; Tue, 14 Jan 2014 15:25:51 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92709294" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 15:25:49 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:25:48 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T6-0006J6-3l; Tue, 14 Jan 2014 14:59:52 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 13/20] xen: introduce flag to disable the local apic Date: Tue, 14 Jan 2014 15:59:35 +0100 Message-ID: <1389711582-66908-14-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:25:53 -0000 PVH guests don't have an emulated lapic. --- sys/amd64/amd64/mp_machdep.c | 10 ++++++---- sys/amd64/include/apicvar.h | 1 + sys/i386/include/apicvar.h | 1 + sys/i386/xen/xen_machdep.c | 2 ++ sys/x86/x86/local_apic.c | 8 +++++--- sys/x86/xen/pv.c | 3 +++ 6 files changed, 18 insertions(+), 7 deletions(-) diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c index 17e957d..cea0306 100644 --- a/sys/amd64/amd64/mp_machdep.c +++ b/sys/amd64/amd64/mp_machdep.c @@ -707,7 +707,8 @@ init_secondary(void) wrmsr(MSR_SF_MASK, PSL_NT|PSL_T|PSL_I|PSL_C|PSL_D); /* Disable local APIC just to be sure. */ - lapic_disable(); + if (lapic_valid) + lapic_disable(); /* signal our startup to the BSP. */ mp_naps++; @@ -733,7 +734,7 @@ init_secondary(void) /* A quick check from sanity claus */ cpuid = PCPU_GET(cpuid); - if (PCPU_GET(apic_id) != lapic_id()) { + if (lapic_valid && (PCPU_GET(apic_id) != lapic_id())) { printf("SMP: cpuid = %d\n", cpuid); printf("SMP: actual apic_id = %d\n", lapic_id()); printf("SMP: correct apic_id = %d\n", PCPU_GET(apic_id)); @@ -749,7 +750,8 @@ init_secondary(void) mtx_lock_spin(&ap_boot_mtx); /* Init local apic for irq's */ - lapic_setup(1); + if (lapic_valid) + lapic_setup(1); /* Set memory range attributes for this CPU to match the BSP */ mem_range_AP_init(); @@ -764,7 +766,7 @@ init_secondary(void) if (cpu_logical > 1 && PCPU_GET(apic_id) % cpu_logical != 0) CPU_SET(cpuid, &logical_cpus_mask); - if (bootverbose) + if (lapic_valid && bootverbose) lapic_dump("AP"); if (smp_cpus == mp_ncpus) { diff --git a/sys/amd64/include/apicvar.h b/sys/amd64/include/apicvar.h index e7423a3..c04a238 100644 --- a/sys/amd64/include/apicvar.h +++ b/sys/amd64/include/apicvar.h @@ -169,6 +169,7 @@ inthand_t extern vm_paddr_t lapic_paddr; extern int apic_cpuids[]; +extern bool lapic_valid; u_int apic_alloc_vector(u_int apic_id, u_int irq); u_int apic_alloc_vectors(u_int apic_id, u_int *irqs, u_int count, diff --git a/sys/i386/include/apicvar.h b/sys/i386/include/apicvar.h index df99ebe..ea8a3c3 100644 --- a/sys/i386/include/apicvar.h +++ b/sys/i386/include/apicvar.h @@ -168,6 +168,7 @@ inthand_t extern vm_paddr_t lapic_paddr; extern int apic_cpuids[]; +extern bool lapic_valid; u_int apic_alloc_vector(u_int apic_id, u_int irq); u_int apic_alloc_vectors(u_int apic_id, u_int *irqs, u_int count, diff --git a/sys/i386/xen/xen_machdep.c b/sys/i386/xen/xen_machdep.c index 09c01f1..25b9cfc 100644 --- a/sys/i386/xen/xen_machdep.c +++ b/sys/i386/xen/xen_machdep.c @@ -59,6 +59,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include @@ -912,6 +913,7 @@ initvalues(start_info_t *startinfo) #endif xen_start_info = startinfo; HYPERVISOR_start_info = startinfo; + lapic_valid = false; xen_phys_machine = (xen_pfn_t *)startinfo->mfn_list; IdlePTD = (pd_entry_t *)((uint8_t *)startinfo->pt_base + PAGE_SIZE); diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c index 41bd602..fddf1fb 100644 --- a/sys/x86/x86/local_apic.c +++ b/sys/x86/x86/local_apic.c @@ -156,6 +156,7 @@ extern inthand_t IDTVEC(rsvd); volatile lapic_t *lapic; vm_paddr_t lapic_paddr; +bool lapic_valid = true; static u_long lapic_timer_divisor; static struct eventtimer lapic_et; @@ -1367,9 +1368,10 @@ apic_setup_io(void *dummy __unused) if (retval != 0) printf("%s: Failed to setup I/O APICs: returned %d\n", best_enum->apic_name, retval); -#ifdef XEN - return; -#endif + + if (!lapic_valid) + return; + /* * Finish setting up the local APIC on the BSP once we know how to * properly program the LINT pins. diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c index 22fd6a6..6ea1e2a 100644 --- a/sys/x86/xen/pv.c +++ b/sys/x86/xen/pv.c @@ -53,6 +53,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -315,4 +316,6 @@ xen_pv_set_init_ops(void) { /* Init ops for Xen PV */ init_ops = xen_init_ops; + /* Disable lapic */ + lapic_valid = false; } -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:25:52 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E19EDBBD; Tue, 14 Jan 2014 15:25:52 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D412710B2; Tue, 14 Jan 2014 15:25:50 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="90590768" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 14 Jan 2014 15:25:47 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:25:45 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T5-0006J6-21; Tue, 14 Jan 2014 14:59:51 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 11/20] xen: changes to hvm code in order to support PVH guests Date: Tue, 14 Jan 2014 15:59:33 +0100 Message-ID: <1389711582-66908-12-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:25:53 -0000 On PVH we don't need to init the shared info page, or disable emulated devices. Also, make sure PV IPIs are set before starting the APs. --- sys/x86/xen/hvm.c | 17 ++++++++++++----- 1 files changed, 12 insertions(+), 5 deletions(-) diff --git a/sys/x86/xen/hvm.c b/sys/x86/xen/hvm.c index 9a0411e..fb1ed79 100644 --- a/sys/x86/xen/hvm.c +++ b/sys/x86/xen/hvm.c @@ -523,7 +523,7 @@ xen_setup_cpus(void) { int i; - if (!xen_hvm_domain() || !xen_vector_callback_enabled) + if (!xen_vector_callback_enabled) return; #ifdef __amd64__ @@ -712,10 +712,13 @@ xen_hvm_init(enum xen_hvm_init_type init_type) } xen_vector_callback_enabled = 0; - xen_domain_type = XEN_HVM_DOMAIN; - xen_hvm_init_shared_info_page(); xen_hvm_set_callback(NULL); - xen_hvm_disable_emulated_devices(); + + if (!xen_pv_domain()) { + xen_domain_type = XEN_HVM_DOMAIN; + xen_hvm_init_shared_info_page(); + xen_hvm_disable_emulated_devices(); + } } void @@ -746,6 +749,9 @@ xen_set_vcpu_id(void) struct pcpu *pc; int i; + if (!xen_hvm_domain()) + return; + /* Set vcpu_id to acpi_id */ CPU_FOREACH(i) { pc = pcpu_find(i); @@ -789,7 +795,8 @@ xen_hvm_cpu_init(void) SYSINIT(xen_hvm_init, SI_SUB_HYPERVISOR, SI_ORDER_FIRST, xen_hvm_sysinit, NULL); #ifdef SMP -SYSINIT(xen_setup_cpus, SI_SUB_SMP, SI_ORDER_FIRST, xen_setup_cpus, NULL); +/* We need to setup IPIs before APs are started */ +SYSINIT(xen_setup_cpus, SI_SUB_SMP-1, SI_ORDER_FIRST, xen_setup_cpus, NULL); #endif SYSINIT(xen_hvm_cpu_init, SI_SUB_INTR, SI_ORDER_FIRST, xen_hvm_cpu_init, NULL); SYSINIT(xen_set_vcpu_id, SI_SUB_CPU, SI_ORDER_ANY, xen_set_vcpu_id, NULL); -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:26:01 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70CB7EFE; Tue, 14 Jan 2014 15:26:01 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CE88710C1; Tue, 14 Jan 2014 15:25:59 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92709361" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 15:25:59 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:25:58 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T4-0006J6-HA; Tue, 14 Jan 2014 14:59:50 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 10/20] xen: add hook for AP bootstrap memory reservation Date: Tue, 14 Jan 2014 15:59:32 +0100 Message-ID: <1389711582-66908-11-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:26:01 -0000 This hook will only be implemented for bare metal, Xen doesn't require any bootstrap code since APs are started in long mode with paging enabled. --- sys/amd64/amd64/machdep.c | 6 +++++- sys/amd64/include/sysarch.h | 1 + 2 files changed, 6 insertions(+), 1 deletions(-) diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index 64df89a..3a2db30 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -178,6 +178,9 @@ struct init_ops init_ops = { .early_delay_init = i8254_init, .early_delay = i8254_delay, .parse_memmap = native_parse_memmap, +#ifdef SMP + .mp_bootaddress = mp_bootaddress, +#endif }; /* @@ -1492,7 +1495,8 @@ getmemsize(caddr_t kmdp, u_int64_t first) #ifdef SMP /* make hole for AP bootstrap code */ - physmap[1] = mp_bootaddress(physmap[1] / 1024); + if (init_ops.mp_bootaddress) + physmap[1] = init_ops.mp_bootaddress(physmap[1] / 1024); #endif /* diff --git a/sys/amd64/include/sysarch.h b/sys/amd64/include/sysarch.h index 084223e..7696064 100644 --- a/sys/amd64/include/sysarch.h +++ b/sys/amd64/include/sysarch.h @@ -16,6 +16,7 @@ struct init_ops { void (*early_delay_init)(void); void (*early_delay)(int); void (*parse_memmap)(caddr_t, vm_paddr_t *, int *); + u_int (*mp_bootaddress)(u_int); }; extern struct init_ops init_ops; -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:25:59 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 08856DAD; Tue, 14 Jan 2014 15:25:59 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 63D2A10BA; Tue, 14 Jan 2014 15:25:57 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92709345" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 15:25:56 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 10:25:56 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W35T8-0006J6-Bu; Tue, 14 Jan 2014 14:59:54 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v10 17/20] xen: add shutdown hook for PVH Date: Tue, 14 Jan 2014 15:59:39 +0100 Message-ID: <1389711582-66908-18-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:25:59 -0000 Add the PV shutdown hook to PVH. --- sys/dev/xen/control/control.c | 37 ++++++++++++++++++------------------- 1 files changed, 18 insertions(+), 19 deletions(-) diff --git a/sys/dev/xen/control/control.c b/sys/dev/xen/control/control.c index bc0609d..78894ba 100644 --- a/sys/dev/xen/control/control.c +++ b/sys/dev/xen/control/control.c @@ -316,21 +316,6 @@ xctrl_suspend() EVENTHANDLER_INVOKE(power_resume); } -static void -xen_pv_shutdown_final(void *arg, int howto) -{ - /* - * Inform the hypervisor that shutdown is complete. - * This is not necessary in HVM domains since Xen - * emulates ACPI in that mode and FreeBSD's ACPI - * support will request this transition. - */ - if (howto & (RB_HALT | RB_POWEROFF)) - HYPERVISOR_shutdown(SHUTDOWN_poweroff); - else - HYPERVISOR_shutdown(SHUTDOWN_reboot); -} - #else /* HVM mode suspension. */ @@ -440,6 +425,21 @@ xctrl_crash() panic("Xen directed crash"); } +static void +xen_pv_shutdown_final(void *arg, int howto) +{ + /* + * Inform the hypervisor that shutdown is complete. + * This is not necessary in HVM domains since Xen + * emulates ACPI in that mode and FreeBSD's ACPI + * support will request this transition. + */ + if (howto & (RB_HALT | RB_POWEROFF)) + HYPERVISOR_shutdown(SHUTDOWN_poweroff); + else + HYPERVISOR_shutdown(SHUTDOWN_reboot); +} + /*------------------------------ Event Reception -----------------------------*/ static void xctrl_on_watch_event(struct xs_watch *watch, const char **vec, unsigned int len) @@ -522,10 +522,9 @@ xctrl_attach(device_t dev) xctrl->xctrl_watch.callback_data = (uintptr_t)xctrl; xs_register_watch(&xctrl->xctrl_watch); -#ifndef XENHVM - EVENTHANDLER_REGISTER(shutdown_final, xen_pv_shutdown_final, NULL, - SHUTDOWN_PRI_LAST); -#endif + if (xen_pv_domain()) + EVENTHANDLER_REGISTER(shutdown_final, xen_pv_shutdown_final, NULL, + SHUTDOWN_PRI_LAST); return (0); } -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:30:36 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 38A2044B for ; Tue, 14 Jan 2014 15:30:36 +0000 (UTC) Received: from mail-ve0-f169.google.com (mail-ve0-f169.google.com [209.85.128.169]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E8F9A1147 for ; Tue, 14 Jan 2014 15:30:35 +0000 (UTC) Received: by mail-ve0-f169.google.com with SMTP id oz11so1306400veb.28 for ; Tue, 14 Jan 2014 07:30:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Dkffv3Bdg/bcXtmbh00LB6dpAG6Zssbgh76zRz3crPU=; b=CNCtWUbLBpdlpCHnci6kmumHM93BpRtPqJtGHdmB7n8FJJX+ktbEfIFCtkG+OI6c0Q 7byF9O6PNifsDRk6d8uNSyz80OwdWzceQvIbMd9dGif7X676F3aEoxiprQcCrO6N10Ia dfsuY79mtj/bkD8vN/J6IFl43vfSQutyDmLpkDmAmsXWBklsfwmwogJjO2DF+hNIwxG5 bWSpsbYW8r2TztNyr5C+RrRRdsGhcLIrrKl5OSNP6RD8CX7PrhQ2mqmiC6BDQOksfYvR OXbeVAFmQIkUgSRpOQp4iBsCUZg2jNzGZfTsRA/0gqjDRA+8LVoWDcT2GrtmOxRM6Y1Y ASjw== X-Gm-Message-State: ALoCoQmCLa0VofZ1q6nuZf6KsfGy6CkuPrWjQ9G4M8cUdZXzxO3MEAtwJ7nCuUsGrqFI5b3J/5lv MIME-Version: 1.0 X-Received: by 10.221.37.1 with SMTP id tc1mr1052759vcb.32.1389713428998; Tue, 14 Jan 2014 07:30:28 -0800 (PST) Received: by 10.58.132.205 with HTTP; Tue, 14 Jan 2014 07:30:28 -0800 (PST) In-Reply-To: <1389711582-66908-6-git-send-email-roger.pau@citrix.com> References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> <1389711582-66908-6-git-send-email-roger.pau@citrix.com> Date: Tue, 14 Jan 2014 08:30:28 -0700 Message-ID: Subject: Re: [PATCH v10 05/20] xen: rework xen timer so it can be used early in boot process From: Will Andrews To: Roger Pau Monne Content-Type: text/plain; charset=UTF-8 Cc: freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:30:36 -0000 Looks good to me. It appears to go with your PATCH v10 06/20, and your commit log should note the dependency directly. Thanks! --Will. On Tue, Jan 14, 2014 at 7:59 AM, Roger Pau Monne wrote: > This should not introduce any functional change, and makes the > functions suitable to be called before we have actually mapped the > vcpu_info struct on a per-cpu basis. > --- > sys/dev/xen/timer/timer.c | 29 ++++++++++++++++++++--------- > 1 files changed, 20 insertions(+), 9 deletions(-) > > diff --git a/sys/dev/xen/timer/timer.c b/sys/dev/xen/timer/timer.c > index 354085b..b2f6bcd 100644 > --- a/sys/dev/xen/timer/timer.c > +++ b/sys/dev/xen/timer/timer.c > @@ -230,22 +230,22 @@ xen_fetch_vcpu_tinfo(struct vcpu_time_info *dst, struct vcpu_time_info *src) > /** > * \brief Get the current time, in nanoseconds, since the hypervisor booted. > * > + * \param vcpu vcpu_info structure to fetch the time from. > + * > * \note This function returns the current CPU's idea of this value, unless > * it happens to be less than another CPU's previously determined value. > */ > static uint64_t > -xen_fetch_vcpu_time(void) > +xen_fetch_vcpu_time(struct vcpu_info *vcpu) > { > struct vcpu_time_info dst; > struct vcpu_time_info *src; > uint32_t pre_version; > uint64_t now; > volatile uint64_t last; > - struct vcpu_info *vcpu = DPCPU_GET(vcpu_info); > > src = &vcpu->time; > > - critical_enter(); > do { > pre_version = xen_fetch_vcpu_tinfo(&dst, src); > barrier(); > @@ -266,16 +266,19 @@ xen_fetch_vcpu_time(void) > } > } while (!atomic_cmpset_64(&xen_timer_last_time, last, now)); > > - critical_exit(); > - > return (now); > } > > static uint32_t > xentimer_get_timecount(struct timecounter *tc) > { > + uint32_t xen_time; > > - return ((uint32_t)xen_fetch_vcpu_time() & UINT_MAX); > + critical_enter(); > + xen_time = (uint32_t)xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)) & UINT_MAX; > + critical_exit(); > + > + return (xen_time); > } > > /** > @@ -305,7 +308,12 @@ xen_fetch_wallclock(struct timespec *ts) > static void > xen_fetch_uptime(struct timespec *ts) > { > - uint64_t uptime = xen_fetch_vcpu_time(); > + uint64_t uptime; > + > + critical_enter(); > + uptime = xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)); > + critical_exit(); > + > ts->tv_sec = uptime / NSEC_IN_SEC; > ts->tv_nsec = uptime % NSEC_IN_SEC; > } > @@ -354,7 +362,7 @@ xentimer_intr(void *arg) > struct xentimer_softc *sc = (struct xentimer_softc *)arg; > struct xentimer_pcpu_data *pcpu = DPCPU_PTR(xentimer_pcpu); > > - pcpu->last_processed = xen_fetch_vcpu_time(); > + pcpu->last_processed = xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)); > if (pcpu->timer != 0 && sc->et.et_active) > sc->et.et_event_cb(&sc->et, sc->et.et_arg); > > @@ -415,7 +423,10 @@ xentimer_et_start(struct eventtimer *et, > do { > if (++i == 60) > panic("can't schedule timer"); > - next_time = xen_fetch_vcpu_time() + first_in_ns; > + critical_enter(); > + next_time = xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)) + > + first_in_ns; > + critical_exit(); > error = xentimer_vcpu_start_timer(cpu, next_time); > } while (error == -ETIME); > > -- > 1.7.7.5 (Apple Git-26) > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 15:41:51 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 045A36CD for ; Tue, 14 Jan 2014 15:41:51 +0000 (UTC) Received: from mail-wg0-f53.google.com (mail-wg0-f53.google.com [74.125.82.53]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 89DB2127F for ; Tue, 14 Jan 2014 15:41:50 +0000 (UTC) Received: by mail-wg0-f53.google.com with SMTP id y10so562936wgg.32 for ; Tue, 14 Jan 2014 07:41:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=eZAf2Jsq8X2YtooXl7ODA3aJHw4fEtu9rY90GKEFVoY=; b=dyN7yBeGNjqeABNm4C02pFXRIbHHd0S+0BGoCGPBg7LJBQEbaCpGjaP/+mtiAlUoyt FrbEWQvcQbWlhswYMchEvuiy0VlDs2EosulwcdTL7FGfNSv2CK2YWssKmZpkgbt7MK8u JGCO7g2n/F9SDwVuXcVmf6ZZIM9UftnwPFdUrx5Q3WvMygTs0bmMeTNYo6l7YzyxxymB xNconNozopHAu6dexApWxoRNPgfd/hJ12pNHsXnDyUfuASxCoHDV8TJjqs112FvW5l4O +HKy8VnCJf0WkcQh7sT6BiHnKQ6bycvzwhncX6ylQP4IoU2X2LTbRXy4yMAXcYq/3gB7 MVsA== X-Gm-Message-State: ALoCoQkDuHhTvZgryJmxRmkEiCyMqx7Dlpd8sTf4oMpydC1MdpfSjh3JNv6W69spS38tRn3g01xr X-Received: by 10.180.8.194 with SMTP id t2mr3606396wia.41.1389714103256; Tue, 14 Jan 2014 07:41:43 -0800 (PST) Received: from [10.80.2.139] ([185.25.64.249]) by mx.google.com with ESMTPSA id f7sm839362wjb.7.2014.01.14.07.41.41 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 14 Jan 2014 07:41:42 -0800 (PST) Message-ID: <52D55AB4.4010504@linaro.org> Date: Tue, 14 Jan 2014 15:41:40 +0000 From: Julien Grall User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20131104 Icedove/17.0.10 MIME-Version: 1.0 To: Roger Pau Monne Subject: Re: [PATCH v10 14/20] xen: introduce xenpv bus and a dummy pvcpu device References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> <1389711582-66908-15-git-send-email-roger.pau@citrix.com> In-Reply-To: <1389711582-66908-15-git-send-email-roger.pau@citrix.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: jhb@freebsd.org, xen-devel@lists.xen.org, julien.grall@citrix.com, freebsd-xen@freebsd.org, freebsd-current@freebsd.org, kib@freebsd.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 15:41:51 -0000 On 01/14/2014 02:59 PM, Roger Pau Monne wrote: > +static int > +xenpv_attach(device_t dev) > +{ > + device_t child; > + > + if (xen_hvm_domain()) { > + device_t xenpci; > + devclass_t dc; > + > + /* Make sure xenpci has been attached */ > + dc = devclass_find("xenpci"); > + if (dc == NULL) > + panic("unable to find xenpci devclass"); > + > + xenpci = devclass_get_device(dc, 0); > + if (xenpci == NULL) > + panic("unable to find xenpci device"); > + > + if (!device_is_attached(xenpci)) > + panic("trying to attach xenpv before xenpci"); > + } Can you use the identify method to add the xenpci device? As I said earlier, I will reuse this code for ARM guest and this device is not used on this architecture. -- Julien Grall From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 16:08:19 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 002E23FD; Tue, 14 Jan 2014 16:08:18 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 525CF14B2; Tue, 14 Jan 2014 16:08:16 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="90613276" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 14 Jan 2014 16:08:13 +0000 Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.80) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 11:08:12 -0500 Message-ID: <52D560EB.8040108@citrix.com> Date: Tue, 14 Jan 2014 17:08:11 +0100 From: =?UTF-8?B?Um9nZXIgUGF1IE1vbm7DqQ==?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Julien Grall Subject: Re: [PATCH v10 14/20] xen: introduce xenpv bus and a dummy pvcpu device References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> <1389711582-66908-15-git-send-email-roger.pau@citrix.com> <52D55AB4.4010504@linaro.org> In-Reply-To: <52D55AB4.4010504@linaro.org> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-DLP: MIA2 Cc: jhb@freebsd.org, xen-devel@lists.xen.org, julien.grall@citrix.com, freebsd-xen@freebsd.org, freebsd-current@freebsd.org, kib@freebsd.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 16:08:19 -0000 On 14/01/14 16:41, Julien Grall wrote: > On 01/14/2014 02:59 PM, Roger Pau Monne wrote: >> +static int >> +xenpv_attach(device_t dev) >> +{ >> + device_t child; >> + >> + if (xen_hvm_domain()) { >> + device_t xenpci; >> + devclass_t dc; >> + >> + /* Make sure xenpci has been attached */ >> + dc = devclass_find("xenpci"); >> + if (dc == NULL) >> + panic("unable to find xenpci devclass"); >> + >> + xenpci = devclass_get_device(dc, 0); >> + if (xenpci == NULL) >> + panic("unable to find xenpci device"); >> + >> + if (!device_is_attached(xenpci)) >> + panic("trying to attach xenpv before xenpci"); >> + } > > Can you use the identify method to add the xenpci device? I don't think so, xenpci is a pci device, it is detected and plugged by the pci bus code. > As I said earlier, I will reuse this code for ARM guest and this device > is not used on this architecture. You could move this chunk of code (the check for xenpci) to a static inline function and make it a noop for ARM. From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 16:14:47 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 295A5A0D; Tue, 14 Jan 2014 16:14:47 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 622831597; Tue, 14 Jan 2014 16:14:44 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92734889" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 16:14:22 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 11:14:21 -0500 Received: from chilopoda.uk.xensource.com ([10.80.2.139]) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W36dB-0007ja-Qb; Tue, 14 Jan 2014 16:14:21 +0000 Message-ID: <52D5625D.7030702@citrix.com> Date: Tue, 14 Jan 2014 16:14:21 +0000 From: Julien Grall User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20131104 Icedove/17.0.10 MIME-Version: 1.0 To: =?UTF-8?B?Um9nZXIgUGF1IE1vbm7DqQ==?= Subject: Re: [PATCH v10 14/20] xen: introduce xenpv bus and a dummy pvcpu device References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> <1389711582-66908-15-git-send-email-roger.pau@citrix.com> <52D55AB4.4010504@linaro.org> <52D560EB.8040108@citrix.com> In-Reply-To: <52D560EB.8040108@citrix.com> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 Cc: jhb@freebsd.org, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-current@freebsd.org, kib@freebsd.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 16:14:47 -0000 On 01/14/2014 04:08 PM, Roger Pau Monné wrote: > On 14/01/14 16:41, Julien Grall wrote: >> On 01/14/2014 02:59 PM, Roger Pau Monne wrote: >>> +static int >>> +xenpv_attach(device_t dev) >>> +{ >>> + device_t child; >>> + >>> + if (xen_hvm_domain()) { >>> + device_t xenpci; >>> + devclass_t dc; >>> + >>> + /* Make sure xenpci has been attached */ >>> + dc = devclass_find("xenpci"); >>> + if (dc == NULL) >>> + panic("unable to find xenpci devclass"); >>> + >>> + xenpci = devclass_get_device(dc, 0); >>> + if (xenpci == NULL) >>> + panic("unable to find xenpci device"); >>> + >>> + if (!device_is_attached(xenpci)) >>> + panic("trying to attach xenpv before xenpci"); >>> + } >> >> Can you use the identify method to add the xenpci device? > > I don't think so, xenpci is a pci device, it is detected and plugged by > the pci bus code. Oups, I though you are trying to add the device. In this case, the check seems pointless. In which case the xenpci couldn't exist? -- Julien Grall From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 16:20:53 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 08221E49; Tue, 14 Jan 2014 16:20:53 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 69C3D15F0; Tue, 14 Jan 2014 16:20:50 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,658,1384300800"; d="scan'208";a="92738398" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 14 Jan 2014 16:20:48 +0000 Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.80) with Microsoft SMTP Server id 14.2.342.4; Tue, 14 Jan 2014 11:20:47 -0500 Message-ID: <52D563DE.1020207@citrix.com> Date: Tue, 14 Jan 2014 17:20:46 +0100 From: =?UTF-8?B?Um9nZXIgUGF1IE1vbm7DqQ==?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Julien Grall Subject: Re: [PATCH v10 14/20] xen: introduce xenpv bus and a dummy pvcpu device References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> <1389711582-66908-15-git-send-email-roger.pau@citrix.com> <52D55AB4.4010504@linaro.org> <52D560EB.8040108@citrix.com> <52D5625D.7030702@citrix.com> In-Reply-To: <52D5625D.7030702@citrix.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 Cc: jhb@freebsd.org, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-current@freebsd.org, kib@freebsd.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 16:20:53 -0000 On 14/01/14 17:14, Julien Grall wrote: > On 01/14/2014 04:08 PM, Roger Pau Monné wrote: >> On 14/01/14 16:41, Julien Grall wrote: >>> On 01/14/2014 02:59 PM, Roger Pau Monne wrote: >>>> +static int >>>> +xenpv_attach(device_t dev) >>>> +{ >>>> + device_t child; >>>> + >>>> + if (xen_hvm_domain()) { >>>> + device_t xenpci; >>>> + devclass_t dc; >>>> + >>>> + /* Make sure xenpci has been attached */ >>>> + dc = devclass_find("xenpci"); >>>> + if (dc == NULL) >>>> + panic("unable to find xenpci devclass"); >>>> + >>>> + xenpci = devclass_get_device(dc, 0); >>>> + if (xenpci == NULL) >>>> + panic("unable to find xenpci device"); >>>> + >>>> + if (!device_is_attached(xenpci)) >>>> + panic("trying to attach xenpv before xenpci"); >>>> + } >>> >>> Can you use the identify method to add the xenpci device? >> >> I don't think so, xenpci is a pci device, it is detected and plugged by >> the pci bus code. > > Oups, I though you are trying to add the device. In this case, the check > seems pointless. In which case the xenpci couldn't exist? It's just a "belt and suspenders", if we attach the xenpv bus without xenpci being attached first a bunch of things are going to fail, I though it might be best to print a clear error message about what went wrong in order to help debug it. From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 16:49:28 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F2B4CA7E for ; Tue, 14 Jan 2014 16:49:27 +0000 (UTC) Received: from mail-we0-f171.google.com (mail-we0-f171.google.com [74.125.82.171]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8368A18BD for ; Tue, 14 Jan 2014 16:49:26 +0000 (UTC) Received: by mail-we0-f171.google.com with SMTP id w61so651882wes.16 for ; Tue, 14 Jan 2014 08:49:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=IrgsaE+czSp8wPiWmzGaD5r9aoqXBgo1RcPgxl/q6nc=; b=eFRPvrHhyJpJtL2AYPG3PIVdeaBWXCUmNpc2AgKU57X+8jhnShrEBRrWeW47UCLjGM mUqi9/nZry2IjN46m0Pxz0ukKn2TTXmyn+wg0wpdWS1W/AERbgcT5A0BAyU/BEDrZoUj 0IntChWGAEIkTfhyhHR1WTPEtKH+C26hBwsB1mon5Xqr7PDH1hIARc0Lycd4gq+YF4Uc 2R7vIQBuOMU3TRgsX4TAt8emIQixsBLlqkLRN003VMbNB2rL1hzeAa5tXToiuuojnbOD hTvHh5Wgy8L8fCdJYxV9txj6AuWPa1Z7HW6W64+AS4jXMp0PNTOt4F0u+FHUiZskj5GG SSMQ== X-Gm-Message-State: ALoCoQmtcv+Q1dHK8TB5OBRNmlp2R8r+wujTRidd/DvKexeSjHu8drZLz9wVavjUT/IxF2sw82RS X-Received: by 10.180.80.103 with SMTP id q7mr11155351wix.14.1389718164890; Tue, 14 Jan 2014 08:49:24 -0800 (PST) Received: from [10.80.2.139] ([185.25.64.249]) by mx.google.com with ESMTPSA id hy8sm1003957wjb.2.2014.01.14.08.49.23 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 14 Jan 2014 08:49:24 -0800 (PST) Message-ID: <52D56A92.4070100@linaro.org> Date: Tue, 14 Jan 2014 16:49:22 +0000 From: Julien Grall User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20131104 Icedove/17.0.10 MIME-Version: 1.0 To: =?UTF-8?B?Um9nZXIgUGF1IE1vbm7DqQ==?= Subject: Re: [PATCH v10 14/20] xen: introduce xenpv bus and a dummy pvcpu device References: <1389711582-66908-1-git-send-email-roger.pau@citrix.com> <1389711582-66908-15-git-send-email-roger.pau@citrix.com> <52D55AB4.4010504@linaro.org> <52D560EB.8040108@citrix.com> <52D5625D.7030702@citrix.com> <52D563DE.1020207@citrix.com> In-Reply-To: <52D563DE.1020207@citrix.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: jhb@freebsd.org, xen-devel@lists.xen.org, Julien Grall , freebsd-xen@freebsd.org, freebsd-current@freebsd.org, kib@freebsd.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 16:49:28 -0000 On 01/14/2014 04:20 PM, Roger Pau Monné wrote: > On 14/01/14 17:14, Julien Grall wrote: >> On 01/14/2014 04:08 PM, Roger Pau Monné wrote: >>> On 14/01/14 16:41, Julien Grall wrote: >>>> On 01/14/2014 02:59 PM, Roger Pau Monne wrote: >>>>> +static int >>>>> +xenpv_attach(device_t dev) >>>>> +{ >>>>> + device_t child; >>>>> + >>>>> + if (xen_hvm_domain()) { >>>>> + device_t xenpci; >>>>> + devclass_t dc; >>>>> + >>>>> + /* Make sure xenpci has been attached */ >>>>> + dc = devclass_find("xenpci"); >>>>> + if (dc == NULL) >>>>> + panic("unable to find xenpci devclass"); >>>>> + >>>>> + xenpci = devclass_get_device(dc, 0); >>>>> + if (xenpci == NULL) >>>>> + panic("unable to find xenpci device"); >>>>> + >>>>> + if (!device_is_attached(xenpci)) >>>>> + panic("trying to attach xenpv before xenpci"); >>>>> + } >>>> >>>> Can you use the identify method to add the xenpci device? >>> >>> I don't think so, xenpci is a pci device, it is detected and plugged by >>> the pci bus code. >> >> Oups, I though you are trying to add the device. In this case, the check >> seems pointless. In which case the xenpci couldn't exist? > > It's just a "belt and suspenders", if we attach the xenpv bus without > xenpci being attached first a bunch of things are going to fail, I > though it might be best to print a clear error message about what went > wrong in order to help debug it. I only see one place which could failed, and we are already protected. It's when we are trying to allocate space from grant-table via xenpci_alloc_space. I think this error should be enough to understand the problem. At the same time, it's the same things with xenstore. If grant-table initialization has failed, an error message is just printed and FreeBSD will likely failed later when it will try to initialized the PV disk. -- Julien Grall From owner-freebsd-xen@FreeBSD.ORG Tue Jan 14 21:01:15 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 92749B9A for ; Tue, 14 Jan 2014 21:01:15 +0000 (UTC) Received: from mail-wg0-f54.google.com (mail-wg0-f54.google.com [74.125.82.54]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1B51618DD for ; Tue, 14 Jan 2014 21:01:14 +0000 (UTC) Received: by mail-wg0-f54.google.com with SMTP id x13so933100wgg.33 for ; Tue, 14 Jan 2014 13:01:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=1TbpAhb4lxt76CWwhJDZBmQx877iqtHxRw5FiCJ8RR4=; b=JD2N9n2u83AQLeoJmc8WvMXlsZPnl2DxSAEV5GHQ7+N6o8xiaPLsOAzRTQAftewkeb EUpAxZ9/0mdxYwZ1F8KmC54WNLQrbPpYBLzo7yxe/ZjEMvLbFltG7q+axPDgdOMYyB2c +Mii2bkoiIxYNgJdBAZs4ApHlGyieTUtfLuVoxeFQnT6CC7jXeqM7F6bJCTOmmT2TYDH WYq9plc5uIw4+2JTItYWtmUBzqChthisLDAFi9MRP8F+DIbpolcKBk7tLKaxfFdA5bSu cNMbYsRjHmiH/CmUndP2FIUS908NBNZF8FTATqJ+dB9CWzlM98i7IQb1fdgU1ran23dT n9tg== X-Gm-Message-State: ALoCoQl8B+MkhEkbceCscs/HbcWcnFnHuCXjGsc+g2XAlEuPbbhILzroxvKt9Wt+9/k67QrX3vfc X-Received: by 10.180.79.38 with SMTP id g6mr4885327wix.60.1389733273106; Tue, 14 Jan 2014 13:01:13 -0800 (PST) Received: from belegaer.uk.xensource.com. ([185.25.64.249]) by mx.google.com with ESMTPSA id md9sm25006951wic.1.2014.01.14.13.01.11 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 Jan 2014 13:01:12 -0800 (PST) From: Julien Grall To: freebsd-xen@freebsd.org, xen-devel@lists.xen.org, gibbs@freebsd.org, freebsd-arm@FreeBSD.org Subject: [RFC] Add support for Xen ARM guest on FreeBSD Date: Tue, 14 Jan 2014 21:01:07 +0000 Message-Id: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> X-Mailer: git-send-email 1.7.10.4 Cc: stefano.stabellini@eu.citrix.com, ian.campbell@citrix.com X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Jan 2014 21:01:15 -0000 Hello, During the last couple of months I have been working on porting FreeBSD on Xen on ARM. It's my first big project on FreeBSD and I'm not sure if I have cc the right MLs and people. If not, could somebody point me to the right email address? With this patch series I'm able to boot FreeBSD with a single VCPU on every 32 bits platform supported by Xen. The patch series is divided in 5 parts: * #1-#12: Clean up and bug fix with some options * #13-#17: Update the Xen interface to Xen 4.4 and fix compilation * #18-#32: Update Xen code to support ARM guest * #33-#38: Update ARM code to support Xen guest * #39-#40: Add platform code to support ARM guest I have pushed a branch on my git based on Royger's pvh work version 10: git clone git://xenbits.xen.org/people/julieng/freebsd.git -b xen-arm http://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm If needed, I can send the patch one by one the different mailing list. This new support brings 2 open questions (for both Xen and FreeBSD community). When a new guest is created, the toolstack will generated a device tree which will contains: - The amount of memory - The description of the platform (gic, timer, hypervisor node) - PSCI node for SMP bringup Until now, Xen on ARM supported only Linux-based OS. When the support of device tree was added in Xen for guest, we chose to use Linux device tree bindings (gic, timer,...). It seems that FreeBSD choose a different way to implement device tree: - strictly respect ePAR (for interrupt-parent property) - gic bindings different (only 1 interrupt cell) I would like to come with a common device tree specification (bindings, ...) across every operating system. I know the Linux community is working on having a device tree bindings out of the kernel tree. Does FreeBSD community plan to work with Linux community for this purpose? As specified early, the device tree can vary accross Xen version and user input (eg the memory). I have noticed few places (mainly the timer) where the IRQs number are harcoded. In the long-term, it would be nice to see FreeBSD booting out-of-box (eg the device tree is directly provided by the board firmware). I plan to add support for Device Tree loading via Linux Boot ABI, it the way that Xen use to boot a new guest. The second question is related to memory attribute for page table. The early page table setup by FreeBSD are using Write-Through memory attribute which result to a likely random (not every time the same place) crash before the real page table are initialized. Replacing Write-Through by Write-Back made FreeBSD boot correctly. Even today, I have no idea if it's a requirement from Xen or a bug (either in Xen or FreeBSD). The code is taking its first faltering steps, therefore the TODO is quite big: - Try the code on x86 architecture. I did lots of rework for the event channels code and didn't even try to compile - Clean up event channel code - Clean up xen control code - Add support for Device Tree loading via Linux Boot ABI - Fixing crashing on userspace. Could be related to the patch series "arm SMP on Cortex-A15" - Add guest SMP support - DOM0 support? Any help, comments, questions are welcomed. Sincerely yours, ============= Instruction to test FreeBSD on Xen on ARM =========== Xen upstream tree doesn't fully support ELF loading for ARM. You will need to recompile the tools with the following 2 patches: - https://patches.linaro.org/22228/ - https://patches.linaro.org/22227/ To compile and boot Xen on your board, you can refer to the wiki page: http://wiki.xen.org/wiki/Xen_ARM_with_Virtualization_Extensions The instruction to compile FreeBSD for Xen on ARM: $ truncate -s 512 xenvm.img $ sudo mdconfig -f xenvm.img -u0 $ sudo newfs /dev/md0 $ sudo mount /dev/md0 /mnt $ sudo make TARGET_ARCH=armv6 kernel-toolchain $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel $ sudo make TARGET_ARCH=armv6 buildworld $ sudo make TARGET_ARCH=armv6 DESTDIR=/mnt installworld distribution $ echo "/dev/xbd0 / ufs rw 1 1" > /mnt/etc/fstab $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on secure") $ sudo umount /mnt $ sudo mdconfig -d u0 Then you can copy the rootfs and the kernel to DOM 0 on your board. To boot the a FreeBSD your will required the following configuration file $ cat freebsd.xl kernel="kernel" memory=64 name="freebsd" vcpus=1 autoballon="off" disk=[ 'phy:/dev/loop0,xvda,w' ] $ losetup /dev/loop0 xenvm.img $ xl create freebsd.xl $ xl console freebsd Julien Grall (40): xen/netfront: Don't need to include machine/intr_machdep.h xen/blkfront: Don't need to include machine/intr_machdep.h xen/control: Remove include machine/intr_machdep.h xen: Define xen_intr_handle_upcall in common headers xen: Remove duplicate features.h header in i386 arch xen/hypercall: Allow HYPERVISOR_console_write to take a const string xen/timer: Make xen timer optional xen/console: Fix build when DDB is enabled xen/console: clean up identify callback xen/xenstore: xs_probe should return BUS_PROBE_NOWILDCARD xen/control: xctlr_probe shoud return BUS_PROBE_NOWILDCARD xen/hypervisor: Be sure to set __XEN_INTERFACE_VERSION__ xen/interface: Update interface to Xen 4.4 headers xen: Bump __XEN_INTERFACE_VERSION__ to 4.3 xen/baloon: Fix compilation with Xen 4.4 headers xen/netfront: Fix compilation with Xen 4.4 headers xen/gnttab: Fix compilation with Xen 4.4 headers xen/ballon: Use correct type for frame list xen/netback: Fix printf format for xen_pfn_t xen/netfront: Use the correct type for rx_pfn_array xen/gnttab: Use the right type for the frames xen/gnttab: Export resume_frames xen/gnttab: Add a guard for xenpci functions xen/netfront: Define PTE flags per architecture xen/control: Implement suspend has panic for ARM xen: Introduce xen_pmap xen: move x86/xen/xenpv.c in dev/xen/xenpv.c xen/xenpv: Only add isa for x86 architecture xen/xenpv: Protect xenpci code by DEV_XENPCI xen: move xen_intr.c to common code xen/evtchn: Rework the event channels handler to make it generic xen/console: handle console for HVM arm/timer: Handle timer with no clock-frequency property arm/timer: WORKAROUND The timer should support DT... arm: Detect new revision for cortex A15 arm: add ELFNOTE macro arm: Implement disable_intr arm: Use Write-Back as memory attribute for early page table dts: Add xenvm.dts arm: Add xenhvm platform sys/amd64/conf/GENERIC | 4 +- sys/amd64/include/apicvar.h | 1 - sys/amd64/include/xen/hypercall.h | 2 +- sys/amd64/include/xen/xen-os.h | 2 + sys/amd64/include/xen/xenpmap.h | 2 +- sys/arm/arm/cpufunc.c | 2 + sys/arm/arm/generic_timer.c | 12 +- sys/arm/arm/locore.S | 4 +- sys/arm/conf/XENHVM | 97 +++ sys/arm/include/armreg.h | 2 + sys/arm/include/asmacros.h | 26 + sys/arm/include/cpufunc.h | 6 + sys/arm/include/xen/hypercall.h | 50 ++ sys/arm/include/xen/synch_bitops.h | 52 ++ sys/arm/include/xen/xen-os.h | 29 + sys/arm/include/xen/xenfunc.h | 4 + sys/arm/include/xen/xenvar.h | 14 + sys/arm/xenhvm/elf.S | 12 + sys/arm/xenhvm/files.xenhvm | 18 + sys/arm/xenhvm/hypercall.S | 107 +++ sys/arm/xenhvm/xen-dt.c | 148 ++++ sys/arm/xenhvm/xenhvm_machdep.c | 132 ++++ sys/boot/fdt/dts/xenvm.dts | 77 ++ sys/conf/files | 4 +- sys/conf/options | 3 + sys/conf/options.arm | 1 + sys/dev/xen/balloon/balloon.c | 6 +- sys/dev/xen/blkfront/blkfront.c | 1 - sys/dev/xen/console/console.c | 28 +- sys/dev/xen/console/xencons_ring.c | 41 +- sys/dev/xen/control/control.c | 19 +- sys/dev/xen/netback/netback.c | 6 +- sys/dev/xen/netfront/netfront.c | 15 +- sys/dev/xen/xenpci/xenpci.c | 2 +- sys/dev/xen/xenpv.c | 133 ++++ sys/i386/conf/GENERIC | 4 +- sys/i386/conf/XEN | 1 + sys/i386/include/apicvar.h | 1 - sys/i386/include/xen/features.h | 22 - sys/i386/include/xen/hypercall.h | 2 +- sys/i386/include/xen/xen-os.h | 2 + sys/i386/include/xen/xenvar.h | 2 +- sys/i386/xen/xen_machdep.c | 2 +- sys/x86/xen/xen_intr.c | 1241 ------------------------------- sys/x86/xen/xenpv.c | 128 ---- sys/xen/gnttab.c | 23 +- sys/xen/hypervisor.h | 3 +- sys/xen/interface/arch-arm.h | 259 +++++-- sys/xen/interface/arch-arm/hvm/save.h | 2 +- sys/xen/interface/arch-x86/hvm/save.h | 25 +- sys/xen/interface/arch-x86/xen-mca.h | 2 +- sys/xen/interface/arch-x86/xen-x86_32.h | 2 +- sys/xen/interface/arch-x86/xen-x86_64.h | 2 +- sys/xen/interface/arch-x86/xen.h | 31 +- sys/xen/interface/callback.h | 4 +- sys/xen/interface/dom0_ops.h | 2 +- sys/xen/interface/domctl.h | 65 +- sys/xen/interface/elfnote.h | 10 +- sys/xen/interface/event_channel.h | 5 +- sys/xen/interface/features.h | 16 +- sys/xen/interface/gcov.h | 115 +++ sys/xen/interface/grant_table.h | 8 +- sys/xen/interface/hvm/hvm_xs_strings.h | 80 ++ sys/xen/interface/hvm/ioreq.h | 20 +- sys/xen/interface/hvm/params.h | 14 +- sys/xen/interface/hvm/pvdrivers.h | 47 ++ sys/xen/interface/hvm/save.h | 4 +- sys/xen/interface/io/blkif.h | 104 ++- sys/xen/interface/io/console.h | 2 +- sys/xen/interface/io/fbif.h | 2 +- sys/xen/interface/io/kbdif.h | 2 +- sys/xen/interface/io/netif.h | 47 +- sys/xen/interface/io/pciif.h | 3 +- sys/xen/interface/io/protocols.h | 5 +- sys/xen/interface/io/tpmif.h | 68 +- sys/xen/interface/io/usbif.h | 1 - sys/xen/interface/io/vscsiif.h | 42 +- sys/xen/interface/io/xenbus.h | 20 +- sys/xen/interface/io/xs_wire.h | 7 +- sys/xen/interface/kexec.h | 7 +- sys/xen/interface/mem_event.h | 4 +- sys/xen/interface/memory.h | 80 +- sys/xen/interface/nmi.h | 7 +- sys/xen/interface/physdev.h | 33 +- sys/xen/interface/platform.h | 27 +- sys/xen/interface/sched.h | 87 ++- sys/xen/interface/sysctl.h | 47 +- sys/xen/interface/tmem.h | 2 +- sys/xen/interface/trace.h | 93 ++- sys/xen/interface/vcpu.h | 2 +- sys/xen/interface/version.h | 6 +- sys/xen/interface/xen-compat.h | 2 +- sys/xen/interface/xen.h | 128 +++- sys/xen/interface/xenoprof.h | 2 +- sys/xen/interface/xsm/flask_op.h | 8 + sys/xen/xen-os.h | 2 +- sys/xen/xen_intr.c | 1072 ++++++++++++++++++++++++++ sys/xen/xen_intr.h | 8 +- sys/xen/xenstore/xenstore.c | 4 +- 99 files changed, 3339 insertions(+), 1791 deletions(-) create mode 100644 sys/arm/conf/XENHVM create mode 100644 sys/arm/include/xen/hypercall.h create mode 100644 sys/arm/include/xen/synch_bitops.h create mode 100644 sys/arm/include/xen/xen-os.h create mode 100644 sys/arm/include/xen/xenfunc.h create mode 100644 sys/arm/include/xen/xenvar.h create mode 100644 sys/arm/xenhvm/elf.S create mode 100644 sys/arm/xenhvm/files.xenhvm create mode 100644 sys/arm/xenhvm/hypercall.S create mode 100644 sys/arm/xenhvm/xen-dt.c create mode 100644 sys/arm/xenhvm/xenhvm_machdep.c create mode 100644 sys/boot/fdt/dts/xenvm.dts create mode 100644 sys/dev/xen/xenpv.c delete mode 100644 sys/i386/include/xen/features.h delete mode 100644 sys/x86/xen/xen_intr.c delete mode 100644 sys/x86/xen/xenpv.c create mode 100644 sys/xen/interface/gcov.h create mode 100644 sys/xen/interface/hvm/hvm_xs_strings.h create mode 100644 sys/xen/interface/hvm/pvdrivers.h create mode 100644 sys/xen/xen_intr.c -- 1.8.0 From owner-freebsd-xen@FreeBSD.ORG Wed Jan 15 01:26:35 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B6E2C72A for ; Wed, 15 Jan 2014 01:26:35 +0000 (UTC) Received: from mail-ie0-f173.google.com (mail-ie0-f173.google.com [209.85.223.173]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6F5DF1F0E for ; Wed, 15 Jan 2014 01:26:35 +0000 (UTC) Received: by mail-ie0-f173.google.com with SMTP id e14so1266771iej.4 for ; Tue, 14 Jan 2014 17:26:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=+4JJGpGzvrdmRsYJC2le9nkdPo2M0GbBqZE4IUfdtLA=; b=FEeXj+Iyc1rdlVqh+zp6/E7VGRUBlWmu59eUnSRj21nk3KTwoctD//DcSqsXYi7nPt TQfamcPKpUT6ch+AXV44AETK1Gd1vzK2OWflzARbxtaItICmWuqKWAWDFTGep7lS1eaV ZJ4Uv1+17f2YzvSbQ5tDS/c8xtRsQOSkdK9WFM8wf310S2I9nDvaepNTtH4f9SlObUvX gn5AvAcyEGmlvLklSFCMeh60pGsJwH3uK7a6cbThjagVHhOzrbybN7wgWWJz4W401F4k gVhiuxTeoG9tv7qxObwgvKtAXAgGti/AGwgpo9lmKViyUpe80phRabGcNcVltFakbueQ YPlQ== X-Gm-Message-State: ALoCoQkZHRRAEo95+Rne0htlhPTZYNkc4Zra/c9k1zlprTBG/qhHNo7H/LrWtUAs7ZOCdJMqFX4K X-Received: by 10.50.50.70 with SMTP id a6mr6024319igo.1.1389749188092; Tue, 14 Jan 2014 17:26:28 -0800 (PST) Received: from fusion-mac.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id gc2sm26851224igd.6.2014.01.14.17.26.27 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 14 Jan 2014 17:26:27 -0800 (PST) Sender: Warner Losh Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> Date: Tue, 14 Jan 2014 18:26:25 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> To: Julien Grall X-Mailer: Apple Mail (2.1085) Cc: ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jan 2014 01:26:35 -0000 On Jan 14, 2014, at 2:01 PM, Julien Grall wrote: > This new support brings 2 open questions (for both Xen and FreeBSD = community). > When a new guest is created, the toolstack will generated a device = tree which > will contains: > - The amount of memory > - The description of the platform (gic, timer, hypervisor node) > - PSCI node for SMP bringup >=20 > Until now, Xen on ARM supported only Linux-based OS. When the support = of > device tree was added in Xen for guest, we chose to use Linux device = tree > bindings (gic, timer,...). It seems that FreeBSD choose a different = way to > implement device tree: > - strictly respect ePAR (for interrupt-parent property) > - gic bindings different (only 1 interrupt cell) >=20 > I would like to come with a common device tree specification = (bindings, ...) > across every operating system. I know the Linux community is working = on having > a device tree bindings out of the kernel tree. Does FreeBSD community = plan to > work with Linux community for this purpose? We generally try to follow the common definitions for the FDT stuff. = There are a few cases where we either lack the feature set of Linux, or = where the Linux folks are moving quickly and changing the underlying = definitions where we wait for the standards to mature before we = implement. In some cases, where maturity hasn't happened, or where the = bindings are overly Linux centric (which in theory they aren't supposed = to be, but sometimes wind up that way). where we've not implemented = things. > As specified early, the device tree can vary accross Xen version and = user input > (eg the memory). I have noticed few places (mainly the timer) where = the IRQs > number are harcoded. These cases should be viewed as bugs in FreeBSD, I believe. One of the = goals that has wide support, at leas tin theory, is that we can boot = with an unaltered Linux FDT. This goal is some time in the future. > In the long-term, it would be nice to see FreeBSD booting out-of-box = (eg the > device tree is directly provided by the board firmware). I plan to add = support > for Device Tree loading via Linux Boot ABI, it the way that Xen use to = boot a > new guest. That would be most welcome. > The second question is related to memory attribute for page table. The = early > page table setup by FreeBSD are using Write-Through memory attribute = which > result to a likely random (not every time the same place) crash before = the > real page table are initialized. > Replacing Write-Through by Write-Back made FreeBSD boot correctly. = Even today, > I have no idea if it's a requirement from Xen or a bug (either in Xen = or > FreeBSD). There were some problems with pages being setup improperly for the = mutexes to operate properly. Have you confirmed this on a recent version = of FreeBSD? > The code is taking its first faltering steps, therefore the TODO is = quite big: > - Try the code on x86 architecture. I did lots of rework for the = event > channels code and didn't even try to compile > - Clean up event channel code > - Clean up xen control code > - Add support for Device Tree loading via Linux Boot ABI > - Fixing crashing on userspace. Could be related to the patch > series "arm SMP on Cortex-A15" > - Add guest SMP support > - DOM0 support? >=20 > Any help, comments, questions are welcomed. I think this is great! We'd love to work with you to make this happen. Warner > Sincerely yours, >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Instruction to test FreeBSD on = Xen on ARM =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D [trimmed]= From owner-freebsd-xen@FreeBSD.ORG Wed Jan 15 16:24:27 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A813D9FD for ; Wed, 15 Jan 2014 16:24:27 +0000 (UTC) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 351B315DA for ; Wed, 15 Jan 2014 16:24:26 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id w62so2029596wes.13 for ; Wed, 15 Jan 2014 08:24:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=oFqgP2bYNTmnHFB2wz8dF2DyDmx5LNBdt4iOM+2CgZ0=; b=X2mVUg0C/nGNXsPW5aHKEIQGNk+iiZ6lmdlZjrvTfRkHGRWl8Fj1BDnDwxqITuhndX 3evYp/Uqc9cMfAX9dg+tn1af6PzDP5QEdzIpVAx4x/znRtNqBlIEbivKylUSeV2owxcq +zSycDJDcs11Okixsg3pvNoHAa8Nf6C9Gd96+bJ7PUPjVuUd7WH9mmFsSmStXqJYT76N pKVuU1x8CtJluLxFvrq8fkHPZX9OZ9uDNR+nkiRGMKgS+Cuqha2e4NVZjQcQzu75Lc1m 4IPoLg9DzYrq7fjyQl1S2DLQp/jIz89S0+wdDYG3fnY9Nfow+nL7lP2lworlm1DZikkm qFow== X-Gm-Message-State: ALoCoQn9wCl+d3N/wk8jUIcgoq4QbCkCU1MpBIWRB/oQFXVgcpeWhARduMzNbRLVbOIcRzSsXLJW X-Received: by 10.194.63.134 with SMTP id g6mr3273698wjs.46.1389803057418; Wed, 15 Jan 2014 08:24:17 -0800 (PST) Received: from [10.80.2.139] ([185.25.64.249]) by mx.google.com with ESMTPSA id y8sm3685838wje.12.2014.01.15.08.24.15 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 15 Jan 2014 08:24:16 -0800 (PST) Message-ID: <52D6B62A.9000208@linaro.org> Date: Wed, 15 Jan 2014 16:24:10 +0000 From: Julien Grall User-Agent: Mozilla/5.0 (X11; Linux i686; rv:17.0) Gecko/20131104 Icedove/17.0.10 MIME-Version: 1.0 To: Warner Losh Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> In-Reply-To: <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, nwhitehorn@freebsd.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jan 2014 16:24:27 -0000 On 01/15/2014 01:26 AM, Warner Losh wrote: > > On Jan 14, 2014, at 2:01 PM, Julien Grall wrote: >> This new support brings 2 open questions (for both Xen and FreeBSD community). >> When a new guest is created, the toolstack will generated a device tree which >> will contains: >> - The amount of memory >> - The description of the platform (gic, timer, hypervisor node) >> - PSCI node for SMP bringup >> >> Until now, Xen on ARM supported only Linux-based OS. When the support of >> device tree was added in Xen for guest, we chose to use Linux device tree >> bindings (gic, timer,...). It seems that FreeBSD choose a different way to >> implement device tree: >> - strictly respect ePAR (for interrupt-parent property) >> - gic bindings different (only 1 interrupt cell) >> >> I would like to come with a common device tree specification (bindings, ...) >> across every operating system. I know the Linux community is working on having >> a device tree bindings out of the kernel tree. Does FreeBSD community plan to >> work with Linux community for this purpose? > > We generally try to follow the common definitions for the FDT stuff. > There are a few cases where we either lack the feature set of Linux, or > where the Linux folks are moving quickly and changing the underlying definitions > where we wait for the standards to mature before we implement. In some cases, where > maturity hasn't happened, or where the bindings are overly Linux centric (which in > theory they aren't supposed to be, but sometimes wind up that way). where we've not > implemented things. As I understand main bindings (gic, timer) are set in stone and won't change. Ian Campbell has a repository with all the ARM bindings here: http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git; a=tree;f=Bindings/arm To compare the difference between the DT provided by Xen, and the one correctly parsed by FreeBSD - Xen: http://xenbits.xen.org/people/julieng/xenvm-4.2.dts - FreeBSD: http://xenbits.xen.org/people/julieng/xenvm-bsd.dts >From Xen side: - Every device should move under a simple-bus. I think it's harmless for Linux side. - How about hypervisor node? IHMO this node should also live under the simple-bus. >From FreeBSD side: - GIC and Timer bindings needs to be handled correctly (see below the problem for the GIC) - Less stricter about interrupt-parent property. Eg, look at the grant-parent if there is no property interrupt-parent - If the hypervisor doesn't live under simple-bus, the interrupt/memory retrieving should be moved from simple-bus to nexus? Before the revision r260282 (I have added Nathan on cc), I was able to use the Linux GIC bindings (see http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;a=blob;f=Bindings/arm/gic.txt) without any issue. It seems the fdt bus, now consider the number of interrupt cells is always 1. >> As specified early, the device tree can vary accross Xen version and user input >> (eg the memory). I have noticed few places (mainly the timer) where the IRQs >> number are harcoded. > > These cases should be viewed as bugs in FreeBSD, I believe. One of the goals > that has wide support, at leas tin theory, is that we can boot with an unaltered > Linux FDT. This goal is some time in the future. The timer interrupt used by FreeBSD doesn't match the one used by Xen (secure vs non-secure interrupt). This should be easily resolved by retrieving the interrupt from the device tree. Is there a particular reason to use the secure interrupt for the timer? > >> The second question is related to memory attribute for page table. The early >> page table setup by FreeBSD are using Write-Through memory attribute which >> result to a likely random (not every time the same place) crash before the >> real page table are initialized. >> Replacing Write-Through by Write-Back made FreeBSD boot correctly. Even today, >> I have no idea if it's a requirement from Xen or a bug (either in Xen or >> FreeBSD). > > There were some problems with pages being setup improperly for the mutexes > to operate properly. Have you confirmed this on a recent version of FreeBSD? I have checkout the freeBSD branch last week. I'm not sure to understand the relation with mutexes. The symptoms I have is mostly data/prefetch abort receive to FreeBSD. Could it be related? >> The code is taking its first faltering steps, therefore the TODO is quite big: >> - Try the code on x86 architecture. I did lots of rework for the event >> channels code and didn't even try to compile >> - Clean up event channel code >> - Clean up xen control code >> - Add support for Device Tree loading via Linux Boot ABI >> - Fixing crashing on userspace. Could be related to the patch >> series "arm SMP on Cortex-A15" >> - Add guest SMP support >> - DOM0 support? >> >> Any help, comments, questions are welcomed. > > I think this is great! We'd love to work with you to make this happen. > > Warner > >> Sincerely yours, >> >> ============= Instruction to test FreeBSD on Xen on ARM =========== > [trimmed] > -- Julien Grall From owner-freebsd-xen@FreeBSD.ORG Thu Jan 16 01:56:42 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 54A1F49B; Thu, 16 Jan 2014 01:56:42 +0000 (UTC) Received: from smtpauth3.wiscmail.wisc.edu (wmauth3.doit.wisc.edu [144.92.197.226]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1D8941EDC; Thu, 16 Jan 2014 01:56:41 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from avs-daemon.smtpauth3.wiscmail.wisc.edu by smtpauth3.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MZH00D001DU6600@smtpauth3.wiscmail.wisc.edu>; Wed, 15 Jan 2014 19:56:34 -0600 (CST) X-Spam-PmxInfo: Server=avs-3, Version=6.0.3.2322014, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2014.1.16.15115, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from comporellon.tachypleus.net (adsl-76-208-68-77.dsl.mdsnwi.sbcglobal.net [76.208.68.77]) by smtpauth3.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MZH00C1I1E6XC00@smtpauth3.wiscmail.wisc.edu>; Wed, 15 Jan 2014 19:56:32 -0600 (CST) Message-id: <52D73C4E.2080306@freebsd.org> Date: Wed, 15 Jan 2014 19:56:30 -0600 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0 To: Julien Grall , Warner Losh Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> In-reply-to: <52D6B62A.9000208@linaro.org> Cc: ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jan 2014 01:56:42 -0000 On 01/15/14 10:24, Julien Grall wrote: > On 01/15/2014 01:26 AM, Warner Losh wrote: >> On Jan 14, 2014, at 2:01 PM, Julien Grall wrote: >>> This new support brings 2 open questions (for both Xen and FreeBSD community). >>> When a new guest is created, the toolstack will generated a device tree which >>> will contains: >>> - The amount of memory >>> - The description of the platform (gic, timer, hypervisor node) >>> - PSCI node for SMP bringup >>> >>> Until now, Xen on ARM supported only Linux-based OS. When the support of >>> device tree was added in Xen for guest, we chose to use Linux device tree >>> bindings (gic, timer,...). It seems that FreeBSD choose a different way to >>> implement device tree: >>> - strictly respect ePAR (for interrupt-parent property) >>> - gic bindings different (only 1 interrupt cell) >>> >>> I would like to come with a common device tree specification (bindings, ...) >>> across every operating system. I know the Linux community is working on having >>> a device tree bindings out of the kernel tree. Does FreeBSD community plan to >>> work with Linux community for this purpose? >> We generally try to follow the common definitions for the FDT stuff. >> There are a few cases where we either lack the feature set of Linux, or >> where the Linux folks are moving quickly and changing the underlying > definitions >> where we wait for the standards to mature before we implement. In some > cases, where >> maturity hasn't happened, or where the bindings are overly Linux > centric (which in >> theory they aren't supposed to be, but sometimes wind up that way). > where we've not >> implemented things. > As I understand main bindings (gic, timer) are set in stone and won't > change. Ian Campbell has a repository with all the ARM bindings here: > http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git; > a=tree;f=Bindings/arm > > To compare the difference between the DT provided by Xen, and the one > correctly parsed by FreeBSD > - Xen: http://xenbits.xen.org/people/julieng/xenvm-4.2.dts > - FreeBSD: http://xenbits.xen.org/people/julieng/xenvm-bsd.dts > > >From Xen side: > - Every device should move under a simple-bus. I think it's harmless > for Linux side. > - How about hypervisor node? IHMO this node should also live under the > simple-bus. > > >From FreeBSD side: > - GIC and Timer bindings needs to be handled correctly (see below the > problem for the GIC) > - Less stricter about interrupt-parent property. Eg, look at the > grant-parent if there is no property interrupt-parent > - If the hypervisor doesn't live under simple-bus, the interrupt/memory > retrieving should be moved from simple-bus to nexus? > > Before the revision r260282 (I have added Nathan on cc), I was able to > use the Linux GIC bindings (see > http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;a=blob;f=Bindings/arm/gic.txt) > without any issue. > > It seems the fdt bus, now consider the number of interrupt cells is > always 1. > > Thanks for the CC. Could you explain what you mean by "grant-parent" etc? "interrupt-parent" is a fundamental part of the way PAPR (and ePAPR) work, so I'm very very hesitant to start guessing. I think things have broken for you because some (a lot, actually) of OF code does not expect #interrupt-cells to be more than 2. Some APIs might need updating, which I'll try to take care of. Could you tell me what the difference between SPI and PPI is, by the way? On the subject of simple-bus, they usually aren't necessary. For example, all hypervisor devices on IBM hardware live under /vdevice, which is attached to the device tree root. They don't use MMIO, so simple-bus doesn't really make sense. How does Xen communicate with the OS in these devices? -Nathan From owner-freebsd-xen@FreeBSD.ORG Thu Jan 16 16:10:59 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 60F1813C; Thu, 16 Jan 2014 16:10:59 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1B49F1AC6; Thu, 16 Jan 2014 16:10:57 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,668,1384300800"; d="scan'208";a="91428093" Received: from accessns.citrite.net (HELO FTLPEX01CL02.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 16 Jan 2014 16:10:50 +0000 Received: from [10.80.2.80] (10.80.2.80) by FTLPEX01CL02.citrite.net (10.13.107.79) with Microsoft SMTP Server id 14.2.342.4; Thu, 16 Jan 2014 11:10:49 -0500 Message-ID: <1389888647.6697.22.camel@kazak.uk.xensource.com> Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD From: Ian Campbell To: Julien Grall Date: Thu, 16 Jan 2014 16:10:47 +0000 In-Reply-To: <52D6B62A.9000208@linaro.org> References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> Organization: Citrix Systems, Inc. Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.4.4-3 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.80.2.80] X-DLP: MIA1 Cc: stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, nwhitehorn@freebsd.org, gibbs@freebsd.org, Warner Losh X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jan 2014 16:10:59 -0000 On Wed, 2014-01-15 at 16:24 +0000, Julien Grall wrote: > As I understand main bindings (gic, timer) are set in stone and won't > change. Ian Campbell has a repository with all the ARM bindings here: > http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;a=tree;f=Bindings/arm FYI this tree is automatically extracted from the bindings in the Linux source, because long term we would like to split them out from Linux and try and standardize some (if not all) of them. This is exactly because these bindings should not be OS specific... Ian. From owner-freebsd-xen@FreeBSD.ORG Thu Jan 16 12:57:08 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 68CD5104; Thu, 16 Jan 2014 12:57:08 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1DF3215DF; Thu, 16 Jan 2014 12:57:06 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,667,1384300800"; d="scan'208";a="91345743" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 16 Jan 2014 12:57:04 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 16 Jan 2014 07:57:03 -0500 Received: from kaball.uk.xensource.com ([10.80.2.59]) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1W3mVL-0004Ut-BY; Thu, 16 Jan 2014 12:57:03 +0000 Date: Thu, 16 Jan 2014 12:56:03 +0000 From: Stefano Stabellini X-X-Sender: sstabellini@kaball.uk.xensource.com To: Nathan Whitehorn Subject: Re: [Xen-devel] [RFC] Add support for Xen ARM guest on FreeBSD In-Reply-To: <52D73C4E.2080306@freebsd.org> Message-ID: References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> <52D73C4E.2080306@freebsd.org> User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" X-DLP: MIA2 X-Mailman-Approved-At: Thu, 16 Jan 2014 16:17:54 +0000 Cc: ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, gibbs@freebsd.org, Warner Losh X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Jan 2014 12:57:08 -0000 On Wed, 15 Jan 2014, Nathan Whitehorn wrote: > On 01/15/14 10:24, Julien Grall wrote: > > On 01/15/2014 01:26 AM, Warner Losh wrote: > > > On Jan 14, 2014, at 2:01 PM, Julien Grall wrote: > > > > This new support brings 2 open questions (for both Xen and FreeBSD > > > > community). > > > > When a new guest is created, the toolstack will generated a device tree > > > > which > > > > will contains: > > > > - The amount of memory > > > > - The description of the platform (gic, timer, hypervisor node) > > > > - PSCI node for SMP bringup > > > > > > > > Until now, Xen on ARM supported only Linux-based OS. When the support of > > > > device tree was added in Xen for guest, we chose to use Linux device > > > > tree > > > > bindings (gic, timer,...). It seems that FreeBSD choose a different way > > > > to > > > > implement device tree: > > > > - strictly respect ePAR (for interrupt-parent property) > > > > - gic bindings different (only 1 interrupt cell) > > > > > > > > I would like to come with a common device tree specification (bindings, > > > > ...) > > > > across every operating system. I know the Linux community is working on > > > > having > > > > a device tree bindings out of the kernel tree. Does FreeBSD community > > > > plan to > > > > work with Linux community for this purpose? > > > We generally try to follow the common definitions for the FDT stuff. > > > There are a few cases where we either lack the feature set of Linux, or > > > where the Linux folks are moving quickly and changing the underlying > > definitions > > > where we wait for the standards to mature before we implement. In some > > cases, where > > > maturity hasn't happened, or where the bindings are overly Linux > > centric (which in > > > theory they aren't supposed to be, but sometimes wind up that way). > > where we've not > > > implemented things. > > As I understand main bindings (gic, timer) are set in stone and won't > > change. Ian Campbell has a repository with all the ARM bindings here: > > http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git; > > a=tree;f=Bindings/arm > > > > To compare the difference between the DT provided by Xen, and the one > > correctly parsed by FreeBSD > > - Xen: http://xenbits.xen.org/people/julieng/xenvm-4.2.dts > > - FreeBSD: http://xenbits.xen.org/people/julieng/xenvm-bsd.dts > > > > >From Xen side: > > - Every device should move under a simple-bus. I think it's harmless > > for Linux side. > > - How about hypervisor node? IHMO this node should also live under the > > simple-bus. > > > > >From FreeBSD side: > > - GIC and Timer bindings needs to be handled correctly (see below the > > problem for the GIC) > > - Less stricter about interrupt-parent property. Eg, look at the > > grant-parent if there is no property interrupt-parent > > - If the hypervisor doesn't live under simple-bus, the > > interrupt/memory > > retrieving should be moved from simple-bus to nexus? > > > > Before the revision r260282 (I have added Nathan on cc), I was able to > > use the Linux GIC bindings (see > > http://xenbits.xen.org/gitweb/?p=people/ianc/device-tree-rebasing.git;a=blob;f=Bindings/arm/gic.txt) > > without any issue. > > > > It seems the fdt bus, now consider the number of interrupt cells is > > always 1. > > > > > > Thanks for the CC. Could you explain what you mean by "grant-parent" etc? > "interrupt-parent" is a fundamental part of the way PAPR (and ePAPR) work, so > I'm very very hesitant to start guessing. I think things have broken for you > because some (a lot, actually) of OF code does not expect #interrupt-cells to > be more than 2. Some APIs might need updating, which I'll try to take care of. > Could you tell me what the difference between SPI and PPI is, by the way? SPI: Shared Peripheral Interrupt PPI: Private Peripheral Interrupt PPIs are per processor interrupts. > On the subject of simple-bus, they usually aren't necessary. For example, all > hypervisor devices on IBM hardware live under /vdevice, which is attached to > the device tree root. They don't use MMIO, so simple-bus doesn't really make > sense. How does Xen communicate with the OS in these devices? The PPI and the memory region advertized under the Xen hypervisor node on device tree are used to setup the basic infrastructure (grant table, event channels) needed by Xen paravirtualized devices. From owner-freebsd-xen@FreeBSD.ORG Fri Jan 17 00:36:48 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 25E0926B for ; Fri, 17 Jan 2014 00:36:48 +0000 (UTC) Received: from mail-wi0-f173.google.com (mail-wi0-f173.google.com [209.85.212.173]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A87F6192D for ; Fri, 17 Jan 2014 00:36:47 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id d13so56431wiw.12 for ; Thu, 16 Jan 2014 16:36:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=4wXhBVEK9bu6kQV4qJr65d7sAYxtH2FVJr6eGR3jehM=; b=OVQFkYw7/8eAu8Kx5mkixisgXMgEpfrnOq+MLaRCTXH9w6E1xX+5tfwEK3Iw3yBNbx vO0XZSFyNgFJN1fbnddK9dthWewyBIih2V/q2yvyUXnUJkgo0Tj9i6hzUATu+apQqA0m oluCy2F7A/ZBihAoYiod08s4ppvtoR/y+SIwCc2EbSSFKo7q5NOfRkBwkomevLXDxmKU 32p3HtvN2TCEDZA9jAakAM/fWkngPmYFUN3pzOyjWAupPAbotix1/etZOY3yA+ousmfE qFHQNjX5UULKg7gw55M6n04Xrks1FLcROgy6xn6DtgQUn4wjVKY5DZ7oE4/Xk+Ua2rxH tw1A== X-Gm-Message-State: ALoCoQlFurdWwCLwS41N9sd27kpcgVseoiPcV1Za+btQrVR3Iwc0Ycd+OV+ZF2QL+v3iaHxMzWXe X-Received: by 10.180.80.103 with SMTP id q7mr30118wix.14.1389919000069; Thu, 16 Jan 2014 16:36:40 -0800 (PST) Received: from [192.168.0.2] (cpc8-cmbg15-2-0-cust169.5-4.cable.virginm.net. [86.30.140.170]) by mx.google.com with ESMTPSA id ea4sm402202wib.7.2014.01.16.16.36.38 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 16 Jan 2014 16:36:39 -0800 (PST) Message-ID: <52D87B15.5090208@linaro.org> Date: Fri, 17 Jan 2014 00:36:37 +0000 From: Julien Grall User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Nathan Whitehorn , Warner Losh Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> <52D73C4E.2080306@freebsd.org> In-Reply-To: <52D73C4E.2080306@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jan 2014 00:36:48 -0000 On 01/16/2014 01:56 AM, Nathan Whitehorn wrote: > > Thanks for the CC. Could you explain what you mean by "grant-parent" > etc? "interrupt-parent" is a fundamental part of the way PAPR (and > ePAPR) work, so I'm very very hesitant to start guessing. I think things > have broken for you because some (a lot, actually) of OF code does not > expect #interrupt-cells to be more than 2. Some APIs might need > updating, which I'll try to take care of. Could you tell me what the > difference between SPI and PPI is, by the way? Sorry, I also made some typoes in my explanation so it was not clear. interrupt-parent is a property in a device node which links this node to an interrupt controller (in our case the GIC controller). The way to handle it on Linux and the ePAR is different: - ePAR (chapter 2.4) says: The physical wiring of an interrupt source to an interrupt controller is represented in the device tree with the interrupt-parent property. Nodes that represent interrupt-generating devices contain an interrupt-parent property which has a phandle value that points to the device to which the device's interrupts are routed, typically an interrupt controller. If an interrupt-generating device does not have an interrupt-parent property, its interrupt parent is assumed to be its device tree parent. From my understanding, it's mandatory to have an interrupt-parent property on each device node which is using interrupts. If it doesn't exist it will assume that the parent is interrupt controller. If I'm mistaken, at least FreeBSD handle the interrupt-parent property in this way. - Linux implementation will look at to the node, if the property doesn't exists, it will check if the ancestor has this property ... So the device tree below is valid on Linux, but not on FreeBSD: / { interrupt-parent = &gic gic: gic@10 { ... } timer@1 { interrupts = <...> } } Most of shipped device tree use this trick. IanC: I was reading the linux binding documentation (devicetree/booting-without-of.txt VII.2) and it seems that the explanation differs from the implementation, right? For the #interrupt-cells property, the problem starts in fdt_intr_to_rl (sys/dev/fdt/fdt_common.c:476). ofw_bus_map_intr is called always with the first cells of the interrupt no matter the number of cells specified by #interrupt-cells. > On the subject of simple-bus, they usually aren't necessary. For > example, all hypervisor devices on IBM hardware live under /vdevice, > which is attached to the device tree root. They don't use MMIO, so > simple-bus doesn't really make sense. How does Xen communicate with the > OS in these devices? > -Nathan As I understand, only the simple bus code (see simplebus_attach) is translating the interrupts in the device on a resource. So if you have a node directly attached to the root node with interrupts and MMIO, the driver won't be able to retrieve and translate the interrupts via bus_alloc_resources. In the Xen device tree, we have an hypervisor node directly attach to the root which contains both MMIO and interrupt used by Xen to communicate with the guest. -- Julien Grall From owner-freebsd-xen@FreeBSD.ORG Fri Jan 17 04:04:52 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D2B6E9AB; Fri, 17 Jan 2014 04:04:52 +0000 (UTC) Received: from smtpauth2.wiscmail.wisc.edu (wmauth2.doit.wisc.edu [144.92.197.222]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9C90E19A0; Fri, 17 Jan 2014 04:04:51 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from avs-daemon.smtpauth2.wiscmail.wisc.edu by smtpauth2.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MZI00E00YYQG100@smtpauth2.wiscmail.wisc.edu>; Thu, 16 Jan 2014 21:04:43 -0600 (CST) X-Spam-PmxInfo: Server=avs-2, Version=6.0.3.2322014, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2014.1.17.25115, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from comporellon.tachypleus.net (adsl-76-208-68-77.dsl.mdsnwi.sbcglobal.net [76.208.68.77]) by smtpauth2.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MZI00CCNZ7TJI00@smtpauth2.wiscmail.wisc.edu>; Thu, 16 Jan 2014 21:04:43 -0600 (CST) Message-id: <52D89DC9.7050303@freebsd.org> Date: Thu, 16 Jan 2014 21:04:41 -0600 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0 To: Julien Grall , Warner Losh Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> <52D73C4E.2080306@freebsd.org> <52D87B15.5090208@linaro.org> In-reply-to: <52D87B15.5090208@linaro.org> Cc: ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jan 2014 04:04:52 -0000 On 01/16/14 18:36, Julien Grall wrote: > > > On 01/16/2014 01:56 AM, Nathan Whitehorn wrote: >> >> Thanks for the CC. Could you explain what you mean by "grant-parent" >> etc? "interrupt-parent" is a fundamental part of the way PAPR (and >> ePAPR) work, so I'm very very hesitant to start guessing. I think things >> have broken for you because some (a lot, actually) of OF code does not >> expect #interrupt-cells to be more than 2. Some APIs might need >> updating, which I'll try to take care of. Could you tell me what the >> difference between SPI and PPI is, by the way? > > Sorry, I also made some typoes in my explanation so it was not clear. > > interrupt-parent is a property in a device node which links this node > to an interrupt controller (in our case the GIC controller). > > The way to handle it on Linux and the ePAR is different: > - ePAR (chapter 2.4) says: > The physical wiring of an interrupt source to an interrupt controller > is represented in the device tree with the interrupt-parent property. > Nodes that represent interrupt-generating devices contain an > interrupt-parent property which has a phandle value that points to the > device to which the device's interrupts are routed, typically an > interrupt controller. If an interrupt-generating device does not have > an interrupt-parent property, its interrupt parent is assumed to be > its device tree parent. > From my understanding, it's mandatory to have an interrupt-parent > property on each device node which is using interrupts. If it doesn't > exist it will assume that the parent is interrupt controller. > If I'm mistaken, at least FreeBSD handle the interrupt-parent property > in this way. > - Linux implementation will look at to the node, if the property > doesn't exists, it will check if the ancestor has this property ... > So the device tree below is valid on Linux, but not on FreeBSD: > > / { > interrupt-parent = &gic > > gic: gic@10 > { > ... > } > > timer@1 > { > interrupts = <...> > } > } > > Most of shipped device tree use this trick. > > IanC: I was reading the linux binding documentation > (devicetree/booting-without-of.txt VII.2) and it seems that the > explanation differs from the implementation, right? > > For the #interrupt-cells property, the problem starts in > fdt_intr_to_rl (sys/dev/fdt/fdt_common.c:476). ofw_bus_map_intr is > called always with the first cells of the interrupt no matter the > number of cells specified by #interrupt-cells. The specification is actually a little unclear on this point, but FreeBSD follows the same rules as Linux in any case. Most, if not all, FreeBSD code should check any ancestor at this point as well. In particular fdt_intr_to_rl does this. What it *doesn't* do is allow #interrupt-cells to be larger than 2. I'll fix this this weekend. >> On the subject of simple-bus, they usually aren't necessary. For >> example, all hypervisor devices on IBM hardware live under /vdevice, >> which is attached to the device tree root. They don't use MMIO, so >> simple-bus doesn't really make sense. How does Xen communicate with the >> OS in these devices? >> -Nathan > > As I understand, only the simple bus code (see simplebus_attach) is > translating the interrupts in the device on a resource. > So if you have a node directly attached to the root node with > interrupts and MMIO, the driver won't be able to retrieve and > translate the interrupts via bus_alloc_resources. Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this. > In the Xen device tree, we have an hypervisor node directly attach to > the root which contains both MMIO and interrupt used by Xen to > communicate with the guest. > OK. This should be fine, though simplebus would also work if you use MMIO. -Nathan From owner-freebsd-xen@FreeBSD.ORG Fri Jan 17 09:29:28 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7AB021D5; Fri, 17 Jan 2014 09:29:28 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 30A361FCD; Fri, 17 Jan 2014 09:29:26 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,670,1384300800"; d="scan'208";a="93794219" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 17 Jan 2014 09:29:24 +0000 Received: from [10.80.2.80] (10.80.2.80) by FTLPEX01CL03.citrite.net (10.13.107.80) with Microsoft SMTP Server id 14.2.342.4; Fri, 17 Jan 2014 04:29:24 -0500 Message-ID: <1389950962.6697.33.camel@kazak.uk.xensource.com> Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD From: Ian Campbell To: Julien Grall Date: Fri, 17 Jan 2014 09:29:22 +0000 In-Reply-To: <52D87B15.5090208@linaro.org> References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> <52D73C4E.2080306@freebsd.org> <52D87B15.5090208@linaro.org> Organization: Citrix Systems, Inc. Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.4.4-3 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Originating-IP: [10.80.2.80] X-DLP: MIA1 Cc: stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, Nathan Whitehorn , gibbs@freebsd.org, Warner Losh X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jan 2014 09:29:28 -0000 On Fri, 2014-01-17 at 00:36 +0000, Julien Grall wrote: > > On 01/16/2014 01:56 AM, Nathan Whitehorn wrote: > > > > Thanks for the CC. Could you explain what you mean by "grant-parent" > > etc? "interrupt-parent" is a fundamental part of the way PAPR (and > > ePAPR) work, so I'm very very hesitant to start guessing. I think things > > have broken for you because some (a lot, actually) of OF code does not > > expect #interrupt-cells to be more than 2. Some APIs might need > > updating, which I'll try to take care of. Could you tell me what the > > difference between SPI and PPI is, by the way? > > Sorry, I also made some typoes in my explanation so it was not clear. > > interrupt-parent is a property in a device node which links this node to > an interrupt controller (in our case the GIC controller). > > The way to handle it on Linux and the ePAR is different: > - ePAR (chapter 2.4) says: > The physical wiring of an interrupt source to an interrupt controller is > represented in the device tree with the interrupt-parent property. Nodes > that represent interrupt-generating devices contain an > interrupt-parent property which has a phandle value that points to the > device to which the device's interrupts are routed, typically an > interrupt controller. If an interrupt-generating device does not have > an interrupt-parent property, its interrupt parent is assumed to be its > device tree parent. > From my understanding, it's mandatory to have an interrupt-parent > property on each device node which is using interrupts. If it doesn't > exist it will assume that the parent is interrupt controller. > If I'm mistaken, at least FreeBSD handle the interrupt-parent property > in this way. > - Linux implementation will look at to the node, if the property > doesn't exists, it will check if the ancestor has this property ... > > So the device tree below is valid on Linux, but not on FreeBSD: > > / { > interrupt-parent = &gic > > gic: gic@10 > { > ... > } > > timer@1 > { > interrupts = <...> > } > } > > Most of shipped device tree use this trick. > > IanC: I was reading the linux binding documentation > (devicetree/booting-without-of.txt VII.2) and it seems that the > explanation differs from the implementation, right? I vaguely recall someone saying that the Linux behaviour was a quirk of some real PPC system which supplied a DTB which required this behaviour which has leaked into the other platforms. It does also sound like a useful extension to the spec which makes the dtb easier to write. The right place to discuss this would probably be either #devicetree on freenode or devicetree@vger.kernel.org. > > On the subject of simple-bus, they usually aren't necessary. For > > example, all hypervisor devices on IBM hardware live under /vdevice, > > which is attached to the device tree root. They don't use MMIO, so > > simple-bus doesn't really make sense. How does Xen communicate with the > > OS in these devices? > > -Nathan > > As I understand, only the simple bus code (see simplebus_attach) is > translating the interrupts in the device on a resource. > So if you have a node directly attached to the root node with interrupts > and MMIO, the driver won't be able to retrieve and translate the > interrupts via bus_alloc_resources. Is the root node not considered to be a "top-level simple-bus" with a 1:1 mapping of MMIO and interrupts? (Linux seems to treat it this way, but I haven't trawled the docs for a spec reference to back that behaviour up). I take it BSD doesn't do this? Ian. From owner-freebsd-xen@FreeBSD.ORG Fri Jan 17 13:45:33 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 358D87D5 for ; Fri, 17 Jan 2014 13:45:33 +0000 (UTC) Received: from mail-wi0-f182.google.com (mail-wi0-f182.google.com [209.85.212.182]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 92E6714C7 for ; Fri, 17 Jan 2014 13:45:32 +0000 (UTC) Received: by mail-wi0-f182.google.com with SMTP id ex4so705488wid.9 for ; Fri, 17 Jan 2014 05:45:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=o20v3u1W/AFMqlLmKwxjEwoJnXnNjVeiKZVFU838MHA=; b=a6HZQsQulsThNKH5rcBbWqDFpNjyVeHMRDAvPGitF9aCmr2YxYqM8+ZolKPlT/wxuD koA+PoX1RUwaIHG90HUpqPjpoWtzH9PO5O5dcmuQVt0JCqkPzVFDeQN+L+Kts9zF+3jn irucXizGtxqn82iMKFO+GB6+8dse4LnfHZImjLSO3vBYU1rYCDCNlv2szHkIMXRmbKdL QUVT0zdXOoEaCFbMsQAVeKmIWBBRDdt+kIrVSwRVqADNrWspFMI2wnoVUAD7jlaPTn8Z bMNOJ0JOqS3GuM9nJGHuoN5pfbjQNYrcEPXmgEMY4WwnC2BJoP5HseVhWATOaFbFQK4X RhxQ== X-Gm-Message-State: ALoCoQkCft3uuX75gR1yAiqDUaxYec8xkQke3Fk8BNqa+uvJzsX7m3UJxBKVv4UStDDsy6hDuUwr X-Received: by 10.180.160.166 with SMTP id xl6mr2472944wib.43.1389966324694; Fri, 17 Jan 2014 05:45:24 -0800 (PST) Received: from [192.168.0.2] (cpc8-cmbg15-2-0-cust169.5-4.cable.virginm.net. [86.30.140.170]) by mx.google.com with ESMTPSA id dm2sm3286200wib.8.2014.01.17.05.45.23 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 17 Jan 2014 05:45:23 -0800 (PST) Message-ID: <52D933F2.8000101@linaro.org> Date: Fri, 17 Jan 2014 13:45:22 +0000 From: Julien Grall User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Nathan Whitehorn , Warner Losh Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> <52D73C4E.2080306@freebsd.org> <52D87B15.5090208@linaro.org> <52D89DC9.7050303@freebsd.org> In-Reply-To: <52D89DC9.7050303@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jan 2014 13:45:33 -0000 On 01/17/2014 03:04 AM, Nathan Whitehorn wrote: > On 01/16/14 18:36, Julien Grall wrote: > > The specification is actually a little unclear on this point, but > FreeBSD follows the same rules as Linux in any case. Most, if not all, > FreeBSD code should check any ancestor at this point as well. In > particular fdt_intr_to_rl does this. What it *doesn't* do is allow > #interrupt-cells to be larger than 2. I'll fix this this weekend. Thanks, for working on this part. Another things to take into account: the first cell doesn't always contain the interrupt. With the Linux binding (#interrupt-cells == 3) - cell 1: 1 or 0 (PPI vs SPI) - cell 2: relative IRQ number to the start of PPI/SPI - cell 3: cpu mask + interrupt flags (edge/level...) >>> On the subject of simple-bus, they usually aren't necessary. For >>> example, all hypervisor devices on IBM hardware live under /vdevice, >>> which is attached to the device tree root. They don't use MMIO, so >>> simple-bus doesn't really make sense. How does Xen communicate with the >>> OS in these devices? >>> -Nathan >> >> As I understand, only the simple bus code (see simplebus_attach) is >> translating the interrupts in the device on a resource. >> So if you have a node directly attached to the root node with >> interrupts and MMIO, the driver won't be able to retrieve and >> translate the interrupts via bus_alloc_resources. > > Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this. I have noticed at least one issue (which is not related to my problem): - When the OFW nexus translate IRQ (with #interrupt-cells > 1), the rid will be equal to 0, 0 + #interrupt-cells, ... So the number will be discontinued. Rather than on simple-bus for the same device, the rid will be 0, 1, 2... For my issue, I will look at it again this week-end. BTW when I look to the FDT (sys/dev/fdt_common.c) and the ofw (sys/dev/ofw_nexus.c) code, I have notice that lots of code are duplicated. It would be nice to have common helper to avoid duplicate code and issue for the future :). -- Julien Grall From owner-freebsd-xen@FreeBSD.ORG Fri Jan 17 13:49:44 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 866F586E for ; Fri, 17 Jan 2014 13:49:44 +0000 (UTC) Received: from mail-we0-f176.google.com (mail-we0-f176.google.com [74.125.82.176]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 121BF14F3 for ; Fri, 17 Jan 2014 13:49:43 +0000 (UTC) Received: by mail-we0-f176.google.com with SMTP id t61so180589wes.21 for ; Fri, 17 Jan 2014 05:49:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=gptWh6SCeZgU4ox3TMfLGNjiaIT4/FIrU9bsOrmbQ3o=; b=lYFfqoHu3Qyv/YPOPhvZgbmqKGBYcNbl9Vi5cvGZt/fquT2ZsW99pO1uAVG5mmm4iI 3p0Pdy82JOQWZWijgdfIYcXTvZCZQ+vQaZZkft3YpMqhToZL5L7i91dAoSP45Cu6IBsD hO81mQhHIuRwABO0KErE5W+IUi37ClA5Nq3bulvBk4/LrFYTSIK8kb2kRr9gRnjvJBDR AVIfdNHus9KQ3+mp8RymnzU6pBLLgjQqze416555pDDOZxukjSrvQhC1tqfOELCble7b NwtZVhXxOrueik7KCWk/6yBPEWQebGe3gnzn2pCxZ4PUa8efst6ZKQa2L6CikJAeBNGX lsYg== X-Gm-Message-State: ALoCoQmiXKQGL5v+jnkmbiq4RqJ4DANyIk6s/XjI7aX3OzS4D+UZGo6ihokuIT8sFZ4VpKjR9+Z5 X-Received: by 10.194.92.7 with SMTP id ci7mr2157335wjb.58.1389966582419; Fri, 17 Jan 2014 05:49:42 -0800 (PST) Received: from [192.168.0.2] (cpc8-cmbg15-2-0-cust169.5-4.cable.virginm.net. [86.30.140.170]) by mx.google.com with ESMTPSA id x4sm3327855wif.0.2014.01.17.05.49.40 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 17 Jan 2014 05:49:41 -0800 (PST) Message-ID: <52D934F4.709@linaro.org> Date: Fri, 17 Jan 2014 13:49:40 +0000 From: Julien Grall User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Ian Campbell Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> <52D73C4E.2080306@freebsd.org> <52D87B15.5090208@linaro.org> <1389950962.6697.33.camel@kazak.uk.xensource.com> In-Reply-To: <1389950962.6697.33.camel@kazak.uk.xensource.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, Nathan Whitehorn , gibbs@freebsd.org, Warner Losh X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jan 2014 13:49:44 -0000 On 01/17/2014 09:29 AM, Ian Campbell wrote: > On Fri, 2014-01-17 at 00:36 +0000, Julien Grall wrote: >> >> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote: >>> On the subject of simple-bus, they usually aren't necessary. For >>> example, all hypervisor devices on IBM hardware live under /vdevice, >>> which is attached to the device tree root. They don't use MMIO, so >>> simple-bus doesn't really make sense. How does Xen communicate with the >>> OS in these devices? >>> -Nathan >> >> As I understand, only the simple bus code (see simplebus_attach) is >> translating the interrupts in the device on a resource. >> So if you have a node directly attached to the root node with interrupts >> and MMIO, the driver won't be able to retrieve and translate the >> interrupts via bus_alloc_resources. > > Is the root node not considered to be a "top-level simple-bus" with a > 1:1 mapping of MMIO and interrupts? (Linux seems to treat it this way, > but I haven't trawled the docs for a spec reference to back that > behaviour up). I take it BSD doesn't do this? There is 2 different paths on FreeBSD to decode interrupt/MMIO (depending if you are under the root node or a simple-bus node). Most of the code is duplicated but there are some parts which differs (for instance interrupt decoding, see my answer to Nathan). I will look at closer to the code this week-end and see if I can fix it. -- Julien Grall From owner-freebsd-xen@FreeBSD.ORG Fri Jan 17 15:27:30 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C00A894C; Fri, 17 Jan 2014 15:27:30 +0000 (UTC) Received: from smtpauth4.wiscmail.wisc.edu (wmauth4.doit.wisc.edu [144.92.197.145]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8879B1E00; Fri, 17 Jan 2014 15:27:30 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from avs-daemon.smtpauth4.wiscmail.wisc.edu by smtpauth4.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MZJ00200X9S0G00@smtpauth4.wiscmail.wisc.edu>; Fri, 17 Jan 2014 09:27:23 -0600 (CST) X-Spam-PmxInfo: Server=avs-4, Version=6.0.3.2322014, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2014.1.17.151815, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from comporellon.tachypleus.net (adsl-76-208-68-77.dsl.mdsnwi.sbcglobal.net [76.208.68.77]) by smtpauth4.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MZJ00D54XLLZ510@smtpauth4.wiscmail.wisc.edu>; Fri, 17 Jan 2014 09:27:23 -0600 (CST) Message-id: <52D94BD9.5020209@freebsd.org> Date: Fri, 17 Jan 2014 09:27:21 -0600 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0 To: Julien Grall , Warner Losh Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> <52D73C4E.2080306@freebsd.org> <52D87B15.5090208@linaro.org> <52D89DC9.7050303@freebsd.org> <52D933F2.8000101@linaro.org> In-reply-to: <52D933F2.8000101@linaro.org> Cc: ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jan 2014 15:27:30 -0000 On 01/17/14 07:45, Julien Grall wrote: > > > On 01/17/2014 03:04 AM, Nathan Whitehorn wrote: >> On 01/16/14 18:36, Julien Grall wrote: >> >> The specification is actually a little unclear on this point, but >> FreeBSD follows the same rules as Linux in any case. Most, if not all, >> FreeBSD code should check any ancestor at this point as well. In >> particular fdt_intr_to_rl does this. What it *doesn't* do is allow >> #interrupt-cells to be larger than 2. I'll fix this this weekend. > > Thanks, for working on this part. > > Another things to take into account: the first cell doesn't always > contain the interrupt. > With the Linux binding (#interrupt-cells == 3) > - cell 1: 1 or 0 (PPI vs SPI) > - cell 2: relative IRQ number to the start of PPI/SPI > - cell 3: cpu mask + interrupt flags (edge/level...) > ` Yep. This will require a little API redesign, but shouldn't be that bad. >>>> On the subject of simple-bus, they usually aren't necessary. For >>>> example, all hypervisor devices on IBM hardware live under /vdevice, >>>> which is attached to the device tree root. They don't use MMIO, so >>>> simple-bus doesn't really make sense. How does Xen communicate with >>>> the >>>> OS in these devices? >>>> -Nathan >>> >>> As I understand, only the simple bus code (see simplebus_attach) is >>> translating the interrupts in the device on a resource. >>> So if you have a node directly attached to the root node with >>> interrupts and MMIO, the driver won't be able to retrieve and >>> translate the interrupts via bus_alloc_resources. >> >> Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this. > > I have noticed at least one issue (which is not related to my problem): > - When the OFW nexus translate IRQ (with #interrupt-cells > 1), the > rid will be equal to 0, 0 + #interrupt-cells, ... So the number will > be discontinued. Rather than on simple-bus for the same device, the > rid will be 0, 1, 2... Interesting. I'll investigate. > For my issue, I will look at it again this week-end. > > BTW when I look to the FDT (sys/dev/fdt_common.c) and the ofw > (sys/dev/ofw_nexus.c) code, I have notice that lots of code are > duplicated. > > It would be nice to have common helper to avoid duplicate code and > issue for the future :). > I'm in the middle of cleaning all this up (which is how I'm on the hook for breaking your case!). Most of fdt_common.c is not long for this world. -Nathan From owner-freebsd-xen@FreeBSD.ORG Sat Jan 18 23:41:54 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F2CD5EF7 for ; Sat, 18 Jan 2014 23:41:54 +0000 (UTC) Received: from mail-wi0-f173.google.com (mail-wi0-f173.google.com [209.85.212.173]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 825781AA2 for ; Sat, 18 Jan 2014 23:41:54 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id d13so2047703wiw.0 for ; Sat, 18 Jan 2014 15:41:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=h76klGEasTqnnGciDyydOIaeOjOmyJ0mgLXev17A2ug=; b=TNjrv7LlC3LaIbGDzdGBdxos61wO33OKn2vzSbpgJZ0hkviKCJ05mk6Et5lYKVYzQt 8REh6SKCpZa2ewkGpukwMbOLz5UrFtEeHWRI8qt1vItRnZ+yrP5rlvgl9QGSKBWmbJRE JygTE+Uo8CZ0MW4YSvBifMCez6cEdrFnw5ABcLncE6ELv2/TSWDjntAzRN0fsnamwTV/ mKY7ce6TcW8O+ND/FENi+leEEGwfrL+gdWyCTxWcauZQQWQ+Od3T5iryAd+UQa62h2OT yINyedbhADeplxLyJgpCUUYCnF818JjLUVMRUOx5ZrQbRtUtStjbYdcmy/kO8ydy7IEs I6mw== X-Gm-Message-State: ALoCoQlOx1SfDZs0o5MAiHAh/yET3CnIoSQGmqKlwuwz1oMKn60Tvw5NtqL6hMZIq3EEm61JBKbE X-Received: by 10.194.175.202 with SMTP id cc10mr8113022wjc.48.1390088506777; Sat, 18 Jan 2014 15:41:46 -0800 (PST) Received: from [192.168.0.2] (cpc8-cmbg15-2-0-cust169.5-4.cable.virginm.net. [86.30.140.170]) by mx.google.com with ESMTPSA id w1sm16375423wix.1.2014.01.18.15.41.45 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 18 Jan 2014 15:41:46 -0800 (PST) Message-ID: <52DB1138.6010804@linaro.org> Date: Sat, 18 Jan 2014 23:41:44 +0000 From: Julien Grall User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Nathan Whitehorn , Warner Losh Subject: Re: [RFC] Add support for Xen ARM guest on FreeBSD References: <1389733267-20822-1-git-send-email-julien.grall@linaro.org> <24851B79-7EC7-4E3A-94DB-4B9B86FDFFFC@bsdimp.com> <52D6B62A.9000208@linaro.org> <52D73C4E.2080306@freebsd.org> <52D87B15.5090208@linaro.org> <52D89DC9.7050303@freebsd.org> In-Reply-To: <52D89DC9.7050303@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: ian.campbell@citrix.com, stefano.stabellini@eu.citrix.com, xen-devel@lists.xen.org, freebsd-xen@freebsd.org, freebsd-arm@FreeBSD.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Jan 2014 23:41:55 -0000 Hello Nathan, On 01/17/2014 03:04 AM, Nathan Whitehorn wrote: > On 01/16/14 18:36, Julien Grall wrote: >> >> >> On 01/16/2014 01:56 AM, Nathan Whitehorn wrote: >> As I understand, only the simple bus code (see simplebus_attach) is >> translating the interrupts in the device on a resource. >> So if you have a node directly attached to the root node with >> interrupts and MMIO, the driver won't be able to retrieve and >> translate the interrupts via bus_alloc_resources. > Why not? nexus on ARM, MIPS, PowerPC, and sparc64 can do this. I have digged into the code to find the reason of my issue. FreeBSD is receiving a VM fault when the driver (xen-dt) is trying to setup the IRQ. This is because the GIC is not yet initialized but FreeBSD asks to unmask the IRQ (sys/arm/arm/gic.c:306). With this problem, all device nodes that are before the GIC in the device tree can't have interrupts. For instance this simple device will segfault on FreeBSD: / { mybus { compatible = "simple-bus"; mynode { interrupt-parent = &gic; interrupts = <...>; }; gic: gic@xxxx { interrupt-controller; } }; }; The node "mynode" will have to move after the GIC to be able to work correctly. -- Julien Grall