From owner-freebsd-xen@FreeBSD.ORG Mon Dec 16 11:07:00 2013 Return-Path: Delivered-To: freebsd-xen@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EE588FA7 for ; Mon, 16 Dec 2013 11:06:59 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D8F481369 for ; Mon, 16 Dec 2013 11:06:59 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id rBGB6xmC019591 for ; Mon, 16 Dec 2013 11:06:59 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id rBGB6xBa019589 for freebsd-xen@FreeBSD.org; Mon, 16 Dec 2013 11:06:59 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 16 Dec 2013 11:06:59 GMT Message-Id: <201312161106.rBGB6xBa019589@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-xen@FreeBSD.org Subject: Current problem reports assigned to freebsd-xen@FreeBSD.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Dec 2013 11:07:00 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/183139 xen [xen] [patch] ifconfig options on xn0 lost after xen v o kern/180788 xen [xen] [panic] XEN PV kernel 9.2-BETA1 panics on boot o kern/180403 xen [xen] Problems with GENERIC and XENHVM kernels with Xe o kern/180402 xen [xen] XEN kernel does not load in XenClient 4.5.5 o kern/179814 xen [xen] mountroot fails with error=19 under Xen on 9-STA o kern/176471 xen [xen] xn driver crash on detach o kern/176053 xen [xen] [patch] i386: Correct wrong usage of vsnprintf() o kern/175954 xen [xen] XENHVM xn network driver extreme packet loss dur o kern/175822 xen [xen] FreeBSD 9.1 does not work with Xen 4.0 o kern/175757 xen [xen] [patch] xen pvhvm looses keyboard input from VNC o kern/171873 xen [xen] xn network device floods warning in dmesg o kern/171118 xen [xen] FreeBSD XENHVM guest doesn't shutdown cleanly o kern/166174 xen [xen] Problems ROOT MOUNT ERROR o kern/165418 xen [xen] Problems mounting root filesystem from XENHVM o kern/164630 xen [xen] XEN HVM kernel: run_interrupt_driven_hooks: stil o kern/164450 xen [xen] Failed to install FreeeBSD 9.0-RELEASE from CD i o kern/162677 xen [xen] FreeBSD not compatible with "Current Stable Xen" o kern/161318 xen [xen] sysinstall crashes with floating point exception o kern/155468 xen [xen] Xen PV i386 multi-kernel CPU system is not worki o kern/155353 xen [xen] [patch] put "nudging TOD" message under boot_ver o kern/154833 xen [xen]: xen 4.0 - DomU freebsd8.2RC3 i386, XEN kernel. o kern/154473 xen [xen] xen 4.0 - DomU freebsd8.1 i386, XEN kernel. Not o kern/154472 xen [xen] xen 4.0 - DomU freebsd8.1 i386 xen kernel reboot o kern/154428 xen [xen] xn0 network interface and PF - Massive performan o kern/153674 xen [xen] i386/XEN idle thread shows wrong percentages o kern/153672 xen [xen] [panic] i386/XEN panics under heavy fork load o kern/153620 xen [xen] Xen guest system clock drifts in AWS EC2 (FreeBS o kern/153477 xen [xen] XEN pmap code abuses vm page queue lock o kern/153150 xen [xen] xen/ec2: disable checksum offloading on interfac o kern/152228 xen [xen] [panic] Xen/PV panic with machdep.idle_mwait=1 o kern/144629 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143398 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143340 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor f kern/143069 xen [xen] [panic] Xen Kernel Panic - Memory modified after f kern/135667 xen ufs filesystem corruption on XEN DomU system f kern/135421 xen [xen] FreeBSD Xen PVM DomU network failure - netfronc. f kern/135178 xen [xen] Xen domU outgoing data transfer stall when TSO i p kern/135069 xen [xen] FreeBSD-current/Xen SMP doesn't function at all f i386/124516 xen [xen] FreeBSD-CURRENT Xen Kernel Segfaults when config o kern/118734 xen [xen] FreeBSD 6.3-RC1 and FreeBSD 7.0-BETA 4 fail to b 40 problems total. From owner-freebsd-xen@FreeBSD.ORG Mon Dec 16 12:48:00 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7A9849C3 for ; Mon, 16 Dec 2013 12:48:00 +0000 (UTC) Received: from mail.tdx.com (mail.tdx.com [62.13.128.18]) by mx1.freebsd.org (Postfix) with ESMTP id 43C941D5B for ; Mon, 16 Dec 2013 12:47:59 +0000 (UTC) Received: from Mail-PC.tdx.co.uk (storm.tdx.co.uk [62.13.130.251]) (authenticated bits=0) by mail.tdx.com (8.14.3/8.14.3/) with ESMTP id rBGCh4iZ043468 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 16 Dec 2013 12:43:04 GMT Date: Mon, 16 Dec 2013 12:43:04 +0000 From: Karl Pielorz To: freebsd-xen@freebsd.org Subject: Issue net booting FreeBSD 9.x under Xen? Message-ID: <37F78C071BE942883D93BF6F@Mail-PC.tdx.co.uk> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Dec 2013 12:48:00 -0000 Hi, We've a small setup with a number of XenServer (6.2) machines - and a PXE boot environment. Real hardware boots fine off the network (the machine we use to boot from is a FreeBSD box itself), as does a VMware guest - but a Xen VM won't boot. Booting 9.2-amd64, it gets as far as: " PXE version 2.1, real mode entry point @9ad4:0255 BIOS 619kB/2090572kB available memory FreeBSD/i386 bootstrap loader, Revision 1.1 (root@bake.isc.freebsd.org, Fri Sep 27 04:58:05 UTC 2013) pxe_open: server addr: 10.0.0.10 pxe_open: server path: /home/netboot/os/9.2-amd64 pxe_open: gateway ip: 10.0.0.1 / " And that's it - it just hangs at that point. There's no errors logged on the server it's booting 'from', and I can't see any obvious network traffic from the client back to the server. Any idea how to troubleshoot / fix this? Thanks, -Karl From owner-freebsd-xen@FreeBSD.ORG Wed Dec 18 00:16:28 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1D1BFEE6 for ; Wed, 18 Dec 2013 00:16:28 +0000 (UTC) Received: from turing.morphism.de (smtp.morphism.de [IPv6:2001:4178:4:204::25]) by mx1.freebsd.org (Postfix) with ESMTP id CB8601B39 for ; Wed, 18 Dec 2013 00:16:27 +0000 (UTC) Received: from localhost (moore.morphism.de [IPv6:2001:4178:4:202::136]) by turing.morphism.de (Postfix) with ESMTP id 59A1D4F89C for ; Wed, 18 Dec 2013 00:16:22 +0000 (UTC) Date: Wed, 18 Dec 2013 00:16:15 +0000 From: Markus Pfeiffer To: freebsd-xen@freebsd.org Subject: Panic with FreeBSD 10 RC2 on Netbsd Xen dom0 Message-ID: <20131218001615.GA10501@moore.morphism.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="tKW2IUtsqtDRztdT" Content-Disposition: inline User-Agent: Mutt/1.5.22 (2013-10-16) X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list Reply-To: Markus Pfeiffer List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2013 00:16:28 -0000 --tKW2IUtsqtDRztdT Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Dear all, I tried installing FreeBSD10-RC2 under Netbsd XEN dom0 (xen 4.2.0). Unfortunately when I try connecting to the guest with ssh, the machine panics. I managed to take a screenshot of the output at the time of the panic: http://imgur.com/MH8Z3Xz If needed I can probably upload the crashdump. Cheers, Markus --tKW2IUtsqtDRztdT Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (DragonFly) iQIcBAEBAgAGBQJSsOlPAAoJEBRHBRYBD4mPkccQALJO6bnKaDaNjnjAMNPpJk/L oviDpYfxRK92LCMfJfPxqTJHdUjNGRYppbn/VLgO8bPltn9+Cyhs9J4+KCzaoW9l F3G8UJQXswOsRKYiCmNcRDW4vU+mmXgBUxacOr2qkhQ77Apa+LzYhED06AvB1/mW 3sB+kKYb8mBebeGr6cyXVLSq1W4Y4KCraxXBA8cdC6gcUqO4YJ3Ryvzm4Qe99iwR HvN9xxQ7jbs6uk0YK7yt4NZIuZa+VG+Y+sFckawUHYv2rxFZGLvqYnHw2Q2lbCPD 3lMjf183am88SC4RGzoEziGVsDNQrxHWKNDEWOlFyHi5/Ym1VUbVxLtBdNM8op6B FV54lM4qY8IUhCeyD1mfa7RoCoGqCQGp4FREb4iYZiPwoB8vtaN5qiQ5PEWNrz2S 3UXxg6q6qc6TfU//GZ2eRDBazgJdBWpvs6OTuFIiTd/7faEN8FbWg5MsGd8yhD2q FmEUBAcnF1+6fK8w2SkVpKPXjGDC+18WaRH6Kq/CC9dQnKVQkWffXhcecSAQg3V7 eSTyiejQEmVDeL1Q5oQbCNpf3zNkG2uJkQ3/Wn1e3NUjiokyf+70HMgt8Zjnafa4 EB7lWPGNgiGYI+s2Ql6DQD/S/39D/Tr09deN3TfVJpf9jRyQJCLakVnpVcplw24r nSzl4S25x0AQYWVx6gtU =Xc0z -----END PGP SIGNATURE----- --tKW2IUtsqtDRztdT-- From owner-freebsd-xen@FreeBSD.ORG Wed Dec 18 22:48:23 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 321B488C for ; Wed, 18 Dec 2013 22:48:23 +0000 (UTC) Received: from mail-yh0-f41.google.com (mail-yh0-f41.google.com [209.85.213.41]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E2F291EC7 for ; Wed, 18 Dec 2013 22:48:22 +0000 (UTC) Received: by mail-yh0-f41.google.com with SMTP id f11so136849yha.28 for ; Wed, 18 Dec 2013 14:48:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=nzM4uiKIyH+iopk6pI1yRrKc65yWGAhYMK3FgnpP61c=; b=jmgs7L3WQvIytvofMDB5yXx5vFIUbivsu5QP1tYspqzY6Gmzou8ztcPpwIpHMzcLHV 9dwIHSbTG00rd6P9fwmH4KzBnltleL0PMlXWHXLG0xkkMAnzQfWSUNV98Uohi/pilhmB p9mOrWp7oqtvxAox2UHMsQ6TBt96EueOM0w7fWGekEpqqew20H+lCGuM3L+yXHo8RKM+ dSm9n9NHn6iLJgJqNFbGlubXuJE/0NmAAFZvsiBudRBLwzsL22qDVqaoKWkR3NrPMNEM tucB5LVuIpNwthyFcYY4Y8ccAuyhkL77YH9KcuLZ7Pi9daBmEdHcM6Wa6MRs983Wnska L25A== X-Gm-Message-State: ALoCoQkc99J9igyZRDPsQAQwlSbahh+n7bLQAnSDKO0t5ON6RQRc/VaruFX+EMmeUEXd3CMJ8dnJ X-Received: by 10.236.45.102 with SMTP id o66mr24631958yhb.13.1387405111690; Wed, 18 Dec 2013 14:18:31 -0800 (PST) Received: from [192.168.25.9] ([177.97.204.135]) by mx.google.com with ESMTPSA id e39sm2752992yhq.15.2013.12.18.14.18.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 18 Dec 2013 14:18:30 -0800 (PST) Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: Panic with FreeBSD 10 RC2 on Netbsd Xen dom0 From: Tiago Ribeiro In-Reply-To: <20131218001615.GA10501@moore.morphism.de> Date: Wed, 18 Dec 2013 20:18:24 -0200 Message-Id: References: <20131218001615.GA10501@moore.morphism.de> To: Markus Pfeiffer X-Mailer: Apple Mail (2.1827) Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Dec 2013 22:48:23 -0000 Em 17/12/2013, =E0(s) 22:16, Markus Pfeiffer = escreveu: > Dear all, >=20 > I tried installing FreeBSD10-RC2 under Netbsd XEN dom0 (xen 4.2.0). > Unfortunately when I try connecting to the guest with ssh, the machine = panics. >=20 > I managed to take a screenshot of the output at the time of the panic: >=20 > http://imgur.com/MH8Z3Xz >=20 > If needed I can probably upload the crashdump. >=20 > Cheers, I'm have same problem Dom0: NetBSD 6.0.1 - Xen 4.0.1 DomU: FreeBSD 10.0 RC2 - i386 Instalation of freebsd is from bootonly iso. Some tests: Domu console - ssh to localhost - Ok DomU console - svnlite co http://svn0.eu.freebsd.org/ports/head = - NOK DomU console - ping www.google.com - OK any host - ping to DomU - OK DomU console - fetch = ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/10.0-RC2/src.txz - OK DomU console - portsnap fetch - NOK -- www.bsdjf.com.br From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 01:38:07 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D910AEAC for ; Thu, 19 Dec 2013 01:38:07 +0000 (UTC) Received: from mail-wg0-x236.google.com (mail-wg0-x236.google.com [IPv6:2a00:1450:400c:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6FD681B14 for ; Thu, 19 Dec 2013 01:38:07 +0000 (UTC) Received: by mail-wg0-f54.google.com with SMTP id n12so422884wgh.21 for ; Wed, 18 Dec 2013 17:38:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=1pAK/9as7ukjFsN0cxdTjdlYkvzBnPwIpnohR0ABqKA=; b=UXTAcVy+v+rzP2Hx84BwzUabEMnz/uI8qf52lrv/dZcXn9YgiVGl8lA96IWPNkBFF2 Y6Mz9ABGunwPY+sLPvnYHKqC3Kf4xj2AzpkUktJiJDI8jI/bQY4pS7CTOqD3BMZ08n8k BOYQgdbBr6BuQv/m8mWiTu0Xr+hkSVy98qWWAewgiEn2tyRX6bJZQdsXDojlS69jdObV 4YeZqjI03s1TgpeaPW/4dxDCNL8ErEY2rBEo9BEvB54TSV9C2s9o0e1nomt3rCj9C263 8P6bXf+aVnTef5qg5POjfrKThgNEomASfj5xykdJSZ3W4Bxc8+392/1zhpER4zzWUVN6 gP4g== X-Received: by 10.194.20.230 with SMTP id q6mr27314160wje.49.1387417085908; Wed, 18 Dec 2013 17:38:05 -0800 (PST) Received: from [10.10.50.70] (247.241.54.77.rev.vodafone.pt. [77.54.241.247]) by mx.google.com with ESMTPSA id p9sm5263630wia.10.2013.12.18.17.38.05 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 18 Dec 2013 17:38:05 -0800 (PST) Message-ID: <52B24E18.7090109@gmail.com> Date: Thu, 19 Dec 2013 01:38:32 +0000 From: "Mike C." User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.1.0 MIME-Version: 1.0 To: freebsd-xen@freebsd.org Subject: Re: Panic with FreeBSD 10 RC2 on Netbsd Xen dom0 References: <20131218001615.GA10501@moore.morphism.de> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 01:38:08 -0000 I've reported this to xen-devel I while ago. It worked in the first FreeBSD-10 current releases but them it stoped, I believe a previous issue was re-introduced somehow! NetBSD Xen backend does not support TSO/GSO at all, there was a very similar problem in FreeBSD 9 a while a go, and my guess is that the code tried to use TSO again, and leads to problems if the Dom0 is NetBSD. Also more recently I had problems with Windows GPLPV drivers, and the dev tracked the issue to the same.. however in windows DomU's if I disabled TSO it would work. In FreeBSD at least until Aplha 5, I tried to disable TSO but it still wouldn't work. Thanks On 12/18/13 22:18, Tiago Ribeiro wrote: > > Em 17/12/2013, à(s) 22:16, Markus Pfeiffer escreveu: > >> Dear all, >> >> I tried installing FreeBSD10-RC2 under Netbsd XEN dom0 (xen 4.2.0). >> Unfortunately when I try connecting to the guest with ssh, the machine panics. >> >> I managed to take a screenshot of the output at the time of the panic: >> >> http://imgur.com/MH8Z3Xz >> >> If needed I can probably upload the crashdump. >> >> Cheers, > > I'm have same problem > > Dom0: NetBSD 6.0.1 - Xen 4.0.1 > DomU: FreeBSD 10.0 RC2 - i386 > > Instalation of freebsd is from bootonly iso. > > Some tests: > > Domu console - ssh to localhost - Ok > DomU console - svnlite co http://svn0.eu.freebsd.org/ports/head - NOK > DomU console - ping www.google.com - OK > any host - ping to DomU - OK > DomU console - fetch ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/10.0-RC2/src.txz - OK > DomU console - portsnap fetch - NOK > > > > -- > www.bsdjf.com.br > > > > > _______________________________________________ > freebsd-xen@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-xen > To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" > From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 08:21:03 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4BA58E54 for ; Thu, 19 Dec 2013 08:21:03 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9B72117F5 for ; Thu, 19 Dec 2013 08:21:02 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,512,1384300800"; d="scan'208";a="86059417" Received: from accessns.citrite.net (HELO FTLPEX01CL02.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 19 Dec 2013 08:19:50 +0000 Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.79) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 03:19:50 -0500 Message-ID: <52B2AC26.2020301@citrix.com> Date: Thu, 19 Dec 2013 09:19:50 +0100 From: =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Mike C. , Subject: Re: Panic with FreeBSD 10 RC2 on Netbsd Xen dom0 References: <20131218001615.GA10501@moore.morphism.de> <52B24E18.7090109@gmail.com> In-Reply-To: <52B24E18.7090109@gmail.com> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 08:21:03 -0000 On 19/12/13 02:38, Mike C. wrote: > I've reported this to xen-devel I while ago. > > > It worked in the first FreeBSD-10 current releases but them it stoped, I > believe a previous issue was re-introduced somehow! > > NetBSD Xen backend does not support TSO/GSO at all, there was a very > similar problem in FreeBSD 9 a while a go, and my guess is that the code > tried to use TSO again, and leads to problems if the Dom0 is NetBSD. > > > Also more recently I had problems with Windows GPLPV drivers, and the > dev tracked the issue to the same.. however in windows DomU's if I > disabled TSO it would work. > In FreeBSD at least until Aplha 5, I tried to disable TSO but it still > wouldn't work. By looking at relevant commits in netfront, could you try to revert r251297 and see if that solves the problem? Also, doing a bisect of the commits in netfront would be very helpful in order to identify the issue. Roger. From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 18:55:24 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 91DCAE15; Thu, 19 Dec 2013 18:55:24 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C3D3B1D6C; Thu, 19 Dec 2013 18:55:22 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="86287392" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 19 Dec 2013 18:55:19 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 13:55:18 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikg-0006gd-OH; Thu, 19 Dec 2013 18:55:18 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 01/19] xen: add PV/PVH kernel entry point Date: Thu, 19 Dec 2013 19:54:38 +0100 Message-ID: <1387479296-33389-2-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 18:55:24 -0000 Add the PV/PVH entry point and the low level functions for PVH initialization. --- sys/amd64/amd64/locore.S | 53 +++++++++++++++++++++++++++++++ sys/amd64/amd64/machdep.c | 72 ++++++++++++++++++++++++++++++++++++++++++ sys/amd64/include/asmacros.h | 26 +++++++++++++++ sys/i386/xen/xen_machdep.c | 2 + sys/x86/xen/hvm.c | 1 + sys/xen/xen-os.h | 4 ++ 6 files changed, 158 insertions(+), 0 deletions(-) diff --git a/sys/amd64/amd64/locore.S b/sys/amd64/amd64/locore.S index 55cda3a..e04cc48 100644 --- a/sys/amd64/amd64/locore.S +++ b/sys/amd64/amd64/locore.S @@ -31,6 +31,12 @@ #include #include +#ifdef XENHVM +#include +#define __ASSEMBLY__ +#include +#endif + #include "assym.s" /* @@ -86,3 +92,50 @@ NON_GPROF_ENTRY(btext) ALIGN_DATA /* just to be sure */ .space 0x1000 /* space for bootstack - temporary stack */ bootstack: + +#ifdef XENHVM +/* Xen */ +.section __xen_guest + ELFNOTE(Xen, XEN_ELFNOTE_GUEST_OS, .asciz, "FreeBSD") + ELFNOTE(Xen, XEN_ELFNOTE_GUEST_VERSION, .asciz, "HEAD") + ELFNOTE(Xen, XEN_ELFNOTE_XEN_VERSION, .asciz, "xen-3.0") + ELFNOTE(Xen, XEN_ELFNOTE_VIRT_BASE, .quad, KERNBASE) + ELFNOTE(Xen, XEN_ELFNOTE_PADDR_OFFSET, .quad, KERNBASE) /* Xen honours elf->p_paddr; compensate for this */ + ELFNOTE(Xen, XEN_ELFNOTE_ENTRY, .quad, xen_start) + ELFNOTE(Xen, XEN_ELFNOTE_HYPERCALL_PAGE, .quad, hypercall_page) + ELFNOTE(Xen, XEN_ELFNOTE_HV_START_LOW, .quad, HYPERVISOR_VIRT_START) + ELFNOTE(Xen, XEN_ELFNOTE_FEATURES, .asciz, "writable_descriptor_tables|auto_translated_physmap|supervisor_mode_kernel|hvm_callback_vector") + ELFNOTE(Xen, XEN_ELFNOTE_PAE_MODE, .asciz, "yes") + ELFNOTE(Xen, XEN_ELFNOTE_L1_MFN_VALID, .long, PG_V, PG_V) + ELFNOTE(Xen, XEN_ELFNOTE_LOADER, .asciz, "generic") + ELFNOTE(Xen, XEN_ELFNOTE_SUSPEND_CANCEL, .long, 0) + ELFNOTE(Xen, XEN_ELFNOTE_BSD_SYMTAB, .asciz, "yes") + + .text +.p2align PAGE_SHIFT, 0x90 /* Hypercall_page needs to be PAGE aligned */ + +NON_GPROF_ENTRY(hypercall_page) + .skip 0x1000, 0x90 /* Fill with "nop"s */ + +NON_GPROF_ENTRY(xen_start) + /* Don't trust what the loader gives for rflags. */ + pushq $PSL_KERNEL + popfq + + /* Parameters for the xen init function */ + movq %rsi, %rdi /* shared_info (arg 1) */ + movq %rsp, %rsi /* xenstack (arg 2) */ + + /* Use our own stack */ + movq $bootstack,%rsp + xorl %ebp, %ebp + + /* u_int64_t hammer_time_xen(start_info_t *si, u_int64_t xenstack); */ + call hammer_time_xen + movq %rax, %rsp /* set up kstack for mi_startup() */ + call mi_startup /* autoconfiguration, mountroot etc */ + + /* NOTREACHED */ +0: hlt + jmp 0b +#endif diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index eae657b..a73e33e 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -146,10 +146,17 @@ __FBSDID("$FreeBSD$"); #include #include +#ifdef XENHVM +#include +#endif + /* Sanity check for __curthread() */ CTASSERT(offsetof(struct pcpu, pc_curthread) == 0); extern u_int64_t hammer_time(u_int64_t, u_int64_t); +#ifdef XENHVM +extern u_int64_t hammer_time_xen(start_info_t *, u_int64_t); +#endif extern void printcpuinfo(void); /* XXX header file */ extern void identify_cpu(void); @@ -1683,6 +1690,71 @@ do_next: msgbufp = (struct msgbuf *)PHYS_TO_DMAP(phys_avail[pa_indx]); } +#ifdef XENHVM +/* + * First function called by the Xen PVH boot sequence. + * + * Set some Xen global variables and prepare the environment so it is + * as similar as possible to what native FreeBSD init function expects. + */ +u_int64_t +hammer_time_xen(start_info_t *si, u_int64_t xenstack) +{ + u_int64_t physfree; + u_int64_t *PT4 = (u_int64_t *)xenstack; + u_int64_t *PT3 = (u_int64_t *)(xenstack + PAGE_SIZE); + u_int64_t *PT2 = (u_int64_t *)(xenstack + 2 * PAGE_SIZE); + int i; + + KASSERT((si != NULL && xenstack != 0), + ("invalid start_info or xenstack")); + + /* We use 3 pages of xen stack for the boot pagetables */ + physfree = xenstack + 3 * PAGE_SIZE - KERNBASE; + + /* Setup Xen global variables */ + HYPERVISOR_start_info = si; + HYPERVISOR_shared_info = + (shared_info_t *)(si->shared_info + KERNBASE); + + /* + * Setup some misc global variables for Xen devices + * + * XXX: devices that need this specific variables should + * be rewritten to fetch this info by themselves from the + * start_info page. + */ + xen_store = (struct xenstore_domain_interface *) + (ptoa(si->store_mfn) + KERNBASE); + + xen_domain_type = XEN_PV_DOMAIN; + vm_guest = VM_GUEST_XEN; + + /* + * Use the stack Xen gives us to build the page tables + * as native FreeBSD expects to find them (created + * by the boot trampoline). + */ + for (i = 0; i < 512; i++) { + /* Each slot of the level 4 pages points to the same level 3 page */ + PT4[i] = ((u_int64_t)&PT3[0]) - KERNBASE; + PT4[i] |= PG_V | PG_RW | PG_U; + + /* Each slot of the level 3 pages points to the same level 2 page */ + PT3[i] = ((u_int64_t)&PT2[0]) - KERNBASE; + PT3[i] |= PG_V | PG_RW | PG_U; + + /* The level 2 page slots are mapped with 2MB pages for 1GB. */ + PT2[i] = i * (2 * 1024 * 1024); + PT2[i] |= PG_V | PG_RW | PG_PS | PG_U; + } + load_cr3(((u_int64_t)&PT4[0]) - KERNBASE); + + /* Now we can jump into the native init function */ + return (hammer_time(0, physfree)); +} +#endif + u_int64_t hammer_time(u_int64_t modulep, u_int64_t physfree) { diff --git a/sys/amd64/include/asmacros.h b/sys/amd64/include/asmacros.h index 1fb592a..ce8dce4 100644 --- a/sys/amd64/include/asmacros.h +++ b/sys/amd64/include/asmacros.h @@ -201,4 +201,30 @@ #endif /* LOCORE */ +#ifdef __STDC__ +#define ELFNOTE(name, type, desctype, descdata...) \ +.pushsection .note.name ; \ + .align 4 ; \ + .long 2f - 1f /* namesz */ ; \ + .long 4f - 3f /* descsz */ ; \ + .long type ; \ +1:.asciz #name ; \ +2:.align 4 ; \ +3:desctype descdata ; \ +4:.align 4 ; \ +.popsection +#else /* !__STDC__, i.e. -traditional */ +#define ELFNOTE(name, type, desctype, descdata) \ +.pushsection .note.name ; \ + .align 4 ; \ + .long 2f - 1f /* namesz */ ; \ + .long 4f - 3f /* descsz */ ; \ + .long type ; \ +1:.asciz "name" ; \ +2:.align 4 ; \ +3:desctype descdata ; \ +4:.align 4 ; \ +.popsection +#endif /* __STDC__ */ + #endif /* !_MACHINE_ASMACROS_H_ */ diff --git a/sys/i386/xen/xen_machdep.c b/sys/i386/xen/xen_machdep.c index 7049be6..fd575ee 100644 --- a/sys/i386/xen/xen_machdep.c +++ b/sys/i386/xen/xen_machdep.c @@ -89,6 +89,7 @@ IDTVEC(div), IDTVEC(dbg), IDTVEC(nmi), IDTVEC(bpt), IDTVEC(ofl), int xendebug_flags; start_info_t *xen_start_info; +start_info_t *HYPERVISOR_start_info; shared_info_t *HYPERVISOR_shared_info; xen_pfn_t *xen_machine_phys = machine_to_phys_mapping; xen_pfn_t *xen_phys_machine; @@ -927,6 +928,7 @@ initvalues(start_info_t *startinfo) HYPERVISOR_vm_assist(VMASST_CMD_enable, VMASST_TYPE_4gb_segments_notify); #endif xen_start_info = startinfo; + HYPERVISOR_start_info = startinfo; xen_phys_machine = (xen_pfn_t *)startinfo->mfn_list; IdlePTD = (pd_entry_t *)((uint8_t *)startinfo->pt_base + PAGE_SIZE); diff --git a/sys/x86/xen/hvm.c b/sys/x86/xen/hvm.c index 72811dc..b397721 100644 --- a/sys/x86/xen/hvm.c +++ b/sys/x86/xen/hvm.c @@ -159,6 +159,7 @@ DPCPU_DEFINE(xen_intr_handle_t, ipi_handle[nitems(xen_ipis)]); /** Hypercall table accessed via HYPERVISOR_*_op() methods. */ char *hypercall_stubs; shared_info_t *HYPERVISOR_shared_info; +start_info_t *HYPERVISOR_start_info; #ifdef SMP /*---------------------------- XEN PV IPI Handlers ---------------------------*/ diff --git a/sys/xen/xen-os.h b/sys/xen/xen-os.h index 87644e9..c7474d8 100644 --- a/sys/xen/xen-os.h +++ b/sys/xen/xen-os.h @@ -51,6 +51,10 @@ void force_evtchn_callback(void); extern shared_info_t *HYPERVISOR_shared_info; +extern start_info_t *HYPERVISOR_start_info; + +/* XXX: we need to get rid of this and use HYPERVISOR_start_info directly */ +extern struct xenstore_domain_interface *xen_store; enum xen_domain_type { XEN_NATIVE, /* running on bare hardware */ -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 18:55:30 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3D53F1CC; Thu, 19 Dec 2013 18:55:30 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4ECC01D75; Thu, 19 Dec 2013 18:55:28 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84054232" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 18:55:23 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 13:55:22 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikk-0006gd-Ob; Thu, 19 Dec 2013 18:55:22 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 08/19] xen: use the same hypercall mechanism for XEN and XENHVM Date: Thu, 19 Dec 2013 19:54:45 +0100 Message-ID: <1387479296-33389-9-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 18:55:30 -0000 --- sys/amd64/include/xen/hypercall.h | 7 ------- sys/i386/i386/locore.s | 9 +++++++++ sys/i386/include/xen/hypercall.h | 8 -------- sys/x86/xen/hvm.c | 24 ++++++++++-------------- 4 files changed, 19 insertions(+), 29 deletions(-) diff --git a/sys/amd64/include/xen/hypercall.h b/sys/amd64/include/xen/hypercall.h index a1b2a5c..499fb4d 100644 --- a/sys/amd64/include/xen/hypercall.h +++ b/sys/amd64/include/xen/hypercall.h @@ -51,15 +51,8 @@ #define CONFIG_XEN_COMPAT 0x030002 #define __must_check -#ifdef XEN #define HYPERCALL_STR(name) \ "call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)" -#else -#define HYPERCALL_STR(name) \ - "mov $("STR(__HYPERVISOR_##name)" * 32),%%eax; "\ - "add hypercall_stubs(%%rip),%%rax; " \ - "call *%%rax" -#endif #define _hypercall0(type, name) \ ({ \ diff --git a/sys/i386/i386/locore.s b/sys/i386/i386/locore.s index 68cb430..bd136b1 100644 --- a/sys/i386/i386/locore.s +++ b/sys/i386/i386/locore.s @@ -898,3 +898,12 @@ done_pde: #endif ret + +#ifdef XENHVM +/* Xen Hypercall page */ + .text +.p2align PAGE_SHIFT, 0x90 /* Hypercall_page needs to be PAGE aligned */ + +NON_GPROF_ENTRY(hypercall_page) + .skip 0x1000, 0x90 /* Fill with "nop"s */ +#endif diff --git a/sys/i386/include/xen/hypercall.h b/sys/i386/include/xen/hypercall.h index edc13f4..16b5ee2 100644 --- a/sys/i386/include/xen/hypercall.h +++ b/sys/i386/include/xen/hypercall.h @@ -39,16 +39,8 @@ #define ENOXENSYS 38 #define CONFIG_XEN_COMPAT 0x030002 - -#if defined(XEN) #define HYPERCALL_STR(name) \ "call hypercall_page + ("STR(__HYPERVISOR_##name)" * 32)" -#else -#define HYPERCALL_STR(name) \ - "mov hypercall_stubs,%%eax; " \ - "add $("STR(__HYPERVISOR_##name)" * 32),%%eax; " \ - "call *%%eax" -#endif #define _hypercall0(type, name) \ ({ \ diff --git a/sys/x86/xen/hvm.c b/sys/x86/xen/hvm.c index b397721..9a0411e 100644 --- a/sys/x86/xen/hvm.c +++ b/sys/x86/xen/hvm.c @@ -157,7 +157,7 @@ DPCPU_DEFINE(xen_intr_handle_t, ipi_handle[nitems(xen_ipis)]); /*------------------ Hypervisor Access Shared Memory Regions -----------------*/ /** Hypercall table accessed via HYPERVISOR_*_op() methods. */ -char *hypercall_stubs; +extern char *hypercall_page; shared_info_t *HYPERVISOR_shared_info; start_info_t *HYPERVISOR_start_info; @@ -559,7 +559,7 @@ xen_hvm_cpuid_base(void) * Allocate and fill in the hypcall page. */ static int -xen_hvm_init_hypercall_stubs(void) +xen_hvm_init_hypercall_stubs(enum xen_hvm_init_type init_type) { uint32_t base, regs[4]; int i; @@ -568,7 +568,7 @@ xen_hvm_init_hypercall_stubs(void) if (base == 0) return (ENXIO); - if (hypercall_stubs == NULL) { + if (init_type == XEN_HVM_INIT_COLD) { do_cpuid(base + 1, regs); printf("XEN: Hypervisor version %d.%d detected.\n", regs[0] >> 16, regs[0] & 0xffff); @@ -578,18 +578,9 @@ xen_hvm_init_hypercall_stubs(void) * Find the hypercall pages. */ do_cpuid(base + 2, regs); - - if (hypercall_stubs == NULL) { - size_t call_region_size; - - call_region_size = regs[0] * PAGE_SIZE; - hypercall_stubs = malloc(call_region_size, M_XENHVM, M_NOWAIT); - if (hypercall_stubs == NULL) - panic("Unable to allocate Xen hypercall region"); - } for (i = 0; i < regs[0]; i++) - wrmsr(regs[1], vtophys(hypercall_stubs + i * PAGE_SIZE) + i); + wrmsr(regs[1], vtophys(&hypercall_page + i * PAGE_SIZE) + i); return (0); } @@ -692,7 +683,12 @@ xen_hvm_init(enum xen_hvm_init_type init_type) if (init_type == XEN_HVM_INIT_CANCELLED_SUSPEND) return; - error = xen_hvm_init_hypercall_stubs(); + if (xen_pv_domain()) { + /* hypercall page is already set in the PV case */ + error = 0; + } else { + error = xen_hvm_init_hypercall_stubs(init_type); + } switch (init_type) { case XEN_HVM_INIT_COLD: -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 18:55:28 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E6086174; Thu, 19 Dec 2013 18:55:27 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 23B3E1D70; Thu, 19 Dec 2013 18:55:25 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84054218" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 18:55:21 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 13:55:20 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtiki-0006gd-Ei; Thu, 19 Dec 2013 18:55:20 +0000 From: Roger Pau Monne To: , , , , , , Subject: =?UTF-8?q?=5BPATCH=20v7=2004/19=5D=20amd64=3A=20introduce=20hook=20for=20custom=20preload=20metadata=20parsers?= Date: Thu, 19 Dec 2013 19:54:41 +0100 Message-ID: <1387479296-33389-5-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 18:55:28 -0000 --- sys/amd64/amd64/machdep.c | 45 +++++++++++++---- sys/amd64/include/sysarch.h | 12 +++++ sys/conf/files.amd64 | 1 + sys/x86/xen/pv.c | 114 +++++++++++++++++++++++++++++++++++++++++++ sys/xen/pv.h | 28 +++++++++++ 5 files changed, 189 insertions(+), 11 deletions(-) create mode 100644 sys/x86/xen/pv.c create mode 100644 sys/xen/pv.h diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index 1880f23..09d9a1a 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -126,6 +126,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #ifdef PERFMON #include #endif @@ -148,6 +149,7 @@ __FBSDID("$FreeBSD$"); #ifdef XENHVM #include +#include #endif /* Sanity check for __curthread() */ @@ -172,6 +174,14 @@ static int set_fpcontext(struct thread *td, const mcontext_t *mcp, char *xfpustate, size_t xfpustate_len); SYSINIT(cpu, SI_SUB_CPU, SI_ORDER_FIRST, cpu_startup, NULL); +/* Preload data parse function */ +static caddr_t native_parse_preload_data(u_int64_t); + +/* Default init_ops implementation. */ +struct init_ops init_ops = { + .parse_preload_data = native_parse_preload_data, +}; + /* * The file "conf/ldscript.amd64" defines the symbol "kernphys". Its value is * the physical address at which the kernel is loaded. @@ -1690,6 +1700,26 @@ do_next: msgbufp = (struct msgbuf *)PHYS_TO_DMAP(phys_avail[pa_indx]); } +static caddr_t +native_parse_preload_data(u_int64_t modulep) +{ + caddr_t kmdp; + + preload_metadata = (caddr_t)(uintptr_t)(modulep + KERNBASE); + preload_bootstrap_relocate(KERNBASE); + kmdp = preload_search_by_type("elf kernel"); + if (kmdp == NULL) + kmdp = preload_search_by_type("elf64 kernel"); + boothowto = MD_FETCH(kmdp, MODINFOMD_HOWTO, int); + kern_envp = MD_FETCH(kmdp, MODINFOMD_ENVP, char *) + KERNBASE; +#ifdef DDB + ksym_start = MD_FETCH(kmdp, MODINFOMD_SSYM, uintptr_t); + ksym_end = MD_FETCH(kmdp, MODINFOMD_ESYM, uintptr_t); +#endif + + return (kmdp); +} + #ifdef XENHVM /* * First function called by the Xen PVH boot sequence. @@ -1754,6 +1784,9 @@ hammer_time_xen(start_info_t *si, u_int64_t xenstack) } load_cr3(((u_int64_t)&PT4[0]) - KERNBASE); + /* Set the hooks for early functions that diverge from bare metal */ + xen_pv_set_init_ops(); + /* Now we can jump into the native init function */ return (hammer_time(0, physfree)); } @@ -1783,17 +1816,7 @@ hammer_time(u_int64_t modulep, u_int64_t physfree) */ proc_linkup0(&proc0, &thread0); - preload_metadata = (caddr_t)(uintptr_t)(modulep + KERNBASE); - preload_bootstrap_relocate(KERNBASE); - kmdp = preload_search_by_type("elf kernel"); - if (kmdp == NULL) - kmdp = preload_search_by_type("elf64 kernel"); - boothowto = MD_FETCH(kmdp, MODINFOMD_HOWTO, int); - kern_envp = MD_FETCH(kmdp, MODINFOMD_ENVP, char *) + KERNBASE; -#ifdef DDB - ksym_start = MD_FETCH(kmdp, MODINFOMD_SSYM, uintptr_t); - ksym_end = MD_FETCH(kmdp, MODINFOMD_ESYM, uintptr_t); -#endif + kmdp = init_ops.parse_preload_data(modulep); /* Init basic tunables, hz etc */ init_param1(); diff --git a/sys/amd64/include/sysarch.h b/sys/amd64/include/sysarch.h index cd380d4..58ac8cd 100644 --- a/sys/amd64/include/sysarch.h +++ b/sys/amd64/include/sysarch.h @@ -4,3 +4,15 @@ /* $FreeBSD$ */ #include + +/* + * Struct containing pointers to init functions whose + * implementation is run time selectable. Selection can be made, + * for example, based on detection of a BIOS variant or + * hypervisor environment. + */ +struct init_ops { + caddr_t (*parse_preload_data)(u_int64_t); +}; + +extern struct init_ops init_ops; diff --git a/sys/conf/files.amd64 b/sys/conf/files.amd64 index d1bdcd9..b3b1319 100644 --- a/sys/conf/files.amd64 +++ b/sys/conf/files.amd64 @@ -566,3 +566,4 @@ x86/x86/nexus.c standard x86/x86/tsc.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm +x86/xen/pv.c optional xenhvm diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c new file mode 100644 index 0000000..bbaca32 --- /dev/null +++ b/sys/x86/xen/pv.c @@ -0,0 +1,114 @@ +/* + * Copyright (c) 2004 Christian Limpach. + * Copyright (c) 2004-2006,2008 Kip Macy + * Copyright (c) 2013 Roger Pau MonnĂ© + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include + +#include + +#include +#include + +/*--------------------------- Forward Declarations ---------------------------*/ +static caddr_t xen_pv_parse_preload_data(u_int64_t); + +/*-------------------------------- Global Data -------------------------------*/ +/* Xen init_ops implementation. */ +struct init_ops xen_init_ops = { + .parse_preload_data = xen_pv_parse_preload_data, +}; + +static struct +{ + const char *ev; + int mask; +} howto_names[] = { + {"boot_askname", RB_ASKNAME}, + {"boot_single", RB_SINGLE}, + {"boot_nosync", RB_NOSYNC}, + {"boot_halt", RB_ASKNAME}, + {"boot_serial", RB_SERIAL}, + {"boot_cdrom", RB_CDROM}, + {"boot_gdb", RB_GDB}, + {"boot_gdb_pause", RB_RESERVED1}, + {"boot_verbose", RB_VERBOSE}, + {"boot_multicons", RB_MULTIPLE}, + {NULL, 0} +}; + +/* + * Functions to convert the "extra" parameters passed by Xen + * into FreeBSD boot options (from the i386 Xen port). + */ +static char * +xen_setbootenv(char *cmd_line) +{ + char *cmd_line_next; + + /* Skip leading spaces */ + for (; *cmd_line == ' '; cmd_line++); + + for (cmd_line_next = cmd_line; strsep(&cmd_line_next, ",") != NULL;); + return (cmd_line); +} + +static int +xen_boothowto(char *envp) +{ + int i, howto = 0; + + /* get equivalents from the environment */ + for (i = 0; howto_names[i].ev != NULL; i++) + if (getenv(howto_names[i].ev) != NULL) + howto |= howto_names[i].mask; + return (howto); +} + +static caddr_t +xen_pv_parse_preload_data(u_int64_t modulep) +{ + /* Parse the extra boot information given by Xen */ + if (HYPERVISOR_start_info->cmd_line) + kern_envp = xen_setbootenv(HYPERVISOR_start_info->cmd_line); + boothowto |= xen_boothowto(kern_envp); + + return (NULL); +} + +void +xen_pv_set_init_ops(void) +{ + /* Init ops for Xen PV */ + init_ops = xen_init_ops; +} diff --git a/sys/xen/pv.h b/sys/xen/pv.h new file mode 100644 index 0000000..71b8776 --- /dev/null +++ b/sys/xen/pv.h @@ -0,0 +1,28 @@ +/* + * Permission is hereby granted, free of charge, to any person obtaining a copy + * of this software and associated documentation files (the "Software"), to + * deal in the Software without restriction, including without limitation the + * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or + * sell copies of the Software, and to permit persons to whom the Software is + * furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER + * DEALINGS IN THE SOFTWARE. + * + * $FreeBSD$ + */ + +#ifndef __XEN_PV_H__ +#define __XEN_PV_H__ + +void xen_pv_set_init_ops(void); + +#endif /* __XEN_PV_H__ */ -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 18:55:26 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 061D6F41; Thu, 19 Dec 2013 18:55:26 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 876ED1D6D; Thu, 19 Dec 2013 18:55:24 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84054213" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 18:55:20 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 13:55:19 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikh-0006gd-AN; Thu, 19 Dec 2013 18:55:19 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 02/19] xen: add macro to detect if running as Dom0 Date: Thu, 19 Dec 2013 19:54:39 +0100 Message-ID: <1387479296-33389-3-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 18:55:26 -0000 --- sys/xen/xen-os.h | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/sys/xen/xen-os.h b/sys/xen/xen-os.h index c7474d8..e8a5a99 100644 --- a/sys/xen/xen-os.h +++ b/sys/xen/xen-os.h @@ -82,6 +82,13 @@ xen_hvm_domain(void) return (xen_domain_type == XEN_HVM_DOMAIN); } +static inline int +xen_initial_domain(void) +{ + return (xen_domain() && HYPERVISOR_start_info && + HYPERVISOR_start_info->flags & SIF_INITDOMAIN); +} + #ifndef xen_mb #define xen_mb() mb() #endif -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 18:55:24 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2C01CE11; Thu, 19 Dec 2013 18:55:24 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A652C1D6B; Thu, 19 Dec 2013 18:55:22 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84054212" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 18:55:19 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 13:55:18 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikg-0006gd-3a; Thu, 19 Dec 2013 18:55:18 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 00/19] FreeBSD PVH DomU support Date: Thu, 19 Dec 2013 19:54:37 +0100 Message-ID: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 18:55:24 -0000 This series is a split of the previous patch "Xen x86 DomU PVH support", with the aim to make the review of the code easier. The series can also be found on my git repo: git://xenbits.xen.org/people/royger/freebsd.git pvh_v7 or http://xenbits.xen.org/gitweb/?p=people/royger/freebsd.git;a=shortlog;h=refs/heads/pvh_v7 PVH mode is basically a PV guest inside an HVM container, and shares a great amount of code with PVHVM. The main difference is the way the guest is started, PVH uses the PV start sequence, jumping directly into the kernel entry point in long mode and with page tables set. The main work of this patch consists in setting the environment as similar as possible to what native FreeBSD expects, and then adding hooks to the PV ops when necessary. sys/amd64/amd64/locore.S: * Add PV entry point, hypervisor_page and the necessary elfnotes. sys/amd64/amd64/machdep.c: * Add hooks to replace bare metal operations that should use a PV helper, this includes: - Preload metadata - i8254_init and i8254_delay - Fetching the e820 memory map - Reserve of the MP bootstrap region * Create a DELAY function that uses the PV hooks. * Introduce a new hammer_time_xen that sets the necessary stuff when running in PVH mode. sys/amd64/amd64/mp_machdep.c: * Introduce a hook to replace start_all_aps. * Use lapic_disabled variable to prevent polluting the code with xen specific gates. sys/amd64/include/asmacros.h: * Copy the ELFNOTE macro from the i386 Xen PV port. sys/amd64/include/clock.h: sys/i386/include/clock.h: * Prototypes for the xen early delay initialization and usage. sys/amd64/include/cpu.h: * Introduce a new cpu hook to init APs. sys/amd64/include/sysarch.h: * Declare the init_ops structure. sys/amd64/include/xen/hypercall.h: sys/i386/include/xen/hypercall.h * Switch to the PV style hypercall mechanism for HVM also. sys/conf/files: * Make the PV console available on XENHVM also. sys/conf/files.amd64: * Include the new files for the PVH port. sys/dev/xen/console/console.c: sys/dev/xen/console/xencons_ring.c: * Remove the identify method and instead add the device from nexus_xen. * Use HYPERVISOR_start_info instead of xen_start_info. * Use HYPERVISOR_event_channel_op to kick the event channel before xen interrupts are setup. * Copy the xc_printf debug function from xen_machdep.c sys/dev/xen/control/control.c: * Use the PV shutdown on PVH. sys/dev/xen/timer/timer.c: * Pass a vcpu_info to xen_fetch_vcpu_time, this allows using this function at very early init, before per-cpu vcpu_info is set. * Remove critical_{enter/exit} from xen_fetch_vcpu_time so it can be used at early boot, instead place them on the callers. * Introduce two new functions, xen_delay_init and xen_delay that can be used at early boot to implement the generic DELAY function. * Remove the identify method that used to add the device, now it is manually added from either xenpci (HVM) or nexus_xen (PV). sys/i386/i386/locore.s: * Reserve space for the hypercall page. sys/i386/i386/machdep.c: * Create a generic DELAY function. sys/i386/xen/xen_machdep.c: * Set HYPERVISOR_start_info. * Move and rename xc_printf debug function to xen console.c sys/x86/isa/clock.c: * Rename the generic DELAY function to i8254_delay. sys/x86/x86/delay.c: * Put generic delay helpers here, get_tsc and delay_tc. sys/x86/x86/local_apic.c: * Prevent the local apic from attaching when running on PVH mode. sys/x86/xen/hvm.c: * Set the start_all_aps hook. * Fix the setting of the hypercall page now that we are using the same mechanism as the PV port. * Initialize Xen CPU hooks for the PVH port. * Initialize APs before SI_SUB_SMP (SI_SUB_SMP-1). sys/x86/xen/mptable.c: * Create a dummy PV CPU enumerator for the PVH port. sys/x86/xen/pv.c: * Implement the PV functions for the early boot hooks, parse_preload_data and fetch_e820_map. * Implement the PV function for the start_all_aps hook. sys/x86/xen/pvcpu.c: * Dummy Xen PV CPU device, that we use to set the per-cpu pc_device. sys/xen/gnttab.c: * Allocate resume_frames for the PVH port. sys/xen/pv.h: * Header that exports the specific PV functions. sys/xen/xen-os.h: * Declare prototypes for the newly added functions. * Include new xen_initial_domain function. sys/xen/xenstore/xenstore.c: * Make the xenstore driver hang from both xenpci and the nexus when running XENHVM, this is because we don't have a xenpci device on the PVH port. * Remove the identify routine that added the device, instead add it from either xenpci (HVM) or nexus_xen (PV). sys/dev/xen/xenpci/xenpci.c: * Add the xenstore and xen_et devices on succesful attach. sys/x86/xen/xen_nexus.c: * Create a specific nexus for Xen PV guests that takes care of adding the top level Xen PV devices. From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 18:55:28 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 737D41B2; Thu, 19 Dec 2013 18:55:28 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A6BC91D73; Thu, 19 Dec 2013 18:55:26 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84054228" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 18:55:22 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 13:55:22 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikk-0006gd-6D; Thu, 19 Dec 2013 18:55:22 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 07/19] xen: implement hook to fetch e820 memory map Date: Thu, 19 Dec 2013 19:54:44 +0100 Message-ID: <1387479296-33389-8-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 18:55:28 -0000 --- sys/amd64/amd64/machdep.c | 50 ++++++++++++++++++++++++++---------------- sys/amd64/include/pc/bios.h | 2 + sys/amd64/include/sysarch.h | 1 + sys/x86/xen/pv.c | 26 ++++++++++++++++++++++ 4 files changed, 60 insertions(+), 19 deletions(-) diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index a2dcb90..6bbfe5a 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -177,11 +177,15 @@ SYSINIT(cpu, SI_SUB_CPU, SI_ORDER_FIRST, cpu_startup, NULL); /* Preload data parse function */ static caddr_t native_parse_preload_data(u_int64_t); +/* Native function to fetch and parse the e820 map */ +static void native_parse_memmap(caddr_t, vm_paddr_t *, int *); + /* Default init_ops implementation. */ struct init_ops init_ops = { .parse_preload_data = native_parse_preload_data, .early_delay_init = i8254_init, .early_delay = i8254_delay, + .parse_memmap = native_parse_memmap, }; /* @@ -1418,21 +1422,12 @@ add_physmap_entry(uint64_t base, uint64_t length, vm_paddr_t *physmap, return (1); } -static void -add_smap_entries(struct bios_smap *smapbase, vm_paddr_t *physmap, - int *physmap_idx) +void +bios_add_smap_entries(struct bios_smap *smapbase, u_int32_t smapsize, + vm_paddr_t *physmap, int *physmap_idx) { struct bios_smap *smap, *smapend; - u_int32_t smapsize; - /* - * Memory map from INT 15:E820. - * - * subr_module.c says: - * "Consumer may safely assume that size value precedes data." - * ie: an int32_t immediately precedes smap. - */ - smapsize = *((u_int32_t *)smapbase - 1); smapend = (struct bios_smap *)((uintptr_t)smapbase + smapsize); for (smap = smapbase; smap < smapend; smap++) { @@ -1449,6 +1444,29 @@ add_smap_entries(struct bios_smap *smapbase, vm_paddr_t *physmap, } } +static void +native_parse_memmap(caddr_t kmdp, vm_paddr_t *physmap, int *physmap_idx) +{ + struct bios_smap *smap; + u_int32_t size; + + /* + * Memory map from INT 15:E820. + * + * subr_module.c says: + * "Consumer may safely assume that size value precedes data." + * ie: an int32_t immediately precedes smap. + */ + + smap = (struct bios_smap *)preload_search_info(kmdp, + MODINFO_METADATA | MODINFOMD_SMAP); + if (smap == NULL) + panic("No BIOS smap info from loader!"); + size = *((u_int32_t *)smap - 1); + + bios_add_smap_entries(smap, size, physmap, physmap_idx); +} + /* * Populate the (physmap) array with base/bound pairs describing the * available physical memory in the system, then test this memory and @@ -1466,19 +1484,13 @@ getmemsize(caddr_t kmdp, u_int64_t first) vm_paddr_t pa, physmap[PHYSMAP_SIZE]; u_long physmem_start, physmem_tunable, memtest; pt_entry_t *pte; - struct bios_smap *smapbase; quad_t dcons_addr, dcons_size; bzero(physmap, sizeof(physmap)); basemem = 0; physmap_idx = 0; - smapbase = (struct bios_smap *)preload_search_info(kmdp, - MODINFO_METADATA | MODINFOMD_SMAP); - if (smapbase == NULL) - panic("No BIOS smap info from loader!"); - - add_smap_entries(smapbase, physmap, &physmap_idx); + init_ops.parse_memmap(kmdp, physmap, &physmap_idx); /* * Find the 'base memory' segment for SMP diff --git a/sys/amd64/include/pc/bios.h b/sys/amd64/include/pc/bios.h index e7d568e..92d4265 100644 --- a/sys/amd64/include/pc/bios.h +++ b/sys/amd64/include/pc/bios.h @@ -106,6 +106,8 @@ struct bios_oem { int bios_oem_strings(struct bios_oem *oem, u_char *buffer, size_t maxlen); uint32_t bios_sigsearch(uint32_t start, u_char *sig, int siglen, int paralen, int sigofs); +void bios_add_smap_entries(struct bios_smap *smapbase, u_int32_t smapsize, + vm_paddr_t *physmap, int *physmap_idx); #endif #endif /* _MACHINE_PC_BIOS_H_ */ diff --git a/sys/amd64/include/sysarch.h b/sys/amd64/include/sysarch.h index 60fa635..084223e 100644 --- a/sys/amd64/include/sysarch.h +++ b/sys/amd64/include/sysarch.h @@ -15,6 +15,7 @@ struct init_ops { caddr_t (*parse_preload_data)(u_int64_t); void (*early_delay_init)(void); void (*early_delay)(int); + void (*parse_memmap)(caddr_t, vm_paddr_t *, int *); }; extern struct init_ops init_ops; diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c index f8d8cfa..db576e0 100644 --- a/sys/x86/xen/pv.c +++ b/sys/x86/xen/pv.c @@ -37,12 +37,17 @@ __FBSDID("$FreeBSD$"); #include #include +#include #include #include +#include + +#define MAX_E820_ENTRIES 128 /*--------------------------- Forward Declarations ---------------------------*/ static caddr_t xen_pv_parse_preload_data(u_int64_t); +static void xen_pv_parse_memmap(caddr_t, vm_paddr_t *, int *); /*-------------------------------- Global Data -------------------------------*/ /* Xen init_ops implementation. */ @@ -50,6 +55,7 @@ struct init_ops xen_init_ops = { .parse_preload_data = xen_pv_parse_preload_data, .early_delay_init = xen_delay_init, .early_delay = xen_delay, + .parse_memmap = xen_pv_parse_memmap, }; static struct @@ -70,6 +76,8 @@ static struct {NULL, 0} }; +static struct bios_smap xen_smap[MAX_E820_ENTRIES]; + /* * Functions to convert the "extra" parameters passed by Xen * into FreeBSD boot options (from the i386 Xen port). @@ -109,6 +117,24 @@ xen_pv_parse_preload_data(u_int64_t modulep) return (NULL); } +static void +xen_pv_parse_memmap(caddr_t kmdp, vm_paddr_t *physmap, int *physmap_idx) +{ + struct xen_memory_map memmap; + u_int32_t size; + int rc; + + /* Fetch the E820 map from Xen */ + memmap.nr_entries = MAX_E820_ENTRIES; + set_xen_guest_handle(memmap.buffer, xen_smap); + rc = HYPERVISOR_memory_op(XENMEM_memory_map, &memmap); + if (rc) + panic("unable to fetch Xen E820 memory map"); + size = memmap.nr_entries * sizeof(xen_smap[0]); + + bios_add_smap_entries(xen_smap, size, physmap, physmap_idx); +} + void xen_pv_set_init_ops(void) { -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 18:55:28 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 472F319B; Thu, 19 Dec 2013 18:55:28 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6003A1D72; Thu, 19 Dec 2013 18:55:26 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84054225" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 18:55:22 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 13:55:21 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikj-0006gd-JU; Thu, 19 Dec 2013 18:55:21 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 06/19] xen: implement an early timer for Xen PVH Date: Thu, 19 Dec 2013 19:54:43 +0100 Message-ID: <1387479296-33389-7-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 18:55:28 -0000 When running as a PVH guest, there's no emulated i8254, so we need to use the Xen PV timer as the early source for DELAY. This change allows for different implementations of the early DELAY function and implements a Xen variant for it. --- sys/amd64/amd64/machdep.c | 15 ++++++- sys/amd64/include/clock.h | 6 +++ sys/amd64/include/sysarch.h | 2 + sys/conf/files.amd64 | 1 + sys/conf/files.i386 | 1 + sys/dev/xen/timer/timer.c | 33 +++++++++++++++ sys/i386/i386/machdep.c | 11 +++++ sys/i386/include/clock.h | 6 +++ sys/x86/isa/clock.c | 53 +----------------------- sys/x86/x86/delay.c | 95 +++++++++++++++++++++++++++++++++++++++++++ sys/x86/xen/pv.c | 3 + 11 files changed, 172 insertions(+), 54 deletions(-) create mode 100644 sys/x86/x86/delay.c diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index 09d9a1a..a2dcb90 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -180,6 +180,8 @@ static caddr_t native_parse_preload_data(u_int64_t); /* Default init_ops implementation. */ struct init_ops init_ops = { .parse_preload_data = native_parse_preload_data, + .early_delay_init = i8254_init, + .early_delay = i8254_delay, }; /* @@ -232,6 +234,15 @@ struct mem_range_softc mem_range_softc; struct mtx dt_lock; /* lock for GDT and LDT */ +void +DELAY(int n) +{ + if (delay_tc(n)) + return; + + init_ops.early_delay(n); +} + static void cpu_startup(dummy) void *dummy; @@ -1900,10 +1911,10 @@ hammer_time(u_int64_t modulep, u_int64_t physfree) lidt(&r_idt); /* - * Initialize the i8254 before the console so that console + * Initialize the early delay before the console so that console * initialization can use DELAY(). */ - i8254_init(); + init_ops.early_delay_init(); /* * Initialize the console before we print anything out. diff --git a/sys/amd64/include/clock.h b/sys/amd64/include/clock.h index d7f7d82..e7817ab 100644 --- a/sys/amd64/include/clock.h +++ b/sys/amd64/include/clock.h @@ -25,6 +25,12 @@ extern int smp_tsc; #endif void i8254_init(void); +void i8254_delay(int); +#ifdef XENHVM +void xen_delay_init(void); +void xen_delay(int); +#endif +int delay_tc(int); /* * Driver to clock driver interface. diff --git a/sys/amd64/include/sysarch.h b/sys/amd64/include/sysarch.h index 58ac8cd..60fa635 100644 --- a/sys/amd64/include/sysarch.h +++ b/sys/amd64/include/sysarch.h @@ -13,6 +13,8 @@ */ struct init_ops { caddr_t (*parse_preload_data)(u_int64_t); + void (*early_delay_init)(void); + void (*early_delay)(int); }; extern struct init_ops init_ops; diff --git a/sys/conf/files.amd64 b/sys/conf/files.amd64 index b3b1319..bdc1517 100644 --- a/sys/conf/files.amd64 +++ b/sys/conf/files.amd64 @@ -564,6 +564,7 @@ x86/x86/mptable_pci.c optional mptable pci x86/x86/msi.c optional pci x86/x86/nexus.c standard x86/x86/tsc.c standard +x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm x86/xen/pv.c optional xenhvm diff --git a/sys/conf/files.i386 b/sys/conf/files.i386 index eb8697c..790296d 100644 --- a/sys/conf/files.i386 +++ b/sys/conf/files.i386 @@ -600,5 +600,6 @@ x86/x86/mptable_pci.c optional apic native pci x86/x86/msi.c optional apic pci x86/x86/nexus.c standard x86/x86/tsc.c standard +x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm diff --git a/sys/dev/xen/timer/timer.c b/sys/dev/xen/timer/timer.c index b2f6bcd..96372ab 100644 --- a/sys/dev/xen/timer/timer.c +++ b/sys/dev/xen/timer/timer.c @@ -59,6 +59,9 @@ __FBSDID("$FreeBSD$"); #include #include +/* For the declaration of clock_lock */ +#include + #include "clock_if.h" static devclass_t xentimer_devclass; @@ -584,6 +587,36 @@ xentimer_suspend(device_t dev) return (0); } +/* + * Xen delay early init + */ +void xen_delay_init(void) +{ + /* Init the clock lock */ + mtx_init(&clock_lock, "clk", NULL, MTX_SPIN | MTX_NOPROFILE); +} +/* + * Xen PV DELAY function + * + * When running on PVH mode we don't have an emulated i8524, so + * make use of the Xen time info in order to code a simple DELAY + * function that can be used during early boot. + */ +void xen_delay(int n) +{ + uint64_t end_ns; + uint64_t current; + + end_ns = xen_fetch_vcpu_time(&HYPERVISOR_shared_info->vcpu_info[0]); + end_ns += n * NSEC_IN_USEC; + + for (;;) { + current = xen_fetch_vcpu_time(&HYPERVISOR_shared_info->vcpu_info[0]); + if (current >= end_ns) + break; + } +} + static device_method_t xentimer_methods[] = { DEVMETHOD(device_identify, xentimer_identify), DEVMETHOD(device_probe, xentimer_probe), diff --git a/sys/i386/i386/machdep.c b/sys/i386/i386/machdep.c index d43abbf..ad2734f 100644 --- a/sys/i386/i386/machdep.c +++ b/sys/i386/i386/machdep.c @@ -253,6 +253,17 @@ struct mtx icu_lock; struct mem_range_softc mem_range_softc; +#ifndef XEN +void +DELAY(int n) +{ + if (delay_tc(n)) + return; + + i8254_delay(n); +} +#endif + static void cpu_startup(dummy) void *dummy; diff --git a/sys/i386/include/clock.h b/sys/i386/include/clock.h index d980ec7..287b2c8 100644 --- a/sys/i386/include/clock.h +++ b/sys/i386/include/clock.h @@ -22,6 +22,12 @@ extern int tsc_is_invariant; extern int tsc_perf_stat; void i8254_init(void); +void i8254_delay(int); +#ifdef XENHVM +void xen_delay_init(void); +void xen_delay(int); +#endif +int delay_tc(int); /* * Driver to clock driver interface. diff --git a/sys/x86/isa/clock.c b/sys/x86/isa/clock.c index a12e175..a5aed1c 100644 --- a/sys/x86/isa/clock.c +++ b/sys/x86/isa/clock.c @@ -247,61 +247,13 @@ getit(void) return ((high << 8) | low); } -#ifndef DELAYDEBUG -static u_int -get_tsc(__unused struct timecounter *tc) -{ - - return (rdtsc32()); -} - -static __inline int -delay_tc(int n) -{ - struct timecounter *tc; - timecounter_get_t *func; - uint64_t end, freq, now; - u_int last, mask, u; - - tc = timecounter; - freq = atomic_load_acq_64(&tsc_freq); - if (tsc_is_invariant && freq != 0) { - func = get_tsc; - mask = ~0u; - } else { - if (tc->tc_quality <= 0) - return (0); - func = tc->tc_get_timecount; - mask = tc->tc_counter_mask; - freq = tc->tc_frequency; - } - now = 0; - end = freq * n / 1000000; - if (func == get_tsc) - sched_pin(); - last = func(tc) & mask; - do { - cpu_spinwait(); - u = func(tc) & mask; - if (u < last) - now += mask - last + u + 1; - else - now += u - last; - last = u; - } while (now < end); - if (func == get_tsc) - sched_unpin(); - return (1); -} -#endif - /* * Wait "n" microseconds. * Relies on timer 1 counting down from (i8254_freq / hz) * Note: timer had better have been programmed before this is first used! */ void -DELAY(int n) +i8254_delay(int n) { int delta, prev_tick, tick, ticks_left; #ifdef DELAYDEBUG @@ -317,9 +269,6 @@ DELAY(int n) } if (state == 1) printf("DELAY(%d)...", n); -#else - if (delay_tc(n)) - return; #endif /* * Read the counter first, so that the rest of the setup overhead is diff --git a/sys/x86/x86/delay.c b/sys/x86/x86/delay.c new file mode 100644 index 0000000..7ea70b1 --- /dev/null +++ b/sys/x86/x86/delay.c @@ -0,0 +1,95 @@ +/*- + * Copyright (c) 1990 The Regents of the University of California. + * Copyright (c) 2010 Alexander Motin + * All rights reserved. + * + * This code is derived from software contributed to Berkeley by + * William Jolitz and Don Ahn. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 4. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * from: @(#)clock.c 7.2 (Berkeley) 5/12/91 + */ + +#include +__FBSDID("$FreeBSD$"); + +/* Generic x86 routines to handle delay */ + +#include +#include +#include +#include +#include +#include + +#include +#include + +static u_int +get_tsc(__unused struct timecounter *tc) +{ + + return (rdtsc32()); +} + +int +delay_tc(int n) +{ + struct timecounter *tc; + timecounter_get_t *func; + uint64_t end, freq, now; + u_int last, mask, u; + + tc = timecounter; + freq = atomic_load_acq_64(&tsc_freq); + if (tsc_is_invariant && freq != 0) { + func = get_tsc; + mask = ~0u; + } else { + if (tc->tc_quality <= 0) + return (0); + func = tc->tc_get_timecount; + mask = tc->tc_counter_mask; + freq = tc->tc_frequency; + } + now = 0; + end = freq * n / 1000000; + if (func == get_tsc) + sched_pin(); + last = func(tc) & mask; + do { + cpu_spinwait(); + u = func(tc) & mask; + if (u < last) + now += mask - last + u + 1; + else + now += u - last; + last = u; + } while (now < end); + if (func == get_tsc) + sched_unpin(); + return (1); +} diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c index bbaca32..f8d8cfa 100644 --- a/sys/x86/xen/pv.c +++ b/sys/x86/xen/pv.c @@ -36,6 +36,7 @@ __FBSDID("$FreeBSD$"); #include #include +#include #include #include @@ -47,6 +48,8 @@ static caddr_t xen_pv_parse_preload_data(u_int64_t); /* Xen init_ops implementation. */ struct init_ops xen_init_ops = { .parse_preload_data = xen_pv_parse_preload_data, + .early_delay_init = xen_delay_init, + .early_delay = xen_delay, }; static struct -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 18:55:26 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4B693FAD; Thu, 19 Dec 2013 18:55:26 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9C6711D6E; Thu, 19 Dec 2013 18:55:24 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84054217" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 18:55:20 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 13:55:19 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikh-0006gd-SP; Thu, 19 Dec 2013 18:55:19 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 03/19] xen: add and enable Xen console for PVH guests Date: Thu, 19 Dec 2013 19:54:40 +0100 Message-ID: <1387479296-33389-4-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 18:55:26 -0000 This adds and enables the console used on XEN kernels. --- sys/amd64/amd64/machdep.c | 4 +++ sys/conf/files | 4 +- sys/dev/xen/console/console.c | 45 ++++++++++++++++++++++------------- sys/dev/xen/console/xencons_ring.c | 15 ++++++++---- sys/i386/include/xen/xen-os.h | 1 - sys/i386/xen/xen_machdep.c | 17 ------------- sys/xen/xen-os.h | 4 +++ 7 files changed, 48 insertions(+), 42 deletions(-) diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index a73e33e..1880f23 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -1709,6 +1709,8 @@ hammer_time_xen(start_info_t *si, u_int64_t xenstack) KASSERT((si != NULL && xenstack != 0), ("invalid start_info or xenstack")); + xc_printf("FreeBSD PVH running on %s\n", si->magic); + /* We use 3 pages of xen stack for the boot pagetables */ physfree = xenstack + 3 * PAGE_SIZE - KERNBASE; @@ -1726,6 +1728,8 @@ hammer_time_xen(start_info_t *si, u_int64_t xenstack) */ xen_store = (struct xenstore_domain_interface *) (ptoa(si->store_mfn) + KERNBASE); + console_page = + (char *)(ptoa(si->console.domU.mfn) + KERNBASE); xen_domain_type = XEN_PV_DOMAIN; vm_guest = VM_GUEST_XEN; diff --git a/sys/conf/files b/sys/conf/files index a73d31e..f55479d 100644 --- a/sys/conf/files +++ b/sys/conf/files @@ -2523,8 +2523,8 @@ dev/xe/if_xe_pccard.c optional xe pccard dev/xen/balloon/balloon.c optional xen | xenhvm dev/xen/blkfront/blkfront.c optional xen | xenhvm dev/xen/blkback/blkback.c optional xen | xenhvm -dev/xen/console/console.c optional xen -dev/xen/console/xencons_ring.c optional xen +dev/xen/console/console.c optional xen | xenhvm +dev/xen/console/xencons_ring.c optional xen | xenhvm dev/xen/control/control.c optional xen | xenhvm dev/xen/netback/netback.c optional xen | xenhvm dev/xen/netfront/netfront.c optional xen | xenhvm diff --git a/sys/dev/xen/console/console.c b/sys/dev/xen/console/console.c index 23eaee2..e8079da 100644 --- a/sys/dev/xen/console/console.c +++ b/sys/dev/xen/console/console.c @@ -69,11 +69,14 @@ struct mtx cn_mtx; static char wbuf[WBUF_SIZE]; static char rbuf[RBUF_SIZE]; static int rc, rp; -static unsigned int cnsl_evt_reg; +unsigned int cnsl_evt_reg; static unsigned int wc, wp; /* write_cons, write_prod */ xen_intr_handle_t xen_intr_handle; device_t xencons_dev; +/* Virt address of the shared console page */ +char *console_page; + #ifdef KDB static int xc_altbrk; #endif @@ -110,9 +113,28 @@ static struct ttydevsw xc_ttydevsw = { .tsw_outwakeup = xcoutwakeup, }; +/*----------------------------- Debug function -------------------------------*/ +#define XC_PRINTF_BUFSIZE 1024 +void +xc_printf(const char *fmt, ...) +{ + __va_list ap; + int retval; + static char buf[XC_PRINTF_BUFSIZE]; + + va_start(ap, fmt); + retval = vsnprintf(buf, XC_PRINTF_BUFSIZE - 1, fmt, ap); + va_end(ap); + buf[retval] = 0; + HYPERVISOR_console_write(buf, retval); +} + static void xc_cnprobe(struct consdev *cp) { + if (!xen_pv_domain()) + return; + cp->cn_pri = CN_REMOTE; sprintf(cp->cn_name, "%s0", driver_name); } @@ -175,7 +197,7 @@ static void xc_cnputc(struct consdev *dev, int c) { - if (xen_start_info->flags & SIF_INITDOMAIN) + if (xen_initial_domain()) xc_cnputc_dom0(dev, c); else xc_cnputc_domu(dev, c); @@ -206,22 +228,12 @@ xcons_putc(int c) xcons_force_flush(); #endif } - if (cnsl_evt_reg) - __xencons_tx_flush(); + __xencons_tx_flush(); /* inform start path that we're pretty full */ return ((wp - wc) >= WBUF_SIZE - 100) ? TRUE : FALSE; } -static void -xc_identify(driver_t *driver, device_t parent) -{ - device_t child; - child = BUS_ADD_CHILD(parent, 0, driver_name, 0); - device_set_driver(child, driver); - device_set_desc(child, "Xen Console"); -} - static int xc_probe(device_t dev) { @@ -245,7 +257,7 @@ xc_attach(device_t dev) cnsl_evt_reg = 1; callout_reset(&xc_callout, XC_POLLTIME, xc_timeout, xccons); - if (xen_start_info->flags & SIF_INITDOMAIN) { + if (xen_initial_domain()) { error = xen_intr_bind_virq(dev, VIRQ_CONSOLE, 0, NULL, xencons_priv_interrupt, NULL, INTR_TYPE_TTY, &xen_intr_handle); @@ -309,7 +321,7 @@ __xencons_tx_flush(void) sz = wp - wc; if (sz > (WBUF_SIZE - WBUF_MASK(wc))) sz = WBUF_SIZE - WBUF_MASK(wc); - if (xen_start_info->flags & SIF_INITDOMAIN) { + if (xen_initial_domain()) { HYPERVISOR_console_io(CONSOLEIO_write, sz, &wbuf[WBUF_MASK(wc)]); wc += sz; } else { @@ -405,7 +417,6 @@ xc_timeout(void *v) } static device_method_t xc_methods[] = { - DEVMETHOD(device_identify, xc_identify), DEVMETHOD(device_probe, xc_probe), DEVMETHOD(device_attach, xc_attach), @@ -424,7 +435,7 @@ xcons_force_flush(void) { int sz; - if (xen_start_info->flags & SIF_INITDOMAIN) + if (xen_initial_domain()) return; /* Spin until console data is flushed through to the domain controller. */ diff --git a/sys/dev/xen/console/xencons_ring.c b/sys/dev/xen/console/xencons_ring.c index 3701551..d826363 100644 --- a/sys/dev/xen/console/xencons_ring.c +++ b/sys/dev/xen/console/xencons_ring.c @@ -32,9 +32,9 @@ __FBSDID("$FreeBSD$"); #define console_evtchn console.domU.evtchn xen_intr_handle_t console_handle; -extern char *console_page; extern struct mtx cn_mtx; extern device_t xencons_dev; +extern int cnsl_evt_reg; static inline struct xencons_interface * xencons_interface(void) @@ -60,6 +60,8 @@ xencons_ring_send(const char *data, unsigned len) struct xencons_interface *intf; XENCONS_RING_IDX cons, prod; int sent; + struct evtchn_send send = { .port = + HYPERVISOR_start_info->console_evtchn }; intf = xencons_interface(); cons = intf->out_cons; @@ -76,7 +78,10 @@ xencons_ring_send(const char *data, unsigned len) wmb(); intf->out_prod = prod; - xen_intr_signal(console_handle); + if (cnsl_evt_reg) + xen_intr_signal(console_handle); + else + HYPERVISOR_event_channel_op(EVTCHNOP_send, &send); return sent; @@ -125,11 +130,11 @@ xencons_ring_init(void) { int err; - if (!xen_start_info->console_evtchn) + if (!HYPERVISOR_start_info->console_evtchn) return 0; err = xen_intr_bind_local_port(xencons_dev, - xen_start_info->console_evtchn, NULL, xencons_handle_input, NULL, + HYPERVISOR_start_info->console_evtchn, NULL, xencons_handle_input, NULL, INTR_TYPE_MISC | INTR_MPSAFE, &console_handle); if (err) { return err; @@ -145,7 +150,7 @@ void xencons_suspend(void) { - if (!xen_start_info->console_evtchn) + if (!HYPERVISOR_start_info->console_evtchn) return; xen_intr_unbind(&console_handle); diff --git a/sys/i386/include/xen/xen-os.h b/sys/i386/include/xen/xen-os.h index a8fba61..3d1ef04 100644 --- a/sys/i386/include/xen/xen-os.h +++ b/sys/i386/include/xen/xen-os.h @@ -45,7 +45,6 @@ static inline void rep_nop(void) #define cpu_relax() rep_nop() #ifndef XENHVM -void xc_printf(const char *fmt, ...); #ifdef SMP extern int gdtset; diff --git a/sys/i386/xen/xen_machdep.c b/sys/i386/xen/xen_machdep.c index fd575ee..09c01f1 100644 --- a/sys/i386/xen/xen_machdep.c +++ b/sys/i386/xen/xen_machdep.c @@ -186,21 +186,6 @@ xen_boothowto(char *envp) return howto; } -#define XC_PRINTF_BUFSIZE 1024 -void -xc_printf(const char *fmt, ...) -{ - __va_list ap; - int retval; - static char buf[XC_PRINTF_BUFSIZE]; - - va_start(ap, fmt); - retval = vsnprintf(buf, XC_PRINTF_BUFSIZE - 1, fmt, ap); - va_end(ap); - buf[retval] = 0; - (void)HYPERVISOR_console_write(buf, retval); -} - #define XPQUEUE_SIZE 128 @@ -745,8 +730,6 @@ void initvalues(start_info_t *startinfo); struct xenstore_domain_interface; extern struct xenstore_domain_interface *xen_store; -char *console_page; - void * bootmem_alloc(unsigned int size) { diff --git a/sys/xen/xen-os.h b/sys/xen/xen-os.h index e8a5a99..e005ccd 100644 --- a/sys/xen/xen-os.h +++ b/sys/xen/xen-os.h @@ -55,6 +55,7 @@ extern start_info_t *HYPERVISOR_start_info; /* XXX: we need to get rid of this and use HYPERVISOR_start_info directly */ extern struct xenstore_domain_interface *xen_store; +extern char *console_page; enum xen_domain_type { XEN_NATIVE, /* running on bare hardware */ @@ -89,6 +90,9 @@ xen_initial_domain(void) HYPERVISOR_start_info->flags & SIF_INITDOMAIN); } +/* Debug function, prints directly to hypervisor console */ +void xc_printf(const char *, ...); + #ifndef xen_mb #define xen_mb() mb() #endif -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 18:55:26 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6DB2AFD5; Thu, 19 Dec 2013 18:55:26 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id ECD1D1D6F; Thu, 19 Dec 2013 18:55:24 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="86287397" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 19 Dec 2013 18:55:21 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 13:55:20 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikj-0006gd-1N; Thu, 19 Dec 2013 18:55:21 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 05/19] xen: rework xen timer so it can be used early in boot process Date: Thu, 19 Dec 2013 19:54:42 +0100 Message-ID: <1387479296-33389-6-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 18:55:26 -0000 This should not introduce any functional change, and makes the functions suitable to be called before we have actually mapped the vcpu_info struct on a per-cpu basis. --- sys/dev/xen/timer/timer.c | 29 ++++++++++++++++++++--------- 1 files changed, 20 insertions(+), 9 deletions(-) diff --git a/sys/dev/xen/timer/timer.c b/sys/dev/xen/timer/timer.c index 354085b..b2f6bcd 100644 --- a/sys/dev/xen/timer/timer.c +++ b/sys/dev/xen/timer/timer.c @@ -230,22 +230,22 @@ xen_fetch_vcpu_tinfo(struct vcpu_time_info *dst, struct vcpu_time_info *src) /** * \brief Get the current time, in nanoseconds, since the hypervisor booted. * + * \param vcpu vcpu_info structure to fetch the time from. + * * \note This function returns the current CPU's idea of this value, unless * it happens to be less than another CPU's previously determined value. */ static uint64_t -xen_fetch_vcpu_time(void) +xen_fetch_vcpu_time(struct vcpu_info *vcpu) { struct vcpu_time_info dst; struct vcpu_time_info *src; uint32_t pre_version; uint64_t now; volatile uint64_t last; - struct vcpu_info *vcpu = DPCPU_GET(vcpu_info); src = &vcpu->time; - critical_enter(); do { pre_version = xen_fetch_vcpu_tinfo(&dst, src); barrier(); @@ -266,16 +266,19 @@ xen_fetch_vcpu_time(void) } } while (!atomic_cmpset_64(&xen_timer_last_time, last, now)); - critical_exit(); - return (now); } static uint32_t xentimer_get_timecount(struct timecounter *tc) { + uint32_t xen_time; - return ((uint32_t)xen_fetch_vcpu_time() & UINT_MAX); + critical_enter(); + xen_time = (uint32_t)xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)) & UINT_MAX; + critical_exit(); + + return (xen_time); } /** @@ -305,7 +308,12 @@ xen_fetch_wallclock(struct timespec *ts) static void xen_fetch_uptime(struct timespec *ts) { - uint64_t uptime = xen_fetch_vcpu_time(); + uint64_t uptime; + + critical_enter(); + uptime = xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)); + critical_exit(); + ts->tv_sec = uptime / NSEC_IN_SEC; ts->tv_nsec = uptime % NSEC_IN_SEC; } @@ -354,7 +362,7 @@ xentimer_intr(void *arg) struct xentimer_softc *sc = (struct xentimer_softc *)arg; struct xentimer_pcpu_data *pcpu = DPCPU_PTR(xentimer_pcpu); - pcpu->last_processed = xen_fetch_vcpu_time(); + pcpu->last_processed = xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)); if (pcpu->timer != 0 && sc->et.et_active) sc->et.et_event_cb(&sc->et, sc->et.et_arg); @@ -415,7 +423,10 @@ xentimer_et_start(struct eventtimer *et, do { if (++i == 60) panic("can't schedule timer"); - next_time = xen_fetch_vcpu_time() + first_in_ns; + critical_enter(); + next_time = xen_fetch_vcpu_time(DPCPU_GET(vcpu_info)) + + first_in_ns; + critical_exit(); error = xentimer_vcpu_start_timer(cpu, next_time); } while (error == -ETIME); -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 18:55:27 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D3FCF168; Thu, 19 Dec 2013 18:55:27 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 443591D71; Thu, 19 Dec 2013 18:55:26 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="86287420" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 19 Dec 2013 18:55:24 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 13:55:23 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikl-0006gd-BJ; Thu, 19 Dec 2013 18:55:23 +0000 From: Roger Pau Monne To: , , , , , , Subject: =?UTF-8?q?=5BPATCH=20v7=2009/19=5D=20xen=3A=20add=20a=20apic=5Fenumerator=20for=20PVH?= Date: Thu, 19 Dec 2013 19:54:46 +0100 Message-ID: <1387479296-33389-10-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 18:55:27 -0000 --- sys/conf/files.amd64 | 1 + sys/x86/xen/mptable.c | 136 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+), 0 deletions(-) create mode 100644 sys/x86/xen/mptable.c diff --git a/sys/conf/files.amd64 b/sys/conf/files.amd64 index bdc1517..3bdc05e 100644 --- a/sys/conf/files.amd64 +++ b/sys/conf/files.amd64 @@ -568,3 +568,4 @@ x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm x86/xen/pv.c optional xenhvm +x86/xen/mptable.c optional xenhvm diff --git a/sys/x86/xen/mptable.c b/sys/x86/xen/mptable.c new file mode 100644 index 0000000..0384886 --- /dev/null +++ b/sys/x86/xen/mptable.c @@ -0,0 +1,136 @@ +/*- + * Copyright (c) 2003 John Baldwin + * Copyright (c) 2013 Roger Pau MonnĂ© + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the author nor the names of any co-contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include + +#include +#include + +#include + +static int xenpv_probe(void); +static int xenpv_probe_cpus(void); +static int xenpv_setup_local(void); +static int xenpv_setup_io(void); + +static struct apic_enumerator xenpv_enumerator = { + "Xen PV", + xenpv_probe, + xenpv_probe_cpus, + xenpv_setup_local, + xenpv_setup_io +}; + +/* + * This enumerator will only be registered on PVH + */ +static int +xenpv_probe(void) +{ + return (-100); +} + +/* + * Test each possible vCPU in order to find the number of vCPUs + */ +static int +xenpv_probe_cpus(void) +{ +#ifdef SMP + int i, ret; + + for (i = 0; i < MAXCPU; i++) { + ret = HYPERVISOR_vcpu_op(VCPUOP_is_up, i, NULL); + if (ret >= 0) + cpu_add((i * 2), (i == 0)); + } +#endif + return (0); +} + +/* + * Initialize the vCPU id of the BSP + */ +static int +xenpv_setup_local(void) +{ + PCPU_SET(vcpu_id, 0); + return (0); +} + +/* + * On PVH guests there's no IO APIC + */ +static int +xenpv_setup_io(void) +{ + return (0); +} + +static void +xenpv_register(void *dummy __unused) +{ + if (xen_pv_domain()) { + apic_register_enumerator(&xenpv_enumerator); + } +} +SYSINIT(xenpv_register, SI_SUB_TUNABLES - 1, SI_ORDER_FIRST, xenpv_register, NULL); + +/* + * Setup per-CPU vCPU IDs + */ +static void +xenpv_set_ids(void *dummy) +{ + struct pcpu *pc; + int i; + + CPU_FOREACH(i) { + pc = pcpu_find(i); + pc->pc_vcpu_id = i; + } +} +SYSINIT(xenpv_set_ids, SI_SUB_CPU, SI_ORDER_MIDDLE, xenpv_set_ids, NULL); -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 19:25:29 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A3AB1FE3; Thu, 19 Dec 2013 19:25:29 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 09C98105D; Thu, 19 Dec 2013 19:25:27 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="86299967" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 19 Dec 2013 19:25:26 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 14:25:25 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikq-0006gd-Jx; Thu, 19 Dec 2013 18:55:28 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 18/19] xen: changes to gnttab for PVH Date: Thu, 19 Dec 2013 19:54:55 +0100 Message-ID: <1387479296-33389-19-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 19:25:29 -0000 --- sys/xen/gnttab.c | 26 +++++++++++++++++++++----- 1 files changed, 21 insertions(+), 5 deletions(-) diff --git a/sys/xen/gnttab.c b/sys/xen/gnttab.c index 03c32b7..6949be5 100644 --- a/sys/xen/gnttab.c +++ b/sys/xen/gnttab.c @@ -25,6 +25,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -607,6 +608,7 @@ gnttab_resume(void) { int error; unsigned int max_nr_gframes, nr_gframes; + void *alloc_mem; nr_gframes = nr_grant_frames; max_nr_gframes = max_nr_grant_frames(); @@ -614,11 +616,25 @@ gnttab_resume(void) return (ENOSYS); if (!resume_frames) { - error = xenpci_alloc_space(PAGE_SIZE * max_nr_gframes, - &resume_frames); - if (error) { - printf("error mapping gnttab share frames\n"); - return (error); + if (xen_pv_domain()) { + /* + * This is a waste of physical memory, + * we should use ballooned pages instead, + * but it will do for now. + */ + alloc_mem = contigmalloc(max_nr_gframes * PAGE_SIZE, + M_DEVBUF, M_NOWAIT, 0, + ULONG_MAX, PAGE_SIZE, 0); + KASSERT((alloc_mem != NULL), + ("unable to alloc memory for gnttab")); + resume_frames = vtophys(alloc_mem); + } else { + error = xenpci_alloc_space(PAGE_SIZE * max_nr_gframes, + &resume_frames); + if (error) { + printf("error mapping gnttab share frames\n"); + return (error); + } } } -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 19:25:27 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A230BEE4; Thu, 19 Dec 2013 19:25:27 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2566F105A; Thu, 19 Dec 2013 19:25:25 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="86299947" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 19 Dec 2013 19:25:24 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 14:25:23 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikr-0006gd-6X; Thu, 19 Dec 2013 18:55:29 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 19/19] isa: allow ISA bus to attach to the nexus Date: Thu, 19 Dec 2013 19:54:56 +0100 Message-ID: <1387479296-33389-20-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 19:25:27 -0000 --- sys/x86/isa/isa.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/sys/x86/isa/isa.c b/sys/x86/isa/isa.c index 1a57137..09d1ab7 100644 --- a/sys/x86/isa/isa.c +++ b/sys/x86/isa/isa.c @@ -241,3 +241,6 @@ isa_release_resource(device_t bus, device_t child, int type, int rid, * On this platform, isa can also attach to the legacy bus. */ DRIVER_MODULE(isa, legacy, isa_driver, isa_devclass, 0, 0); +#ifdef XENHVM +DRIVER_MODULE(isa, nexus, isa_driver, isa_devclass, 0, 0); +#endif -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 19:25:24 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 17A8BDE5; Thu, 19 Dec 2013 19:25:24 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 73A451058; Thu, 19 Dec 2013 19:25:21 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84066098" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 19:25:19 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 14:25:18 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikq-0006gd-1J; Thu, 19 Dec 2013 18:55:28 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 17/19] xen: xenstore changes to support PVH Date: Thu, 19 Dec 2013 19:54:54 +0100 Message-ID: <1387479296-33389-18-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 19:25:24 -0000 --- sys/xen/xenstore/xenstore.c | 21 ++++++++++----------- 1 files changed, 10 insertions(+), 11 deletions(-) diff --git a/sys/xen/xenstore/xenstore.c b/sys/xen/xenstore/xenstore.c index bcf6357..2893c84 100644 --- a/sys/xen/xenstore/xenstore.c +++ b/sys/xen/xenstore/xenstore.c @@ -229,13 +229,11 @@ struct xs_softc { */ struct sx xenwatch_mutex; -#ifdef XENHVM /** * The HVM guest pseudo-physical frame number. This is Xen's mapping * of the true machine frame number into our "physical address space". */ unsigned long gpfn; -#endif /** * The event channel for communicating with the @@ -1141,13 +1139,15 @@ xs_attach(device_t dev) /* Initialize the interface to xenstore. */ struct proc *p; -#ifdef XENHVM - xs.evtchn = hvm_get_parameter(HVM_PARAM_STORE_EVTCHN); - xs.gpfn = hvm_get_parameter(HVM_PARAM_STORE_PFN); - xen_store = pmap_mapdev(xs.gpfn * PAGE_SIZE, PAGE_SIZE); -#else - xs.evtchn = xen_start_info->store_evtchn; -#endif + if (xen_hvm_domain()) { + xs.evtchn = hvm_get_parameter(HVM_PARAM_STORE_EVTCHN); + xs.gpfn = hvm_get_parameter(HVM_PARAM_STORE_PFN); + xen_store = pmap_mapdev(xs.gpfn * PAGE_SIZE, PAGE_SIZE); + } else if (xen_pv_domain()) { + xs.evtchn = HYPERVISOR_start_info->store_evtchn; + } else { + panic("Unknown domain type, cannot initialize xenstore\n"); + } TAILQ_INIT(&xs.reply_list); TAILQ_INIT(&xs.watch_events); @@ -1256,9 +1256,8 @@ static devclass_t xenstore_devclass; #ifdef XENHVM DRIVER_MODULE(xenstore, xenpci, xenstore_driver, xenstore_devclass, 0, 0); -#else -DRIVER_MODULE(xenstore, nexus, xenstore_driver, xenstore_devclass, 0, 0); #endif +DRIVER_MODULE(xenstore, nexus, xenstore_driver, xenstore_devclass, 0, 0); /*------------------------------- Sysctl Data --------------------------------*/ /* XXX Shouldn't the node be somewhere else? */ -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 19:25:31 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 365B8FE8; Thu, 19 Dec 2013 19:25:31 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 940111060; Thu, 19 Dec 2013 19:25:29 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="86299990" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 19 Dec 2013 19:25:29 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 14:25:28 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikn-0006gd-3k; Thu, 19 Dec 2013 18:55:25 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 12/19] xen: add a hook to perform AP startup Date: Thu, 19 Dec 2013 19:54:49 +0100 Message-ID: <1387479296-33389-13-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 19:25:31 -0000 AP startup on PVH follows the PV method, so we need to add a hook in order to diverge from bare metal. --- sys/amd64/amd64/mp_machdep.c | 16 ++++--- sys/amd64/include/cpu.h | 1 + sys/x86/xen/hvm.c | 17 +++++++- sys/x86/xen/pv.c | 90 ++++++++++++++++++++++++++++++++++++++++++ sys/xen/pv.h | 1 + 5 files changed, 117 insertions(+), 8 deletions(-) diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c index 4ef4b3d..e302886 100644 --- a/sys/amd64/amd64/mp_machdep.c +++ b/sys/amd64/amd64/mp_machdep.c @@ -90,7 +90,7 @@ extern struct pcpu __pcpu[]; /* AP uses this during bootstrap. Do not staticize. */ char *bootSTK; -static int bootAP; +int bootAP; /* Free these after use */ void *bootstacks[MAXCPU]; @@ -122,9 +122,12 @@ u_long *ipi_rendezvous_counts[MAXCPU]; static u_long *ipi_hardclock_counts[MAXCPU]; #endif +int native_start_all_aps(void); + /* Default cpu_ops implementation. */ struct cpu_ops cpu_ops = { - .ipi_vectored = lapic_ipi_vectored + .ipi_vectored = lapic_ipi_vectored, + .start_all_aps = native_start_all_aps, }; extern inthand_t IDTVEC(fast_syscall), IDTVEC(fast_syscall32); @@ -138,7 +141,7 @@ extern int pmap_pcid_enabled; static volatile cpuset_t ipi_nmi_pending; /* used to hold the AP's until we are ready to release them */ -static struct mtx ap_boot_mtx; +struct mtx ap_boot_mtx; /* Set to 1 once we're ready to let the APs out of the pen. */ static volatile int aps_ready = 0; @@ -165,7 +168,6 @@ static int cpu_cores; /* cores per package */ static void assign_cpu_ids(void); static void set_interrupt_apic_ids(void); -static int start_all_aps(void); static int start_ap(int apic_id); static void release_aps(void *dummy); @@ -569,7 +571,7 @@ cpu_mp_start(void) assign_cpu_ids(); /* Start each Application Processor */ - start_all_aps(); + cpu_ops.start_all_aps(); set_interrupt_apic_ids(); } @@ -908,8 +910,8 @@ assign_cpu_ids(void) /* * start each AP in our list */ -static int -start_all_aps(void) +int +native_start_all_aps(void) { vm_offset_t va = boot_address + KERNBASE; u_int64_t *pt4, *pt3, *pt2; diff --git a/sys/amd64/include/cpu.h b/sys/amd64/include/cpu.h index 3d9ff531..ed9f1db 100644 --- a/sys/amd64/include/cpu.h +++ b/sys/amd64/include/cpu.h @@ -64,6 +64,7 @@ struct cpu_ops { void (*cpu_init)(void); void (*cpu_resume)(void); void (*ipi_vectored)(u_int, int); + int (*start_all_aps)(void); }; extern struct cpu_ops cpu_ops; diff --git a/sys/x86/xen/hvm.c b/sys/x86/xen/hvm.c index fb1ed79..5ec9f3a 100644 --- a/sys/x86/xen/hvm.c +++ b/sys/x86/xen/hvm.c @@ -53,6 +53,9 @@ __FBSDID("$FreeBSD$"); #include #include #include +#ifdef __amd64__ +#include +#endif #include #include @@ -97,6 +100,11 @@ extern void pmap_lazyfix_action(void); /* Variables used by mp_machdep to perform the bitmap IPI */ extern volatile u_int cpu_ipi_pending[MAXCPU]; +#ifdef __amd64__ +/* Native AP start used on PVHVM */ +extern int native_start_all_aps(void); +#endif + /*---------------------------------- Macros ----------------------------------*/ #define IPI_TO_IDX(ipi) ((ipi) - APIC_IPI_INTS) @@ -119,7 +127,10 @@ enum xen_domain_type xen_domain_type = XEN_NATIVE; struct cpu_ops xen_hvm_cpu_ops = { .ipi_vectored = lapic_ipi_vectored, .cpu_init = xen_hvm_cpu_init, - .cpu_resume = xen_hvm_cpu_resume + .cpu_resume = xen_hvm_cpu_resume, +#ifdef __amd64__ + .start_all_aps = native_start_all_aps, +#endif }; static MALLOC_DEFINE(M_XENHVM, "xen_hvm", "Xen HVM PV Support"); @@ -698,6 +709,10 @@ xen_hvm_init(enum xen_hvm_init_type init_type) setup_xen_features(); cpu_ops = xen_hvm_cpu_ops; vm_guest = VM_GUEST_XEN; +#ifdef __amd64__ + if (xen_pv_domain()) + cpu_ops.start_all_aps = xen_pv_start_all_aps; +#endif break; case XEN_HVM_INIT_RESUME: if (error != 0) diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c index db576e0..7e45a83 100644 --- a/sys/x86/xen/pv.c +++ b/sys/x86/xen/pv.c @@ -34,21 +34,43 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include +#include +#include +#include + +#include +#include +#include +#include #include #include #include +#include #include #include #include +#include + #define MAX_E820_ENTRIES 128 /*--------------------------- Forward Declarations ---------------------------*/ static caddr_t xen_pv_parse_preload_data(u_int64_t); static void xen_pv_parse_memmap(caddr_t, vm_paddr_t *, int *); +/*---------------------------- Extern Declarations ---------------------------*/ +/* Variables used by amd64 mp_machdep to start APs */ +extern struct mtx ap_boot_mtx; +extern void *bootstacks[]; +extern char *doublefault_stack; +extern char *nmi_stack; +extern void *dpcpu; +extern int bootAP; +extern char *bootSTK; + /*-------------------------------- Global Data -------------------------------*/ /* Xen init_ops implementation. */ struct init_ops xen_init_ops = { @@ -78,6 +100,74 @@ static struct static struct bios_smap xen_smap[MAX_E820_ENTRIES]; +static int +start_xen_ap(int cpu) +{ + struct vcpu_guest_context *ctxt; + int ms, cpus = mp_naps; + + ctxt = malloc(sizeof(*ctxt), M_TEMP, M_NOWAIT | M_ZERO); + if (ctxt == NULL) + panic("unable to allocate memory"); + + ctxt->flags = VGCF_IN_KERNEL; + ctxt->user_regs.rip = (unsigned long) init_secondary; + ctxt->user_regs.rsp = (unsigned long) bootSTK; + + /* Set the AP to use the same page tables */ + ctxt->ctrlreg[3] = KPML4phys; + + if (HYPERVISOR_vcpu_op(VCPUOP_initialise, cpu, ctxt)) + panic("unable to initialize AP#%d\n", cpu); + + free(ctxt, M_TEMP); + + /* Launch the vCPU */ + if (HYPERVISOR_vcpu_op(VCPUOP_up, cpu, NULL)) + panic("unable to start AP#%d\n", cpu); + + /* Wait up to 5 seconds for it to start. */ + for (ms = 0; ms < 5000; ms++) { + if (mp_naps > cpus) + return (1); /* return SUCCESS */ + DELAY(1000); + } + + return (0); +} + +int +xen_pv_start_all_aps(void) +{ + int cpu; + + mtx_init(&ap_boot_mtx, "ap boot", NULL, MTX_SPIN); + + for (cpu = 1; cpu < mp_ncpus; cpu++) { + + /* allocate and set up an idle stack data page */ + bootstacks[cpu] = (void *)kmem_malloc(kernel_arena, + KSTACK_PAGES * PAGE_SIZE, M_WAITOK | M_ZERO); + doublefault_stack = (char *)kmem_malloc(kernel_arena, + PAGE_SIZE, M_WAITOK | M_ZERO); + nmi_stack = (char *)kmem_malloc(kernel_arena, PAGE_SIZE, + M_WAITOK | M_ZERO); + dpcpu = (void *)kmem_malloc(kernel_arena, DPCPU_SIZE, + M_WAITOK | M_ZERO); + + bootSTK = (char *)bootstacks[cpu] + KSTACK_PAGES * PAGE_SIZE - 8; + bootAP = cpu; + + /* attempt to start the Application Processor */ + if (!start_xen_ap(cpu)) + panic("AP #%d failed to start!", cpu); + + CPU_SET(cpu, &all_cpus); /* record AP in CPU map */ + } + + return (mp_naps); +} + /* * Functions to convert the "extra" parameters passed by Xen * into FreeBSD boot options (from the i386 Xen port). diff --git a/sys/xen/pv.h b/sys/xen/pv.h index 71b8776..60d9def 100644 --- a/sys/xen/pv.h +++ b/sys/xen/pv.h @@ -24,5 +24,6 @@ #define __XEN_PV_H__ void xen_pv_set_init_ops(void); +int xen_pv_start_all_aps(void); #endif /* __XEN_PV_H__ */ -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 19:25:35 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3FA4A19B; Thu, 19 Dec 2013 19:25:35 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9294E1063; Thu, 19 Dec 2013 19:25:33 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84066172" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 19:25:31 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 14:25:30 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtiko-0006gd-92; Thu, 19 Dec 2013 18:55:26 +0000 From: Roger Pau Monne To: , , , , , , Subject: =?UTF-8?q?=5BPATCH=20v7=2014/19=5D=20xen=3A=20introduce=20a=20dummy=20pvcpu=20device?= Date: Thu, 19 Dec 2013 19:54:51 +0100 Message-ID: <1387479296-33389-15-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 19:25:35 -0000 Since Xen PVH guests doesn't have ACPI, we need to create a dummy pvcpu device that will be used to fill the pcpu->pc_device field. --- sys/conf/files.amd64 | 1 + sys/x86/xen/pvcpu.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 85 insertions(+), 0 deletions(-) create mode 100644 sys/x86/xen/pvcpu.c diff --git a/sys/conf/files.amd64 b/sys/conf/files.amd64 index 3bdc05e..4a3f612 100644 --- a/sys/conf/files.amd64 +++ b/sys/conf/files.amd64 @@ -569,3 +569,4 @@ x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm x86/xen/pv.c optional xenhvm x86/xen/mptable.c optional xenhvm +x86/xen/pvcpu.c optional xenhvm diff --git a/sys/x86/xen/pvcpu.c b/sys/x86/xen/pvcpu.c new file mode 100644 index 0000000..8725e76 --- /dev/null +++ b/sys/x86/xen/pvcpu.c @@ -0,0 +1,84 @@ +/* + * Copyright (c) 2013 Roger Pau MonnĂ© + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include + +/* + * Dummy Xen cpu device + * + * Since there's no ACPI on PVH guests, we need to create a dummy + * CPU device in order to fill the pcpu->pc_device field. + */ + +static int +xenpvcpu_probe(device_t dev) +{ + if (!xen_pv_domain()) + return (ENXIO); + + device_set_desc(dev, "Xen PV CPU"); + return (0); +} + +static int +xenpvcpu_attach(device_t dev) +{ + struct pcpu *pc; + int cpu; + + cpu = device_get_unit(dev); + pc = pcpu_find(cpu); + pc->pc_device = dev; + return (0); +} + +static device_method_t xenpvcpu_methods[] = { + DEVMETHOD(device_probe, xenpvcpu_probe), + DEVMETHOD(device_attach, xenpvcpu_attach), + DEVMETHOD_END +}; + +static driver_t xenpvcpu_driver = { + "pvcpu", + xenpvcpu_methods, + 0, +}; + +devclass_t xenpvcpu_devclass; + +DRIVER_MODULE(xenpvcpu, nexus, xenpvcpu_driver, xenpvcpu_devclass, 0, 0); +MODULE_DEPEND(xenpvcpu, nexus, 1, 1, 1); -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 19:25:40 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 788EB2C9; Thu, 19 Dec 2013 19:25:40 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 01CA3106C; Thu, 19 Dec 2013 19:25:38 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="86300020" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 19 Dec 2013 19:25:36 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 14:25:35 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikp-0006gd-EJ; Thu, 19 Dec 2013 18:55:27 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 16/19] xen: add shutdown hook for PVH Date: Thu, 19 Dec 2013 19:54:53 +0100 Message-ID: <1387479296-33389-17-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 19:25:40 -0000 Add the PV shutdown hook to PVH. --- sys/dev/xen/control/control.c | 37 ++++++++++++++++++------------------- 1 files changed, 18 insertions(+), 19 deletions(-) diff --git a/sys/dev/xen/control/control.c b/sys/dev/xen/control/control.c index bc0609d..78894ba 100644 --- a/sys/dev/xen/control/control.c +++ b/sys/dev/xen/control/control.c @@ -316,21 +316,6 @@ xctrl_suspend() EVENTHANDLER_INVOKE(power_resume); } -static void -xen_pv_shutdown_final(void *arg, int howto) -{ - /* - * Inform the hypervisor that shutdown is complete. - * This is not necessary in HVM domains since Xen - * emulates ACPI in that mode and FreeBSD's ACPI - * support will request this transition. - */ - if (howto & (RB_HALT | RB_POWEROFF)) - HYPERVISOR_shutdown(SHUTDOWN_poweroff); - else - HYPERVISOR_shutdown(SHUTDOWN_reboot); -} - #else /* HVM mode suspension. */ @@ -440,6 +425,21 @@ xctrl_crash() panic("Xen directed crash"); } +static void +xen_pv_shutdown_final(void *arg, int howto) +{ + /* + * Inform the hypervisor that shutdown is complete. + * This is not necessary in HVM domains since Xen + * emulates ACPI in that mode and FreeBSD's ACPI + * support will request this transition. + */ + if (howto & (RB_HALT | RB_POWEROFF)) + HYPERVISOR_shutdown(SHUTDOWN_poweroff); + else + HYPERVISOR_shutdown(SHUTDOWN_reboot); +} + /*------------------------------ Event Reception -----------------------------*/ static void xctrl_on_watch_event(struct xs_watch *watch, const char **vec, unsigned int len) @@ -522,10 +522,9 @@ xctrl_attach(device_t dev) xctrl->xctrl_watch.callback_data = (uintptr_t)xctrl; xs_register_watch(&xctrl->xctrl_watch); -#ifndef XENHVM - EVENTHANDLER_REGISTER(shutdown_final, xen_pv_shutdown_final, NULL, - SHUTDOWN_PRI_LAST); -#endif + if (xen_pv_domain()) + EVENTHANDLER_REGISTER(shutdown_final, xen_pv_shutdown_final, NULL, + SHUTDOWN_PRI_LAST); return (0); } -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 19:25:37 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2483B28A; Thu, 19 Dec 2013 19:25:37 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8B56E1065; Thu, 19 Dec 2013 19:25:35 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84066191" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 19:25:34 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 14:25:32 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtiko-0006gd-SL; Thu, 19 Dec 2013 18:55:26 +0000 From: Roger Pau Monne To: , , , , , , Subject: =?UTF-8?q?=5BPATCH=20v7=2015/19=5D=20xen=3A=20create=20a=20Xen=20nexus=20to=20use=20in=20PV/PVH?= Date: Thu, 19 Dec 2013 19:54:52 +0100 Message-ID: <1387479296-33389-16-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 19:25:37 -0000 Introduce a Xen specific nexus that is going to be in charge for attaching Xen specific devices. Remove the identify routine from Xen devices and instead attach them from the nexus (PV/PVH) or xenpci (HVM). --- sys/conf/files.amd64 | 1 + sys/conf/files.i386 | 1 + sys/dev/xen/timer/timer.c | 14 ------ sys/dev/xen/xenpci/xenpci.c | 5 ++ sys/x86/xen/xen_nexus.c | 99 +++++++++++++++++++++++++++++++++++++++++++ sys/xen/xenstore/xenstore.c | 7 --- 6 files changed, 106 insertions(+), 21 deletions(-) create mode 100644 sys/x86/xen/xen_nexus.c diff --git a/sys/conf/files.amd64 b/sys/conf/files.amd64 index 4a3f612..8bb0b5c 100644 --- a/sys/conf/files.amd64 +++ b/sys/conf/files.amd64 @@ -570,3 +570,4 @@ x86/xen/xen_intr.c optional xen | xenhvm x86/xen/pv.c optional xenhvm x86/xen/mptable.c optional xenhvm x86/xen/pvcpu.c optional xenhvm +x86/xen/xen_nexus.c optional xenhvm diff --git a/sys/conf/files.i386 b/sys/conf/files.i386 index 790296d..751db32 100644 --- a/sys/conf/files.i386 +++ b/sys/conf/files.i386 @@ -603,3 +603,4 @@ x86/x86/tsc.c standard x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm +x86/xen/xen_nexus.c optional xen | xenhvm diff --git a/sys/dev/xen/timer/timer.c b/sys/dev/xen/timer/timer.c index 96372ab..b047786 100644 --- a/sys/dev/xen/timer/timer.c +++ b/sys/dev/xen/timer/timer.c @@ -98,19 +98,6 @@ struct xentimer_softc { /* Last time; this guarantees a monotonically increasing clock. */ volatile uint64_t xen_timer_last_time = 0; -static void -xentimer_identify(driver_t *driver, device_t parent) -{ - if (!xen_domain()) - return; - - /* Handle all Xen PV timers in one device instance. */ - if (devclass_get_device(xentimer_devclass, 0)) - return; - - BUS_ADD_CHILD(parent, 0, "xen_et", 0); -} - static int xentimer_probe(device_t dev) { @@ -618,7 +605,6 @@ void xen_delay(int n) } static device_method_t xentimer_methods[] = { - DEVMETHOD(device_identify, xentimer_identify), DEVMETHOD(device_probe, xentimer_probe), DEVMETHOD(device_attach, xentimer_attach), DEVMETHOD(device_detach, xentimer_detach), diff --git a/sys/dev/xen/xenpci/xenpci.c b/sys/dev/xen/xenpci/xenpci.c index dd2ad92..cac8022 100644 --- a/sys/dev/xen/xenpci/xenpci.c +++ b/sys/dev/xen/xenpci/xenpci.c @@ -270,6 +270,11 @@ xenpci_attach(device_t dev) goto errexit; } + if (BUS_ADD_CHILD(dev, 0, "xenstore", 0) == NULL) + panic("xenpci: unable to add xenstore device"); + if (BUS_ADD_CHILD(nexus, 0, "xen_et", 0) == NULL) + panic("xenpci: unable to add xen pv timer device"); + return (bus_generic_attach(dev)); errexit: diff --git a/sys/x86/xen/xen_nexus.c b/sys/x86/xen/xen_nexus.c new file mode 100644 index 0000000..288e6b6 --- /dev/null +++ b/sys/x86/xen/xen_nexus.c @@ -0,0 +1,99 @@ +/* + * Copyright (c) 2013 Roger Pau MonnĂ© + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include + +static const char *xen_devices[] = +{ + "xenstore", /* XenStore bus */ + "xen_et", /* Xen PV timer (provides: tc, et, clk) */ + "xc", /* Xen PV console */ + "isa", /* Dummy ISA bus for sc to attach */ +}; + +/* + * Xen nexus(4) driver. + */ +static int +nexus_xen_probe(device_t dev) +{ + if (!xen_pv_domain()) + return (ENXIO); + + return (BUS_PROBE_DEFAULT); +} + +static int +nexus_xen_attach(device_t dev) +{ + int i, error = 0; + + nexus_init_resources(); + bus_generic_probe(dev); + + /* + * Since we have no ACPI, we need to create a dummy CPU device + * in order to set pcpu->pc_device. + */ + CPU_FOREACH(i) + if (BUS_ADD_CHILD(dev, 0, "pvcpu", i) == NULL) + panic("unable to add pvcpu#%d device", i); + + for (i = 0; i < nitems(xen_devices); i++) { + if (BUS_ADD_CHILD(dev, 0, xen_devices[i], 0) == NULL) + panic("%s: could not add", xen_devices[i]); + } + + bus_generic_attach(dev); + + return (error); +} + +static device_method_t nexus_xen_methods[] = { + /* Device interface */ + DEVMETHOD(device_probe, nexus_xen_probe), + DEVMETHOD(device_attach, nexus_xen_attach), + + { 0, 0 } +}; + +DEFINE_CLASS_1(nexus, nexus_xen_driver, nexus_xen_methods, 1, nexus_driver); +static devclass_t nexus_devclass; + +DRIVER_MODULE(nexus_xen, root, nexus_xen_driver, nexus_devclass, 0, 0); diff --git a/sys/xen/xenstore/xenstore.c b/sys/xen/xenstore/xenstore.c index d404862..bcf6357 100644 --- a/sys/xen/xenstore/xenstore.c +++ b/sys/xen/xenstore/xenstore.c @@ -1079,12 +1079,6 @@ xs_init_comms(void) } /*------------------ Private Device Attachment Functions --------------------*/ -static void -xs_identify(driver_t *driver, device_t parent) -{ - - BUS_ADD_CHILD(parent, 0, "xenstore", 0); -} /** * Probe for the existance of the XenStore. @@ -1240,7 +1234,6 @@ xs_resume(device_t dev __unused) /*-------------------- Private Device Attachment Data -----------------------*/ static device_method_t xenstore_methods[] = { /* Device interface */ - DEVMETHOD(device_identify, xs_identify), DEVMETHOD(device_probe, xs_probe), DEVMETHOD(device_attach, xs_attach), DEVMETHOD(device_detach, bus_generic_detach), -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 19:25:43 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 52D4C2FE; Thu, 19 Dec 2013 19:25:43 +0000 (UTC) Received: from SMTP.CITRIX.COM (smtp.citrix.com [66.165.176.89]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D16491075; Thu, 19 Dec 2013 19:25:41 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="86300046" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO01.CITRIX.COM with ESMTP; 19 Dec 2013 19:25:41 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 14:25:40 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikn-0006gd-M5; Thu, 19 Dec 2013 18:55:25 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 13/19] xen: introduce flag to disable the local apic Date: Thu, 19 Dec 2013 19:54:50 +0100 Message-ID: <1387479296-33389-14-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 19:25:43 -0000 PVH guests don't have an emulated lapic. --- sys/amd64/amd64/mp_machdep.c | 10 ++++++---- sys/amd64/include/apicvar.h | 1 + sys/i386/include/apicvar.h | 1 + sys/i386/xen/xen_machdep.c | 2 ++ sys/x86/x86/local_apic.c | 8 +++++--- sys/x86/xen/pv.c | 3 +++ 6 files changed, 18 insertions(+), 7 deletions(-) diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c index e302886..0296a92 100644 --- a/sys/amd64/amd64/mp_machdep.c +++ b/sys/amd64/amd64/mp_machdep.c @@ -709,7 +709,8 @@ init_secondary(void) wrmsr(MSR_SF_MASK, PSL_NT|PSL_T|PSL_I|PSL_C|PSL_D); /* Disable local APIC just to be sure. */ - lapic_disable(); + if (!lapic_disabled) + lapic_disable(); /* signal our startup to the BSP. */ mp_naps++; @@ -735,7 +736,7 @@ init_secondary(void) /* A quick check from sanity claus */ cpuid = PCPU_GET(cpuid); - if (PCPU_GET(apic_id) != lapic_id()) { + if (!lapic_disabled && (PCPU_GET(apic_id) != lapic_id())) { printf("SMP: cpuid = %d\n", cpuid); printf("SMP: actual apic_id = %d\n", lapic_id()); printf("SMP: correct apic_id = %d\n", PCPU_GET(apic_id)); @@ -751,7 +752,8 @@ init_secondary(void) mtx_lock_spin(&ap_boot_mtx); /* Init local apic for irq's */ - lapic_setup(1); + if (!lapic_disabled) + lapic_setup(1); /* Set memory range attributes for this CPU to match the BSP */ mem_range_AP_init(); @@ -766,7 +768,7 @@ init_secondary(void) if (cpu_logical > 1 && PCPU_GET(apic_id) % cpu_logical != 0) CPU_SET(cpuid, &logical_cpus_mask); - if (bootverbose) + if (!lapic_disabled && bootverbose) lapic_dump("AP"); if (smp_cpus == mp_ncpus) { diff --git a/sys/amd64/include/apicvar.h b/sys/amd64/include/apicvar.h index e7423a3..84e01d4 100644 --- a/sys/amd64/include/apicvar.h +++ b/sys/amd64/include/apicvar.h @@ -169,6 +169,7 @@ inthand_t extern vm_paddr_t lapic_paddr; extern int apic_cpuids[]; +extern bool lapic_disabled; u_int apic_alloc_vector(u_int apic_id, u_int irq); u_int apic_alloc_vectors(u_int apic_id, u_int *irqs, u_int count, diff --git a/sys/i386/include/apicvar.h b/sys/i386/include/apicvar.h index df99ebe..24c99f0 100644 --- a/sys/i386/include/apicvar.h +++ b/sys/i386/include/apicvar.h @@ -168,6 +168,7 @@ inthand_t extern vm_paddr_t lapic_paddr; extern int apic_cpuids[]; +extern bool lapic_disabled; u_int apic_alloc_vector(u_int apic_id, u_int irq); u_int apic_alloc_vectors(u_int apic_id, u_int *irqs, u_int count, diff --git a/sys/i386/xen/xen_machdep.c b/sys/i386/xen/xen_machdep.c index 09c01f1..7039b4a 100644 --- a/sys/i386/xen/xen_machdep.c +++ b/sys/i386/xen/xen_machdep.c @@ -59,6 +59,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include @@ -912,6 +913,7 @@ initvalues(start_info_t *startinfo) #endif xen_start_info = startinfo; HYPERVISOR_start_info = startinfo; + lapic_disabled = true; xen_phys_machine = (xen_pfn_t *)startinfo->mfn_list; IdlePTD = (pd_entry_t *)((uint8_t *)startinfo->pt_base + PAGE_SIZE); diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c index 41bd602..85736c8 100644 --- a/sys/x86/x86/local_apic.c +++ b/sys/x86/x86/local_apic.c @@ -156,6 +156,7 @@ extern inthand_t IDTVEC(rsvd); volatile lapic_t *lapic; vm_paddr_t lapic_paddr; +bool lapic_disabled; static u_long lapic_timer_divisor; static struct eventtimer lapic_et; @@ -1367,9 +1368,10 @@ apic_setup_io(void *dummy __unused) if (retval != 0) printf("%s: Failed to setup I/O APICs: returned %d\n", best_enum->apic_name, retval); -#ifdef XEN - return; -#endif + + if (lapic_disabled) + return; + /* * Finish setting up the local APIC on the BSP once we know how to * properly program the LINT pins. diff --git a/sys/x86/xen/pv.c b/sys/x86/xen/pv.c index 7e45a83..e783bb8 100644 --- a/sys/x86/xen/pv.c +++ b/sys/x86/xen/pv.c @@ -48,6 +48,7 @@ __FBSDID("$FreeBSD$"); #include #include #include +#include #include #include @@ -230,4 +231,6 @@ xen_pv_set_init_ops(void) { /* Init ops for Xen PV */ init_ops = xen_init_ops; + /* Disable lapic */ + lapic_disabled = true; } -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 19:25:25 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DC121E54; Thu, 19 Dec 2013 19:25:25 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 465661059; Thu, 19 Dec 2013 19:25:24 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84066105" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 19:25:21 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 14:25:20 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikm-0006gd-Gk; Thu, 19 Dec 2013 18:55:24 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 11/19] xen: changes to hvm code in order to support PVH guests Date: Thu, 19 Dec 2013 19:54:48 +0100 Message-ID: <1387479296-33389-12-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA2 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 19:25:25 -0000 On PVH we don't need to init the shared info page, or disable emulated devices. Also, make sure PV IPIs are set before starting the APs. --- sys/x86/xen/hvm.c | 17 ++++++++++++----- 1 files changed, 12 insertions(+), 5 deletions(-) diff --git a/sys/x86/xen/hvm.c b/sys/x86/xen/hvm.c index 9a0411e..fb1ed79 100644 --- a/sys/x86/xen/hvm.c +++ b/sys/x86/xen/hvm.c @@ -523,7 +523,7 @@ xen_setup_cpus(void) { int i; - if (!xen_hvm_domain() || !xen_vector_callback_enabled) + if (!xen_vector_callback_enabled) return; #ifdef __amd64__ @@ -712,10 +712,13 @@ xen_hvm_init(enum xen_hvm_init_type init_type) } xen_vector_callback_enabled = 0; - xen_domain_type = XEN_HVM_DOMAIN; - xen_hvm_init_shared_info_page(); xen_hvm_set_callback(NULL); - xen_hvm_disable_emulated_devices(); + + if (!xen_pv_domain()) { + xen_domain_type = XEN_HVM_DOMAIN; + xen_hvm_init_shared_info_page(); + xen_hvm_disable_emulated_devices(); + } } void @@ -746,6 +749,9 @@ xen_set_vcpu_id(void) struct pcpu *pc; int i; + if (!xen_hvm_domain()) + return; + /* Set vcpu_id to acpi_id */ CPU_FOREACH(i) { pc = pcpu_find(i); @@ -789,7 +795,8 @@ xen_hvm_cpu_init(void) SYSINIT(xen_hvm_init, SI_SUB_HYPERVISOR, SI_ORDER_FIRST, xen_hvm_sysinit, NULL); #ifdef SMP -SYSINIT(xen_setup_cpus, SI_SUB_SMP, SI_ORDER_FIRST, xen_setup_cpus, NULL); +/* We need to setup IPIs before APs are started */ +SYSINIT(xen_setup_cpus, SI_SUB_SMP-1, SI_ORDER_FIRST, xen_setup_cpus, NULL); #endif SYSINIT(xen_hvm_cpu_init, SI_SUB_INTR, SI_ORDER_FIRST, xen_hvm_cpu_init, NULL); SYSINIT(xen_set_vcpu_id, SI_SUB_CPU, SI_ORDER_ANY, xen_set_vcpu_id, NULL); -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Thu Dec 19 19:25:42 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DD5E32FA; Thu, 19 Dec 2013 19:25:42 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 457BF1070; Thu, 19 Dec 2013 19:25:41 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,514,1384300800"; d="scan'208";a="84066208" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 19:25:39 +0000 Received: from ukmail1.uk.xensource.com (10.80.16.128) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 14:25:38 -0500 Received: from gateway-cbg.eng.citrite.net ([10.80.16.17] helo=localhost.localdomain) by ukmail1.uk.xensource.com with esmtp (Exim 4.69) (envelope-from ) id 1Vtikl-0006gd-TX; Thu, 19 Dec 2013 18:55:23 +0000 From: Roger Pau Monne To: , , , , , , Subject: [PATCH v7 10/19] xen: add hook for AP bootstrap memory reservation Date: Thu, 19 Dec 2013 19:54:47 +0100 Message-ID: <1387479296-33389-11-git-send-email-roger.pau@citrix.com> X-Mailer: git-send-email 1.7.7.5 (Apple Git-26) In-Reply-To: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: text/plain X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 19:25:42 -0000 This hook will only be implemented for bare metal, Xen doesn't require any bootstrap code since APs are started in long mode with paging enabled. --- sys/amd64/amd64/machdep.c | 6 +++++- sys/amd64/include/sysarch.h | 3 +++ 2 files changed, 8 insertions(+), 1 deletions(-) diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c index 6bbfe5a..a811a9b 100644 --- a/sys/amd64/amd64/machdep.c +++ b/sys/amd64/amd64/machdep.c @@ -186,6 +186,9 @@ struct init_ops init_ops = { .early_delay_init = i8254_init, .early_delay = i8254_delay, .parse_memmap = native_parse_memmap, +#ifdef SMP + .mp_bootaddress = mp_bootaddress, +#endif }; /* @@ -1507,7 +1510,8 @@ getmemsize(caddr_t kmdp, u_int64_t first) #ifdef SMP /* make hole for AP bootstrap code */ - physmap[1] = mp_bootaddress(physmap[1] / 1024); + if (init_ops.mp_bootaddress) + physmap[1] = init_ops.mp_bootaddress(physmap[1] / 1024); #endif /* diff --git a/sys/amd64/include/sysarch.h b/sys/amd64/include/sysarch.h index 084223e..77f4b29 100644 --- a/sys/amd64/include/sysarch.h +++ b/sys/amd64/include/sysarch.h @@ -16,6 +16,9 @@ struct init_ops { void (*early_delay_init)(void); void (*early_delay)(int); void (*parse_memmap)(caddr_t, vm_paddr_t *, int *); +#ifdef SMP + u_int (*mp_bootaddress)(u_int); +#endif }; extern struct init_ops init_ops; -- 1.7.7.5 (Apple Git-26) From owner-freebsd-xen@FreeBSD.ORG Fri Dec 20 07:36:54 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EF7F049E; Fri, 20 Dec 2013 07:36:54 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5D7F71663; Fri, 20 Dec 2013 07:36:54 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id rBK7ah2j025516; Fri, 20 Dec 2013 09:36:43 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua rBK7ah2j025516 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id rBK7agkV025515; Fri, 20 Dec 2013 09:36:42 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 20 Dec 2013 09:36:42 +0200 From: Konstantin Belousov To: Roger Pau Monne Subject: Re: [PATCH v7 01/19] xen: add PV/PVH kernel entry point Message-ID: <20131220073642.GP59496@kib.kiev.ua> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> <1387479296-33389-2-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/NOItKPPfAGZtKZd" Content-Disposition: inline In-Reply-To: <1387479296-33389-2-git-send-email-roger.pau@citrix.com> User-Agent: Mutt/1.5.22 (2013-10-16) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: jhb@freebsd.org, xen-devel@lists.xen.org, julien.grall@citrix.com, freebsd-xen@freebsd.org, freebsd-current@freebsd.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Dec 2013 07:36:55 -0000 --/NOItKPPfAGZtKZd Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Thank you for doing the split. On Thu, Dec 19, 2013 at 07:54:38PM +0100, Roger Pau Monne wrote: > Add the PV/PVH entry point and the low level functions for PVH > initialization. > --- > sys/amd64/amd64/locore.S | 53 +++++++++++++++++++++++++++++++ > sys/amd64/amd64/machdep.c | 72 ++++++++++++++++++++++++++++++++++++++++++ > sys/amd64/include/asmacros.h | 26 +++++++++++++++ The changes to the three files above all protected by #ifdef XENHVM. IMO it is more logical to move the code into xen-specific files. The support would be localized and somewhat postpone morphing the sys/amd64 into the maze of #ifdefs, which is the sys/i386 already. --/NOItKPPfAGZtKZd Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSs/OJAAoJEJDCuSvBvK1B+2cQAJKoJaZryNNJuOiN67ocHEHy R3PyKtisqo0b2ABgobYhfv1wxRXfr3fhW2Hqzd9GJfEcg/7hV9KSjilBIDnWrbii 720/VuGFh3vQRIWSu6WqylVPxkPkNrEW6pbzcja/FuxYeyheFanxMT5Fu3/uWxGG CadiT8VErkh6Z4HZnOi0goURiXkzpvU4kc/q0MeeUuPWd1KEA+OPxBmcaOz9+lfi FYmRJ0u3BqmPHvbXQpWMeaTAOS5MdhJkp3v5Q2tFlqNHXNYesm3JpwdaiceHAzUG gfpvBKxxVGw1TP3TcBuxN3v8NMlzsFdaO0EfNXL2jR12x21rs67t4LZxWpsF/5yr 6pAp+1DHWsY4FSKTAvRXI5XHlrA1Zb8IP/egIrtEVC0FoCKdgjgzS1mr3EOIYp4t 126R5fvc7TIcC1yZF6YfdlYrmT6RLducDYptnhibQ9DOixYV1iwiBpCGK5FX5etp Bg8i5SBdxDcqPYQvbg7PH+BaWct1K+Y48a/CDAb9/ACtwk8G+OHHmTEHLpx9NEs5 D38iuAi+bdRPjBVYW8n6CFGXgvSsLzLoYPUshPo9BZt2uB0opo5Og4nnoigSxk/L /hL+SLz3gaYmVNSUGu15UkTRLDuep5Sq4vHba09jmoptLoItxAPA3x4Drt9Je7zK CA4TxTE0C2l+BKLpYjuV =bBCJ -----END PGP SIGNATURE----- --/NOItKPPfAGZtKZd-- From owner-freebsd-xen@FreeBSD.ORG Fri Dec 20 08:10:36 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 248D2E82; Fri, 20 Dec 2013 08:10:36 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 8528C1984; Fri, 20 Dec 2013 08:10:35 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id rBK89Ae0031929; Fri, 20 Dec 2013 10:09:10 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua rBK89Ae0031929 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id rBK89Alt031928; Fri, 20 Dec 2013 10:09:10 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 20 Dec 2013 10:09:10 +0200 From: Konstantin Belousov To: Roger Pau Monne Subject: Re: [PATCH v7 04/19] amd64: introduce hook for custom preload metadata parsers Message-ID: <20131220080910.GQ59496@kib.kiev.ua> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> <1387479296-33389-5-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="hVxjDcWLALlH2Fxo" Content-Disposition: inline In-Reply-To: <1387479296-33389-5-git-send-email-roger.pau@citrix.com> User-Agent: Mutt/1.5.22 (2013-10-16) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: jhb@freebsd.org, xen-devel@lists.xen.org, julien.grall@citrix.com, freebsd-xen@freebsd.org, freebsd-current@freebsd.org, kib@freebsd.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Dec 2013 08:10:36 -0000 --hVxjDcWLALlH2Fxo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Thu, Dec 19, 2013 at 07:54:41PM +0100, Roger Pau Monne wrote: > --- > sys/amd64/amd64/machdep.c | 45 +++++++++++++---- > sys/amd64/include/sysarch.h | 12 +++++ > sys/conf/files.amd64 | 1 + This looks fine for me. --hVxjDcWLALlH2Fxo Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSs/slAAoJEJDCuSvBvK1B/g0P/RKm3rkVmHf17jzTwa/Ox02i FBUWVLpWIG7AcJrR48MUAyy0tTjsO31PFvjoY/5V0ydvW+Ym2k7Qqznt2BQ+41xX odSVxtlFdDhtcICU4WJhvDtxtP/3jYVqoZyAf4USZ8cSksgPtUjH5mK4F2AxqkQx GICsKr8ODYK6+pv/STjNPtkB6mssWaAiykqGEtT/GaLMtYKEv7NeAgsoanTSFQxU 1wQCmw0CoeXjZmuC9asJkO9FxcMaPMhmz1xsLbRc4ZDIavL3q0Q1ceSZOW2yIGK7 Mx6MTLnM/KfTBn610YtPhY+x9WDh/ZUmNGzQWco/nymv0CTD8dvlW9Rh7MSYZz4M 6766T7ufjQfBUcI7YgNHwkOEUSTZcNmdWfHK8sJoox/QFkcrZLIPpkyC8fFk+LPe kIrxsjnDdqcgjNDfQuMLCNR2myYiGHnNltjBYf6o5Eld2jNKY0fj8c/qmf3VC0Ya sqpvkseEx8UQRBsKQR5cyIGXS/fGKXUEYj76JmXWVVGqrYJVQUFmmVmxLu5lh/PL rXzjuzWlID11d/9LqXPJQFYYNIZoinWEwWi17C0SLupinFhjpLQhK0XDwA33JnPF Bf/sd8z9UEsTgLTMjaLzY63FUu4dbLKsEngwVlPpafT+fwtcwuZzRpvp1DdbYl4s LPMDW3d3tu6hBqTFRVPs =08Zf -----END PGP SIGNATURE----- --hVxjDcWLALlH2Fxo-- From owner-freebsd-xen@FreeBSD.ORG Fri Dec 20 15:15:44 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 824A9EB4 for ; Fri, 20 Dec 2013 15:15:44 +0000 (UTC) Received: from mail-we0-f172.google.com (mail-we0-f172.google.com [74.125.82.172]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0F8D61D5C for ; Fri, 20 Dec 2013 15:15:43 +0000 (UTC) Received: by mail-we0-f172.google.com with SMTP id p61so2626748wes.31 for ; Fri, 20 Dec 2013 07:15:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :subject:references:in-reply-to:content-type :content-transfer-encoding; bh=pDbyOqmTxnsX9at/P6fQnqhK9TX8H8XGEq/224LjTrs=; b=Hp3KLf8g0D/R0iS6Ttz+5lcuP1GF+e1L5oaFsU1d+ATQFlwZbA01qbK4cw7tiD+Hfp gfU+v56qlh5XyXuaBP6N79qBDlA0qLshZcUbNjeZOaTCcUpJP5yjbywZmffep4IuEqZa 8bsZtXPlJND8kWyamigqQW6YEOW01Dq/Yvqch/rOdo2rsFvmvHFmSXPHDdTRmZe7I48b //uaHyQG7bJ2DuXx1dEGzQQjEJkeXWmznf8+w3sUZjbCO3oDW474YpU3OjspU7XsqSr6 BPjo8bfSofHpBjamm/k0dqrHowSg0wSkSMP3sY5/X1N9Hw79utVwXs/yTRRR1ldIpyc4 Bgxg== X-Gm-Message-State: ALoCoQl9au65nJK9p45+eKVeBhaX19yuyscGHD4kV75hAsf+1UZ/sjx3OuylDqmisKoAw0DEs3Ky X-Received: by 10.194.10.34 with SMTP id f2mr2369870wjb.77.1387552536580; Fri, 20 Dec 2013 07:15:36 -0800 (PST) Received: from [192.168.42.155] ([80.215.9.126]) by mx.google.com with ESMTPSA id pl7sm2770199wjc.16.2013.12.20.07.15.32 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 20 Dec 2013 07:15:34 -0800 (PST) Message-ID: <52B45F11.1030504@linaro.org> Date: Fri, 20 Dec 2013 15:15:29 +0000 From: Julien Grall User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Roger Pau Monne , freebsd-xen@freebsd.org, freebsd-current@freebsd.org, xen-devel@lists.xen.org, gibbs@freebsd.org, jhb@freebsd.org, kib@freebsd.org, julien.grall@citrix.com Subject: Re: [Xen-devel] [PATCH v7 01/19] xen: add PV/PVH kernel entry point References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> <1387479296-33389-2-git-send-email-roger.pau@citrix.com> In-Reply-To: <1387479296-33389-2-git-send-email-roger.pau@citrix.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Dec 2013 15:15:44 -0000 On 12/19/2013 06:54 PM, Roger Pau Monne wrote: > Add the PV/PVH entry point and the low level functions for PVH > initialization. > --- > sys/amd64/amd64/locore.S | 53 +++++++++++++++++++++++++++++++ > sys/amd64/amd64/machdep.c | 72 ++++++++++++++++++++++++++++++++++++++++++ > sys/amd64/include/asmacros.h | 26 +++++++++++++++ > sys/i386/xen/xen_machdep.c | 2 + > sys/x86/xen/hvm.c | 1 + > sys/xen/xen-os.h | 4 ++ > 6 files changed, 158 insertions(+), 0 deletions(-) > [..] > diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c > index eae657b..a73e33e 100644 > --- a/sys/amd64/amd64/machdep.c > +++ b/sys/amd64/amd64/machdep.c > @@ -146,10 +146,17 @@ __FBSDID("$FreeBSD$"); > #include > #include > > +#ifdef XENHVM > +#include > +#endif > + > /* Sanity check for __curthread() */ > CTASSERT(offsetof(struct pcpu, pc_curthread) == 0); > > extern u_int64_t hammer_time(u_int64_t, u_int64_t); > +#ifdef XENHVM > +extern u_int64_t hammer_time_xen(start_info_t *, u_int64_t); > +#endif > > extern void printcpuinfo(void); /* XXX header file */ > extern void identify_cpu(void); > @@ -1683,6 +1690,71 @@ do_next: > msgbufp = (struct msgbuf *)PHYS_TO_DMAP(phys_avail[pa_indx]); > } > > +#ifdef XENHVM > +/* > + * First function called by the Xen PVH boot sequence. > + * > + * Set some Xen global variables and prepare the environment so it is > + * as similar as possible to what native FreeBSD init function expects. > + */ > +u_int64_t > +hammer_time_xen(start_info_t *si, u_int64_t xenstack) > +{ > + u_int64_t physfree; > + u_int64_t *PT4 = (u_int64_t *)xenstack; > + u_int64_t *PT3 = (u_int64_t *)(xenstack + PAGE_SIZE); > + u_int64_t *PT2 = (u_int64_t *)(xenstack + 2 * PAGE_SIZE); > + int i; > + > + KASSERT((si != NULL && xenstack != 0), > + ("invalid start_info or xenstack")); If I'm not mistaken, the user won't see the error if the assert failed (the console is not yet initialized). Perhaps xc_printf is more suitable here. -- Julien Grall From owner-freebsd-xen@FreeBSD.ORG Sat Dec 21 13:57:14 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DBB7055A; Sat, 21 Dec 2013 13:57:13 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5D3FE16EB; Sat, 21 Dec 2013 13:57:13 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id rBLDtdEf027049; Sat, 21 Dec 2013 15:55:39 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua rBLDtdEf027049 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id rBLDtdLk027047; Sat, 21 Dec 2013 15:55:39 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 21 Dec 2013 15:55:39 +0200 From: Konstantin Belousov To: Roger Pau Monne , royger@freebsd.org Subject: Re: [PATCH v7 07/19] xen: implement hook to fetch e820 memory map Message-ID: <20131221135539.GW59496@kib.kiev.ua> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> <1387479296-33389-8-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="CT8mOM7I+A1mxHvt" Content-Disposition: inline In-Reply-To: <1387479296-33389-8-git-send-email-roger.pau@citrix.com> User-Agent: Mutt/1.5.22 (2013-10-16) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: jhb@freebsd.org, xen-devel@lists.xen.org, julien.grall@citrix.com, freebsd-xen@freebsd.org, freebsd-current@freebsd.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Dec 2013 13:57:14 -0000 --CT8mOM7I+A1mxHvt Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable This also looks fine, with the style nit noted in the patch. On Thu, Dec 19, 2013 at 07:54:44PM +0100, Roger Pau Monne wrote: > --- > sys/amd64/amd64/machdep.c | 50 ++++++++++++++++++++++++++-----------= ----- > sys/amd64/include/pc/bios.h | 2 + > sys/amd64/include/sysarch.h | 1 + > sys/x86/xen/pv.c | 26 ++++++++++++++++++++++ > 4 files changed, 60 insertions(+), 19 deletions(-) >=20 > diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c > index a2dcb90..6bbfe5a 100644 > --- a/sys/amd64/amd64/machdep.c > +++ b/sys/amd64/amd64/machdep.c > @@ -177,11 +177,15 @@ SYSINIT(cpu, SI_SUB_CPU, SI_ORDER_FIRST, cpu_startu= p, NULL); > /* Preload data parse function */ > static caddr_t native_parse_preload_data(u_int64_t); > =20 > +/* Native function to fetch and parse the e820 map */ > +static void native_parse_memmap(caddr_t, vm_paddr_t *, int *); > + > /* Default init_ops implementation. */ > struct init_ops init_ops =3D { > .parse_preload_data =3D native_parse_preload_data, > .early_delay_init =3D i8254_init, > .early_delay =3D i8254_delay, > + .parse_memmap =3D native_parse_memmap, > }; > =20 > /* > @@ -1418,21 +1422,12 @@ add_physmap_entry(uint64_t base, uint64_t length,= vm_paddr_t *physmap, > return (1); > } > =20 > -static void > -add_smap_entries(struct bios_smap *smapbase, vm_paddr_t *physmap, > - int *physmap_idx) > +void > +bios_add_smap_entries(struct bios_smap *smapbase, u_int32_t smapsize, > + vm_paddr_t *physmap, int *physmap_idx) > { > struct bios_smap *smap, *smapend; > - u_int32_t smapsize; > =20 > - /* > - * Memory map from INT 15:E820. > - * > - * subr_module.c says: > - * "Consumer may safely assume that size value precedes data." > - * ie: an int32_t immediately precedes smap. > - */ > - smapsize =3D *((u_int32_t *)smapbase - 1); > smapend =3D (struct bios_smap *)((uintptr_t)smapbase + smapsize); > =20 > for (smap =3D smapbase; smap < smapend; smap++) { > @@ -1449,6 +1444,29 @@ add_smap_entries(struct bios_smap *smapbase, vm_pa= ddr_t *physmap, > } > } > =20 > +static void > +native_parse_memmap(caddr_t kmdp, vm_paddr_t *physmap, int *physmap_idx) > +{ > + struct bios_smap *smap; > + u_int32_t size; > + > + /* > + * Memory map from INT 15:E820. > + * > + * subr_module.c says: > + * "Consumer may safely assume that size value precedes data." > + * ie: an int32_t immediately precedes smap. > + */ > + > + smap =3D (struct bios_smap *)preload_search_info(kmdp, > + MODINFO_METADATA | MODINFOMD_SMAP); > + if (smap =3D=3D NULL) > + panic("No BIOS smap info from loader!"); > + size =3D *((u_int32_t *)smap - 1); > + > + bios_add_smap_entries(smap, size, physmap, physmap_idx); > +} > + > /* > * Populate the (physmap) array with base/bound pairs describing the > * available physical memory in the system, then test this memory and > @@ -1466,19 +1484,13 @@ getmemsize(caddr_t kmdp, u_int64_t first) > vm_paddr_t pa, physmap[PHYSMAP_SIZE]; > u_long physmem_start, physmem_tunable, memtest; > pt_entry_t *pte; > - struct bios_smap *smapbase; > quad_t dcons_addr, dcons_size; > =20 > bzero(physmap, sizeof(physmap)); > basemem =3D 0; > physmap_idx =3D 0; > =20 > - smapbase =3D (struct bios_smap *)preload_search_info(kmdp, > - MODINFO_METADATA | MODINFOMD_SMAP); > - if (smapbase =3D=3D NULL) > - panic("No BIOS smap info from loader!"); > - > - add_smap_entries(smapbase, physmap, &physmap_idx); > + init_ops.parse_memmap(kmdp, physmap, &physmap_idx); > =20 > /* > * Find the 'base memory' segment for SMP > diff --git a/sys/amd64/include/pc/bios.h b/sys/amd64/include/pc/bios.h > index e7d568e..92d4265 100644 > --- a/sys/amd64/include/pc/bios.h > +++ b/sys/amd64/include/pc/bios.h > @@ -106,6 +106,8 @@ struct bios_oem { > int bios_oem_strings(struct bios_oem *oem, u_char *buffer, size_t maxlen= ); > uint32_t bios_sigsearch(uint32_t start, u_char *sig, int siglen, int par= alen, > int sigofs); > +void bios_add_smap_entries(struct bios_smap *smapbase, u_int32_t smapsiz= e, > + vm_paddr_t *physmap, int *physmap_idx); This uses the GNU formatting conventions, please indent the continuation line same as the previous line. --CT8mOM7I+A1mxHvt Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJStZ3aAAoJEJDCuSvBvK1BgfQP/jpj4Oh1SEQ7UlPkAowY+3+s 3cPDxrepWMtJrYuYWpPJTtNYbn8VmMlnOo97YTdEUQZ1idflGJufZco6kgSRXvxt XTH5CaZuQ2FEgMP6iU+pGGEMknZj4BXa5Tfiv6b9eYB99f87Mvp2lC5E8mrDBqQx 7hYC1fz3B0LGG0Z/kGGDFijjWFo/sVZUDLvUgP5nV6tYYfWrkf/8PTazxDtw+Jaw YhDXo9qSDmNbrKQ0AepECdPsdTX2dSd7AK85Px+fpwuclgza9B/ZDp1256rxOg+6 MdASfC51pWzr1FYEevUi0xujGPjKo17aIMnaxVluVTmfY7drSgEzq0ye2u9ogTHf ISXtni7DQisePagC20VlQQv26G1tGVLqsPIjWG/UoI033YCqW7LwlmUnuHjoaC2c 3e5aM5Nht+3fJ1WjMW+2EH9c99ecep+WnLxpFXt9jdcrjtERBmr4fhXnuwtI6P8r 4fBTFOVF2Bvsbx/VyXqigVVNPnrK6u2PcWIRnwPbKAJTZFpAVWC7wQPQLYI9Ca7O kI2/WJQFVV2LOuHLR0EwpEZaDncMccRyc8pyUzVlaf6Blogy+F3iWk/lDYo0kPpi OZFmtsz0rvWTDrqK6iT4a7RoMHUB5UuZHnoJtuT96aedTpkFYmPn6WMWFyXfPRiD 10vwIYGTVQvvVJPyapod =GSIk -----END PGP SIGNATURE----- --CT8mOM7I+A1mxHvt-- From owner-freebsd-xen@FreeBSD.ORG Sat Dec 21 14:00:05 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 583FC7FB; Sat, 21 Dec 2013 14:00:05 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C18AE171A; Sat, 21 Dec 2013 14:00:04 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id rBLDwfo9027305; Sat, 21 Dec 2013 15:58:41 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua rBLDwfo9027305 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id rBLDwfcL027304; Sat, 21 Dec 2013 15:58:41 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 21 Dec 2013 15:58:41 +0200 From: Konstantin Belousov To: Roger Pau Monne , royger@freebsd.org Subject: Re: [PATCH v7 10/19] xen: add hook for AP bootstrap memory reservation Message-ID: <20131221135841.GX59496@kib.kiev.ua> References: <1387479296-33389-1-git-send-email-roger.pau@citrix.com> <1387479296-33389-11-git-send-email-roger.pau@citrix.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="n3hFPmSitNO6nIHR" Content-Disposition: inline In-Reply-To: <1387479296-33389-11-git-send-email-roger.pau@citrix.com> User-Agent: Mutt/1.5.22 (2013-10-16) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: jhb@freebsd.org, xen-devel@lists.xen.org, julien.grall@citrix.com, freebsd-xen@freebsd.org, freebsd-current@freebsd.org, kib@freebsd.org, gibbs@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Dec 2013 14:00:05 -0000 --n3hFPmSitNO6nIHR Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Dec 19, 2013 at 07:54:47PM +0100, Roger Pau Monne wrote: > This hook will only be implemented for bare metal, Xen doesn't require > any bootstrap code since APs are started in long mode with paging > enabled. > --- > sys/amd64/amd64/machdep.c | 6 +++++- > sys/amd64/include/sysarch.h | 3 +++ > 2 files changed, 8 insertions(+), 1 deletions(-) >=20 > diff --git a/sys/amd64/amd64/machdep.c b/sys/amd64/amd64/machdep.c > index 6bbfe5a..a811a9b 100644 > --- a/sys/amd64/amd64/machdep.c > +++ b/sys/amd64/amd64/machdep.c > @@ -186,6 +186,9 @@ struct init_ops init_ops =3D { > .early_delay_init =3D i8254_init, > .early_delay =3D i8254_delay, > .parse_memmap =3D native_parse_memmap, > +#ifdef SMP > + .mp_bootaddress =3D mp_bootaddress, > +#endif > }; > =20 > /* > @@ -1507,7 +1510,8 @@ getmemsize(caddr_t kmdp, u_int64_t first) > =20 > #ifdef SMP > /* make hole for AP bootstrap code */ > - physmap[1] =3D mp_bootaddress(physmap[1] / 1024); > + if (init_ops.mp_bootaddress) > + physmap[1] =3D init_ops.mp_bootaddress(physmap[1] / 1024); > #endif > =20 > /* > diff --git a/sys/amd64/include/sysarch.h b/sys/amd64/include/sysarch.h > index 084223e..77f4b29 100644 > --- a/sys/amd64/include/sysarch.h > +++ b/sys/amd64/include/sysarch.h > @@ -16,6 +16,9 @@ struct init_ops { > void (*early_delay_init)(void); > void (*early_delay)(int); > void (*parse_memmap)(caddr_t, vm_paddr_t *, int *); > +#ifdef SMP > + u_int (*mp_bootaddress)(u_int); > +#endif > }; > =20 > extern struct init_ops init_ops; I suggest to make the init_ops contain the mp_bootaddress unconditionally, instead of making it depended on SMP. The #ifdef makes the kernel binary interface fragile and depended on the config, which would cause problems in future if any module needs any interation with init_ops. --n3hFPmSitNO6nIHR Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJStZ6QAAoJEJDCuSvBvK1B1lIQAJZBaZKW30oWuiwH0kswmBgE uBi/aWXUO9ENY9Cw1xjmqWLL2s1Xw3joA4kTJth6I45SfYcifUihKZlpYdr+WEf/ Gut/qF7zZyPLQrfEtrpAr1HgFopgQyBw+7zQvMCHaJAKeAP0AKXE6tsdzeNvSan2 zQ7WrgmH054Fraqnq4w6ebJD0uIeOZdmGA4Cifv8XVSXZVhr9XpuPrlIEe66x5Al 2YBlqaHFpZwTbh3SCCfaFpgv0p55VxW3VtX+u+2gb9m2DFbPF67U9CLXKKyupkjc zbTE8kDU+X+eS005wcC0LcVMBFLXJ+kAdP9pMMsMukkmUFCDpWvjvYn3g+9TXN9E lQbE72H7FJs7InHtgXeKADOMyqMsJUg+lQ5hGyR9MShaahrr6E/LMiwncYQLluZl S/6JoL3uqI5BjuZYUWhzIFbckccvRnqPowLmJbIjY1/7qsFTdM/tny9CjuXGGcIW bEhQ46Pi7GIZG/ATGDd5b/BLJvXDdrNsXA06Ro7bdoc1YInu/hrpz9DN+Oyg37ld edANo0xiwEGpPYl2rl87y4ftNfev2X9srEd9MSSCpqWmeMeFJoVQwyk8/R4a9rvL AVGnC0WX6oXcYFpOsNqYeyHntgTmgB7XxwdMCW8odJdr81fOE+WsdAACKn+8K4LD gGEmtbZlRqTIiPqOGgBi =qPyT -----END PGP SIGNATURE----- --n3hFPmSitNO6nIHR--