From nobody Wed Sep 7 07:08:57 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MMtf16X7vz4bmD4; Wed, 7 Sep 2022 07:08:57 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MMtf15k3fz3hMq; Wed, 7 Sep 2022 07:08:57 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1662534537; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=piiLSyzPcczbfX2KNhnKa7T8KHsQNCRjRSqkPU2lnco=; b=LH8kO3jck7k3bWLodeGdeJTDlWY3dzdKtyrlodlIU1wC/nlT0Ou0fTPjTDXjkmyv66WaFi WCT9pUpPMKLxdy8SkYYlRDZfkGGJ+4kyZW9ShVGa9L0VuJ3yVj95figAz/NygKwi/+7EQz ujrb2nFyGNYrQ2V7H3oCogI6bzwI4AJyHZzFb8dlizbbEIgeQlXBTpZpisVDD9upD60fGL F2Lz3V++QAzjwZWKq/tOs6LjNPR+fFWCXik4Scfd/dd1Xgv504yXFuHFKc2LdCMACd2phN 0hjqYZLORuLjGPUFz2q64lLkOGhyvl+MYuOCTF/JRBKVErYxDASOBiMfkmJTVQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4MMtf14lN0zRfr; Wed, 7 Sep 2022 07:08:57 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 28778v5Z018204; Wed, 7 Sep 2022 07:08:57 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 28778vOG018203; Wed, 7 Sep 2022 07:08:57 GMT (envelope-from git) Date: Wed, 7 Sep 2022 07:08:57 GMT Message-Id: <202209070708.28778vOG018203@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Emmanuel Vadot Subject: git: 9cc9abf409cc - main - bhyve: create all vcpus on startup List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: manu X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 9cc9abf409cc9d761f42b16b0dc8f5c0cc4f9a44 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1662534537; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=piiLSyzPcczbfX2KNhnKa7T8KHsQNCRjRSqkPU2lnco=; b=qDQDoIkVWlNYfgdmKjKEeEygOmzrtKwnTLi3ee8ilWJIuRk2WJKRSY8crj8/wwlI3Thi7m TY5l1SMOXKyd5eWTJoKtJH2oJacaUwI7ODi+nqb+QIc/n87KHq3aC+7kCtEOepON9uG8w3 LbOvEFkRidqoDUAJUL5nOXrrXvT4LAWf8yJAh75oRvkYwE4s79HPwB1uvEdL8azTnGONl/ kvlSdMqrYOmOKBquGin28DnIlow7lEs+Ok3pDVuyI/TApzCqgcS66EOAstpF9fOQMAUk/V crH6+rEaPPXh0DbTjItutMLJ9EBl8UEhnSfuEMgjtszThZ0+1U6TrB+FSzvnBw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1662534537; a=rsa-sha256; cv=none; b=S7vxgG2l5tAcnhi0uSuDwl6Hryk995iqpA2qnSxq5ySfMo2SKZoXcDIrud/HONQC9UWd0N XtMYv/yeEeC1GwUSsPC4sT+VXOt4I4fPako6gt6WhclAjeq6DohZWvVD/PZIpdOFOOa3XW sMmgGZRgu0NXjuoCOxWphQnb0gNj8aHo76UMPHyBMwUWXLjPmM7XReWD86KOsXDfFU3YDX PEVUTYJVAU2q6b3wF7IoX3GmzcAJUzPOw6szkFxR+ZNEG0VZiOtDEBDcD+FdW8z4R78/41 LrSG+rwdxkLvI+Cpd60066iA3nVuZ48iwFJme2oLQBKHvPJTCSMF4bZuRkx1tA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by manu: URL: https://cgit.FreeBSD.org/src/commit/?id=9cc9abf409cc9d761f42b16b0dc8f5c0cc4f9a44 commit 9cc9abf409cc9d761f42b16b0dc8f5c0cc4f9a44 Author: Corvin Köhne AuthorDate: 2022-09-07 07:05:36 +0000 Commit: Emmanuel Vadot CommitDate: 2022-09-07 07:05:36 +0000 bhyve: create all vcpus on startup vcpus could be restarted by the guest by sending an INIT SIPI SIPI sequence to a vcpu. That's not supported by bhyve yet but it will be supported in a future commit. So, create the vcpu threads only once on startup to make restarting a vcpu easier. MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D35621 Sponsored by: Beckhoff Automation GmbH & Co. KG --- usr.sbin/bhyve/bhyverun.c | 41 ++++++++++++++++------------------------- usr.sbin/bhyve/bhyverun.h | 1 - usr.sbin/bhyve/spinup_ap.c | 4 ++-- usr.sbin/bhyve/spinup_ap.h | 2 +- 4 files changed, 19 insertions(+), 29 deletions(-) diff --git a/usr.sbin/bhyve/bhyverun.c b/usr.sbin/bhyve/bhyverun.c index 4f9fc4a2b050..550cc9d15477 100644 --- a/usr.sbin/bhyve/bhyverun.c +++ b/usr.sbin/bhyve/bhyverun.c @@ -546,13 +546,11 @@ fbsdrun_start_thread(void *param) return (NULL); } -void -fbsdrun_addcpu(struct vmctx *ctx, int fromcpu, int newcpu, uint64_t rip) +static void +fbsdrun_addcpu(struct vmctx *ctx, int newcpu, uint64_t rip, bool suspend) { int error; - assert(fromcpu == BSP); - /* * The 'newcpu' must be activated in the context of 'fromcpu'. If * vm_activate_cpu() is delayed until newcpu's pthread starts running @@ -565,6 +563,9 @@ fbsdrun_addcpu(struct vmctx *ctx, int fromcpu, int newcpu, uint64_t rip) CPU_SET_ATOMIC(newcpu, &cpumask); + if (suspend) + vm_suspend_cpu(ctx, newcpu); + /* * Set up the vmexit struct to allow execution to start * at the given RIP @@ -687,8 +688,7 @@ static int vmexit_spinup_ap(struct vmctx *ctx, struct vm_exit *vme, int *pvcpu) { - (void)spinup_ap(ctx, *pvcpu, - vme->u.spinup_ap.vcpu, vme->u.spinup_ap.rip); + (void)spinup_ap(ctx, vme->u.spinup_ap.vcpu, vme->u.spinup_ap.rip); return (VMEXIT_CONTINUE); } @@ -1138,8 +1138,8 @@ do_open(const char *vmname) return (ctx); } -void -spinup_vcpu(struct vmctx *ctx, int vcpu) +static void +spinup_vcpu(struct vmctx *ctx, int vcpu, bool suspend) { int error; uint64_t rip; @@ -1151,7 +1151,7 @@ spinup_vcpu(struct vmctx *ctx, int vcpu) error = vm_set_capability(ctx, vcpu, VM_CAP_UNRESTRICTED_GUEST, 1); assert(error == 0); - fbsdrun_addcpu(ctx, BSP, vcpu, rip); + fbsdrun_addcpu(ctx, vcpu, rip, suspend); } static bool @@ -1584,25 +1584,16 @@ main(int argc, char *argv[]) mt_vmm_info = calloc(guest_ncpus, sizeof(*mt_vmm_info)); /* - * Add CPU 0 + * Add all vCPUs. */ - fbsdrun_addcpu(ctx, BSP, BSP, rip); - + for (int vcpu = 0; vcpu < guest_ncpus; vcpu++) { + bool suspend = (vcpu != BSP); #ifdef BHYVE_SNAPSHOT - /* - * If we restore a VM, start all vCPUs now (including APs), otherwise, - * let the guest OS to spin them up later via vmexits. - */ - if (restore_file != NULL) { - for (vcpu = 0; vcpu < guest_ncpus; vcpu++) { - if (vcpu == BSP) - continue; - - fprintf(stdout, "spinning up vcpu no %d...\r\n", vcpu); - spinup_vcpu(ctx, vcpu); - } - } + if (restore_file != NULL) + suspend = false; #endif + spinup_vcpu(ctx, vcpu, suspend); + } /* * Head off to the main event dispatch loop diff --git a/usr.sbin/bhyve/bhyverun.h b/usr.sbin/bhyve/bhyverun.h index 9b0c9fb533e2..3ca252a8ffc2 100644 --- a/usr.sbin/bhyve/bhyverun.h +++ b/usr.sbin/bhyve/bhyverun.h @@ -44,7 +44,6 @@ uintptr_t paddr_host2guest(struct vmctx *ctx, void *addr); #endif void fbsdrun_set_capabilities(struct vmctx *ctx, int cpu); -void fbsdrun_addcpu(struct vmctx *ctx, int fromcpu, int newcpu, uint64_t rip); int fbsdrun_virtio_msix(void); #endif diff --git a/usr.sbin/bhyve/spinup_ap.c b/usr.sbin/bhyve/spinup_ap.c index 7c4186f5ed15..2b7e602f8003 100644 --- a/usr.sbin/bhyve/spinup_ap.c +++ b/usr.sbin/bhyve/spinup_ap.c @@ -77,7 +77,7 @@ spinup_ap_realmode(struct vmctx *ctx, int newcpu, uint64_t *rip) } int -spinup_ap(struct vmctx *ctx, int vcpu, int newcpu, uint64_t rip) +spinup_ap(struct vmctx *ctx, int newcpu, uint64_t rip) { int error; @@ -100,7 +100,7 @@ spinup_ap(struct vmctx *ctx, int vcpu, int newcpu, uint64_t rip) spinup_ap_realmode(ctx, newcpu, &rip); - fbsdrun_addcpu(ctx, vcpu, newcpu, rip); + vm_resume_cpu(ctx, newcpu); return (newcpu); } diff --git a/usr.sbin/bhyve/spinup_ap.h b/usr.sbin/bhyve/spinup_ap.h index 226542f6c331..e67bac42944b 100644 --- a/usr.sbin/bhyve/spinup_ap.h +++ b/usr.sbin/bhyve/spinup_ap.h @@ -31,6 +31,6 @@ #ifndef _SPINUP_AP_H_ #define _SPINUP_AP_H_ -int spinup_ap(struct vmctx *ctx, int vcpu, int newcpu, uint64_t rip); +int spinup_ap(struct vmctx *ctx, int newcpu, uint64_t rip); #endif