From nobody Tue Sep 30 15:23:50 2025 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cbhfb1Frvz69DxK; Tue, 30 Sep 2025 15:23:51 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cbhfZ5QG0z3M3j; Tue, 30 Sep 2025 15:23:50 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1759245830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mUuk8gOAzEYxBdp05PW/bQ2FZqDX9ZVylTHO38wtxY8=; b=hKIfU5jdkv7CacopIxIWAEg0I9F31doDsRFeDGOasXsQROenZme1EHby2SHF1OQV3L1LRK JyU/nVAoL+TxMjhLa5wtgT90luuWCcq0acpEk+FqPeRd1wj/pZpUQ8js/D2N4sPtmenOfP w9/HFJM8kTYb8Q4oHXS0VpMK0kYtC5SUzm8Jrq5DpaqwYSdqbvmtSt+EH4iFSIIiHiz+Ke Vp+NPyQH+PVF/pARSjFlqwuELcyT9ITumqQVKWStVJzKgkyK7ILkEvPQVQz7SvmsCWP8ac iZHoshThZMyyehG7lVRxxEeJUK1Y23EUrmxMiju6IPB5Zxj9OfeZu1xxJ9QAnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1759245830; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=mUuk8gOAzEYxBdp05PW/bQ2FZqDX9ZVylTHO38wtxY8=; b=kpBBQ2srYZnFSCCru/S2G5vuFFb2A8/iOAD4dKD85c0iEjkW2rXOYcI7zjUYMf389ws/5A h86vzdQzBTG18fr9+oCMLdLr3/IBY7XDdUlO5EEkHlBNnYpHwpFbX0MJLgX+zK+weL2YeO DpigeSffti7q9i3Yl0dG/wfa9EztoA2f5Zy1o50etHvjDoPzmDKJDBnLEJ5eZzJ+hGK7Y+ mrajYTQCHt7d5Mc1zg6gV5bII3DSkZ6V1LD4nkIWONPU0Tw4sbGvyWR0v6mVuDtaffXpkg vXnG3XjUblqK5rHOrmvesyZ1X9CDeQJ8ar8NWLxrid3haNG3yodGSkG3YMbSQA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1759245830; a=rsa-sha256; cv=none; b=gdUYre1OpYQmxHunB4qUPswojMxHZAJN/vVf9biB2mU+F7IbYLTTHDPRR+9oFkHjtIAlLB VwblrzqHWOf5uQo5vZwYDZywMySyn2CA3YI7G/ybGd1/+zgsg3gi91grrryM3YlMjqqxhZ W7j/HcRUJSCUlTiS+yNxHW93eMZAEAKpyCAroK2UBeECFCr3wzhHltfuynTiotx747Jd08 JZ+ya1rXJLEXmu1LfMyeNR4SWg/1BFoJ8Ak3EuiimyBN2JldgBtdT4ViDq8gkru9sTHZom KKZTvBuOnwkPXdcDoJh+TXp+ywQAqYHH+jef75ibaAVTlzABAwKMYBexdQFZKg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cbhfZ4ylGzj07; Tue, 30 Sep 2025 15:23:50 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 58UFNoh7046083; Tue, 30 Sep 2025 15:23:50 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 58UFNom4046078; Tue, 30 Sep 2025 15:23:50 GMT (envelope-from git) Date: Tue, 30 Sep 2025 15:23:50 GMT Message-Id: <202509301523.58UFNom4046078@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: f7bce8e1826c - stable/15 - vmm: Suspend the VM before destroying it List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: f7bce8e1826c362d848d9fb2caffbc80da2c73a3 Auto-Submitted: auto-generated The branch stable/15 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=f7bce8e1826c362d848d9fb2caffbc80da2c73a3 commit f7bce8e1826c362d848d9fb2caffbc80da2c73a3 Author: Mark Johnston AuthorDate: 2025-09-10 16:00:36 +0000 Commit: Mark Johnston CommitDate: 2025-09-30 09:43:08 +0000 vmm: Suspend the VM before destroying it Otherwise we don't do anything to kick vcpu threads out of a sleep state when destroying a VM. For instance, suppose a guest executes hlt on amd64 or wfi on arm64 with interrupts disabled. Then, bhyvectl --destroy will hang until the vcpu thread somehow comes out of vm_handle_hlt()/vm_handle_wfi() since destroy_dev() is waiting for vCPU threads to drain. Note that on amd64, if hw.vmm.halt_detection is set to 1 (the default), the guest will automatically exit in this case since it's treated as a shutdown. But, the above should not hang if halt_detection is set to 0. Here, vm_suspend() wakes up vcpu threads, and a subsequent attempt to run the vCPU will result in an error which gets propagated to userspace, allowing destroy_dev() to proceed. Add a new suspend code for this purpose. Modify bhyve to exit with status 4 ("exited due to an error") when it's received, since that's what'll happen generally when the VM is destroyed asynchronously. Reported by: def MFC after: 2 weeks Sponsored by: Innovate UK Differential Revision: https://reviews.freebsd.org/D51761 (cherry picked from commit 3d39856d4dfeab5b5a5e6bbdb6ce965db5bc4dc1) --- sys/amd64/include/vmm.h | 1 + sys/arm64/include/vmm.h | 1 + sys/arm64/vmm/vmm.c | 6 ++++++ sys/dev/vmm/vmm_dev.c | 1 + sys/riscv/include/vmm.h | 1 + sys/riscv/vmm/vmm.c | 6 +++++- usr.sbin/bhyve/aarch64/vmexit.c | 2 ++ usr.sbin/bhyve/amd64/vmexit.c | 2 ++ usr.sbin/bhyve/bhyve.8 | 2 +- usr.sbin/bhyve/bhyverun.c | 6 ++---- usr.sbin/bhyve/riscv/vmexit.c | 2 ++ 11 files changed, 24 insertions(+), 6 deletions(-) diff --git a/sys/amd64/include/vmm.h b/sys/amd64/include/vmm.h index 0b3daed4f69e..e35119af8572 100644 --- a/sys/amd64/include/vmm.h +++ b/sys/amd64/include/vmm.h @@ -46,6 +46,7 @@ enum vm_suspend_how { VM_SUSPEND_POWEROFF, VM_SUSPEND_HALT, VM_SUSPEND_TRIPLEFAULT, + VM_SUSPEND_DESTROY, VM_SUSPEND_LAST }; diff --git a/sys/arm64/include/vmm.h b/sys/arm64/include/vmm.h index 73b5b4a09591..e839b5dd92c9 100644 --- a/sys/arm64/include/vmm.h +++ b/sys/arm64/include/vmm.h @@ -42,6 +42,7 @@ enum vm_suspend_how { VM_SUSPEND_RESET, VM_SUSPEND_POWEROFF, VM_SUSPEND_HALT, + VM_SUSPEND_DESTROY, VM_SUSPEND_LAST }; diff --git a/sys/arm64/vmm/vmm.c b/sys/arm64/vmm/vmm.c index 3082d2941221..1dcefa1489e9 100644 --- a/sys/arm64/vmm/vmm.c +++ b/sys/arm64/vmm/vmm.c @@ -1342,8 +1342,14 @@ vm_handle_smccc_call(struct vcpu *vcpu, struct vm_exit *vme, bool *retu) static int vm_handle_wfi(struct vcpu *vcpu, struct vm_exit *vme, bool *retu) { + struct vm *vm; + + vm = vcpu->vm; vcpu_lock(vcpu); while (1) { + if (vm->suspend) + break; + if (vgic_has_pending_irq(vcpu->cookie)) break; diff --git a/sys/dev/vmm/vmm_dev.c b/sys/dev/vmm/vmm_dev.c index 9f2b009d02ec..460a508a60dc 100644 --- a/sys/dev/vmm/vmm_dev.c +++ b/sys/dev/vmm/vmm_dev.c @@ -901,6 +901,7 @@ vmmdev_lookup_and_destroy(const char *name, struct ucred *cred) sc->cdev = NULL; sx_xunlock(&vmmdev_mtx); + vm_suspend(sc->vm, VM_SUSPEND_DESTROY); destroy_dev(cdev); vmmdev_destroy(sc); diff --git a/sys/riscv/include/vmm.h b/sys/riscv/include/vmm.h index 1221521be368..de7119dd534a 100644 --- a/sys/riscv/include/vmm.h +++ b/sys/riscv/include/vmm.h @@ -49,6 +49,7 @@ enum vm_suspend_how { VM_SUSPEND_RESET, VM_SUSPEND_POWEROFF, VM_SUSPEND_HALT, + VM_SUSPEND_DESTROY, VM_SUSPEND_LAST }; diff --git a/sys/riscv/vmm/vmm.c b/sys/riscv/vmm/vmm.c index 7528ef6e4698..ec4514f70fa6 100644 --- a/sys/riscv/vmm/vmm.c +++ b/sys/riscv/vmm/vmm.c @@ -1036,10 +1036,14 @@ vm_raise_msi(struct vm *vm, uint64_t msg, uint64_t addr, int bus, int slot, static int vm_handle_wfi(struct vcpu *vcpu, struct vm_exit *vme, bool *retu) { + struct vm *vm; + vm = vcpu->vm; vcpu_lock(vcpu); - while (1) { + if (vm->suspend) + break; + if (aplic_check_pending(vcpu->cookie)) break; diff --git a/usr.sbin/bhyve/aarch64/vmexit.c b/usr.sbin/bhyve/aarch64/vmexit.c index 3acad4020a3c..2457cbe76b5e 100644 --- a/usr.sbin/bhyve/aarch64/vmexit.c +++ b/usr.sbin/bhyve/aarch64/vmexit.c @@ -122,6 +122,8 @@ vmexit_suspend(struct vmctx *ctx, struct vcpu *vcpu, struct vm_run *vmrun) exit(1); case VM_SUSPEND_HALT: exit(2); + case VM_SUSPEND_DESTROY: + exit(4); default: fprintf(stderr, "vmexit_suspend: invalid reason %d\n", how); exit(100); diff --git a/usr.sbin/bhyve/amd64/vmexit.c b/usr.sbin/bhyve/amd64/vmexit.c index 944f5de34645..14f89563fd0f 100644 --- a/usr.sbin/bhyve/amd64/vmexit.c +++ b/usr.sbin/bhyve/amd64/vmexit.c @@ -418,6 +418,8 @@ vmexit_suspend(struct vmctx *ctx, struct vcpu *vcpu, struct vm_run *vmrun) exit(2); case VM_SUSPEND_TRIPLEFAULT: exit(3); + case VM_SUSPEND_DESTROY: + exit(4); default: EPRINTLN("vmexit_suspend: invalid reason %d", how); exit(100); diff --git a/usr.sbin/bhyve/bhyve.8 b/usr.sbin/bhyve/bhyve.8 index 89c0b23961a8..c902c265da9e 100644 --- a/usr.sbin/bhyve/bhyve.8 +++ b/usr.sbin/bhyve/bhyve.8 @@ -1126,7 +1126,7 @@ powered off .It 2 halted .It 3 -triple fault +triple fault (amd64 only) .It 4 exited due to an error .El diff --git a/usr.sbin/bhyve/bhyverun.c b/usr.sbin/bhyve/bhyverun.c index 9ead49582a7d..bfc0b949a75d 100644 --- a/usr.sbin/bhyve/bhyverun.c +++ b/usr.sbin/bhyve/bhyverun.c @@ -561,10 +561,8 @@ fbsdrun_start_thread(void *param) #endif vm_loop(vi->ctx, vi->vcpu); - - /* not reached */ - exit(1); - return (NULL); + /* We get here if the VM was destroyed asynchronously. */ + exit(4); } void diff --git a/usr.sbin/bhyve/riscv/vmexit.c b/usr.sbin/bhyve/riscv/vmexit.c index 3bc83b3bef4e..985f8e4e9065 100644 --- a/usr.sbin/bhyve/riscv/vmexit.c +++ b/usr.sbin/bhyve/riscv/vmexit.c @@ -121,6 +121,8 @@ vmexit_suspend(struct vmctx *ctx, struct vcpu *vcpu, struct vm_run *vmrun) exit(1); case VM_SUSPEND_HALT: exit(2); + case VM_SUSPEND_DESTROY: + exit(4); default: fprintf(stderr, "vmexit_suspend: invalid reason %d\n", how); exit(100);