From nobody Thu Nov 13 11:20:22 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d6d9Q2g7Vz6GhqY for ; Thu, 13 Nov 2025 11:20:26 +0000 (UTC) (envelope-from bojan.novkovic@kset.org) Received: from mail-ed1-f46.google.com (mail-ed1-f46.google.com [209.85.208.46]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4d6d9P0xnMz3l8n for ; Thu, 13 Nov 2025 11:20:25 +0000 (UTC) (envelope-from bojan.novkovic@kset.org) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of bojan.novkovic@kset.org designates 209.85.208.46 as permitted sender) smtp.mailfrom=bojan.novkovic@kset.org Received: by mail-ed1-f46.google.com with SMTP id 4fb4d7f45d1cf-64074f01a6eso1212875a12.2 for ; Thu, 13 Nov 2025 03:20:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763032823; x=1763637623; h=content-transfer-encoding:autocrypt:subject:from:cc:to :content-language:user-agent:mime-version:date:message-id:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cR+tpdhjdHP53nnfoIJpuzwMtrOUrrFZUffaGKMlOVA=; b=Pt6eSj59eo0l12/XnORLQXUy7k9rucSQ06vQI9hVQBk15mIBbGS2sm5Uul66sLbtNu RUNE79p+2fPSHJnHzHVrrNvW0jh28ExCWP4t3awS5MPxD8yPRFXwA0jTOLXtMLmRJfKc hj3biGdORQPq8HAPMC2T5haiFOacMMgCh+eiXd9pX/wVJ+G2uDY/j80AaWCNHQtC7WTC 0AVSr7nztKK/Zk5K1fPjzIIDFYjuLMsL5L40e4K8ujxsHoFwsqAqJIqKAdfk+6KPFAVv diK36r2r3qX4HEyvQNPXE0ntBOkQ8D/d7WxBgE3ckWhPxsu01iQIRyARB5Ge7j+paTX4 YMcQ== X-Gm-Message-State: AOJu0YzTrJRE91jjF815PLpFiAP2P05NCNlbWE2+cK0Ykd4g2XNK4rP5 Cr7Lj/tFRayifZ5LNoLtytP2bh1/frEbUd9Rs7er+CJdVoFCY7shOaQtgP5ar1i9l8Wi0TvkZCh gJvPr X-Gm-Gg: ASbGncsXpiquVaO1cF1YycL6UhzCBqzkSp3ptAVUUvuMBgmzb+agbGVdFPqMJA7pF7K SFa6hGI8bTQT78akD1esZPZqa/S6b9zBGOM7RRETWbMKuCiLpwtpPlNgE4hLplFXb4fT/FNPz5O jvqehGcAbl+AM4pJvjFST7fe/ril++IKo8hUOCIZyE/Jyp4zmbjHLPq71+xHJpji4XG2rQaJ6H0 NymFHRMVolmOi+b4vkB1TXSQLA0nI0m/q4zY+kz1xwAxWHkfs7rRg2JBftGmtqSmbVXElnGSpIG 44LxNv+2W3N+0ICKyiHrMMgpy9OLBqxm6n+8dwH1lyKZ0luL3eA1IL2RRBi8aHHf/7GKZd98O35 7XZ2ycCtSFRaEYRGcqsw4PsNxf1qjuKdXWRYW9MjSIbf7T+UtWM46NxaOfhVyZ8SNIguiI0fOHv xF7pLDBCoSk0FDPFSxwwpwlILwbvmx84Y/2aHt X-Google-Smtp-Source: AGHT+IFYe5THFtjbuxiJs2te/8gKlK+CKPHXzYjvNi+Rpo2pTIYT5plK0tUA4p84mtl6dO0rAaY6Ow== X-Received: by 2002:a05:6402:254c:b0:641:9aac:e4bd with SMTP id 4fb4d7f45d1cf-6431a5756b6mr6149404a12.26.1763032823294; Thu, 13 Nov 2025 03:20:23 -0800 (PST) Received: from [192.168.0.91] (cpe-188-252-141-250.zg5.cable.xnet.hr. [188.252.141.250]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-6433a497ffdsm1240393a12.21.2025.11.13.03.20.22 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 13 Nov 2025 03:20:22 -0800 (PST) Message-ID: <9d28570c-0700-4d24-8977-6f5eb68c23eb@freebsd.org> Date: Thu, 13 Nov 2025 12:20:22 +0100 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: freebsd-virtualization@freebsd.org Cc: Mark Johnston From: =?UTF-8?Q?Bojan_Novkovi=C4=87?= Subject: RFC: transient virtual machines and "monitor mode" Autocrypt: addr=bnovkov@freebsd.org; keydata= xjMEZdIQsxYJKwYBBAHaRw8BAQdA/V2uVmdN7VY2Ps8wDgLlWU3b9+kPdg9bf+FHgEEX49TN JUJvamFuIE5vdmtvdmnEhyA8Ym5vdmtvdkBGcmVlQlNELm9yZz7CmQQTFgoAQRYhBLAb6L2d hfD6hKflVB43npi7IZ8rBQJl0hCzAhsDBQkFo5qABQsJCAcCAiICBhUKCQgLAgQWAgMBAh4H AheAAAoJEB43npi7IZ8rzb0A/0aY3c/XubbtQzNyA0xzyKNZlDc9zesxEMVi6rOAZNz/AQC2 QmBTBEcbyOKDfJ5m02LpudVi9thZxlrr2n0ZX9kgA844BGXSELMSCisGAQQBl1UBBQEBB0Dn 3m+8g7KTp3yC4CLICis/CIonFfNqQcJOVv6Gd73adQMBCAfCfgQYFgoAJhYhBLAb6L2dhfD6 hKflVB43npi7IZ8rBQJl0hCzAhsMBQkFo5qAAAoJEB43npi7IZ8ruPcBAJM5wq5j64RFu4sc zrryK4FeCTt/Xhfyn3UhT2hHuYkPAQDWHDN6XZ097C5wUkWUr8ywHDlMM5gWIDbr9TMUudoc Aw== Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: + X-Spamd-Result: default: False [1.41 / 15.00]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; NEURAL_SPAM_LONG(1.00)[0.996]; NEURAL_HAM_SHORT(-0.68)[-0.682]; FORGED_SENDER(0.30)[bnovkov@freebsd.org,bojan.novkovic@kset.org]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.208.46:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[bnovkov@freebsd.org,bojan.novkovic@kset.org]; RCVD_IN_DNSWL_NONE(0.00)[209.85.208.46:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[] X-Rspamd-Queue-Id: 4d6d9P0xnMz3l8n Hi, As part of the work towards making unprivileged bhyve viable I made some changes to the VM lifecycle, so I'd like to get some feedback on the new VM lifecycle scheme. The first big change was to add mechanisms to automatically reclaim a virtual machine's resources when the owning process goes away, especially when it's killed by the OOM killer. To address this we decided to tie a virtual machine's lifecycle to a file descriptor. vmmctl's VMMCTL_VM_CREATE ioctl was extended with a new flag (VMMCTL_CREATE_DESTROY_ON_CLOSE) that instructs it to destroy all virtual machines associated with a specific vmmctl file descriptor when it is closed. vmmctl uses the 'devfs_xxx_cdevpriv' functions to maintain a per-descriptor list of such virtual machines. Under this regime the virtual machine gets destroyed once the owning process dies, which is not very useful when one just wants to restart a virtual machine. The second big change addresses this by introducing "monitor mode" to bhyve (for lack of a better term), which is enabled by passing the '-M' flag. This will instruct bhyve to create a transient virtual machine using the mechanism described in the above paragraph. At a certain point during startup, it will then fork itself and let the child process continue with the setup and start the VM. The parent process waits for the virtual machine to exit and will automatically restart it if the child exited with a "reboot" status (i.e., 0). Exiting with any other status code or killing the parent process will automatically destroy the VM. Monitor mode also works with non-transient VMs, but it will not destroy them when bhyve exits. Please feel free to leave comments on any of the following reviews:  - https://reviews.freebsd.org/D53729 (vmm: Add ability to destroy VMs on close)  - https://reviews.freebsd.org/D53731 (bhyve: Introduce monitor mode) Thanks, Bojan