From nobody Sat Nov 29 14:20:05 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4dJXS31XyWz6Hxjp for ; Sat, 29 Nov 2025 14:22:27 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4dJXS15dNjz3ZRs for ; Sat, 29 Nov 2025 14:22:25 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=NafyjqL2; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of bogorodskiy@gmail.com designates 2a00:1450:4864:20::534 as permitted sender) smtp.mailfrom=bogorodskiy@gmail.com Received: by mail-ed1-x534.google.com with SMTP id 4fb4d7f45d1cf-640a3317b89so4190697a12.0 for ; Sat, 29 Nov 2025 06:22:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1764426138; x=1765030938; darn=freebsd.org; h=content-disposition:mime-version:message-id:subject:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=VKvJ3cbNYLxu11j5zmAgs2BVHxKUDu19Sho1HNRZJn8=; b=NafyjqL2HHev42rbaTHRRauBA/GsK+XrkLjDsFt/P+5TvVWTaZBUScOc8O7u4c9uTi vuZ99NcW+i1Dh0hFsS7cEPtt31VwfIQvCFc6y497Fhtvb53sG3qGgYfSdXPMKuNCkKEB Y1hQhmIsmRHzc4LJ5XssDytGrePvadGXToHxw53XZ/XZLCoaG/+FIKqPwscykhBoghdn mc1LDflDC4z7i0bNrBxPmBwahyQjodsgdni+7K07MKtfXzX4ajf5TfUIRW+KA5b0K2M+ ln0HsUKWL3gL5tbSa5aSC0O8GVyQ5xa18x3MaRa+GrGvA6xIwchvqVZenu3b5/QC+t2u k2hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764426138; x=1765030938; h=content-disposition:mime-version:message-id:subject:to:from:date :sender:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=VKvJ3cbNYLxu11j5zmAgs2BVHxKUDu19Sho1HNRZJn8=; b=bnf7mF1arjRK3hM3fhNf2PGkxy6Xix2kucn1lNzT0DvDZCa7As+YBUFBxSVlzqGEPc 2N5S+5W3TnN4bcgxIH+jDEmYQ+l5ySXqNz5aBChihTr45btlzU8Tk8I6oxCEXWBtZtZ4 HPMK7YnYousl/UfUzY+jll2QrFhLc3UWWGm+S6BimNR2seahKrS8PWDg67hGKGFf54Tg 5kZLDxH941w3gK3FwsyUdeEE6T4N4BiIUcC0z9E7DKRoREzJvdk+b+i10517kULqraw5 VuL8zmS+WCkht7EDFsaow+PvfsUbyx3BYkZ070IA1KHDlOulA1CXcQg4Y50f2GJQazw1 +SfA== X-Gm-Message-State: AOJu0Yz26Vla751FZ3DFdn4CgmM/vAJIfZngkjjyu1VXJNeDG4t7REVc vv+uJxocALT619NPyMSF9n7JUwm5ph01CQy+/5RkzIbtqZoPuQWN1PROWbfKWBzN X-Gm-Gg: ASbGncu9nj7/9DNrY+b3qkzSYw7XALz/ahRdWw9n1vUMPAI8A6a6Tk2vlOul9ezLtrA eA1TO1XvPQWTBLowuNTAwokfvJM3gpKS30pljsy2BEo5ZgBW0JFBqGdatmO9rhRwb1l+8V1USMZ SOJVMiJPuvPJ8maAlD1u6FvZhj7Snz7NTaOsOXC4b44axU48WOC7ZgVz0hjOShpnGRyi9xZnEo0 DEcfpIkt4NmAC24EinX1k8cjBeBkiATcxEcuMnbPx0OtNkppnOC2Ml1xInpYkUEzWX7y6jdXHlW kniLUtf3lcT3DtkW6KlazE1vCJZZRk25TWesfarFbmnba4+djWeNqkXY7IBgRcNnAy/3WwBHhF5 +UKqhwHM4JUyFpyvig/rWsMQZz9kDv3E3SEVxVTgxLaysMWUw+6Z/JuTKnDqyYFWks2qTY9Nkbc pqtIDxI0mwvi18k+wdk7NqDre08Whp/4Q= X-Google-Smtp-Source: AGHT+IHAYGw30fb2yl96CytiSveuiL3q69TQ+Pc6Q8bMYmWho2Yfyk+asUXDC449avUSJm50jxJx1w== X-Received: by 2002:a17:907:d1f:b0:b77:bd1:22c6 with SMTP id a640c23a62f3a-b770bd12995mr637858066b.48.1764426137390; Sat, 29 Nov 2025 06:22:17 -0800 (PST) Received: from tulp (84-25-144-101.cable.dynamic.v4.ziggo.nl. [84.25.144.101]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-b76f5162d26sm718605866b.3.2025.11.29.06.22.16 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 29 Nov 2025 06:22:16 -0800 (PST) Date: Sat, 29 Nov 2025 15:20:05 +0100 From: Roman Bogorodskiy To: virtualization@freebsd.org Subject: bhyve(8) suspend interface Message-ID: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.60 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[novel@freebsd.org,bogorodskiy@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[novel@freebsd.org,bogorodskiy@gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_DOM_EQ_FROM_DOM(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; MISSING_XM_UA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::534:from] X-Rspamd-Queue-Id: 4dJXS15dNjz3ZRs Hi, While experimenting with the bhyve suspend/resume feature I noticed a couple of things which seem a little confusing. The first thing is bhyve(8) exit code after suspend. If I run: bhyve -c 2 -m 4096 ... my_vm And then run: bhyvectl --suspend=/var/run/vms/my_vm --vm my_vm Once suspend completes, the bhyve process exits with retcode 0. According to the bhyve(8) manual page, this exit code corresponds to a VM reboot. It also seems inconvenient from management software/scripting point of view, because reaction to this exit code is to boot a VM again (assuming that if a user initiated a reboot, they expect their VM to boot again soon). My impression is that it would be more natural to group it with other events like 1:power off or 2:halted, which do not imply immediate VM restart. Or add a dedicated exit code, say, 5:suspended? The second question is that, apparently, calling bhyvectl --suspend ... is not enough, as after suspend completion the VM entity is still present in /dev/vmm, so it looks like bhyvectl --destroy is necessary. It's not obvious from the bhyvectl(8) manual page and generally it's not obvious why two bhyvectl(8) commands are necessary to suspend a VM. Interestingly, bhyvectl(8) does not complain early about using "--suspend" and "--destroy" at the same time, i.e.: bhyvectl --suspend=/var/run/vms/my_vm --destroy --vm my_vm But fails with obscure messages: bhyvectl(8): connect() failed: Connection refused bhyve(8): vm_run error -1, errno 6 Failed to emulate instruction sequence [ 8b800040000089c048018108060000 ] at 0xffffffffc06d85f3 And dumps a core. Roman