From nobody Tue Jun 16 15:47:01 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gfrw75Pd8z6hN2j for ; Tue, 16 Jun 2026 15:47:19 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [204.27.62.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4gfrw66Qg5z48MQ for ; Tue, 16 Jun 2026 15:47:18 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b="bhfdB/RZ"; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.57 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx1.shrew.net (8.18.1/8.18.1) with ESMTP id 65GFl7t2080415 for ; Tue, 16 Jun 2026 10:47:07 -0500 (CDT) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1781624827; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vgOYULBiJwaa1iBREdCEpF+7GKxmpRA0xH961drRsQ8=; b=bhfdB/RZ+57qmV1NZW++jrHFfIZlrorwqc1emcoXSdF9+zU/oXMmTZxeJ2rPEhfJHm130S BEGqyc3laP54tfGWLkelhSlvspEqWZKj4mUp2NX/QR+EBUUcgKr2jksGP71nDZ6NXS7tkK lPlnzYo5EBUGjdTXZ5aZvqydKlmKSxlEF2oPj1IBfm70E6leSq7YI929ZsXb9KUwLT2IJW senKahp8S1f6zpMg4L3dNMcfOCTc0RvEPVuZLJ+hkBOG1Mf0ZU9Nz/WmsggrkejQc2nJBp 0aejAdHph5pjuWStqba/dkr6OVJGUqXCmbM+7NjNCr0+YQO4sftJs23GjTDw5g== Received: from [10.22.200.32] (unknown [136.60.75.165]) by mail.shrew.net (Postfix) with ESMTPSA id CBDFA3AB05 for ; Tue, 16 Jun 2026 10:47:07 -0500 (CDT) Message-ID: <381ad530-f921-496d-b636-14ad28aa1e2d@shrew.net> Date: Tue, 16 Jun 2026 10:47:01 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: =?UTF-8?Q?Re=3A_**=5BHelp=5D_bhyve=3A_bootrom=5Falloc=3A_vm=5Fmmap?= =?UTF-8?Q?=5Fmapseg=3A_Invalid_argument_=E2=80=94_NVIDIA_passthrough_with_C?= =?UTF-8?Q?orvin=27s_branch_on_FreeBSD_15=2E0**?= To: virtualization@freebsd.org References: <1bbf28a7-7025-4d83-937d-72d8583048ca@shrew.net> Content-Language: en-US From: Matthew Grooms In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[shrew.net]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[shrew.net:+] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4gfrw66Qg5z48MQ Thanks Corvin. The passthrough feature appears to be working well. I appreciate the time and effort you've put into this. Once passed, I can use the device in a Debian 13 guest to execute cuda workloads. There do appear to be a few issues: 1) The nvidia device is always claimed as the primary GPU by the guest. 2) The guest hangs on shutdown and/or reboot. This only happens when the GPU is passed through. The first issue prevents me from performing a device reset if the GPU gets wedged in a CUDA operation. Reboot required. The second issue prevents a clean reboot ( always a dirty FS at startup ). Any ideas how to force UEFI to prefer the virtual frame buffer device? I've tried everything I can think of but no luck so far. Thanks again, -Matthew On 6/8/26 01:39, Corvin Köhne wrote: > On Fri, 2026-06-05 at 11:20 -0500, Matthew Grooms wrote: >> On 6/5/26 01:34, Corvin Köhne wrote: >>> On Thu, 2026-06-04 at 13:23 +0200, Mario Marietto wrote: >>>>          Hi everyone, >>>>   I'm trying to get NVIDIA GPU passthrough working with bhyve on FreeBSD >>>> 15.0- >>>> RELEASE-p5, using Corvin Köhne's nvidia-wip branch: >>>> >>>> https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/15.0/nvidia-wip >>>>   The VM fails to start with the following error before the guest even >>>> boots: >>>>   bhyve: bootrom_alloc: vm_mmap_mapseg: Invalid argument >>> NVIDIA GPU passthrough should work with stock 15.0, no patches required. >> I was about to attempt this myself. Are all the required patches present >> in 14.4 as well? >> >> Thanks, >> >> -Matthew >> > Yes, the required patch is present in 14.4 as well. > > From nobody Tue Jun 16 16:16:02 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gfsYX2rz6z6hPvQ for ; Tue, 16 Jun 2026 16:16:16 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4gfsYV6Cxcz4CmN for ; Tue, 16 Jun 2026 16:16:14 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=vW1zROUJ; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.58 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.18.1/8.18.1) with ESMTP id 65GGG2or070915 for ; Tue, 16 Jun 2026 11:16:02 -0500 (CDT) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1781626562; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VsPFRy6PcXo4sTlhEoT4z6r+tWhAQP9JCaS4P4NsEfE=; b=vW1zROUJKNTwkN5B+De+kY90+ELp5zfKcYgwOOmEV21IpjYphvr/9o7+l4tCIHX15v+zeE 7yh/hwpNVMKZuaTTy6XlvxnesO9k4HeLjs41l+s/2raBVnIP777RJFXU3g7d71KwwTh0bP frRh5CJg5P3NWujRpSCzFn/Wl+W3JlYUR4JNYGYZz3pckv2CMZPndPUOngA1IqapGpL0Ty MbhglFDCb8azywA+AJfd5clXH+lloqAzdgk5QaKXOOHk5hnCnLtgZh1SGrHVkgOgxnQ8A4 gdX1CUK9/MF9b5FYss4zSFwqGEagwbdr4KHZVZnHaw8WEl+unljdXs6jbgPq6Q== Received: from [10.22.200.32] (unknown [136.60.75.165]) by mail.shrew.net (Postfix) with ESMTPSA id BF5483AB05 for ; Tue, 16 Jun 2026 11:16:02 -0500 (CDT) Message-ID: <561d0b98-2183-4b0c-9843-38f897e65af5@shrew.net> Date: Tue, 16 Jun 2026 11:16:02 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: =?UTF-8?Q?Re=3A_**=5BHelp=5D_bhyve=3A_bootrom=5Falloc=3A_vm=5Fmmap?= =?UTF-8?Q?=5Fmapseg=3A_Invalid_argument_=E2=80=94_NVIDIA_passthrough_with_C?= =?UTF-8?Q?orvin=27s_branch_on_FreeBSD_15=2E0**?= To: virtualization@freebsd.org References: <1bbf28a7-7025-4d83-937d-72d8583048ca@shrew.net> <381ad530-f921-496d-b636-14ad28aa1e2d@shrew.net> Content-Language: en-US From: Matthew Grooms In-Reply-To: <381ad530-f921-496d-b636-14ad28aa1e2d@shrew.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.990]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[shrew.net]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[shrew.net:+] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4gfsYV6Cxcz4CmN If anyone else experiences this, disabling the frame buffer device appears to work around the issue for me. I can now shutdown and reboot the guest without a hang. Thanks again, -Matthew On 6/16/26 10:47, Matthew Grooms wrote: > Thanks Corvin. > > The passthrough feature appears to be working well. I appreciate the > time and effort you've put into this. Once passed, I can use the > device in a Debian 13 guest to execute cuda workloads. There do appear > to be a few issues: > > 1) The nvidia device is always claimed as the primary GPU by the guest. > 2) The guest hangs on shutdown and/or reboot. This only happens when > the GPU is passed through. > > The first issue prevents me from performing a device reset if the GPU > gets wedged in a CUDA operation. Reboot required. The second issue > prevents a clean reboot ( always a dirty FS at startup ). Any ideas > how to force UEFI to prefer the virtual frame buffer device? I've > tried everything I can think of but no luck so far. > > Thanks again, > > -Matthew > > On 6/8/26 01:39, Corvin Köhne wrote: >> On Fri, 2026-06-05 at 11:20 -0500, Matthew Grooms wrote: >>> On 6/5/26 01:34, Corvin Köhne wrote: >>>> On Thu, 2026-06-04 at 13:23 +0200, Mario Marietto wrote: >>>>>           Hi everyone, >>>>>    I'm trying to get NVIDIA GPU passthrough working with bhyve on >>>>> FreeBSD >>>>> 15.0- >>>>> RELEASE-p5, using Corvin Köhne's nvidia-wip branch: >>>>> https://github.com/Beckhoff/freebsd-src/tree/phab/corvink/15.0/nvidia-wip >>>>>    The VM fails to start with the following error before the guest >>>>> even >>>>> boots: >>>>>    bhyve: bootrom_alloc: vm_mmap_mapseg: Invalid argument >>>> NVIDIA GPU passthrough should work with stock 15.0, no patches >>>> required. >>> I was about to attempt this myself. Are all the required patches >>> present >>> in 14.4 as well? >>> >>> Thanks, >>> >>> -Matthew >>> >> Yes, the required patch is present in 14.4 as well. >> >> > From nobody Tue Jun 16 19:28:33 2026 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gfxqY1fXGz6hgZp for ; Tue, 16 Jun 2026 19:28:41 +0000 (UTC) (envelope-from wmckenzie@rhelitpro.com) Received: from mail-yw1-x112d.google.com (mail-yw1-x112d.google.com [IPv6:2607:f8b0:4864:20::112d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4gfxqX1HqZz3Tpv for ; Tue, 16 Jun 2026 19:28:40 +0000 (UTC) (envelope-from wmckenzie@rhelitpro.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=rhelitpro.com header.s=rhelitpro header.b=sDHj8wR+; dmarc=none; spf=pass (mx1.freebsd.org: domain of wmckenzie@rhelitpro.com designates 2607:f8b0:4864:20::112d as permitted sender) smtp.mailfrom=wmckenzie@rhelitpro.com Received: by mail-yw1-x112d.google.com with SMTP id 00721157ae682-7e3b2a435ecso54196127b3.1 for ; Tue, 16 Jun 2026 12:28:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rhelitpro.com; s=rhelitpro; t=1781638114; x=1782242914; darn=freebsd.org; h=mime-version:content-language:accept-language:message-id:date :thread-index:thread-topic:subject:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XNaDTG3YZyrOou312g9NKQDK0Q3u6MI2KL5DPdkl8eg=; b=sDHj8wR+cKir1uScuH19jQ0ufKXEZ3LILc3JkzK95WfrgRtsewNND2xw156Ym4+uwh arGl/TcgMbj7HR88xe/2PNccjeYLBONqVDmY9u1n9zU/dfk540ai0/RRi44UrEs1wRrS /K+rD5U9lKUJVvAG8o61h9X1zZTNZs1wMR1eQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781638114; x=1782242914; h=mime-version:content-language:accept-language:message-id:date :thread-index:thread-topic:subject:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XNaDTG3YZyrOou312g9NKQDK0Q3u6MI2KL5DPdkl8eg=; b=DbccWzfk7pZAwPDLnjIfgcOEZw6qRdEN9ujOzLN6sj0JgMBXrq/1Y2V3OersD+8khV 9gilJfhR740cuPYjC0GMp5JQkyTy17pUk0Eo5fXY5ZpdXDUdajRUo+ZBY12grzcSVFDu OHRBvGL3K0Kj0pce2vR4bwynsRm7p3pMrs39MesXHpI9XITft7hS3Bmi/MYDKx0r7SHr 7KKym9k0LwqbXo++vN8EfhEw0v1Dzw0ZHLO1d4A9/9nWkiWsL16Xuf6uopx3KFp7d1Z9 K/yKqhBOX4O0OJfwIohgzPUn0I3jdPHM9zjjrohjPY3wzQSdVGdqn1xe3N9tIccFIS4j iI8Q== X-Gm-Message-State: AOJu0Yy3egUdxz8q8+HkMzplkb81MPgq09PK5KbWd5YszmkaEmv9J2oL NdAJbmvnc1/3WFIOMXPC7LsfiLalnRhnMRRzgxiw85itmzvyOvNnEOjdE1mr2PEXXX9Gzs00pQ1 t8o6mPw== X-Gm-Gg: AfdE7cmiugZW34+honItfpCYP+PG4EyzeQxp/rC1dfHtniRVmyQJCTFoJDtGRktr3GM MEnG5qnkNkpYsPtzFzcw0ePovyO3vfyMuv0Ez0Cu9Ujk/VF2LrRgyv2WHevZWC3vNGkB761P/wd eaogGbqHnqFZmgnb2cS1ni/hptSQyZCBamM3YBd8i7+wcLayH0dawalwV0YAZdZLuJ/faRZ87VX zOH6v/PfD02437zjY6cXlzEJXDqEsV6OZdFkYU33OXjkZL9taJ3JVH04qf2V2rcJjeyOiK0e289 CUX0B9eQb8qC2eKK/xVfRCEXPQtucXNOmXTzdHwc1uOMGve9AWD1ZqHV+rLhij6RuY+7Kjqh+lR aN9TUNxJPhmJeV4QAtuwTawbVtkSRq3G0RHnSsIujk6WyawjFPnyCewQIuVuK6fSFxjfHJIlyVg Oj2Bn/8F59beBK8LPPGK6AgsOT2j12L5bDbdlDejpmCQY13ZwSW9MACqINkiYrMyJK2z7kvh4kZ zl5kmVZtJZcNZvYHli5 X-Received: by 2002:a05:690c:ed5:b0:79a:5fb9:62ad with SMTP id 00721157ae682-7fe5e4997famr4066747b3.43.1781638114330; Tue, 16 Jun 2026 12:28:34 -0700 (PDT) Received: from CH3PR12MB8187.namprd12.prod.outlook.com ([2603:1036:304:3005::5]) by smtp.gmail.com with ESMTPSA id 00721157ae682-7fd33743bc6sm18221897b3.18.2026.06.16.12.28.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jun 2026 12:28:34 -0700 (PDT) From: William Mckenzie To: "freebsd-virtualization@FreeBSD.org" Subject: Bhyve live migration, virtio-ballooning, kvm-clock Thread-Topic: Bhyve live migration, virtio-ballooning, kvm-clock Thread-Index: AQHc/cQXwI75fn0BTkOmVX/aGFkWIA== X-MS-Exchange-MessageSentRepresentingType: 1 Date: Tue, 16 Jun 2026 19:28:33 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-Exchange-Organization-SCL: -1 X-MS-TNEF-Correlator: X-MS-Exchange-Organization-RecordReviewCfmType: 0 x-ms-reactions: allow Content-Type: multipart/alternative; boundary="_000_CH3PR12MB8187F88C7668D06E506DEB5CFEE52CH3PR12MB8187namp_" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4864::/56]; R_DKIM_ALLOW(-0.20)[rhelitpro.com:s=rhelitpro]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[rhelitpro.com]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::112d:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[rhelitpro.com:+] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4gfxqX1HqZz3Tpv --_000_CH3PR12MB8187F88C7668D06E506DEB5CFEE52CH3PR12MB8187namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Hi all, For some time we have been working on getting bhyve live vm-migration worki= ng. We have developed, deployed, and validated three feature series against= the FreeBSD base system (15.0) and we would like to contribute them upstre= am. I=92m writing to ask whether a member of the virtualization team would = be willing to act as champion/mentor for these series through the review pr= ocess. What we=92ve done: 1) bhyve live migration (vmm + libvmmapi + bhyve + bhyvectl; 10 commits, th= e kernel engine decomposed into four buildable commits). Live migration of a running guest between two hosts: a versioned VM_MIGRATE_* ioctl surface in vmm(4), iterative RAM precopy driven by EPT/NPT dirty-bit harvesting, vCPU/device/timer state transfer reusing t= he existing vm_snapshot machinery, "bhyve -M send/recv" as the userland mover, and a set of restore-correctness fixes (vCPU allocation order, authoritative RIP, PIT re-arm, vm_restore_time on finalize, TSC/vHPET co-anchoring). The PCI BAR re-registration fix is a standalone commit because it also repairs a pre-existing bug in stock bhyvectl(8) --checkpoint/restore, independent of migration. Validated end-to-end on = a two-host physical Intel lab as a transparent live handoff: a running Rocky Linux = 9 guest migrates in both directions keeping its boot_id, uptime, processes= , AND live network sessions across the cutover, at ~0.4 s idle downtime; 2= 0/20 bidirectional runs with zero failures, and a stress run (4 GB / 24 GB gu= est under ~2 GB/s memory churn during the migration) stayed correct with downtime scaling as expected with the at-pause dirty set. One read-only ioctl is added to the capsicum allow-list; all state-changing ioctls sta= y outside the sandbox. 2) bhyve virtio-balloon (usr.sbin/bhyve; 1 commit). A virtio-balloon (type 5) device emulation: inflate/deflate virtqueues with host reclaim via paddr_guest2host() + madvise(MADV_FREE), standard num_pages/actual config space, VIRTIO_BALLOON_F_STATS_VQ guest telemetry= , and a per-VM control socket created before cap_enter(). Guest-validated against FreeBSD virtio_balloon(4) on two nodes (inflate/deflate tracked exactly; mid-flight readings prove the values are guest-driven) and a Linux guest for the stats queue. 3) bhyve kvm-clock (vmm; 4 commits, gated behind hw.vmm.kvmclock, default off). A KVM-compatible paravirtual clock: KVM CPUID signature at 0x40000100 (bhyve's own signature leaf untouched), MSR_KVM_SYSTEM_TIME_NEW / MSR_KVM_WALL_CLOCK_NEW on both VMX and SVM paths, publishing standard pvclock structures through vm_gpa_hold_global(). This is the durable fix for Linux guests marking the TSC unstable and degrading to hpet after any snapshot/restore or migration. Validated on hardware: guests register kvm-clock and survive repeated bidirectional migrations with zero TSC-unstable events (the pre-kvmclock baseline reliably degraded on the same hardware). I=92ve got a full submission document (design, per-failure bring-up history= , complete test matrix, untested-areas inventory, and security analysis) an= d the git-format-patch series (against releng/15.0, where they are validate= d). I=92ve tested many rounds of live vm-migrations across hosts (AMD using KVM= nested virtualization and Intel physical systems) and have finally gotten = it to a stable state with 30+ live migrations without packets dropping. I = intend to do further testing (specifically with AMD physical boxes). Bhyve is phenomenal. If there is no interest in a champion, I still intend = to at least attempt to see the process through (acceptance or not). Happy t= o provide the documentation/requested info. Thanks for the consideration. William Mckenzie wmckenzie@rhelitpro.com --_000_CH3PR12MB8187F88C7668D06E506DEB5CFEE52CH3PR12MB8187namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable
Hi all, 

For some time we have been working on getting bhyve live vm-migration worki= ng. We have developed, deployed, and validated three feature series against= the FreeBSD base system (15.0) and we would like to contribute them upstre= am. I=92m writing to ask whether a member of the virtualization team would be willing to act as champion/ment= or for these series through the review process.

What we=92ve done: 

1) bhyve live migration (vmm + libvmmapi + bhyve + bhyvectl; 10 commits, th= e
   kernel engine decomposed into four buildable commits).
   Live migration of a running guest between two hosts: a version= ed
   VM_MIGRATE_* ioctl surface in vmm(4), iterative RAM precopy dr= iven by
   EPT/NPT dirty-bit harvesting, vCPU/device/timer state transfer= reusing the
   existing vm_snapshot machinery, "bhyve -M send/recv"= as the userland
   mover, and a set of restore-correctness fixes (vCPU allocation= order,
   authoritative RIP, PIT re-arm, vm_restore_time on finalize, TS= C/vHPET
   co-anchoring). The PCI BAR re-registration fix is a standalone= commit
   because it also repairs a pre-existing bug in stock bhyvectl(8= )
   --checkpoint/restore, independent of migration. Validated end-= to-end on a two-host
   physical Intel lab as a transparent live handoff: a running Ro= cky Linux 9
   guest migrates in both directions keeping its boot_id, uptime,= processes,
   AND live network sessions across the cutover, at ~0.4 s idle d= owntime; 20/20
   bidirectional runs with zero failures, and a stress run (4 GB = / 24 GB guest
   under ~2 GB/s memory churn during the migration) stayed correc= t with
   downtime scaling as expected with the at-pause dirty set. One = read-only
   ioctl is added to the capsicum allow-list; all state-changing = ioctls stay
   outside the sandbox.

2) bhyve virtio-balloon (usr.sbin/bhyve; 1 commit).
   A virtio-balloon (type 5) device emulation: inflate/deflate vi= rtqueues
   with host reclaim via paddr_guest2host() + madvise(MADV_FREE),= standard
   num_pages/actual config space, VIRTIO_BALLOON_F_STATS_VQ guest= telemetry,
   and a per-VM control socket created before cap_enter(). Guest-= validated
   against FreeBSD virtio_balloon(4) on two nodes (inflate/deflat= e tracked
   exactly; mid-flight readings prove the values are guest-driven= ) and a
   Linux guest for the stats queue.

3) bhyve kvm-clock (vmm; 4 commits, gated behind hw.vmm.kvmclock, default
   off). A KVM-compatible paravirtual clock: KVM CPUID signature = at
   0x40000100 (bhyve's own signature leaf untouched),
   MSR_KVM_SYSTEM_TIME_NEW / MSR_KVM_WALL_CLOCK_NEW on both VMX a= nd SVM
   paths, publishing standard pvclock structures through
   vm_gpa_hold_global(). This is the durable fix for Linux guests= marking
   the TSC unstable and degrading to hpet after any snapshot/rest= ore or
   migration. Validated on hardware: guests register kvm-clock an= d survive
   repeated bidirectional migrations with zero TSC-unstable event= s (the
   pre-kvmclock baseline reliably degraded on the same hardware).=


I=92ve got a full submission document (design, per-failure bring-up history= , complete test matrix, untested-areas inventory, and security analysis) an= d the git-format-patch series (against releng/15.0, where they are validate= d).

I=92ve tested many rounds of live vm-migrations across hosts (AMD using KVM= nested virtualization and Intel physical systems) and have finally gotten = it to a stable state with 30+ live migrations without packets dropping. &nb= sp;I intend to do further testing (specifically with AMD physical boxes).  

Bhyve is phenomenal. If there is no interest in a champion, I still intend = to at least attempt to see the process through (acceptance or not). Happy t= o provide the documentation/requested info.

Thanks for the consideration.

William Mckenzie
wmckenzie@rhelitpro.com

--_000_CH3PR12MB8187F88C7668D06E506DEB5CFEE52CH3PR12MB8187namp_-- From nobody Tue Jun 16 20:26:42 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gfz6f3MS0z6hlZl for ; Tue, 16 Jun 2026 20:26:50 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4gfz6c3LcJz3gv9 for ; Tue, 16 Jun 2026 20:26:48 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=wp3YWYGP; dmarc=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.58 as permitted sender) smtp.mailfrom=mgrooms@shrew.net Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.18.1/8.18.1) with ESMTP id 65GKQgpX073158 for ; Tue, 16 Jun 2026 15:26:42 -0500 (CDT) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1781641602; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=9kWkTmnXoAUoY9uVFZ/fdONs/bvy8tnxwIu7ahRYkNU=; b=wp3YWYGPXQsqsNhXpxJ9JXC4qNW57csbduWtUYqMERawPfrvrWsgfNBGEFvaxkieT77q/K kWMiFBfdFZ8IsQBa0aoL67llCJXy/KETH8tJG6eC7l1/2Y9poJnznbpSiQumaXR//9DJ+P VI+gfRhID/rmqO+WwYm5RPWaa34bq5VDMrzenZ/+qe7b9vgAkaIvnwdbAjfTX6aaMiq/qv Ew13bniI/2U6i4b+KN5ftc+qli2q5bhstMFaBvhzda3qkUgwKRRXcHwrcX+7/f3MgQ737U /BJ6ishbPz7vkZ7AXHzBOfaL7ZuHU7rxZWJ4arCC5IsrNgW5KQSXpaCpdsxKAA== Received: from [10.22.200.32] (unknown [136.60.75.165]) by mail.shrew.net (Postfix) with ESMTPSA id CD0AA3B715 for ; Tue, 16 Jun 2026 15:26:42 -0500 (CDT) Content-Type: multipart/alternative; boundary="------------6p08OhymQA68uCWgAhnqyFir" Message-ID: Date: Tue, 16 Jun 2026 15:26:42 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Bhyve live migration, virtio-ballooning, kvm-clock To: virtualization@freebsd.org References: Content-Language: en-US From: Matthew Grooms In-Reply-To: X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DKIM_TRACE(0.00)[shrew.net:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[shrew.net]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4gfz6c3LcJz3gv9 This is a multi-part message in MIME format. --------------6p08OhymQA68uCWgAhnqyFir Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 6/16/26 14:28, William Mckenzie wrote: > Hi all, > > For some time we have been working on getting bhyve live vm-migration > working. We have developed, deployed, and validated three feature > series against the FreeBSD base system (15.0) and we would like to > contribute them upstream. I’m writing to ask whether a member of the > virtualization team would be willing to act as champion/mentor for > these series through the review process. > > What we’ve done: > > 1) bhyve live migration (vmm + libvmmapi + bhyve + bhyvectl; 10 > commits, the >    kernel engine decomposed into four buildable commits). >    Live migration of a running guest between two hosts: a versioned >    VM_MIGRATE_* ioctl surface in vmm(4), iterative RAM precopy driven by >    EPT/NPT dirty-bit harvesting, vCPU/device/timer state transfer > reusing the >    existing vm_snapshot machinery, "bhyve -M send/recv" as the userland >    mover, and a set of restore-correctness fixes (vCPU allocation order, >    authoritative RIP, PIT re-arm, vm_restore_time on finalize, TSC/vHPET >    co-anchoring). The PCI BAR re-registration fix is a standalone commit >    because it also repairs a pre-existing bug in stock bhyvectl(8) >    --checkpoint/restore, independent of migration. Validated > end-to-end on a two-host >    physical Intel lab as a transparent live handoff: a running Rocky > Linux 9 >    guest migrates in both directions keeping its boot_id, uptime, > processes, >    AND live network sessions across the cutover, at ~0.4 s idle > downtime; 20/20 >    bidirectional runs with zero failures, and a stress run (4 GB / 24 > GB guest >    under ~2 GB/s memory churn during the migration) stayed correct with >    downtime scaling as expected with the at-pause dirty set. One read-only >    ioctl is added to the capsicum allow-list; all state-changing > ioctls stay >    outside the sandbox. > > 2) bhyve virtio-balloon (usr.sbin/bhyve; 1 commit). >    A virtio-balloon (type 5) device emulation: inflate/deflate virtqueues >    with host reclaim via paddr_guest2host() + madvise(MADV_FREE), standard >    num_pages/actual config space, VIRTIO_BALLOON_F_STATS_VQ guest > telemetry, >    and a per-VM control socket created before cap_enter(). Guest-validated >    against FreeBSD virtio_balloon(4) on two nodes (inflate/deflate tracked >    exactly; mid-flight readings prove the values are guest-driven) and a >    Linux guest for the stats queue. > > 3) bhyve kvm-clock (vmm; 4 commits, gated behind hw.vmm.kvmclock, default >    off). A KVM-compatible paravirtual clock: KVM CPUID signature at >    0x40000100 (bhyve's own signature leaf untouched), >    MSR_KVM_SYSTEM_TIME_NEW / MSR_KVM_WALL_CLOCK_NEW on both VMX and SVM >    paths, publishing standard pvclock structures through >    vm_gpa_hold_global(). This is the durable fix for Linux guests marking >    the TSC unstable and degrading to hpet after any snapshot/restore or >    migration. Validated on hardware: guests register kvm-clock and survive >    repeated bidirectional migrations with zero TSC-unstable events (the >    pre-kvmclock baseline reliably degraded on the same hardware). > > > I’ve got a full submission document (design, per-failure bring-up > history, complete test matrix, untested-areas inventory, and security > analysis) and the git-format-patch series (against releng/15.0, where > they are validated). > > I’ve tested many rounds of live vm-migrations across hosts (AMD using > KVM nested virtualization and Intel physical systems) and have finally > gotten it to a stable state with 30+ live migrations without packets > dropping.  I intend to do further testing (specifically with AMD > physical boxes). > > Bhyve is phenomenal. If there is no interest in a champion, I still > intend to at least attempt to see the process through (acceptance or > not). Happy to provide the documentation/requested info. > Thanks for working on this. Live migration patch sets have been proposed a few times before. You can find the most recent attempt sitting in reviews from 2022 ... https://reviews.freebsd.org/D34722 https://reviews.freebsd.org/D34811 https://reviews.freebsd.org/D34719 https://reviews.freebsd.org/D34720 https://reviews.freebsd.org/D34721 You should also be able to locate several email threads related to the topic on the public freebsd mailing list archives. I won't rehash that here, but there was resistance. The orignal work for that and other bhyve related projects ( libvdsk w/ qcow2+vmdk support, user mode usb pass-through, etc ... ) were hosted here ... https://github.com/orgs/FreeBSD-UPB/repositories You should probably also have a look at this ... https://www.freebsd.org/status/report-2025-10-2025-12/bhyve-cpuid/ From what I gather from his Zagreb presentation, the feature is being developed as a foundational layer to import illumos bhyve migration code with an eye towards feature parity and potential interoperability. Good luck! -Matthew --------------6p08OhymQA68uCWgAhnqyFir Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit


On 6/16/26 14:28, William Mckenzie wrote:
Hi all, 

For some time we have been working on getting bhyve live vm-migration working. We have developed, deployed, and validated three feature series against the FreeBSD base system (15.0) and we would like to contribute them upstream. I’m writing to ask whether a member of the virtualization team would be willing to act as champion/mentor for these series through the review process.

What we’ve done: 

1) bhyve live migration (vmm + libvmmapi + bhyve + bhyvectl; 10 commits, the
   kernel engine decomposed into four buildable commits).
   Live migration of a running guest between two hosts: a versioned
   VM_MIGRATE_* ioctl surface in vmm(4), iterative RAM precopy driven by
   EPT/NPT dirty-bit harvesting, vCPU/device/timer state transfer reusing the
   existing vm_snapshot machinery, "bhyve -M send/recv" as the userland
   mover, and a set of restore-correctness fixes (vCPU allocation order,
   authoritative RIP, PIT re-arm, vm_restore_time on finalize, TSC/vHPET
   co-anchoring). The PCI BAR re-registration fix is a standalone commit
   because it also repairs a pre-existing bug in stock bhyvectl(8)
   --checkpoint/restore, independent of migration. Validated end-to-end on a two-host
   physical Intel lab as a transparent live handoff: a running Rocky Linux 9
   guest migrates in both directions keeping its boot_id, uptime, processes,
   AND live network sessions across the cutover, at ~0.4 s idle downtime; 20/20
   bidirectional runs with zero failures, and a stress run (4 GB / 24 GB guest
   under ~2 GB/s memory churn during the migration) stayed correct with
   downtime scaling as expected with the at-pause dirty set. One read-only
   ioctl is added to the capsicum allow-list; all state-changing ioctls stay
   outside the sandbox.

2) bhyve virtio-balloon (usr.sbin/bhyve; 1 commit).
   A virtio-balloon (type 5) device emulation: inflate/deflate virtqueues
   with host reclaim via paddr_guest2host() + madvise(MADV_FREE), standard
   num_pages/actual config space, VIRTIO_BALLOON_F_STATS_VQ guest telemetry,
   and a per-VM control socket created before cap_enter(). Guest-validated
   against FreeBSD virtio_balloon(4) on two nodes (inflate/deflate tracked
   exactly; mid-flight readings prove the values are guest-driven) and a
   Linux guest for the stats queue.

3) bhyve kvm-clock (vmm; 4 commits, gated behind hw.vmm.kvmclock, default
   off). A KVM-compatible paravirtual clock: KVM CPUID signature at
   0x40000100 (bhyve's own signature leaf untouched),
   MSR_KVM_SYSTEM_TIME_NEW / MSR_KVM_WALL_CLOCK_NEW on both VMX and SVM
   paths, publishing standard pvclock structures through
   vm_gpa_hold_global(). This is the durable fix for Linux guests marking
   the TSC unstable and degrading to hpet after any snapshot/restore or
   migration. Validated on hardware: guests register kvm-clock and survive
   repeated bidirectional migrations with zero TSC-unstable events (the
   pre-kvmclock baseline reliably degraded on the same hardware).


I’ve got a full submission document (design, per-failure bring-up history, complete test matrix, untested-areas inventory, and security analysis) and the git-format-patch series (against releng/15.0, where they are validated).

I’ve tested many rounds of live vm-migrations across hosts (AMD using KVM nested virtualization and Intel physical systems) and have finally gotten it to a stable state with 30+ live migrations without packets dropping.  I intend to do further testing (specifically with AMD physical boxes).  

Bhyve is phenomenal. If there is no interest in a champion, I still intend to at least attempt to see the process through (acceptance or not). Happy to provide the documentation/requested info.


Thanks for working on this. Live migration patch sets have been proposed a few times before. You can find the most recent attempt sitting in reviews from 2022 ...

https://reviews.freebsd.org/D34722
https://reviews.freebsd.org/D34811
https://reviews.freebsd.org/D34719
https://reviews.freebsd.org/D34720
https://reviews.freebsd.org/D34721

You should also be able to locate several email threads related to the topic on the public freebsd mailing list archives. I won't rehash that here, but there was resistance. The orignal work for that and other bhyve related projects ( libvdsk w/ qcow2+vmdk support, user mode usb pass-through, etc ... ) were hosted here ...

https://github.com/orgs/FreeBSD-UPB/repositories

You should probably also have a look at this ...

https://www.freebsd.org/status/report-2025-10-2025-12/bhyve-cpuid/

From what I gather from his Zagreb presentation, the feature is being developed as a foundational layer to import illumos bhyve migration code with an eye towards feature parity and potential interoperability.

Good luck!

-Matthew

--------------6p08OhymQA68uCWgAhnqyFir-- From nobody Tue Jun 16 20:40:28 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gfzQQ1HR9z6hmsJ for ; Tue, 16 Jun 2026 20:40:30 +0000 (UTC) (envelope-from wmckenzie@rhelitpro.com) Received: from mail-qt1-x830.google.com (mail-qt1-x830.google.com [IPv6:2607:f8b0:4864:20::830]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4gfzQP57Vrz3kZ4 for ; Tue, 16 Jun 2026 20:40:29 +0000 (UTC) (envelope-from wmckenzie@rhelitpro.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-x830.google.com with SMTP id d75a77b69052e-5177945a279so56622271cf.0 for ; Tue, 16 Jun 2026 13:40:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1781642423; cv=none; d=google.com; s=arc-20240605; b=SZD3KkxwclrxzTHhKCs8341vf/4e/Af9loep0QLzWornoy7H1h6kG4MTYYn/0QELRn qx/oXEsTnBgjfC7zhWBfm+kAYFzjhWb0FhGrP4bSD+fv3ytrJHVTzG1HF+p4/pAqYQiQ wCmh/INhu5hk/lerQqmtbPTL0WwXyS9zqeFaAv/YpDoh7DW7nBwBnBpm/LchWRPb9SGk joaH0tskiqiVO9WMlvrNgVhIQEAr50Wx8QR5koYDnjEhmYTHDZwybdX9ZGUusHrGaChP OdmrMyhC+84lBL2XYPbHzNADIF0AWMgqjobKvYr9DlpMvy4+3VlmhBtegScoR25cI26L SBYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=vinntj9XamN/AdXXauFeDB/68YrrwBtpTHs6jO/fsIg=; fh=lDuWU0Le42xP677gly0d2S07EEycXbmLN7JFYYIPYuE=; b=bpN9BkVA7fEeDDIEeJTgnZjjHjj+ZSPTGcHCAadT/+/VeJmNGqCeIfZL0f9w34VlMX UedDhwMcqtu0+a9S+4h/u+5xncLK5mt+B3kzBRuhZgBqY2MznNcuIK43wH3BJmLPLMV8 JOptRJV6zmH1VF3e2hXe6Y2C4HaXDEY/gEJ0QdwDfBuUGHUdbD69s0nFMfiHu+ldYMjJ 3Ghzbpz34IUD64kAWRhwB/MR9DTraXyEzCHx1ErbulE9XuQSqMa31D8/esOnqmFY9os5 osLS9WiqWwlgG4BKmRMIusd31sw2BM4hqt9tGba/Lm2XTzlxoro/+R0VhHL5GjC9sL5R zTPw==; darn=freebsd.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rhelitpro.com; s=rhelitpro; t=1781642423; x=1782247223; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=vinntj9XamN/AdXXauFeDB/68YrrwBtpTHs6jO/fsIg=; b=A/TS8lwQU8hRB8foQMm8LrQE4/9t8chDC9EH8XJsM8CRRZUyyY7ElZY5R+AttJYsf8 INbfhIPWE6BS0HPHsfZfE1bYw5J/Tcvf/TdDbs4K0qPSy16WlIa6p42v8bZadiEwpU39 18g0BMDEirdeACeXrkGygaQPEtPwdNIktvagM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781642423; x=1782247223; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=vinntj9XamN/AdXXauFeDB/68YrrwBtpTHs6jO/fsIg=; b=dqWcGWjC/De+iETSNiC8SUsK4r4JJZYB2BTHeDx3CrYaL0+6tXSkk/K8XEx4vuYe90 QI8igO5wilsZ4GZE7T7IIeQT9w/7JRt+aZr0AQRjioFs3FpKzPx6dCdH0ajJPYbbl+Nn qVUjbPNCv7LE+cggTzRktz4odFHP1pJVcYzwdUbdTAYfTxHhuWSUzNkXOga+O4cB+gHQ sZGoet7GEE7vBDe1bt4ImfpxyGhGWmPUSND5EA+EuaLeLu88IM7cwnd1sFVtgEED/3vE 11EhxbTZqrSBsO3huNg1f3qPlEvJYD5VsmACZ+9fWBAwBqF9SaciMFKghkJXMzvdBV1U KfNQ== X-Gm-Message-State: AOJu0YxTT8GJkmAc2FFUyV7FB+JXnez9bjNdSzIzQEEuqlSoflunaIex kSAtnXtvNl+CILCHqJsecZs8lLXqemXXIzFJBTcyfE/VdKyc4dWgXgEaVbBNzOoQp/QIwu1BWXc IJvlrikOzdypROYlNuzNzyzUU66DMfVylUu3+mMyfPhOiIRrKei3jbAW9 X-Gm-Gg: Acq92OHD7/9/kMgWOE1HVKsw6bNMiKcJYm46xB7gTn748PGBGIMBm9JXRMuFXmxb5TO R8Gb9OnE3G0ldu8cEF4SEDSPHXc9dfRsEzj1iHz6d3iBlsJrhvHGsS5EIWwXw20cberXZNGqaHp 1PbA8g99H+9ikLavawSEEyOQOysjfl0hjFxUEoBV86mw40ZkFFLrLVMaTAYW/hkZ9yIL392vQCz 5y0LhZPrkG8bRe8JmXZSfovWH/7sTv4jY67x8TC8RToTP99qvz6aW0DkKzb5l8FQkJqDyIpLFyb /oE8QO/1 X-Received: by 2002:a05:622a:8d04:b0:519:5680:1b5 with SMTP id d75a77b69052e-519a8e00989mr16399931cf.21.1781642423205; Tue, 16 Jun 2026 13:40:23 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 References: In-Reply-To: From: William Mckenzie Date: Tue, 16 Jun 2026 16:40:28 -0400 X-Gm-Features: AVVi8Ce4w22QY7UnWvjvgmtIluXdei7Y0FVG6hbLwABn_d_xYyABE4pLcAMalgM Message-ID: Subject: Re: Bhyve live migration, virtio-ballooning, kvm-clock To: Matthew Grooms Cc: virtualization@freebsd.org Content-Type: multipart/alternative; boundary="000000000000c08e9b065464f301" X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4gfzQP57Vrz3kZ4 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated --000000000000c08e9b065464f301 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank you Matthew for the quick response. I certainly don't want to be a point of contention for the direction the team is already headed in. Ill read over what you sent out and if there's anyway I can contribute (lab hardware, resources, testing), i'd be more than happy to. Much appreciated! On Tue, Jun 16, 2026 at 4:27=E2=80=AFPM Matthew Grooms = wrote: > > On 6/16/26 14:28, William Mckenzie wrote: > > Hi all, > > For some time we have been working on getting bhyve live vm-migration > working. We have developed, deployed, and validated three feature series > against the FreeBSD base system (15.0) and we would like to contribute th= em > upstream. I=E2=80=99m writing to ask whether a member of the virtualizati= on team > would be willing to act as champion/mentor for these series through the > review process. > > What we=E2=80=99ve done: > > 1) bhyve live migration (vmm + libvmmapi + bhyve + bhyvectl; 10 commits, > the > kernel engine decomposed into four buildable commits). > Live migration of a running guest between two hosts: a versioned > VM_MIGRATE_* ioctl surface in vmm(4), iterative RAM precopy driven by > EPT/NPT dirty-bit harvesting, vCPU/device/timer state transfer reusing > the > existing vm_snapshot machinery, "bhyve -M send/recv" as the userland > mover, and a set of restore-correctness fixes (vCPU allocation order, > authoritative RIP, PIT re-arm, vm_restore_time on finalize, TSC/vHPET > co-anchoring). The PCI BAR re-registration fix is a standalone commit > because it also repairs a pre-existing bug in stock bhyvectl(8) > --checkpoint/restore, independent of migration. Validated end-to-end o= n > a two-host > physical Intel lab as a transparent live handoff: a running Rocky Linu= x > 9 > guest migrates in both directions keeping its boot_id, uptime, > processes, > AND live network sessions across the cutover, at ~0.4 s idle downtime; > 20/20 > bidirectional runs with zero failures, and a stress run (4 GB / 24 GB > guest > under ~2 GB/s memory churn during the migration) stayed correct with > downtime scaling as expected with the at-pause dirty set. One read-onl= y > ioctl is added to the capsicum allow-list; all state-changing ioctls > stay > outside the sandbox. > > 2) bhyve virtio-balloon (usr.sbin/bhyve; 1 commit). > A virtio-balloon (type 5) device emulation: inflate/deflate virtqueues > with host reclaim via paddr_guest2host() + madvise(MADV_FREE), standar= d > num_pages/actual config space, VIRTIO_BALLOON_F_STATS_VQ guest > telemetry, > and a per-VM control socket created before cap_enter(). Guest-validate= d > against FreeBSD virtio_balloon(4) on two nodes (inflate/deflate tracke= d > exactly; mid-flight readings prove the values are guest-driven) and a > Linux guest for the stats queue. > > 3) bhyve kvm-clock (vmm; 4 commits, gated behind hw.vmm.kvmclock, default > off). A KVM-compatible paravirtual clock: KVM CPUID signature at > 0x40000100 (bhyve's own signature leaf untouched), > MSR_KVM_SYSTEM_TIME_NEW / MSR_KVM_WALL_CLOCK_NEW on both VMX and SVM > paths, publishing standard pvclock structures through > vm_gpa_hold_global(). This is the durable fix for Linux guests marking > the TSC unstable and degrading to hpet after any snapshot/restore or > migration. Validated on hardware: guests register kvm-clock and surviv= e > repeated bidirectional migrations with zero TSC-unstable events (the > pre-kvmclock baseline reliably degraded on the same hardware). > > > I=E2=80=99ve got a full submission document (design, per-failure bring-up= history, > complete test matrix, untested-areas inventory, and security analysis) an= d > the git-format-patch series (against releng/15.0, where they are validate= d). > > I=E2=80=99ve tested many rounds of live vm-migrations across hosts (AMD u= sing KVM > nested virtualization and Intel physical systems) and have finally gotten > it to a stable state with 30+ live migrations without packets dropping. = I > intend to do further testing (specifically with AMD physical boxes). > > Bhyve is phenomenal. If there is no interest in a champion, I still inten= d > to at least attempt to see the process through (acceptance or not). Happy > to provide the documentation/requested info. > > > Thanks for working on this. Live migration patch sets have been proposed = a > few times before. You can find the most recent attempt sitting in reviews > from 2022 ... > > https://reviews.freebsd.org/D34722 > https://reviews.freebsd.org/D34811 > https://reviews.freebsd.org/D34719 > https://reviews.freebsd.org/D34720 > https://reviews.freebsd.org/D34721 > > You should also be able to locate several email threads related to the > topic on the public freebsd mailing list archives. I won't rehash that > here, but there was resistance. The orignal work for that and other bhyve > related projects ( libvdsk w/ qcow2+vmdk support, user mode usb > pass-through, etc ... ) were hosted here ... > > https://github.com/orgs/FreeBSD-UPB/repositories > > You should probably also have a look at this ... > > https://www.freebsd.org/status/report-2025-10-2025-12/bhyve-cpuid/ > > From what I gather from his Zagreb presentation, the feature is being > developed as a foundational layer to import illumos bhyve migration code > with an eye towards feature parity and potential interoperability. > > Good luck! > > -Matthew > --000000000000c08e9b065464f301 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thank you Matthew for the quick response. I certainly= don't want to be a point of contention for the direction the team is a= lready headed in. Ill read over what you sent out and if there's anyway= I can contribute (lab hardware, resources, testing), i'd be more than = happy=C2=A0to.

Much appreciated!

On Tue, = Jun 16, 2026 at 4:27=E2=80=AFPM Matthew Grooms <mgrooms@shrew.net> wrote:
=20 =20 =20


On 6/16/26 14:28, William Mckenzie wrote:
=20
Hi all,=C2=A0

For some time we have been working on getting bhyve live vm-migration working. We have developed, deployed, and validated three feature series against the FreeBSD base system (15.0) and we would like to contribute them upstream. I=E2=80=99m writing to a= sk whether a member of the virtualization team would be willing to act as champion/mentor for these series through the review process.

What we=E2=80=99ve done:=C2=A0

1) bhyve live migration (vmm + libvmmapi + bhyve + bhyvectl; 10 commits, the
=C2=A0 =C2=A0kernel engine decomposed into four buildable commits).=
=C2=A0 =C2=A0Live migration of a running guest between two hosts: a versioned
=C2=A0 =C2=A0VM_MIGRATE_* ioctl surface in vmm(4), iterative RAM pr= ecopy driven by
=C2=A0 =C2=A0EPT/NPT dirty-bit harvesting, vCPU/device/timer state transfer reusing the
=C2=A0 =C2=A0existing vm_snapshot machinery, "bhyve -M send/re= cv" as the userland
=C2=A0 =C2=A0mover, and a set of restore-correctness fixes (vCPU allocation order,
=C2=A0 =C2=A0authoritative RIP, PIT re-arm, vm_restore_time on fina= lize, TSC/vHPET
=C2=A0 =C2=A0co-anchoring). The PCI BAR re-registration fix is a standalone commit
=C2=A0 =C2=A0because it also repairs a pre-existing bug in stock bhyvectl(8)
=C2=A0 =C2=A0--checkpoint/restore, independent of migration. Valida= ted end-to-end on a two-host
=C2=A0 =C2=A0physical Intel lab as a transparent live handoff: a ru= nning Rocky Linux 9
=C2=A0 =C2=A0guest migrates in both directions keeping its boot_id, uptime, processes,
=C2=A0 =C2=A0AND live network sessions across the cutover, at ~0.4 = s idle downtime; 20/20
=C2=A0 =C2=A0bidirectional runs with zero failures, and a stress ru= n (4 GB / 24 GB guest
=C2=A0 =C2=A0under ~2 GB/s memory churn during the migration) staye= d correct with
=C2=A0 =C2=A0downtime scaling as expected with the at-pause dirty s= et. One read-only
=C2=A0 =C2=A0ioctl is added to the capsicum allow-list; all state-c= hanging ioctls stay
=C2=A0 =C2=A0outside the sandbox.

2) bhyve virtio-balloon (usr.sbin/bhyve; 1 commit).
=C2=A0 =C2=A0A virtio-balloon (type 5) device emulation: inflate/de= flate virtqueues
=C2=A0 =C2=A0with host reclaim via paddr_guest2host() + madvise(MADV_FREE), standard
=C2=A0 =C2=A0num_pages/actual config space, VIRTIO_BALLOON_F_STATS_= VQ guest telemetry,
=C2=A0 =C2=A0and a per-VM control socket created before cap_enter()= . Guest-validated
=C2=A0 =C2=A0against FreeBSD virtio_balloon(4) on two nodes (inflate/deflate tracked
=C2=A0 =C2=A0exactly; mid-flight readings prove the values are guest-driven) and a
=C2=A0 =C2=A0Linux guest for the stats queue.

3) bhyve kvm-clock (vmm; 4 commits, gated behind hw.vmm.kvmclock, default
=C2=A0 =C2=A0off). A KVM-compatible paravirtual clock: KVM CPUID si= gnature at
=C2=A0 =C2=A00x40000100 (bhyve's own signature leaf untouched),=
=C2=A0 =C2=A0MSR_KVM_SYSTEM_TIME_NEW / MSR_KVM_WALL_CLOCK_NEW on bo= th VMX and SVM
=C2=A0 =C2=A0paths, publishing standard pvclock structures through<= /div>
=C2=A0 =C2=A0vm_gpa_hold_global(). This is the durable fix for Linu= x guests marking
=C2=A0 =C2=A0the TSC unstable and degrading to hpet after any snapshot/restore or
=C2=A0 =C2=A0migration. Validated on hardware: guests register kvm-= clock and survive
=C2=A0 =C2=A0repeated bidirectional migrations with zero TSC-unstab= le events (the
=C2=A0 =C2=A0pre-kvmclock baseline reliably degraded on the same hardware).


I=E2=80=99ve got a full submission document (design, per-failure bring-up history, complete test matrix, untested-areas inventory, and security analysis) and the git-format-patch series (against releng/15.0, where they are validated).

I=E2=80=99ve tested many rounds of live vm-migrations across hosts = (AMD using KVM nested virtualization and Intel physical systems) and have finally gotten it to a stable state with 30+ live migrations without packets dropping.=C2=A0 I intend to do further testing (specifically with AMD physical boxes). =C2=A0

Bhyve is phenomenal. If there is no interest in a champion, I still intend to at least attempt to see the process through (acceptance or not). Happy to provide the documentation/requested info.


Thanks for working on this. Live migration patch sets have been proposed a few times before. You can find the most recent attempt sitting in reviews from 2022 ...

http= s://reviews.freebsd.org/D34722
http= s://reviews.freebsd.org/D34811
http= s://reviews.freebsd.org/D34719
http= s://reviews.freebsd.org/D34720
http= s://reviews.freebsd.org/D34721

You should also be able to locate several email threads related to the topic on the public freebsd mailing list archives. I won't rehash that here, but there was resistance. The orignal work for that and other bhyve related projects ( libvdsk w/ qcow2+vmdk support, user mode usb pass-through, etc ... ) were hosted here ...

https://github.com/orgs/FreeBSD-UPB/repositories

You should probably also have a look at this ...

https://www.freebsd.org/status/report-2025-10-2= 025-12/bhyve-cpuid/

From what I gather from his Zagreb presentation, the feature is being developed as a foundational layer to import illumos bhyve migration code with an eye towards feature parity and potential interoperability.

Good luck!

-Matthew

--000000000000c08e9b065464f301-- From nobody Fri Jun 19 06:04:09 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ghRqs708Wz6hcGj for ; Fri, 19 Jun 2026 06:04:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ghRqs45zGz3pSs for ; Fri, 19 Jun 2026 06:04:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1781849049; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kBbW331fPF5mcGWYxmpD57UUfZcXWY8AV2MvO2iVfzc=; b=Yl4piMGyoSq9knTLyOEcSiOkhlKP9ih5OY6P33iQzhx5AUhfnPIysuihzX+w2fEOXTM3KK FzrEHuzg7FFMbOkuh5upZLquIKyIgF/lXe+dGO0nwx9ZIOz8PsF4MDRW2hcjXjP8R+3evk 4m8rF3IpH5SxczcEdxWMzKn7eRrLUeNO9q+Qo3lQVt3RWWw9T8s+Umcxdn4d80B0s+BJE8 e10Sd3GgQqk7weiRq69FpfeckeeHT25kPlAgjxwPIDr4VxeqZ1wPj+kOMrsLGDQlS3kjso 9rDgOXzdwVcixwBOj0tqrastbIv5rhKT33xh0K3OukHK8O9fpbIzea1MoFMdnw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1781849049; a=rsa-sha256; cv=none; b=d5KzdaSoiUKok33bLswa23isoA1veBKXHr9y9zexsuiKumToi7ygdJ5xyA9+6EWmChVqDQ 29OLjAkfhsOIEPLW6KftKxVxCjyiev6RW2JEhv4lkbPYPbOGIxkigffjdqcZJaNmHkQ8p+ X5nphSyiRCSWY6bDn7St4IDvHGTqwo63vt70PoTE/Az0jVRNLFjNC4g94nLsL3cmGN40GA 6P8353PhalII1v7/7BELmefJDo10o7QqJ6Cpl4zstSkZSwBObhv9aNSX50ADDxG78tKyrf XSTarauE9wq1TDi4e7vlmxBAX0hXZAEiykBxr3fzetRhh9+P4738+3iWc9xbpw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1781849049; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kBbW331fPF5mcGWYxmpD57UUfZcXWY8AV2MvO2iVfzc=; b=vUZBcBmFJjjkuM5PvG2H9/QwJ3Bgu7X5ZtQBAw7fTqGNZUSYwRxY8a2wRkpDBR5XvkAaGa k2Tb3dLT7XR8pHkSb+/nnCUMB0nDcmmqKfkpy2VQ4x9i0divhmqhDRwiFMtxyIQqJJC3nT DGHB005iCBqQ5YSB3SY2QJnqzxEyISiFIF/KBV9SL/m7bN8ABz7MfGDwI5Y6vSmObq3wfD IcquSMnTWzKwdsly05xABJEbqblyY9nEzIl4J1ZVeSvsrheB3O1aNTD1o1RPPHvuLAYyIz votoD41zQEOIYudUysmjNf4kUSIqDD+YdLGuOLIJwz6RaOR/VjeCkVdkjWHGjQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ghRqs31H6zZCG for ; Fri, 19 Jun 2026 06:04:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 65J6492S075506 for ; Fri, 19 Jun 2026 06:04:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 65J649pQ075505 for virtualization@FreeBSD.org; Fri, 19 Jun 2026 06:04:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 215258] Update openstack.conf options Date: Fri, 19 Jun 2026 06:04:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: conf X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugmeister@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution assigned_to Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215258 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Status|Open |Closed Resolution|--- |FIXED Assignee|virtualization@FreeBSD.org |bugmeister@FreeBSD.org --- Comment #6 from Mark Linimon --- ^Triage: after careful examination, it seems that all of the requested chan= ges were either a) made or b) obsoleted by subsequent events. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Fri Jun 19 18:11:35 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ghlzF6r0yz6jltQ for ; Fri, 19 Jun 2026 18:11:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ghlzF6Gq5z3gYn for ; Fri, 19 Jun 2026 18:11:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1781892697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+Nd/XijIw46s1pQRQViPI2VNcm0uGS0aSGOPRW7A29Q=; b=EkjV5UOzQNT3iKp8ni5nWnO4n7kr0I3XslQUUVPwGQCs0nuQWvlCGZVWLIm8Ng6tIP/8Cz eokQgbjkDhY5G1E032d/EOC1NOQqKRTYB6SU86U8vSQLTsxdZJ3FepR1PSif/L/ms8jH+T Dn85YRsqZtP8DIxZxbnrp6YR2tJVrKf13LnreDeZd5xvQWH8wNBVmBaBWwxt+cq3uVN/Cb QdTER2P+OWfF/nC8wvFniRxz2auQGZBkKskv7QyPU1lXfHQRlVRpysfhWzRy6wiATVQtcZ upYraAacet75Eys0ADaN8VOeX8maK9aCWBg+PCJjEPMvLd8fDRPkEp9+aNDWvg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1781892697; a=rsa-sha256; cv=none; b=UwJ66/xzUXJwMe2ta1WITUTc6xxtXDvKZRWEI4FNj5xtb3ceiaJKPq6i73v+uD86Xsyj6R 60eIs49581jtHEba6B8DC6AGLiJAD54w+ue1ZbcPEr6MWGjKy4EiixkZ9O2lWCkQO8arsG xzYBWOxJxyd9YhJwQbdR/a7fmL4gd7hiaAg7Wqn4QWefQTnAGLSjD1bNRq7SJqTEhEmouw LTV09tkohF/f5tSMEUxSoEvdP2g82ZEoMJvXKycnnPyTDS71mUJlP9jU9GjThzzF2NGU93 67tzXTEdsD867uQOARZzCQfg0u68ozptBpc2vZ98rrorR15RZpb2H9niUUpd1w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1781892697; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+Nd/XijIw46s1pQRQViPI2VNcm0uGS0aSGOPRW7A29Q=; b=sjcBW4LhsotJfiEQW3IXX0Y4WuxKnFm9DDd1K9vSJxRSiFBMJ8rPRdGCTTxLnoCbczlmJR YmLk6xV65+k7rJOeE1x3Iybp0C4QSkGx0MgdU946IhztfNii9S6Dvx9YcZDBtiUmOwZbRI IaosloynzgPVyw1nJFKC5DAs04rj6e+lfWXgYbRp3FjsjRekSlIqyGGnJZM0IGwfLummmh VeyLlfR1EjPCHmb53MxqCy7/byTHV71EH4+oPhhss0UojZWB9Zge9DR7v83ZVN+J3LeQRU 75GHrGe45Un43LC7UEbvg6NHbEYKrRbKhetjls7PNsLu/bpKLFEoJ+dIrsoD3g== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ghlzF4hnyzwq7 for ; Fri, 19 Jun 2026 18:11:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 65JIBbLQ062620 for ; Fri, 19 Jun 2026 18:11:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 65JIBbNe062615 for virtualization@FreeBSD.org; Fri, 19 Jun 2026 18:11:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 291768] bhyve won't boot anything on 15.0 - AMD EPYC 9124 Date: Fri, 19 Jun 2026 18:11:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 15.0-RELEASE X-Bugzilla-Keywords: regression X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: deimos_bsd@mailbox.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D291768 --- Comment #37 from deimos --- (In reply to ShengYi Hung from comment #35) Hello. The 9950 seems to be a Ryzen model number. Is this correct? If so, I expect this to be the case because the bug only appears on EPYC 90= 00 series processors. My Ryzen CPUs run current linux kernels in bhyve in 15.0 just fine.=20 I have not yet tried the 15.1 release nor the linux 7.x kernels in bhyve. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Jun 21 21:00:48 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gk3dY30Zfz6hmgF for ; Sun, 21 Jun 2026 21:00:49 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4gk3dY012hz3s72 for ; Sun, 21 Jun 2026 21:00:49 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1782075649; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ZVdfTOOxt2p9UZUEafJuzwuEvzmvxRyyOqu/p+cPxhU=; b=i7H5Ef17tkUeO0lKI/FW7Elf9hn1ssbVAm57GdE5IjUL0wcXEezLBSyzIhGMWbAAhlHeXY mj48DX1uZd5e4g6NClRG96Ae7dN4i7Qq17USpQeoOch2GrxfRU6/oiBJMhcFMODKoO6NKE v/7a0wL+HL2fl8EXK1wp6yP01XpwEbMDeNYUY3bvu/DAZ6DBuDebx7SOWwkm41zsNhRzWK CyCjMjFWohS88x2J+OPubplRlmjwzFgnJsFpf8bSrmvnlfIkYrn2flt9Cod55GXvrJLZUm 89P5bq70MzI6h4G9clMK10SsYVWtMCPgXiCTqIKVdsuHTfdsS26D6c5WqxDY3g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1782075649; a=rsa-sha256; cv=none; b=q+K6VXjXqOV5hqdtgmuySCIW8+TyKvye9feQRU+abvSEf6k/zlfLcCfyMcGknaPvcv8IlL sdBQ/hcRb4J0Lg8IO3gnLYalj6jI+pAdaGzoNdOUsvtl03MxNLLhFXXRiLRnpRh1mitT8X lGD1FjuyD/RkMsy5Lqd72D8+rKcKQRvfADCqyFLq9KFxYV5wY/FSCSZjqzlszFpRqzbAzr btORGRK1rH5qz1KTLuLsjTENFJWMkIGYKTNYndY3pqk4ZGX8TxfhJLRhj3MT/AT7KDNsIB Llhr4UT07gRE1oDm9uvffbfkc52de5pVn3xLuhbtDR5oJzZN7Sr4KrRxVLDgPg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1782075649; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ZVdfTOOxt2p9UZUEafJuzwuEvzmvxRyyOqu/p+cPxhU=; b=mHa8w0M4gt7gUjyfXB1Q1dMlKH91UnQqOwZepbzvK1YfkbjbjxYApAVv3Ie79VX+wJhltt JmfIAdKGPqRONvdhsSsbiNMSH7HlSmUHxGk0iPEyzC2YLZbuQgzEcQM3h0sQoCBR5SomoS uWUFqMgJ+ygSR7u666U1Wtj30BPizSuWyd2XUAv2hIeMWBw3qgopgstusEeahP0jxjlOMm AgT5iwiiEJQZSpi5zLrY36yRr/9mH/2Djrs8tyjeAAuymDMfUkj1TsT5+MgcFT9FkAt5+u vQRt+KRZzuaeapP6PO7W/kRMrtY6oERbs9/6vJ4ZKPzYwFKko9fLNCrFka8Seg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4gk3dX6BBqz2xL for ; Sun, 21 Jun 2026 21:00:48 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 65LL0mqf009638 for ; Sun, 21 Jun 2026 21:00:48 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 65LL0mSL009637 for virtualization@FreeBSD.org; Sun, 21 Jun 2026 21:00:48 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202606212100.65LL0mSL009637@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 21 Jun 2026 21:00:48 +0000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17820756487.46c68a8.5265" Content-Transfer-Encoding: 7bit --17820756487.46c68a8.5265 Date: Sun, 21 Jun 2026 21:00:48 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 290735 | bhyve resume hangs New | 289848 | Enable by default BHYVE_SNAPSHOT 2 problems total for which you should take action. --17820756487.46c68a8.5265 Date: Sun, 21 Jun 2026 21:00:48 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
In Progress |    290735 | bhyve resume hangs
New         |    289848 | Enable by default BHYVE_SNAPSHOT

2 problems total for which you should take action.
--17820756487.46c68a8.5265--