From nobody Sat Jul 18 11:22:52 2026 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4h2PXD6bmLz6lK4H for ; Sat, 18 Jul 2026 11:22:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4h2PXD5jlJz3dsN for ; Sat, 18 Jul 2026 11:22:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1784373772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=53pwefgdSbIvztmuu9z6fjdcw/lJzqOP8ne2XQsR5Fw=; b=H2aPw4mScfmBUo8o6/OwfD9/sMzQxl42SJzS1asTGIpANH6Omp1wLasLRPnrjoAWBaVu5T MhhWWWrcvSFgEx/Cme1kehzfpfe5Vrb1U9ysPlbvk4QwmlEaqAlXDASTqk7CCXFVLT5arl 5XeacbHO7E8wDiuDKqLO+W2JsSakGfltHyPLkq0XhJk8kDsJKy2yzDqTl3cpk0ivsHb5+l 1HuGJ/fdU0CMmo6q7I7HcneoIAiK/DKrzD3zlkOAkALa7d9OPr5MQ3rc9dczGBnbqLRAdf YsCa987IWhnJJ1GfWZFU/DCeWzS9SLjJGc4LdanAaAeTEpMGhqI4hITTmJ+HXA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1784373772; a=rsa-sha256; cv=none; b=cMlF4JLaosEayfFm0VAHlh26jUqt1Lkzzzsye+s3jklIszefJQ3euVRL/Zj+IW3dUuNhSU RMlOj5rs4CjB3a6WPr2eoSb6zGEWJZWfbGJi0GF/k7t4Cxm+iwmysGGXuf4kW+KUephUwk RB0YBpJOq+rkbWWirr3eXdAHh1iElODVN9xAgtXiEQF6Yp/+Aq0AnAXrc4ogUBj3Bi1NfT GssbmLnnNYlXX4a1ghwpfyEuRwOj3pR0tfL5IDyu75a+THt1OcGhgIuAUq79RVVEdsSpOS 9/5fHksOk+pfAtef0mxXoC2L+C+MA4ZlZFS+S0HhGh9QKn4IYnP6/hGmieqm4A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1784373772; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=53pwefgdSbIvztmuu9z6fjdcw/lJzqOP8ne2XQsR5Fw=; b=JPepS4wnQLh8omEYI9ifarRj7qok2Tmtmc1PNqRHwc91liQqe0kScibwGJLcMO8iEklOcb GcTs64APSTTmPmQbMXX+nMnrXL5ToXed1iiuayNz3Zgo7KFplmcYZmG7ect0Josg0xEJk9 Nf0PSuZGg52CM75fNljZ2yrjxtP3hWGydyXWOWIrhslxe+ggw+njKZNADY+eggiZFV/384 wN9NzCvmOHdmFQDFBP0/tNwsRzl90oyDdkX7BlLxn+JALULDNQqoHN2gy3VlfbwWI5YR7z T1Y19LG/XcQp6lOmRsbm26W404sKuQ1jIN+eJWAsT4dP9DKsYI8kkjUac46FGQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4h2PXD3sgSzTW7 for ; Sat, 18 Jul 2026 11:22:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 66IBMqOi038629 for ; Sat, 18 Jul 2026 11:22:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 66IBMqfs038628 for virtualization@FreeBSD.org; Sat, 18 Jul 2026 11:22:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 296872] bhyve guest hangs at start_init (100%+ vCPU spin in lock_delay) on AMD Ryzen (Zen2/SVM), 15.1-RELEASE host; identical guest boots on 14.3 Intel host Date: Sat, 18 Jul 2026 11:22:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 15.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ozgur@kazancci.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D296872 Bug ID: 296872 Summary: bhyve guest hangs at start_init (100%+ vCPU spin in lock_delay) on AMD Ryzen (Zen2/SVM), 15.1-RELEASE host; identical guest boots on 14.3 Intel host Product: Base System Version: 15.1-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: ozgur@kazancci.com Created attachment 272954 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D272954&action= =3Dedit The zip file contains: 01-host-info.txt - 02-bhyve-cmdline.txt - 03-bhyvectl-getall-1 - 03-bhyvectl-getall-2 - 03-bhyvectl-getall-3 - 04-guest-console Overview -------- A FreeBSD 15.1-RELEASE amd64 guest under bhyve reaches userland (the kernel prints "start_init: trying /sbin/init") and then hangs permanently. The bhyve proc= ess pins ~100% CPU per vCPU (~213% for 4 vCPUs, ~120% for 1 vCPU). The guest never reaches multi-user; no login prompt, no network. The identical guest disk image boo= ts normally on a different host (Intel Haswell, 14.3-RELEASE), isolating the problem to the 15.1 host's vmm/SVM path on AMD. bhyvectl shows the guest spinning in lock_delay() with millions of intercep= ted PAUSE instructions and no interrupt-window activity =E2=80=94 the classic signatu= re of a waited-for timer/interrupt never being delivered to the guest. Host environment ---------------- - Host: FreeBSD 15.1-RELEASE-p1 amd64 (GENERIC) - CPU: AMD Ryzen 7 3700X (Zen2), Family=3D0x17 Model=3D0x71, AMD Features2=3D<...,SVM,...> - vmm.ko loaded; managed via vm-bhyve 1.7.4 (also reproduced with 1.7.99.87) - hw.vmm.svm.num_asids=3D32768, features=3D1293567, vmcb_clean=3D1023 - hw.vmm.amdvi.enable=3D0 (no PCI passthrough / IOMMU in use) Guest ----- - FreeBSD 15.1-RELEASE amd64 GENERIC (official 15.1 dist sets; also repro f= rom the stock 15.1 bootonly install ISO's own installer kernel). - loader: bhyveload (NOT UEFI -> not an edk2 firmware issue). - Reproduced with disk =3D nvme AND virtio-blk; backing =3D zvol AND raw fi= le; cpu=3D4 AND cpu=3D1. bhyve invocation (verbatim, from vm-bhyve) ------------------------------------------ bhyve -c 4 -m 8G -AHPw -U -u \ -s 0,hostbridge -s 31,lpc \ -s 0:4:0,virtio-blk,/zroot/vm/web1/disk0.img \ -s 0:5:0,virtio-net,tap0,mac=3D... \ -l com1,/dev/nmdm-web1.1A web1 (-H yield-on-HLT and -w ignore-unimplemented-MSR are both present.) Steps to reproduce ------------------ 1. On the AMD Ryzen 3700X / 15.1-RELEASE-p1 host, create a FreeBSD 15.1 amd= 64 guest (bhyveload loader, virtio-blk or nvme disk, virtio-net). 2. Boot the guest. Actual result ------------- Kernel loads, probes devices, mounts root, prints: Trying to mount root from ufs:/dev/ufs/rootfs []... start_init: trying /sbin/init ...then NO further output. bhyve spins ~100% CPU per vCPU indefinitely. No /etc/rc output, no login, no network. Guest is unresponsive (destroyable via bhyvec= tl --destroy). Expected result --------------- Guest completes /etc/rc and reaches login (as it does on a 14.3 Intel host = with the same image). Diagnostic evidence (bhyvectl --get-all, twice ~4s apart while hung) ------------------------------------------------------------------- rip[0] 0xffffffff80bd77b2 (sample 1) rip[0] 0xffffffff80bd77b0 (sample 2) -> RIP oscillates within 2 bytes =3D=3D tight PAUSE spin loop. -> nm on guest kernel: 0xffffffff80bd77a0 =3D lock_delay() =3D> spinni= ng in lock_delay(). number of vm exits due to exceptions : 0 -> 0 (rules out MSR/#GP loop) vm exits due to external interrupt : 11390 -> 13712 (~580/s) number of times pause was intercepted : 9415553 -> 10025216 (~600k/s) vm exits due to interrupt window opening: 0 -> 0 vm exits due to nested page fault : 11657 -> 11657 (no growth) Interpretation: ~600k PAUSE/s with RIP fixed in lock_delay() =3D> guest stu= ck on a spinlock; exceptions=3D0 =3D> not an MSR loop; interrupt-window=3D0 with a = lock owner waiting =3D> a timer/callout interrupt is not being delivered; no forward progress. Cross-check / isolation (key data point) ---------------------------------------- The SAME 15.1 amd64 guest image, created with the identical procedure, boots CLEANLY to a login prompt on: Intel Core i7-4790 (Haswell, VT-x), FreeBSD 14.3-RELEASE-p15. =3D> Guest image/config is correct; the fault is on the AMD/15.1 host side. Additional observations ------------------------ - Reproduced with cpu=3D1 (rules out AP-launch/SMP-startup-only bugs; point= s at the interrupt/timer delivery layer). - Warm reboot AND cold power-cycle of the AMD host do not change behaviour (expected for a deterministic software issue; argues against stuck hardware/firmware state). - Preceded by guest-side "nvme: Resetting controller due to a timeout" duri= ng heavy fsync I/O =E2=80=94 a SYMPTOM (I/O completion interrupts not arriving), n= ot the cause; reproduces afterwards on every boot including with virtio-blk (no nvme). Workarounds attempted that did NOT help (guest still hangs in lock_delay) ------------------------------------------------------------------------- guest hw.x2apic_enable=3D0; guest kern.eventtimer.timer=3DHPET; cpu=3D1; nv= me & virtio-blk; zvol & raw file; vm-bhyve 1.7.4 & 1.7.99.87; vmm kldunload/kldload. (hw.vmm.svm.num_asids=3D1 made the guest fail to run entirely =E2=80=94 inc= onclusive.) Probable root cause ------------------- A regression in the 15.x vmm(4) AMD SVM path delivering a timer/APIC interr= upt (or in event-injection / interrupt-window handling) to the guest. The guest reaches early userland, a thread blocks on a callout/timer, the interrupt is never inject= ed, the lock is never released, another context spins in lock_delay() forever. The = 15.x vmm refactor is known to have produced at least one AMD-relevant regression (FreeBSD-EN-25:20.vmm, PCI-passthru IOMMU); this appears to be a separate regression in the same subsystem on the non-passthru interrupt path. Willing to test patches / a rebuilt vmm.ko and provide additional bhyvectl --get-all dumps or a bhyve -G gdb-stub session on request. --=20 You are receiving this mail because: You are the assignee for the bug.=