From nobody Thu Oct 9 16:28:17 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cjFhF3kMqz6Bjbl for ; Thu, 09 Oct 2025 16:29:33 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cjFhD5b8lz427H for ; Thu, 09 Oct 2025 16:29:32 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b="bQw/aYjq"; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of bogorodskiy@gmail.com designates 2a00:1450:4864:20::531 as permitted sender) smtp.mailfrom=bogorodskiy@gmail.com Received: by mail-ed1-x531.google.com with SMTP id 4fb4d7f45d1cf-634cef434beso3525654a12.1 for ; Thu, 09 Oct 2025 09:29:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1760027363; x=1760632163; darn=freebsd.org; h=content-disposition:mime-version:message-id:subject:to:from:date :sender:from:to:cc:subject:date:message-id:reply-to; bh=Gllk/JMkI2lQSxJNIQLaNZdgHoWugal9zOBf2AQERws=; b=bQw/aYjqRAOe5sS+SjvjoRxIC+/MF0JXnloKOMkbJl6Yc/nvPlWrVtlzFH1RY7WbCX ZstihZRAUF7mlF2Gq0b49mg8P9CCTfBopniv1/I8wgc660ELwS1VvuyIfwCsWTD1t6pt ZfcbdgI/fVglqmO6AuIKL18JRuecg69AREL3PaQNVnfII0WbVh8RRVdHKJ9XD5DyUrGA oyAwIQNaSFkeMbj3NHuhsvxYTev1a+2RUy0bBFJl9Sn3gHNR5S9ekNl8lyUj3+aPoXmq uSZR+P4swn2LYyQv7LV7Zd6CCs8Pers6Kmg49sCv6wvMsBgySCDc9xX8KG4s9uUuCP6q pQNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1760027363; x=1760632163; h=content-disposition:mime-version:message-id:subject:to:from:date :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Gllk/JMkI2lQSxJNIQLaNZdgHoWugal9zOBf2AQERws=; b=URv8tPqx1JpXjAA8lHY3fWd2rytTMRXOeQs0WfFQlGLhP54GaRkD2TiUTbFCEVefXq MMlJaS6OBFdWLNtrtxFgudHwPERlo9ZJR2lALzcLzfHaaCde+3k46Kqr4ipdJ74JgZkj QKOMCXs70/Xoy7JxeOU8INvk0V6U1PvXtWAcq58Ut5efOC+z+XzsCkMSP4DTNPAFs7qK BiSQM6WLWTmcO4qvVhcEirCB46MbWN5TNXhX3vBq1/YCVlWRYg5KikQq0mSfYuOBCenu vYcb9+jDkxmidbQ9/mVWuukcaLc3jBt0F/yyEmgyOKjp32t3SlWUxDxYzJnrVWXoGjOl fTaA== X-Gm-Message-State: AOJu0YwkpYGeLxqp0i8ggbz/7z5fJ9GnuNBNg8YhH0JUd2QwhKG1EFEt SL0Iw7+HlEIatNS+vaI7ZZLhzgOlwbacdWnmoHl91+rPpNynDYHpp3BaXBKkcfcHwxaWUw== X-Gm-Gg: ASbGncvj2jrL6SZ26eSJB+C99yeXwv3W0iPi86EhsE8mfOsGTp/bjsYv/eKDmCgrPjI U43KfrUFoaqPO00CkyZXqsvmEi3X1Cc9VlyJjCL0t4HXgDGWovJ/52QRe9OM4bZfDNGuoSxWq+i KCmIfaOa9RYyZ5eFCI/oYAYcNwcBhk39oaPTVDCujiq342NkOyeASOBgFoPoMGkCCc23VMuKtFv y3csUBMrAqi+AhaQ6Z6XczvASEaTISDwzBzBH2L1W+YAYMJGMS3hmCL/b3DSIaPTNW8qHCOocvi j9gef5z0cxgHL8fzbCO41J2WCIi5Y4B2YTnJNZKmEP5YvdmkNlvGE8Oj1QtehEZDMhpFde2HyrN ExNiGpzbyPgL7H/auwhfEGfLe4EArkJV1GKu1TLeTbogWbuFgq1XM1mVTQbhlTddY+2HAqIgBbw == X-Google-Smtp-Source: AGHT+IHXYQbMTnfTIus2EvieOtmFijeD40mK8kDcVD8W8BSj7LtO7VKS2Sg3o9n/fYrUYQW9FD2ZPA== X-Received: by 2002:a17:907:d7ca:b0:b3d:a295:5445 with SMTP id a640c23a62f3a-b4f4116a343mr1668683066b.13.1760027363064; Thu, 09 Oct 2025 09:29:23 -0700 (PDT) Received: from tulp (84-25-144-101.cable.dynamic.v4.ziggo.nl. [84.25.144.101]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-b55d67d2ce9sm11051766b.35.2025.10.09.09.29.21 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Oct 2025 09:29:21 -0700 (PDT) Date: Thu, 9 Oct 2025 18:28:17 +0200 From: Roman Bogorodskiy To: virtualization@freebsd.org Subject: Suspend/resume question with Ubuntu guest question Message-ID: List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.56 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.964]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[novel@freebsd.org,bogorodskiy@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), DKIM not aligned (relaxed),none]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_NEQ_ENVFROM(0.00)[novel@freebsd.org,bogorodskiy@gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_DOM_EQ_FROM_DOM(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::531:from] X-Rspamd-Queue-Id: 4cjFhD5b8lz427H Hi, I'm trying to get suspend/resume working without success so far. My setup is: - 16.0-CURRENT #0 main-n280778-f45608124286: Sun Oct 5 10:30:44 CEST 2025 - Ubuntu 25.10 guest I run the guest with: bhyve -c 2 -m 4096 -A -I -u -H -P -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -s 2:0,xhci,tablet -s 1:0,lpc \ -s 3:0,ahci,hd:/data/img/ubuntu2510.img \ -s 4:0,virtio-net,tap5,mac=52:54:00:02:93:f1 \ -s 5:0,fbuf,tcp=127.0.0.1:5955 -l com1,/dev/nmdm200A ubuntu2510 Guest starts fine and works as expected. I suspend it using: bhyvectl --vm=ubuntu2510 --suspend=/var/run/libvirt/bhyve/ubuntu2510 That also goes well, bhyve terminates, VNC and network connections drop, no instance in /dev/vmm, and files are there: # du -h /var/run/libvirt/bhyve/ubuntu2510* 4,0G /var/run/libvirt/bhyve/ubuntu2510 32M /var/run/libvirt/bhyve/ubuntu2510.kern 4,0K /var/run/libvirt/bhyve/ubuntu2510.meta Then I resume the guesting using the following command: bhyve -r /var/run/libvirt/bhyve/ubuntu2510 \ -c 2 -m 4096 -A -I -u -H -P -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -s 2:0,xhci,tablet -s 1:0,lpc -s 3:0,ahci,hd:/data/img/ubuntu2510.img \ -s 4:0,virtio-net,tap5,mac=52:54:00:02:93:f1 \ -s 5:0,fbuf,tcp=127.0.0.1:5955 -l com1,/dev/nmdm200A ubuntu2510 Here issues begin: I can see the bhyve process running and there is an entry in /dev/vmm for this guest, but it does not respond to network connections. I can connect via VNC but it just shows a static image of the screen at the point when the guest was suspended. I don't seem to see any related error messages either, except probably: vatpit_snapshot: snapshot restore does not reset timers! In /var/log/messages for both suspend and resume events. Another interesting thing: if I destroy the guest now, it prints: fbuf frame buffer base: 0x1d60d9600000 [sz 33554432] Pausing pci devs... Restoring vm mem... Restoring pci devs... Restoring kernel structs... Resuming pci devs... And then exits. Am I doing something wrong/using incompatible device configuration, or is that a bug? PS Also checked with a FreeBSD guest: same result. Thanks, Roman