From nobody Thu May 18 11:36:01 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QMSbQ3qrcz4BSBm for ; Thu, 18 May 2023 11:36:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QMSbQ2bL7z3w9C for ; Thu, 18 May 2023 11:36:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684409762; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rb/Yw+JBi0YyrpHVh3VmiAQrTaT0WuMW6kkBRYGOZbQ=; b=gpe9pLIwNSJR8IJyXvfZAfotBkK8Fx+xLsFp6evbxRB3/aH6vewFnIe6n2Zl1rTq8g579n lIy82pj/m9xgWjpcBJ6XLUKYaeSJfmIobKDYfaZSShPuSKl2LELQrCTPcQH3mZp2Jt9mnA TMUXndg2BoZGqCVxo7wIwPCK/Ls6wsG/RF5tS3Dg+j7GUegVIyjJ0iZzRu4G4yeV2ngd4F saW++8PNF9eCjHSwFpn5t3PMla1ntnaO/F5SZ6Od5B6dLEXKfGYoDOtekCJO5BjIoJP5i8 ybQXUwNAUe9Dsl+2+5Vr4shXfNAYZlLP0LqEOXrIO9grs732r/ATKwmlkI4tKA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684409762; a=rsa-sha256; cv=none; b=lEaHwaZth4Z0DO+Gh+CmCk36tfM1pIpmikzVoBrtM0rbYhR9IwTGuY3so+jspv4bF/bVzg C459XHyHYD/HK6jd0OC5AKNWSEKdrr0tmnbq2dAeSqn/tmSUZni850uXd7EfY/Dp3AeXo9 qVDLAHS/Gf8nnKMWWejv4frVhlIzsGt5RTcjaIoHSsbNmPyUroGwC/AVaP7TgY08yJSiks cq4h0ECuKOUAvIwuSmwvoTp/S+3fVHS77jRDjCTcGwHjYc5n7eq6h24MIz2HEH9Zf5EtBh 42TLpYpAcqJDsgI7h8D/zEWHS+hGCw5YjFoo7M5Lm3BDZw1GjrJx0g68yugyGA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QMSbQ1ZvnzkLl for ; Thu, 18 May 2023 11:36:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 34IBa2jt031317 for ; Thu, 18 May 2023 11:36:02 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 34IBa2PA031316 for virtualization@FreeBSD.org; Thu, 18 May 2023 11:36:02 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 271484] Guest OS Rocky Linux 9.1 freezes sporadically Date: Thu, 18 May 2023 11:36:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bubnovky@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271484 --- Comment #6 from Konstantin --- (In reply to Vitaliy Gusev from comment #5) Okay, I'll try, I'll report the result. Thanks a lot --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun May 21 21:00:03 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QPXyr1JLhz4CT0Z for ; Sun, 21 May 2023 21:00:04 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QPXyq6qWfz48hw for ; Sun, 21 May 2023 21:00:03 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684702804; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=J6n/geNki3/D0DeILztDjvWOpU4DZlX2aNYIqb7QAwM=; b=idjeQSDX8unevFQKp26HCqnjW7yXlzcbCt8BC0E1y0q5vR8x/plI604gB8aLelQPshcb8n bRwXHx4Iklpy8PPKbG9aK6x4PDImnM5bNZM1VNhGUkQvI0oPS5nC8eOJPByIzlLwYP/vPN wLrZgLecWcgCH0Arv3CK0DjgNIodDzmqDaLIbYZZisETdjnPIxovJzqmfztYyFiACLHiCY yQ9wvxm9pMmRqCBLMMkLrYNYPRp510BuZFv80w+Pa/X4busocSErXpaqK6lEapaulv74dC jf+EtJs2ahNDEpmwnR5Hr/FHeR8O0Wxfsq7fpOjP6652znpSik88fsBFzx3PWg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684702804; a=rsa-sha256; cv=none; b=i6/Muc0V/C3J/1rW/0WfUjFRLTGvFtN6CdibheE9wPdnFgrYH1hr5+4tYxvnSOTFQh/Rtp CZ8B1HlveaApVfYPY6FwsJCNzNPkT1ZFlym6mU73ZNIRI71gOyermCl+9Sjy5VS+cK7/tB 44Oe6atZIJcolibtpoRPLbMxrUxDKhvVchrkLsx+1Ha1nuqs6RSfqTdtIB31pyjdk9bciX br6YriY/KI0RMwIfLKozPGOr6pNo9eZFlHq1yu70fuAuS/zkwpgSbVH5MhOeaFJGECZ4Qd hGkWdoOQFDV6UUkO3os/umiFc60HnBHjwto0/OF17ignrn/DJ0Z8gJ8EjSwsuw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QPXyq5wr4z15WS for ; Sun, 21 May 2023 21:00:03 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 34LL033B039081 for ; Sun, 21 May 2023 21:00:03 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 34LL03Nq039074 for virtualization@FreeBSD.org; Sun, 21 May 2023 21:00:03 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202305212100.34LL03Nq039074@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 21 May 2023 21:00:03 +0000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16847028035.c349d1dF2.37781" Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N --16847028035.c349d1dF2.37781 Date: Sun, 21 May 2023 21:00:03 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 247208 | mpt(4): VMWare virtualized LSI controller panics New | 240945 | [hyper-v] [netvsc] hn network driver incorrectly Open | 244838 | "bectl activate -t" does not honor the -t flag in 3 problems total for which you should take action. --16847028035.c349d1dF2.37781 Date: Sun, 21 May 2023 21:00:03 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
In Progress |    247208 | mpt(4): VMWare virtualized LSI controller panics 
New         |    240945 | [hyper-v] [netvsc] hn network driver incorrectly 
Open        |    244838 | "bectl activate -t" does not honor the -t flag in

3 problems total for which you should take action.
--16847028035.c349d1dF2.37781-- From nobody Tue May 23 06:05:13 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQP1P2gDSz4T62M for ; Tue, 23 May 2023 06:05:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQP1P0HlFz42ZR for ; Tue, 23 May 2023 06:05:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684821913; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iecAJnrK/+oN2FbxpIXtuQ4FZ/IOL/AqMqhkR+lXAH8=; b=WjgmpVOR+/JOhOJ2bXeeAillavAs/s8wmtNGOOFlDPx8RpyvmGv02+B+LliVwnS9EEIa7G wV5/hQ2rkOrwGhb7M3t6Uy6U1Iz+8EB+I5kNqlizFrii9gZuc4PCZFBOkrHn2VJOSccPd4 /+n6wx+AuHezM/kC+vmJHp0kZjiEtjLNj9eoTBEUJDveKpoQ66/fnVjRv6Nd4TrFVapbRl doSmWp2oLl61b88ebqMTtHjmj1HdnkMGW5e0DL/teQUKS/38LRxCDeja4ArMjuNBEZtvh9 VsL46vUBlQmTtOSUrecFwWQrwVCOb5lpNmJmT8zxwonB6U1PcjubddkzNHI23g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684821913; a=rsa-sha256; cv=none; b=wtCZDBwIK3TVIzIsjHrNFlPi6RIuCTyJX9MuXSBGjaz4siA+QrGDqpv9/XlOibXOAZMR6K R04U07MvbTTPuDlsK5Wgnl8VL8zfwKVCD5kwjRLWLtQgk1g2UlFnl+ENqKVK0x75NwUkcc fSSI3LsOnEE8RdmvhJaV+zKBRyoKa3NHK4UBY+GtfvFQBKS/Z03m7si7Tra1jZ+hYH+bAd 7M4r0AjAY5WUsBjk3Vsj0LCUGOQeJgbnb0vuQ1Bg3NheRZLh8RrJ0LFEZrAOn6tRmZZ1hB O/veSmO8Hvs3CxkCJmu6CzVF8R8we/nzIuXpdLLoVBnVxo41RLMPe998Hf4wog== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QQP1N6Sp3zpQL for ; Tue, 23 May 2023 06:05:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 34N65CcP045821 for ; Tue, 23 May 2023 06:05:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 34N65Cgj045820 for virtualization@FreeBSD.org; Tue, 23 May 2023 06:05:12 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 269872] system freeze/crash with kldload vmm.ko Date: Tue, 23 May 2023 06:05:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: bhyve, crash, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: z4m40i@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D269872 --- Comment #2 from z4m40i@gmail.com --- I installed a "minimal" FreeBSD-14.0-CURRENT-amd64-20230518-743516d51fa7-263002-memstick.img and tr= ied kldload vmm.ko , still error. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue May 23 09:39:35 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQTmp3Lncz4CFlf for ; Tue, 23 May 2023 09:39:38 +0000 (UTC) (envelope-from fbsd-lists@dudes.ch) Received: from mail.dudes.ch (mail.dudes.ch [193.73.211.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.dudes.ch", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQTmn5vhxz3BwS for ; Tue, 23 May 2023 09:39:37 +0000 (UTC) (envelope-from fbsd-lists@dudes.ch) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of fbsd-lists@dudes.ch has no SPF policy when checking 193.73.211.25) smtp.mailfrom=fbsd-lists@dudes.ch; dmarc=none Received: from amd64.dudes.ch (amd64.dudes.ch [193.73.211.16]) (authenticated bits=0) by mail.dudes.ch (8.16.0.41/8.16.0.41) with ESMTPSA id 34N9dZhn086397 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Tue, 23 May 2023 11:39:36 +0200 (CEST) (envelope-from fbsd-lists@dudes.ch) Date: Tue, 23 May 2023 11:39:35 +0200 From: Markus Wild To: freebsd-virtualization@FreeBSD.org Subject: bhyve with qemu-ga (guest agent) Message-ID: <20230523113935.5dd1c779@amd64.dudes.ch> X-Mailer: Claws Mail 4.1.1 (GTK 3.24.34; x86_64-unknown-freebsd13.1) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.83 on 193.73.211.25 X-Spamd-Result: default: False [-1.67 / 15.00]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.992]; NEURAL_HAM_LONG(-0.91)[-0.906]; NEURAL_HAM_SHORT(-0.67)[-0.671]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8758, ipnet:193.73.211.0/24, country:CH]; MLMMJ_DEST(0.00)[freebsd-virtualization@FreeBSD.org]; R_DKIM_NA(0.00)[]; R_SPF_NA(0.00)[no SPF record]; DMARC_NA(0.00)[dudes.ch]; FROM_EQ_ENVFROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4QQTmn5vhxz3BwS X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N Hello, I'm trying to use qemu guest agent to for example make use of freeze/thaw for consistent live zfs-snapshots on the host (to backup VMs). qemu-ga wants to use a specific virtio-serial port by default, so I tried to start bhyve using: bhyve -c 2 -m 4G -Hwl bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -U 54ac18e6-f643-11ed-a64e-7085c2b3862f -s 0,hostbridge -s 31,lpc \ -s 4:0,nvme,/bhyve/test2019/disk0.img \ -s 5:0,virtio-net,tap2,mac=58:9c:fc:04:aa:53 -s 6:0,fbuf,tcp=0.0.0.0:5901 -s 7:0,xhci,tablet \ -s 8:0,virtio-console,port1=/bhyve/test2019/org.qemu.guest_agent.0 \ test2019 this causes windows to show "VirtIO Serial Driver" under "System Devices", but bhyve on the host does not create the Unix Domain Socket at the requested path. I also don't know how to query the Windows namespace to check whether the proper named pipe has been created. When I try the same thing with an ubuntu20 VM, it's looking for the port at /dev/virtio-ports/org.qemu.guest_agent.0, but no such directory even exists within /dev. So: - is the bhyve virtio-console device something different than virtio-serial? - why does this device not open a unix socket on the host? BTW: if I use isa-serial, I can communicate with the qemu-ga, but I'd prefer to get it working with the default method. Cheers, Markus From nobody Tue May 23 16:05:31 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQfLL6j8bz4CdWY; Tue, 23 May 2023 16:05:46 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQfLK6kZmz4Nwf; Tue, 23 May 2023 16:05:45 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=N7L+vi5t; spf=pass (mx1.freebsd.org: domain of gusev.vitaliy@gmail.com designates 2a00:1450:4864:20::22b as permitted sender) smtp.mailfrom=gusev.vitaliy@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-x22b.google.com with SMTP id 38308e7fff4ca-2af29e51722so239631fa.1; Tue, 23 May 2023 09:05:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684857943; x=1687449943; h=to:cc:date:message-id:subject:mime-version:from:from:to:cc:subject :date:message-id:reply-to; bh=b1A/pwsrBKPrpnOnXMwIjxXuyJjG5a3N5tKA4NqjFPQ=; b=N7L+vi5taGmxOj5jG7/O1uHMQfELuW/wFxufMcpBkonKduQpO8whXPRE3hn22ObW5k JXkBqNomIQxm2252iSo3XlM8FHNzyZtQIpZ3z4JHvcB6bYlyZraH4QWkbV4pyhw4BwU3 JtQma1/grVgnpkmGWBMSiTaGuWnErU7FlyetBwN57UFpPyBNMISWWWhRD0lEBD0707je ElUw57mig33AKaU5XTPoftZDhjXB/ekfATI/6ZxsBYagBRf+Xy+LpPVDFeq8CdRVmC1a V4o7RqDX+DDkxT8MHECol38JaR2x2XwGjTLk39xioXXfFREBA1HtpoH7FStnGvQ+Ovdc mRhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684857943; x=1687449943; h=to:cc:date:message-id:subject:mime-version:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=b1A/pwsrBKPrpnOnXMwIjxXuyJjG5a3N5tKA4NqjFPQ=; b=b9oRNixYJqKqQqwyHStWe/GbGm50U09CP7GP6aQvIt6oGYqdfrFoglVVQL2yJ+yezW bQ4kgB636Txp/nWAoAbZzs/r5FIPdMQ6vIue/SAfQ4Jobx0nGXH4OxZjzESyYskjL0CQ ocWsESPp3Tt6gjIM7b+xRy+fqu69AvEkg+9sXr0alrKhBtDMoEN9OWHR/z2UZY3/Zrl9 QKBzezVkelAN0itf2bZKe32HuYgFPmS+96x0L69KxMJfLiBs3VZTJEYUGRXUBLPr7G5D gsSzsJNq0bE4nmM/aEYPyx258MsJMymfpxk0RpSYN7GU5twoai4uAoOIi+9Dgo3Xbiry /HAQ== X-Gm-Message-State: AC+VfDy2oC69NAaN0MITqUach21OR41UbgAxl6fmPhGm0oCtJkW3pa1H zT4cRmxMVrEYpw6fiyhDU7UFiZzaBCA= X-Google-Smtp-Source: ACHHUZ7Ht9U8+nVs76wvqwgPUZr66GEma0XvKqoh13Sa+5eDUOrMjmxnF6bsoG8r1LYeSMmwyoLWRQ== X-Received: by 2002:a2e:a318:0:b0:2a8:a6a9:4303 with SMTP id l24-20020a2ea318000000b002a8a6a94303mr5184581lje.8.1684857942775; Tue, 23 May 2023 09:05:42 -0700 (PDT) Received: from smtpclient.apple ([78.140.234.98]) by smtp.gmail.com with ESMTPSA id l7-20020a2e3e07000000b002af25598f07sm1659025lja.78.2023.05.23.09.05.42 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 May 2023 09:05:42 -0700 (PDT) From: Vitaliy Gusev Content-Type: multipart/alternative; boundary="Apple-Mail=_574E1CEA-F86F-46F0-AC0E-FD372008C2F5" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: BHYVE SNAPSHOT image format proposal Message-Id: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> Date: Tue, 23 May 2023 19:05:31 +0300 Cc: freebsd-hackers@freebsd.org To: virtualization@freebsd.org X-Mailer: Apple Mail (2.3731.500.231) X-Spamd-Result: default: False [-3.44 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.945]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; BLOCKLISTDE_FAIL(0.00)[2a00:1450:4864:20::22b:server fail,78.140.234.98:server fail]; TAGGED_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22b:from]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org,freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4QQfLK6kZmz4Nwf X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_574E1CEA-F86F-46F0-AC0E-FD372008C2F5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, Here is a proposal for bhyve snapshot/checkpoint image format = improvements. It implies moving snapshot code to nvlist engine.=20 Current snapshot implementation has disadvantages: 3 files per snapshot: .meta, .kern, vram Binary Stream format of data. Adding optional variable - breaks resume Removing variable - breaks resume Changing saved order of variables - breaks resume Hard to get information about what is saved and decode. Hard to debug if somethings goes wrong No versions. If change code, resume of an old images can be passed, but with UB. New nvlist implementation should solve all things above. The first step = - improve snapshot/checkpoint saving format. It eliminates three files = usage per a snapshot. 1. BHYVE SNAPSHOT image format: =20 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ | HEADER PHYS - 4096 BYTES | = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ | | | DATA | | | = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ 2. HEADER PHYS format:=20 0 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+=20= | IDENT STRING - 64 BYTES | 64 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ =20= | NVLIST SIZE - 4 BYTES | NVLIST DATA | 72 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ | | | NVLIST DATA | | | 4096 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ IDENT STRING - Each producer can set its own value to specify image. NVLIST SIZE - The following packed header nvlist data size. NVLIST DATA - Packed nvlist header data. 4KB should be enough for the HEADER to keep basic information about = Sections. However, it can be enlarged lately, without breaking backward compatibility.=20 3. NVLIST HEADER consists of Sections in the following format: Name - string Type: string: =E2=80=9Ctext, - plain text, =E2=80=9Cnvlist=E2=80=9D - packed nvlist, =E2=80=9Cbinary=E2=80=9D - raw binary data. Size - Size of section - uint64 Offset - Offset in image format - uint64 Predefined sections: =E2=80=9Cconfig=E2=80=9D, =E2=80=9Cdevices=E2=80= =9D, =E2=80=9Ckernel=E2=80=9D, =E2=80=9Cmemory=E2=80=9D.=20 4. EXAMPLE: IDENT STRING: "BHYVE CHECKPOINT IMAGE VERSION 1" NVLIST HEADER:=20 [config] config.offset =3D 0x1000 (4096) config.size =3D 0x1f6 (502) config.type =3D "text" [kernel] kernel.offset =3D 0x11f6 (4598) kernel.size =3D 0x19a7 (6567) kernel.type =3D =E2=80=9Cnvlist" [devices] devices.offset =3D 0x2b9d (11165) devices.size =3D 0x10145ba (16860602) devices.type =3D "nvlist" [memory] memory.offset =3D 0x1200000 (18874368) memory.size =3D 0x3ce00000 (1021313024) memory.type =3D =E2=80=9Cbinary" SECTIONS: [section "config" size 0x1f6 offset 0x1000]: memory.size=3D1024M x86.strictmsr=3Dtrue x86.vmexit_on_hlt=3Dtrue cpus=3D2 acpi_tables=3Dtrue pci.0.0.0.device=3Dhostbridge pci.0.31.0.device=3Dlpc pci.0.4.0.device=3Dvirtio-net pci.0.4.0.backend=3Dtap0 pci.0.7.0.device=3Dfbuf pci.0.7.0.tcp=3D10.42.0.78:5900 pci.0.7.0.w=3D1024 pci.0.7.0.h=3D768 pci.0.5.0.device=3Dahci pci.0.5.0.port.0.type=3Dcd pci.0.5.0.port.0.path=3D/ISO/ubuntu-22.04.1-live-server-amd64.iso lpc.bootrom=3D/usr/local/share/uefi-firmware/BHYVE_UEFI.fd checkpoint.date=3D"Wed Jan 25 23:48:29 2023" name=3Dubuntu22 [section "kernel" size 0x19a7 offset 0x11f6]: [vm] vm.vds_version =3D 0x1 (1) vm.cpu0.data(BINARY): 00 00 00 00 0D 00 00 00 01 00 00 00 00 00 = 00 00 ... size=3D0x28 vm.cpu1.data(BINARY): 00 00 00 00 0D 00 00 00 01 00 00 00 00 00 = 00 00 ... size=3D0x28 vm.checkpoint_tsc =3D 0xe2e0ac6fbe456 (3991273496896598) [hpet] hpet.vds_version =3D 0x1 (1) hpet.data(BINARY): 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = 00 ... size=3D0x118 [vmx] vmx.vds_version =3D 0x1 (1) vmx.cpu_features =3D 0 (0) vmx.cpu0.vmx_data(BINARY): F0 CC 15 B8 FF FF FF FF 40 B4 21 B9 = FF FF FF FF ... size=3D0x288 vmx.cpu1.vmx_data(BINARY): F0 CC 15 B8 FF FF FF FF 00 00 67 41 = D8 9B FF FF ... size=3D0x288 [ioapic] ioapic.vds_version =3D 0x1 (1) ioapic.data(BINARY): 00 00 01 00 00 00 00 00 00 00 00 00 00 00 = 00 00 ... size=3D0x208 [lapic] lapic.vds_version =3D 0x1 (1) lapic.cpu0.data(BINARY): 00 00 00 00 00 00 00 00 00 00 00 00 00 = 00 00 00 ... size=3D0x460 lapic.cpu1.data(BINARY): 00 00 00 00 00 00 00 00 00 00 00 00 00 = 00 00 00 ... size=3D0x460 [atpit] atpit.vds_version =3D 0x1 (1) atpit.data(BINARY): 00 00 00 00 00 00 00 00 54 AD 51 97 0F 0E 00 = 00 ... size=3D0xa0 [atpic] atpic.vds_version =3D 0x1 (1) atpic.data(BINARY): 01 00 00 00 00 00 00 00 00 00 00 00 01 00 00 = 00 ... size=3D0x84 [pmtimer] pmtimer.vds_version =3D 0x1 (1) pmtimer.uptime =3D 0x26fd133e5cc (2679274464716) [rtc] rtc.vds_version =3D 0x1 (1) rtc.data(BINARY): 0A 00 00 00 00 FE FF FF 10 35 13 3D 40 F7 14 = 00 ... size=3D0x98 =E2=80=94 Thanks, Vitaliy Gusev --Apple-Mail=_574E1CEA-F86F-46F0-AC0E-FD372008C2F5 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi,

Here is a proposal for bhyve = snapshot/checkpoint image format = improvements.

It implies moving snapshot code = to nvlist engine. 

Current snapshot = implementation has disadvantages:

  • 3 files per snapshot: .meta, .kern, = vram
  • Binary Stream format of data.
  • Adding  optional = variable - breaks resume
  • Removing variable - breaks = resume
  • Changing saved order of variables - breaks = resume
  • Hard to get information about what is saved and = decode.
  • Hard to debug if somethings goes wrong
  • No = versions. If change code, resume of an old images can be
    passed, but = with UB.

New nvlist implementation = should solve all things above. The first step -
improve = snapshot/checkpoint saving format. It eliminates three files = usage
per a = snapshot.


1. BHYVE = SNAPSHOT image format:  

+=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94+
| =      HEADER PHYS - 4096 BYTES         = |
+=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+
|             =                     =       |
|=                DATA   =                 = |
|     =                     =               = |
+=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+


2. HEADER PHYS = format: 

 0   =  +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= + 
    =   |        IDENT STRING  - 64 BYTES   =       |
 64   = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ =   
      | NVLIST SIZE  - 4 BYTES =   |  NVLIST DATA |
 72   = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+
      | =                     =                     = |
      = |               NVLIST DATA   =             |
      |       =                     =               = |
 4096 = +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+


IDENT STRING - Each producer = can set its own value to specify image.
NVLIST SIZE  - = The following packed header nvlist data size.
NVLIST DATA - = Packed nvlist header data.

4KB should be enough = for the HEADER to keep basic information about Sections. However, it = can
be enlarged lately, without breaking backward = compatibility. 

3. NVLIST = HEADER consists of Sections in the following = format:

  • Name - = string
  • Type:    string:
    • =E2=80=9Ctext, =   - plain text,
    • =E2=80=9Cnvlist=E2=80=9D - packed = nvlist,
    • =E2=80=9Cbinary=E2=80=9D - raw binary = data.
  • Size - Size of section - uint64
  • Offset - = Offset in image format - uint64

    Predefined sections: =  =E2=80=9Cconfig=E2=80=9D, =E2=80=9Cdevices=E2=80=9D, =E2=80=9Ckernel= =E2=80=9D, =E2=80=9Cmemory=E2=80=9D. 


= 4. EXAMPLE:


 IDENT = STRING:

  =      "BHYVE CHECKPOINT IMAGE VERSION = 1"

 NVLIST = HEADER: 

  = [config]

    =     config.offset =3D 0x1000 (4096)

    =     config.size =3D 0x1f6 (502)

    =     config.type =3D "text"

  = [kernel]

    =     kernel.offset =3D 0x11f6 (4598)

    =     kernel.size =3D 0x19a7 (6567)

    =     kernel.type =3D =E2=80=9Cnvlist"

  = [devices]

    =     devices.offset =3D 0x2b9d (11165)

    =     devices.size =3D 0x10145ba (16860602)

    =     devices.type =3D "nvlist"

  = [memory]

    =     memory.offset =3D 0x1200000 (18874368)

    =     memory.size =3D 0x3ce00000 (1021313024)

    =     memory.type =3D =E2=80=9Cbinary"


 SECTIONS:


 [section = "config" size 0x1f6 offset 0x1000]:

memory.size=3D1024M

x86.strictmsr=3Dtrue

x86.vmexit_on_hlt=3Dtrue

cpus=3D2

acpi_tables=3Dtrue

pci.0.0.0.device=3Dhostbridge

pci.0.31.0.device=3Dlpc

pci.0.4.0.device=3Dvirtio-net

pci.0.4.0.backend=3Dtap0

pci.0.7.0.device=3Dfbuf

pci.0.7.0.tcp=3D10.42.0.78:5900

pci.0.7.0.w=3D1024

pci.0.7.0.h=3D768

pci.0.5.0.device=3Dahci

pci.0.5.0.port.0.type=3Dcd

pci.0.5.0.port.0.path=3D/ISO/ubuntu-22.04.1-live-serv= er-amd64.iso

lpc.bootrom=3D/usr/local/share/uefi-firmware/BHYVE_UE= FI.fd

checkpoint.date=3D"Wed Jan 25 23:48:29 = 2023"

name=3Dubuntu22


 [section "kernel" size 0x19a7 offset = 0x11f6]:

  =  [vm]

    =     vm.vds_version =3D 0x1 (1)

    =     vm.cpu0.data(BINARY): 00 00 00 00 0D 00 00 00 01 00 00 00 = 00 00 00 00 ...  size=3D0x28

    =     vm.cpu1.data(BINARY): 00 00 00 00 0D 00 00 00 01 00 00 00 = 00 00 00 00 ...  size=3D0x28

    =     vm.checkpoint_tsc =3D 0xe2e0ac6fbe456 = (3991273496896598)

  =  [hpet]

    =     hpet.vds_version =3D 0x1 (1)

    =     hpet.data(BINARY): 01 00 00 00 00 00 00 00 00 00 00 00 00 = 00 00 00 ...  size=3D0x118

  =  [vmx]

    =     vmx.vds_version =3D 0x1 (1)

    =     vmx.cpu_features =3D 0 (0)

    =     vmx.cpu0.vmx_data(BINARY): F0 CC 15 B8 FF FF FF FF 40 B4 = 21 B9 FF FF FF FF ...  size=3D0x288

    =     vmx.cpu1.vmx_data(BINARY): F0 CC 15 B8 FF FF FF FF 00 00 = 67 41 D8 9B FF FF ...  size=3D0x288

  =  [ioapic]

    =     ioapic.vds_version =3D 0x1 (1)

    =     ioapic.data(BINARY): 00 00 01 00 00 00 00 00 00 00 00 00 = 00 00 00 00 ...  size=3D0x208

  =  [lapic]

    =     lapic.vds_version =3D 0x1 (1)

    =     lapic.cpu0.data(BINARY): 00 00 00 00 00 00 00 00 00 00 00 = 00 00 00 00 00 ...  size=3D0x460

    =     lapic.cpu1.data(BINARY): 00 00 00 00 00 00 00 00 00 00 00 = 00 00 00 00 00 ...  size=3D0x460

  =  [atpit]

    =     atpit.vds_version =3D 0x1 (1)

    =     atpit.data(BINARY): 00 00 00 00 00 00 00 00 54 AD 51 97 0F = 0E 00 00 ...  size=3D0xa0

  =  [atpic]

    =     atpic.vds_version =3D 0x1 (1)

    =     atpic.data(BINARY): 01 00 00 00 00 00 00 00 00 00 00 00 01 = 00 00 00 ...  size=3D0x84

  =  [pmtimer]

    =     pmtimer.vds_version =3D 0x1 (1)

    =     pmtimer.uptime =3D 0x26fd133e5cc = (2679274464716)

  =  [rtc]

    =     rtc.vds_version =3D 0x1 (1)

    =     rtc.data(BINARY): 0A 00 00 00 00 FE FF FF 10 35 13 3D 40 = F7 14 00 ...  = size=3D0x98


=E2=80=94
Thanks,
Vitaliy = Gusev



= --Apple-Mail=_574E1CEA-F86F-46F0-AC0E-FD372008C2F5-- From nobody Tue May 23 16:38:09 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQg3j5j0hz4Cg3G for ; Tue, 23 May 2023 16:38:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQg3j3Dwxz4SmP for ; Tue, 23 May 2023 16:38:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684859889; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=71mI8nygiC4wX3nu3pILbMNTcq6thvzLQdpeUVwfGNg=; b=MPqLvygg2zUDSofpFKWN7QMrkvuueOFH+FYxTPDGLV4kqrfIMaqfHW2nXlwr5Ogt+w46jg 2NLqj399IKDUn2aKDKQuGtD0zmAG2TVteh69f8i7sebqwvukN4v8I1FzEHaFd2yE+4rFO2 uIMMNJZK6JkRDXbNvmocK129eV3DclafnplkYnk6v9L3+4kprqWP6IJERhn8FENV3j5dzz 3cHEQctXRSuQT8847YfesfAbkw1BpfPbiYBAuPC5e0fkiHxJSaVl1YvvTgn3/Hr7VnJVCe K0nLjWhBOlsYsJ3/eIMG7qS10bffkkTF6XdHH04c2oW6USXTLj5nEX+8iMznXQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684859889; a=rsa-sha256; cv=none; b=Kx5UDP+iiPDrVMpdp4mO1KvtLlzOLtXgd9VgENmJTUMAxWSKPYNyyH3EpqX8GUi5LPo36E MmfgvCIJzPDNgrFJC7Ijk6LpIqk0uGul2Nmx3ujNPPCJt0KpD4j724mC+YaCGZD8ctIgnZ Tbn54lgKVZrLStI/cG8WwGHEF2wTymEzjK2dC7VYNKIrkmpg6obXw98iPCR03K93PWZf5P mpb6v4fUUZcWADNwvADwULIjGlMi9S16syRL1/vw7tml/8bO9PASm83jgDlGt0zw5dnSps xfDa4o1jWj1vKbJbhd92A/2xHBB4KJqpvl1f2SGUHy0rgmwkkZuzg3JynTbb/w== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QQg3j2JGhz16q9 for ; Tue, 23 May 2023 16:38:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 34NGc9G1074787 for ; Tue, 23 May 2023 16:38:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 34NGc9cr074786 for virtualization@FreeBSD.org; Tue, 23 May 2023 16:38:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 271456] SR-IOV: mce(4) with bhyve: can't ping the vm Date: Tue, 23 May 2023 16:38:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: benoitc@enki-multimedia.eu X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271456 --- Comment #3 from benoitc --- Created attachment 242345 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D242345&action= =3Dedit ip offered by the router is marked as active attached is a screenshot showing that IP received by the vm has been offere= d by the router and is active.=20 I can't ping from the router to the vm, and from the vm to the bridge. I tried with a different vlan but same result. I can't understand why the v= m is able to get an IPV4 from from the router using DHCP but can't ping it... Let me know if you need more information to debug this issue. I --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue May 23 17:26:27 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQh7k04WZz4CjvF; Tue, 23 May 2023 17:26:42 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQh7j4Z2Kz4Y7d; Tue, 23 May 2023 17:26:41 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lj1-x22d.google.com with SMTP id 38308e7fff4ca-2af2e1725bdso1509091fa.0; Tue, 23 May 2023 10:26:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684862799; x=1687454799; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=S/ZTmuVyoeziujL0N7pYts/kBHYF9tHqFLcN7FXr8vU=; b=nJ+4q9iZHWMWJkkggN/MPQY8GNZYDBJDnNWqK/eKOpMjstRl7uYljQRnr+c5ld6btZ X58e8V0pHllashGAj7buySXF+5649mB3wf05nwidswMHco4P9/4EpjI5c3kXuuQAjdBX z4M4KD0hZo2HDwF0YSthbMCTVye1fJwfZG6j7wfT8+vlZ1uaC0/wrxW9nUskZC/voLEM R95gIDl64rWep7ZgZXVl4XcoX5mPBh6tDmufZuwKzWNj7nMRiHpqGRJoayEae4pV4uhs a+GV/SWgrx9K4NXLHa1bMtvzIfGsXKOOp1/4cyK4hMjNpFqpq3bxVoF/IDM/sY26/bUn dfww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684862799; x=1687454799; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=S/ZTmuVyoeziujL0N7pYts/kBHYF9tHqFLcN7FXr8vU=; b=Gls7fgDEpL4DMWxLj8fFIN/RtBU6FTPV8S5yDdy4kTVFCfEMDZG/HyxLid45ajVvhw pfqzaYRAh8rBfOz0SB7BUjEg7hAC4cYMNlzvxnBjl31QZeC0E/nsK5bIwL/awG0q6QEG DE6+PXYsNtlNmFS7mJKic6ndM14ek+/G8oQs/l/bSdVNFFwoR3eMsXxUD3AdSjdQaMiS 33o4nf1Y/blEEo+mfs0KXzygBesvbHjHbho2Aq29kVYvQdHdJPLD0oHqvoddE8meigpq NCgdDwu8dI+0Uf7ovuePF2Aohecgw14/B0BOnj0tFnzSGueB1/L5XwZWndDqC1Y6BJTa TkpA== X-Gm-Message-State: AC+VfDxA4Tni3+OUzsUrBrX9wmaxgy8xm+LZQO8vA9SnA4fI+ZXmjhy4 qHIpTPUAGjY81X+cE7BgbHMyuhPt/78= X-Google-Smtp-Source: ACHHUZ78yzYis+Rr48+V4neR82NEcMTfJFgJBCwE7JG6uidgOpTb5w7PVZ/ewDsuW32fDM+CWauuIw== X-Received: by 2002:a2e:b907:0:b0:2ad:9edd:4e2 with SMTP id b7-20020a2eb907000000b002ad9edd04e2mr4875567ljb.20.1684862798504; Tue, 23 May 2023 10:26:38 -0700 (PDT) Received: from smtpclient.apple ([78.140.234.98]) by smtp.gmail.com with ESMTPSA id j21-20020a2e8015000000b002aeee2a093csm1697061ljg.59.2023.05.23.10.26.38 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 May 2023 10:26:38 -0700 (PDT) Content-Type: text/plain; charset=utf-8 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE SNAPSHOT image format proposal From: Vitaliy Gusev In-Reply-To: <202305231645.34NGj2Bq081239@critter.freebsd.dk> Date: Tue, 23 May 2023 20:26:27 +0300 Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <5544A8DA-4E91-4384-B72D-8C91B32B6D69@gmail.com> References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <202305231645.34NGj2Bq081239@critter.freebsd.dk> To: Poul-Henning Kamp X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4QQh7j4Z2Kz4Y7d X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Hi, > On 23 May 2023, at 19:45, Poul-Henning Kamp = wrote: >=20 >=20 >> 1. BHYVE SNAPSHOT image format: >=20 > Please do not invent Yet Another Format, please ? >=20 > Why not make it a tar(5) file ? >=20 Tar cannot solve issues mentioned in =E2=80=9Cdisadvantages=E2=80=9D. = Tar doesn=E2=80=99t have versions, it is just container for files that would introduce another level of indirection. Snapshot/resume = doesn=E2=80=99t need just container. It needs information what is saved and in what format. For example, virtual = memory can be saved in different ways: binary, diff pages, etc. Virtual memory of VM should be saved faster without additional cost. The = same for restore stage. Do you like an idea to have tar file with size 8 GB ? And how it can be saved = efficiently without double copying of data? Yes, tar is powerful and convenient for many purposes, but it is not so = suitable to suspend/resume process and would introduce just another level of complexity. =E2=80=94=E2=80=94 Vitaliy Gusev= From nobody Tue May 23 17:57:38 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQhqQ1CzMz4ClMN for ; Tue, 23 May 2023 17:57:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQhqP6lxSz4lPf for ; Tue, 23 May 2023 17:57:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684864657; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sGzPQjb5cQ90nRth1SEfGDsQMUjGV3t5tfMhuaqM08Y=; b=eD1yEcokJlHf2LyO3qAhPk5JynuPgdyjpG/yfGugxyq48VjphatNzF/lshm2y/DdhRzmFM 7qDsa+iBGk1CnuOrtxwNENijhd7l5C/hd55fL9YyfIFOEUVu3FWPlvJW6QHFOpS8Pjph2T 87KYrLWAVACuEhS62q818jYY4c0w420ZR9U27dkL3xbHhJIddGWQZuFkDairaO8DPvYddn I60sNFy6/Bf0HMyTgf5p3lcII7ZEBGIqKUKfxH+GfJ1ABdnXxZMO6DXa2zkHSIUDgxWC78 0xY8wGCYQQpfXBvNEV8Ld5+soMpL6ynI8nAzVgsypQ7FKSCKGeYBCyAjXuY0Bw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684864657; a=rsa-sha256; cv=none; b=sin3KiuIx9pd2qYQEFaonVK2PjcWLMEknINVLOmxxNNnNs4TaifjDhSfjTsk2yYWPXcD69 GPbu+wVToud3y3QnnznNLQnztCbuvXBzoWKgL9DvGlCGjQdB+WuaKT76plCHHhPJLRCXn2 cHsvG66N8Ex3U/gjMTkuuGuydotWxKD0RMNCXWubl8TdZXKmlP29rMwIMx3HWEDbXVBdOu UbqwoAyGZoOhIpv4/tktZx4SSjC041tI7J52/xBJOWWtFpIg/+XG3lE8KtXiJbq2kupNzY rQ+twrNLC2muv5wpsPVMx8i2jSpNLMYNsGF6J5aK22rImE2Mb7FsB8W3xfAdSA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QQhqP5r0Dz18WF for ; Tue, 23 May 2023 17:57:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 34NHvbRY091734 for ; Tue, 23 May 2023 17:57:37 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 34NHvb4m091733 for virtualization@FreeBSD.org; Tue, 23 May 2023 17:57:37 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 271456] SR-IOV: mce(4) with bhyve: can't ping the vm Date: Tue, 23 May 2023 17:57:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: benoitc@enki-multimedia.eu X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271456 --- Comment #4 from benoitc --- same result with jails. Are virtual devices created with srv-io working in a bridge? I can set the ip of the interface, but once I put it in the bridge = it doesn't work. With jail, here is the bastille configuration: ``` testing { devfs_ruleset =3D 13; enforce_statfs =3D 2; exec.clean; exec.consolelog =3D /var/log/bastille/testing_console.log; exec.start =3D '/bin/sh /etc/rc'; exec.stop =3D '/bin/sh /etc/rc.shutdown'; host.hostname =3D testing; mount.devfs; mount.fstab =3D /usr/local/bastille/jails/testing/fstab; path =3D /usr/local/bastille/jails/testing/root; securelevel =3D 2; vnet; vnet.interface =3D e0b_bastille0; exec.prestart +=3D "jib addm bastille0 mce2"; exec.prestart +=3D "ifconfig e0a_bastille0 description \"vnet host interf= ace for Bastille jail testing\""; exec.poststop +=3D "jib destroy bastille0"; } ``` /etc/rc.conf: ``` syslogd_flags=3D"-ss" sendmail_enable=3D"NO" sendmail_submit_enable=3D"NO" sendmail_outbound_enable=3D"NO" sendmail_msp_queue_enable=3D"NO" cron_flags=3D"-J 60" ifconfig_e0b_bastille0_name=3D"vnet0" ifconfig_vnet0=3D"inet6 XXXX:XXXX:XXXX:102::20/64" defaultrouter=3D"XXXX:XXXX:XXXX:102::1" ``` On the host network s configured like this: ``` ce2: flags=3D8963 metric 0 = mtu 9000 =20=20=20=20=20=20=20 options=3D7ead00b9 ether a2:5b:2e:82:9f:09 inet 0.0.5.220 netmask 0xff000000 broadcast 0.255.255.255 inet6 fe80::a05b:2eff:fe82:9f09%mce2 prefixlen 64 tentative scopeid= 0x6 media: Ethernet 25GBase-SR status: active nd6 options=3D21 mce2bridge: flags=3D8843 metric 0 m= tu 9000 ether 58:9c:fc:10:fc:41 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: e0a_bastille0 flags=3D143 ifmaxaddr 0 port 8 priority 128 path cost 2000 member: mce2 flags=3D143 ifmaxaddr 0 port 6 priority 128 path cost 800 groups: bridge nd6 options=3D9 e0a_bastille0: flags=3D8963 metric 0 mtu 9000 description: vnet host interface for Bastille jail testing options=3D8 ether 0a:20:98:82:9f:09 hwaddr 02:00:91:24:1e:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T ) status: active nd6 options=3D29 ``` --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue May 23 17:59:09 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQhs96QmVz4ClHS for ; Tue, 23 May 2023 17:59:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQhs94b6Fz3CLt for ; Tue, 23 May 2023 17:59:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684864749; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ylN4UXRDHFVDvavwK1PNHx8SwurjQXY0ozh54L+MSZw=; b=vSFkg4NTWGtxofTtMM+voNIIcxT0mzsUZ4f5cbDSVLp7kb3YoJmGr9yPUFe8KCbbtPLBAO Ai+xrESkOV+JJXqHvucHuvKrwHdErBfCtuRTwWPqKuXDhbde4v1I3cvs1miesHKyW3bmiK cvXYDTZf4WYaCDJudAHY1i7yVacGb1GbA8nQKe6JTBvo82C9xXRIM1sg9qqJ5A5dwZ1QwI cSI8FF3KoW4iqbyoqQyaFqP+oLW/bzt4oBZEfFkZR4rgQGfAgKgR4w4HlGH47h5bjXGKud qmigy+DKraWbrtsJge8PZrKsVLrjJKvbZkufvs6gVxAxgPr0bsGcwjuAcRD14w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684864749; a=rsa-sha256; cv=none; b=SysRML/9SH7K2pZv1DYU0ViRoof3fM6FmJFDGRXOP9avwoZ3cXgUB96K0D31A2LHaheIPG 1cCOGdCjt01WomW/uUJkD76/B+zYUWO0/1x/Ki6kZm8Yvya8emQH1sC/ZzgoBhHssGyItn XcAa1Bxij4EFeFFptW4KqfMmWej9ClfCJJ/kRovLannH3/E5ZkRdhifdKhlwCxKBHPpD6T 0/WW8WSXTkuz1b/wsZLQ4BmzfRRc4hCtH6spu4sWChnCGwTlB0PsN1D7seRw+8JraVV2m5 e7PnEHMwgwboY0G7VSG5TAIjPZMTfxf7doIQvXhDtBVurJM5ntpiURPfRNT8Qg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QQhs93fQ1z18nL for ; Tue, 23 May 2023 17:59:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 34NHx9BY092265 for ; Tue, 23 May 2023 17:59:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 34NHx9hY092264 for virtualization@FreeBSD.org; Tue, 23 May 2023 17:59:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 271456] SR-IOV: mce(4) srv-io VF interface in a bridge can't be pinged or ping IPs outside of a VM Date: Tue, 23 May 2023 17:59:09 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: benoitc@enki-multimedia.eu X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271456 benoitc changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|SR-IOV: mce(4) with bhyve: |SR-IOV: mce(4) srv-io VF |can't ping the vm |interface in a bridge can't | |be pinged or ping IPs | |outside of a VM --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue May 23 17:59:24 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQhsS5DGzz4ClFC for ; Tue, 23 May 2023 17:59:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQhsS2wkqz3CcL for ; Tue, 23 May 2023 17:59:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684864764; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xQNuiyKjMbbS8zsQCYUDi9+d6GIG2TAnLPMMN2N6Mnw=; b=FyLHCmJwwFqLiKxbf+0/OsAsYekmPEGMhFxcMCNPUmJjLDQA3hgswl5mPRUdgbneuEu32D LMjur5JK8i9LBd4AhvnbjSgKr1VsPmAcWo1NPNcS5eGXCWKbEZvZNoDccpYC2nt37wnLyW WgmwRKVXKpZ6LcMY8jDZ2n8tRl9tFKoh8V3DAC2D/wScoUuARbTHGbkBzCP1Zjb8HSdpr5 u6hNzzfRPc8JGs0FZLm9lsX8V3TUsX004y187OBO4BDl+aXrk+iZDTHhV411A3gS0fARss R9D6fsw98Z1bSPgdB/sIzRLI7TcWj2V3UYKWpL1N6Q1uSHmDBF6aF74SelFkEQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684864764; a=rsa-sha256; cv=none; b=b74uqlqYoF1DI0HilzdYFP0IrcOwv1KP3nlzgMc/1LBuScoERwXRA/n46z//f5gH4wET8y w+x+PKO2fViGVW93vokxpEvC8sNOsahMXvAV94dK9i7qiudu/RWRygBdIDrMaEFgiZ+mRh Nm4PTMU4JdMeGvQ/+iGJVIB34bG2qgQvIOjD6VrUIp0LUgea6hNVuuka5ltBM1PsiJ3zz4 0zgOJKPewCNw8YwxuVnXCtDpwOLIONYKu91bPiVhhXPFK+jKp0hyuxuhA58jem9pkPYmV+ zIEwZebOu1221KND9cpPYQe9XIIkpKlzXXU/bWUiSxvDvBmKXoE7EnwHIS5SEg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QQhsS1ss3z18nM for ; Tue, 23 May 2023 17:59:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 34NHxO8m092341 for ; Tue, 23 May 2023 17:59:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 34NHxOxD092340 for virtualization@FreeBSD.org; Tue, 23 May 2023 17:59:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 271456] SR-IOV: mce(4) VF interface in a bridge can't be pinged or ping IPs outside of a VM Date: Tue, 23 May 2023 17:59:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: benoitc@enki-multimedia.eu X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271456 benoitc changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|SR-IOV: mce(4) srv-io VF |SR-IOV: mce(4) VF interface |interface in a bridge can't |in a bridge can't be pinged |be pinged or ping IPs |or ping IPs outside of a VM |outside of a VM | --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue May 23 18:58:27 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQk9w74kmz4CpGN for ; Tue, 23 May 2023 18:58:44 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yb1-xb33.google.com (mail-yb1-xb33.google.com [IPv6:2607:f8b0:4864:20::b33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQk9v6Knpz3Kyk for ; Tue, 23 May 2023 18:58:43 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cedro.info header.s=google header.b=B7uEzEwP; spf=none (mx1.freebsd.org: domain of tomek@cedro.info has no SPF policy when checking 2607:f8b0:4864:20::b33) smtp.mailfrom=tomek@cedro.info; dmarc=none Received: by mail-yb1-xb33.google.com with SMTP id 3f1490d57ef6-ba1815e12efso45082276.3 for ; Tue, 23 May 2023 11:58:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1684868322; x=1687460322; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=nB+KETuhNWgh2rcrgLzJEn5b97eKXdyGdxRX3fKaMCc=; b=B7uEzEwP8SQldtubkIrU6NtvitAadiDiEB685dR5n5F2HlelFlg7a4YpTR/WNDvp+D ESrK4fRWMXshH+b38oaZR4fOWAwF6lFNAJHyRONeImAO7FPuA10RI9LZ7xLQnPfT1DdN +rfjpfYy8FBLbF4qemBVRMYYyM1NblaYqOJJJJywuNXTd5BJjhesPO0crz4eRwpFqYtP wqZnNHD9UmsKU5jkKdOO2StwJdbV0qz9lQ2omlrilsODgT12mumvqx8Ene0vQ7hyAJBe JPCAF6Eo/LHOY4+o3P2RCUjXeSeUCBLmbj68ifuJecXSNTY/FYwIpxCluZ73yQBDC7sE a29w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684868322; x=1687460322; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nB+KETuhNWgh2rcrgLzJEn5b97eKXdyGdxRX3fKaMCc=; b=AMvupWZmk2QVAF9JDdxvff0DcncDOxQXhEVejjqkUNsDbILrXtZHXRM01xeeyL+khG 1DFOkAY6N1wOgBF5PZtcxqjXHN/36sHDXOMXTdEkPYFUcQEb0OAJF9QgNqA7M0BQ2XDq CScvmiWaFPOxda4Tzxb5CENeGe669xtgfN1vAHsQZ3yeeQkNyJEgzgJMRcANouxWB2Zv AQ77d6hcCXy3ENJFvd5HivmXpGRlbUJekTSoU5etc7U+Lsl10n11AbCHWhjxF2wmnykw cGwfhvpKKjNuSiGNEG9EZtvlSsNqPnqLinkRqQq6FcqzCwoQBmAKCC4COrknuS+0Ty9H dqLg== X-Gm-Message-State: AC+VfDy+T7x0nUV3MCs2A4+ECbLxGKglZsP0u5Sg8VLoFyQftwvPXPjY FOGJ0k6jZirhOlHOM0SzHDKzjp6eXDXs3IE1LoU= X-Google-Smtp-Source: ACHHUZ4LaXPVxWPuZHqIBC38l4A+rOX47kIxSJJIHSOA/49K+n6SJjMf6M0uBrGBl2RC8PDKmw4aNQ== X-Received: by 2002:a25:6ac2:0:b0:b72:4ca:b3ce with SMTP id f185-20020a256ac2000000b00b7204cab3cemr14863980ybc.16.1684868322509; Tue, 23 May 2023 11:58:42 -0700 (PDT) Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com. [209.85.219.171]) by smtp.gmail.com with ESMTPSA id b185-20020a0df2c2000000b0056507de3d82sm1669836ywf.104.2023.05.23.11.58.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 23 May 2023 11:58:42 -0700 (PDT) Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-ba6d024a196so46993276.2; Tue, 23 May 2023 11:58:42 -0700 (PDT) X-Received: by 2002:a81:4e4a:0:b0:55a:985e:8ad1 with SMTP id c71-20020a814e4a000000b0055a985e8ad1mr16906355ywb.33.1684868321800; Tue, 23 May 2023 11:58:41 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> In-Reply-To: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> From: Tomek CEDRO Date: Tue, 23 May 2023 20:58:27 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: BHYVE SNAPSHOT image format proposal To: Vitaliy Gusev Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.29 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.995]; R_DKIM_ALLOW(-0.20)[cedro.info:s=google]; MIME_GOOD(-0.10)[text/plain]; BLOCKLISTDE_FAIL(0.00)[209.85.219.171:server fail,2607:f8b0:4864:20::b33:server fail]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b33:from,209.85.219.171:received]; R_SPF_NA(0.00)[no SPF record]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; DKIM_TRACE(0.00)[cedro.info:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TAGGED_RCPT(0.00)[]; DMARC_NA(0.00)[cedro.info]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4QQk9v6Knpz3Kyk X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Tue, May 23, 2023 at 6:06=E2=80=AFPM Vitaliy Gusev wrote: > Hi, > Here is a proposal for bhyve snapshot/checkpoint image format improvement= s. > It implies moving snapshot code to nvlist engine. Hey there Vitaliy :-) bhyve getting more and more traction, I am new user of bhyve and no expert, but new and missing features are welcome I guess.. there was a discussion on the mailing lists recently on better snapshots mechanism :-) > Current snapshot implementation has disadvantages: > 3 files per snapshot: .meta, .kern, vram No problem, unless new single file will be protected against corruption (filesystem, transfer, application crash) and possible to be easily and cheaply modified in place? > Binary Stream format of data. This is small and fast? Will new format too? > Adding optional variable - breaks resume > Removing variable - breaks resume > Changing saved order of variables - breaks resume Obviously need improvement :-) > Hard to get information about what is saved and decode. > Hard to debug if somethings goes wrong Additional tools missing? Will new format allow text editor interaction? > No versions. If change code, resume of an old images can be > passed, but with UB. Is new format future proof and provides backward compatibility? > New nvlist implementation should solve all things above. The first step - > improve snapshot/checkpoint saving format. It eliminates three files usag= e > per a snapshot. > > 1. BHYVE SNAPSHOT image format: > > +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ > | HEADER PHYS - 4096 BYTES | > +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ > | | > | DATA | > | | > +=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2= =80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80= =94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94+ > (..) > Predefined sections: =E2=80=9Cconfig=E2=80=9D, =E2=80=9Cdevices=E2= =80=9D, =E2=80=9Ckernel=E2=80=9D, =E2=80=9Cmemory=E2=80=9D. > 4. EXAMPLE: > IDENT STRING: > "BHYVE CHECKPOINT IMAGE VERSION 1" > NVLIST HEADER: > [config] > config.offset =3D 0x1000 (4096) > config.size =3D 0x1f6 (502) > config.type =3D "text" > [kernel] > kernel.offset =3D 0x11f6 (4598) > kernel.size =3D 0x19a7 (6567) > kernel.type =3D =E2=80=9Cnvlist" > (..) So this will be new text config based format with variable =3D value and se= ctions? How much bigger will be the overal file size increase? How much longer it will take do decode/encode/process files? What is the possibility of format change and backward/foward compatibility? Have you considered efficiency comparison of current format, proposed format, and maybe using SQLITE or JSON storage/parsers? For instance sqlite would be blazingly fast but hard to migrate. json would be most versatile but more time/memory consuming? Maybe EFL approach of storing configuration files for limited resources embedded system storage that use binary storage data but can be decompressed in chunks that can be replaced in place? https://www.enlightenment.org/develop/efl/start Sorry for asking those questions but there may be already good and verified solutions out there not to reinvent the wheel? :-) -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From nobody Tue May 23 19:21:52 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QQkhd2VxTz4Cql2 for ; Tue, 23 May 2023 19:21:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QQkhd1Nv5z3PTn for ; Tue, 23 May 2023 19:21:53 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1684869713; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tw/7ZqapdQEIZc+UTlTeop4P8BzjFB+y7GAxFXdKxFA=; b=HhnjOmhTASaUBy4SHm0RSpI+wnJbmG29gcEkVMC7Q6BckA6FXnxekHgASQ9Oi4JEXfsOQx cEoEkXgCZv8wqjxghOTOj78zUB+vZQzvYMwKsgStXHtWffXJDjH1sDHaF5efSK1TTRnN3T 2eZvvAk4LLnkUkYatNg3F4U8umyC8Y8YiRHxVo176hQbC2sB87hIGToblUHaqf4l0+Oq2g c0KW/BnNjJvMl50KkCCVYYCatS7gR5DNdLeLkW/ZWXCQlzMM62tIcw8J2Sf2iMWks/5ePg fOR7oOiGdMx+W5yrNHhPlh6UC85r8Nj5/mo0jdBtrKjOyzogwX/05jay3DUC7A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1684869713; a=rsa-sha256; cv=none; b=j8iiX1Cf/LSSvTmGe22WY5hIVCHF+ItKSwPdk6RCErrqVBdWqsIJTs9J6/Wm/wViH8AVbA vt7G/Dr+FXO+IvZMy9FkvTzlVrBRxNovMr4VM8sDz9U2xYOI1oTKkPDlulU7CJS35YtdvU GOOLKnVRT5QXCiI9Od9aL1OTBW4bIrfVpuWst0lY3D3KlDmZMvq64HRIQPsBzSzxf7dbUS TlndwCQb0oylCEkxD4qB2lctQVSY6DlAXhfF5UO90zeMD1hqY+3o2+FOp5DG+JymGxsebv acP+Jupts+a4LRp7VrZDk/Ji3SYg+3zW2Bl/Foclg3njHMlBlNeDRY9oeOquDA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QQkhc6qHWz1C40 for ; Tue, 23 May 2023 19:21:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 34NJLq1I016335 for ; Tue, 23 May 2023 19:21:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 34NJLqrX016302 for virtualization@FreeBSD.org; Tue, 23 May 2023 19:21:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 271578] FreeBSD fails to init SMP and hits panic post install in Azure ARM64 Date: Tue, 23 May 2023 19:21:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271578 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|bugs@FreeBSD.org |virtualization@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed May 24 07:12:15 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QR2SV50w4z4CL4S for ; Wed, 24 May 2023 07:12:26 +0000 (UTC) (envelope-from elk-freebsd@wetznet.de) Received: from relay.yourmailgateway.de (relay.yourmailgateway.de [46.38.247.119]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.netcup.net", Issuer "RapidSSL Global TLS RSA4096 SHA256 2022 CA1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QR2ST5Mkhz3vnp for ; Wed, 24 May 2023 07:12:25 +0000 (UTC) (envelope-from elk-freebsd@wetznet.de) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=wetznet.de header.s=key2 header.b=azfTi1KC; spf=pass (mx1.freebsd.org: domain of elk-freebsd@wetznet.de designates 46.38.247.119 as permitted sender) smtp.mailfrom=elk-freebsd@wetznet.de; dmarc=none Received: from mors-relay-8404.netcup.net (localhost [127.0.0.1]) by mors-relay-8404.netcup.net (Postfix) with ESMTPS id 4QR2SS1qWLz820M for ; Wed, 24 May 2023 09:12:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=wetznet.de; s=key2; t=1684912344; bh=T+flYtyvkjg13kFciPd0h1kG8UkC9GXkIEtsblG+o+c=; h=Date:From:To:Subject:From; b=azfTi1KCSXZtTVAv4lyQlIMQacN0NwQvwqT/TjnFJb0BpNSep7ZY0m8ujtzDYBJrX y5adVpuX3+uDSRCxWF+bfCdYSRwQtzUziQT4s2xJiw9rumlDZIUu+yLChwwM3zcE/p Ih4t75TEoLh97EmYZKUjlLHZiH/wvgQhgxRP9o2oVYU9b27tmej5Xl6j7+Q0RJCaQ/ gfYi46DgO/XSLya6jWhCMyyRqNY929kaucEgam8UoLGObpnbHoTXi+CpomgFc4/zdZ VdU8QqD7s0Tp6jOO9t4b7V08OI9Jv0+XUqlYiYe99yDd+nkQ/WbVF9fMyTaaaV/sFd n/dqNc9X2o3Ew== Received: from policy02-mors.netcup.net (unknown [46.38.225.35]) by mors-relay-8404.netcup.net (Postfix) with ESMTPS id 4QR2SS1RZYz4xsw for ; Wed, 24 May 2023 09:12:24 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at policy02-mors.netcup.net X-Spam-Flag: NO X-Spam-Score: -2.901 X-Spam-Level: X-Spam-Status: No, score=-2.901 required=6.31 tests=[ALL_TRUSTED=-1, BAYES_00=-1.9, SPF_PASS=-0.001] autolearn=ham autolearn_force=no Received: from mxe85d.netcup.net (unknown [10.243.12.53]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by policy02-mors.netcup.net (Postfix) with ESMTPS id 4QR2SH6cFxz8sh8 for ; Wed, 24 May 2023 09:12:15 +0200 (CEST) Received: from webmail.wetznet.de (unknown [46.38.249.153]) by mxe85d.netcup.net (Postfix) with ESMTPA id 655421C08C7 for ; Wed, 24 May 2023 09:12:15 +0200 (CEST) Received-SPF: pass (mxe85d: connection is authenticated) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Date: Wed, 24 May 2023 09:12:15 +0200 From: "elk-wetznet.de" To: freebsd-virtualization@freebsd.org Subject: bhyve HEVC/H.265 encoding support in ubuntu vm User-Agent: Roundcube Webmail/1.4.13 Message-ID: <3c1de1f44be2bb9bc1fab961e8a34722@wetznet.de> X-Sender: elk-freebsd@wetznet.de Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-PPP-Message-ID: <168491233562.22968.9068129034579101543@mxe85d.netcup.net> X-Rspamd-Server: rspamd-worker-8404 X-NC-CID: Kac6e7EyqVEjZry4+cLQUHuzzCeAE41hnsHXfc26Jw== X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.997]; R_SPF_ALLOW(-0.20)[+ip4:46.38.247.112/28]; R_DKIM_ALLOW(-0.20)[wetznet.de:s=key2]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[46.38.247.119:from]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; BLOCKLISTDE_FAIL(0.00)[46.38.247.119:server fail,46.38.249.153:server fail,46.38.225.35:server fail]; DKIM_TRACE(0.00)[wetznet.de:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:197540, ipnet:46.38.240.0/21, country:DE]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; ARC_NA(0.00)[]; DMARC_NA(0.00)[wetznet.de]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4QR2ST5Mkhz3vnp X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N HI, I run bhyve at Freebsd on a AMD Pro Ryzen 5 and still dont know a way to get the encoding support of the GPU inside the VM. Any hints ? Thanks a lot From nobody Wed May 24 15:10:49 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRF4l5BmPz4Cpqy; Wed, 24 May 2023 15:11:03 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRF4l11xkz3sk2; Wed, 24 May 2023 15:11:03 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x130.google.com with SMTP id 2adb3069b0e04-4f00d41df22so1050881e87.1; Wed, 24 May 2023 08:11:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684941061; x=1687533061; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=1yB/FsRx+5zL23ybcm4qwD8TsIzyLeLgWMfyeSKOMNQ=; b=W64aUx+oW8OFnUJF3wWuKRG1vTSmDPAiBXzEX0pM4LBBt2Tm4blg64TDToqpWfd4Q2 z/eDUpeY+uAbAqJUWQtsMBAWu5b1NAGjZts5hOzMEBVioi9WGqMccekwbq7xDf9k0k8B LyPhSpJTOLq/yYp1EXc1zNLxDRofWXdZ8XbC8Am5KNxQfRdv2rm+7BI7LEqDUn6Ocp98 M2IPQ42UAw2ysevxDxUOwsJhOfbwtd84MAOpy9kelpqAymCE4qscsHmD1rZgZJCmrV/E /tCibY/cecEeVWH0QAvvn4j2duzILlTEMwqLWFWggqnnwUB6gByukFyU2eABcvO84kaX fmOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684941061; x=1687533061; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1yB/FsRx+5zL23ybcm4qwD8TsIzyLeLgWMfyeSKOMNQ=; b=NfAeMxluefkN82EO7KJUgQu0ITu96TASE7JKZVK1AJZCtfza4rkHDzl4LSDoMeKhsp 9FnetyRFPiV3svd2n9sKFCOCcACN4iSWGUL9qQl0FJ7FTFWFMDAUA6VPXcP2/Lng0DEJ /l/9WlpSq1f0qYkl0p1pUYZCHZREb7lmjBLwaPoMpgH8uM0IMoRYMnEQF3vvrU3o1Fqi FuMu/oBHUpb/Y5bCGmpZelZUZbn9pJoEGv9wmw3t/DIajn4EERU6OMfFYxVuoRWklmpH IZkr5+q4wZDrJ3dSxByJnkHFq196r9fqgKnIjUzn2joHUCHCBKFN6fcTq6xaQpPD35dA mrqQ== X-Gm-Message-State: AC+VfDwFqVbeJhwSTUOWStmGdf6acNlWbbIs//5T7ujP5LtHBTmmgHuB qEokVUgZfL/eHVHe3VMA6BlgIDPuY3M= X-Google-Smtp-Source: ACHHUZ4Il2yAc46NVzC54B0e5E+NWEnNKOiuhzubO8uSZeDitSezIDGRNVlIBjMt1pgsw2jqmhQniQ== X-Received: by 2002:ac2:488b:0:b0:4f4:ca61:82b3 with SMTP id x11-20020ac2488b000000b004f4ca6182b3mr775605lfc.21.1684941060833; Wed, 24 May 2023 08:11:00 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id b10-20020a056512024a00b004f13634da05sm1749710lfo.180.2023.05.24.08.11.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 May 2023 08:11:00 -0700 (PDT) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_B2C31D91-4697-4C26-ADE9-9F456DBBFFC5" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE SNAPSHOT image format proposal Date: Wed, 24 May 2023 18:10:49 +0300 In-Reply-To: Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org To: Tomek CEDRO References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4QRF4l11xkz3sk2 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_B2C31D91-4697-4C26-ADE9-9F456DBBFFC5 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Tomek, Try to answer to the all questions below, please let me know if I miss = some important. > On 23 May 2023, at 21:58, Tomek CEDRO wrote: >=20 > On Tue, May 23, 2023 at 6:06=E2=80=AFPM Vitaliy Gusev wrote: >> Hi, >> Here is a proposal for bhyve snapshot/checkpoint image format = improvements. >> It implies moving snapshot code to nvlist engine. >=20 > Hey there Vitaliy :-) bhyve getting more and more traction, I am new > user of bhyve and no expert, but new and missing features are welcome > I guess.. there was a discussion on the mailing lists recently on > better snapshots mechanism :-) >=20 >=20 >> Current snapshot implementation has disadvantages: >> 3 files per snapshot: .meta, .kern, vram >=20 > No problem, unless new single file will be protected against > corruption (filesystem, transfer, application crash) and possible to > be easily and cheaply modified in place? Current snapshot implementation doesn=E2=80=99t have it. I would say = more, current pkg implementation doesn=E2=80=99t track/notify if some of files are = changed. Binary files on a system can be changed, for example ELF files, without any notification. Tar doesn=E2=80=99t have protection for keeping data. Some filesystems = like ZFS guarantee that data is not modified by underlying disks. Protecting requires more efforts and it should be clearly defined: what = is purpose. If purpose is having checksum with 99.9% reliability, NVLIST HEADER can be = widen to have =E2=80=9Cchecksum=E2=80=9D key/value for a Section. If purpose is having crypto verification - I believe sha256 program = should be your choice. >=20 >> Binary Stream format of data. >=20 > This is small and fast? Will new format too? Small is not so perfect. As the first attempt snapshot code is good. But = if you want to get values related to some specific device, for example, for NIC or HPET, = you cannot get it easily. Please try :) Stream doesn=E2=80=99t have flexibility. It is good for well specified = and long long time discussed protocols like XDR (NFS), when it has RFC and each position in the stream is = described. Example: RFC1813. New format with NVLIST has flexibility and is fast enough. Note, ZFS = uses nvlist for keeping attributes=20 and more another things. >> Adding optional variable - breaks resume >> Removing variable - breaks resume >> Changing saved order of variables - breaks resume >=20 > Obviously need improvement :-) >=20 >> Hard to get information about what is saved and decode. >> Hard to debug if somethings goes wrong >=20 > Additional tools missing? Will new format allow text editor = interaction? Why do you need modify snapshot image ? Could you describe more? Do you modify current 3 snapshot files? >> No versions. If change code, resume of an old images can be >> passed, but with UB. >=20 > Is new format future proof and provides backward compatibility? Intention of moving to the new format - to have backward compatibility = if some code is changed: Adding optional variable=20 Removing variable that is not used anymore Change order of saving variables =E2=80=9CHot Fixes=E2=80=9D. If changes are critical and are incompatible, restore stage should have = clear information about incompatibility and break resume. Ideally it should be able to get = informed even before starting restore process. For this purpose, the new format introduce versions. >=20 >> New nvlist implementation should solve all things above. The first = step - >> improve snapshot/checkpoint saving format. It eliminates three files = usage >> per a snapshot. >>=20 >> (..) >=20 > So this will be new text config based format with variable =3D value = and sections? This is NVLIST approach with key=3Dvalue, where key is string, and value = can be Integer, array, string, etc. >=20 > How much bigger will be the overal file size increase? Not so huge. NVLIST internals is well specified. For example, for my VM [kernel] kernel.offset =3D 0x11f6 (4598) kernel.size =3D 0x19a7 (6567) kernel.type =3D =E2=80=9Cnvlist" [devices] devices.offset =3D 0x2b9d (11165) devices.size =3D 0x10145ba (16860602) devices.type =3D =E2=80=9Cnvlist=E2=80=9D So packed size for kernel is 6567 bytes, for devices is 16860602 = including framebuffer 16MB. If remove fbuf, packed nvlist devices Section has size = 83386 bytes. >=20 > How much longer it will take do decode/encode/process files? It is fast, just several milliseconds. NVLIST is very fast format. It is = already integrated into bhyve as Config engine. >=20 > What is the possibility of format change and backward/foward = compatibility? If you are talking about compatibility of a Image format - it should be = compatible in both directions, at least for not so big format changes. If consider overall snapshot/resume compatibility - I believe forward = compatibility is not case and target. Indeed, why do you need to resume an image = created by a higher version of a program?=20 The most important thing - backward compatibility, i.e. when an image is = created by an older version of a program, but should be resumed on a new one. This is target and and intention of this improvement. >=20 > Have you considered efficiency comparison of current format, proposed > format, and maybe using SQLITE or JSON storage/parsers? For instance > sqlite would be blazingly fast but hard to migrate. json would be most > versatile but more time/memory consuming? Yes, I know about another formats, like JSON or others. NVLIST is the = most effective and suitable for the current purposes. >=20 > Maybe EFL approach of storing configuration files for limited > resources embedded system storage that use binary storage data but can > be decompressed in chunks that can be replaced in place? > https://www.enlightenment.org/develop/efl/start There are many things that can be used, but it should be well known, = easy, stable, fast and supportable. I believe NVLIST is the best choice. >=20 > Sorry for asking those questions but there may be already good and > verified solutions out there not to reinvent the wheel? :-) Thank you for your questions. If you would like, you can try to test the = new implementation and give feedback. =E2=80=94=E2=80=94=E2=80=94 Vitaliy Gusev --Apple-Mail=_B2C31D91-4697-4C26-ADE9-9F456DBBFFC5 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi = Tomek,

Try to answer to the all questions below, = please let me know if I miss some = important.


On 23 May 2023, at 21:58, Tomek CEDRO = <tomek@cedro.info> wrote:

On Tue, May 23, 2023 at = 6:06=E2=80=AFPM Vitaliy Gusev wrote:
Hi,
Here is a proposal for bhyve snapshot/checkpoint = image format improvements.
It implies moving snapshot code to nvlist = engine.

Hey there Vitaliy :-) bhyve getting more and = more traction, I am new
user of bhyve and no expert, but new and = missing features are welcome
I guess.. there was a discussion on the = mailing lists recently on
better snapshots mechanism = :-)


Current snapshot implementation = has disadvantages:
3 files per snapshot: .meta, .kern, = vram

No problem, unless new single file will be = protected against
corruption (filesystem, transfer, application = crash) and possible to
be easily and cheaply modified in = place?

Current snapshot = implementation doesn=E2=80=99t have it. I would say more, = current
pkg implementation doesn=E2=80=99t track/notify if = some of files are changed.  Binary files on a
system can = be changed, for example ELF files, without any = notification.

Tar doesn=E2=80=99t have = protection for keeping data.  Some filesystems like = ZFS
guarantee that data is not modified by underlying = disks.

Protecting requires more efforts = and it should be clearly defined: what is purpose. If
purpose = is having checksum with 99.9% reliability, NVLIST HEADER can be = widen
to have =E2=80=9Cchecksum= =E2=80=9D key/value for a Section.

If = purpose is having crypto verification - I believe sha256 program = should be your = choice.


Binary Stream = format of data.

This is small and fast? Will new = format too?

Small is not so = perfect. As the first attempt snapshot code is good. But if you want to = get
values related to some specific device, for example, for = NIC or HPET, you cannot get it easily. Please
try = :)

Stream doesn=E2=80=99t have = flexibility. It is good for well specified  and long long time = discussed protocols
like XDR (NFS), when it has RFC and each = position in the stream is described. Example: = RFC1813.

New format with NVLIST has flexibility = and is fast enough. Note, ZFS uses nvlist for keeping = attributes 
and more another = things.


Adding  optional = variable - breaks resume
Removing variable - breaks = resume
Changing saved order of variables - breaks = resume

Obviously need improvement = :-)

Hard to get information about what = is saved and decode.
Hard to debug if somethings goes = wrong

Additional tools missing? Will new format = allow text editor = interaction?

Why do you need = modify snapshot image ? Could you describe more? Do you
modify = current 3 snapshot files?


No versions. If change = code, resume of an old images can be
passed, but with = UB.

Is new format future proof and provides backward = compatibility?

Intention of = moving to the new format - to have backward compatibility if some = code
is changed:
  • Adding optional = variable 
  • Removing variable that is not used = anymore
  • Change order of saving variables
  • =E2=80=9CHot = Fixes=E2=80=9D.

If = changes are critical and are incompatible, restore stage should have = clear information about
incompatibility and break resume. = Ideally it should be able to get informed even before = starting
restore process. For this purpose, the new format = introduce versions.



New nvlist = implementation should solve all things above. The first step = -
improve snapshot/checkpoint saving format. It eliminates three = files usage
per a snapshot.

(..)

So this = will be new text config based format with variable =3D value and = sections?

This is NVLIST = approach with key=3Dvalue, where key is string, and value can = be
Integer, array, string, etc.


How much bigger will be the overal file size = increase?

Not so huge. NVLIST = internals is well specified. For example, for my = VM

  [kernel]

    =     kernel.offset =3D 0x11f6 (4598)

        kernel.size =3D 0x19a7 = (6567)

        kernel.type =3D = =E2=80=9Cnvlist"

  [devices]

    =     devices.offset =3D 0x2b9d (11165)

        devices.size =3D = 0x10145ba (16860602)

        devices.type =3D = =E2=80=9Cnvlist=E2=80=9D


So packed size for = kernel  is 6567 = bytes, for devices  is 16860602 = including
framebuffer 16MB. If remove fbuf, packed nvlist = devices Section has size 83386 bytes.



How much longer it will = take do decode/encode/process = files?

It is fast, just = several milliseconds. NVLIST is very fast format. It is already = integrated
into bhyve as Config = engine.



What is the possibility of format change and = backward/foward = compatibility?

If you = are talking about compatibility of a Image format - it should be = compatible in
both directions, at least for not so big format = changes.

If consider overall = snapshot/resume compatibility - I believe  forward = compatibility
is not case and target. Indeed, why do you need =  to resume an image created by
a higher version of a = program? 

The most important thing - = backward compatibility, i.e. when an image is created
by an = older version of a program, but should be resumed on a new = one.

This is target and and intention of = this improvement.


Have you considered efficiency comparison of = current format, proposed
format, and maybe using SQLITE or JSON = storage/parsers?  For instance
sqlite would be blazingly fast = but hard to migrate. json would be most
versatile but more = time/memory = consuming?

Yes, I know = about another formats, like JSON or others. NVLIST is the = most
effective and suitable for the current = purposes.


Maybe EFL approach of storing configuration = files for limited
resources embedded system storage that use binary = storage data but can
be decompressed in chunks that can be replaced = in = place?
https://www.enlightenment.org/develop/efl/start
<= /blockquote>

There are many things that can be used, = but it should be well known, easy, stable,
fast and = supportable. I believe NVLIST is the best choice.


Sorry for asking those questions but there = may be already good and
verified solutions out there not to reinvent = the wheel? :-)

Thank you for = your questions. If you would like, you can try to test the new = implementation and give = feedback.

=E2=80=94=E2=80=94=E2=80=94
V= italiy Gusev

= --Apple-Mail=_B2C31D91-4697-4C26-ADE9-9F456DBBFFC5-- From nobody Wed May 24 17:33:48 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRJGB2MMVz4TJyg; Wed, 24 May 2023 17:34:26 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-yb1-xb2f.google.com (mail-yb1-xb2f.google.com [IPv6:2607:f8b0:4864:20::b2f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRJG946Pcz4FFD; Wed, 24 May 2023 17:34:25 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=IIFeit2R; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::b2f as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-yb1-xb2f.google.com with SMTP id 3f1490d57ef6-ba81deea9c2so1137434276.2; Wed, 24 May 2023 10:34:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684949664; x=1687541664; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=daVIsXoWCxfEdp0Yy3GFJd1A7zl9NWnIHW82T+Gn7hU=; b=IIFeit2Roa2ZjoByscqIlT4bIzJoSA8fkM4ggPvIFCXKEY8uFM2DpB7PgfOdLDUIPN fQyc6C6Eu1mYDHaQosld2Gm9nnWk2NxTM/eU2a4jS0g/GJbtX+8Rq028unIRSR+5kED3 ItBDZHazs0D3jGXb8Vp/XdlZ8XXWDvAtDiwBEOoCda5b5XVJe8VqFM3tOIjWkbq3IAGI B16Y5CyPtnhFBNexmW/bYFXLbj8tsKW7PInhsxy6u825tr0K9e8cE0Uxg5SXFBhXPB5m WCD+jwpkj9Pna8UW8IDL1W0iPvFJ3eRId3IwZHzQitogC7LQLBPJrw2JjbiNpucSELax tYdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684949664; x=1687541664; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=daVIsXoWCxfEdp0Yy3GFJd1A7zl9NWnIHW82T+Gn7hU=; b=XoMcSgRu7iRw67oJnUMTaLLfgnLmyog7RdzZbBSxxC+cfnyoqGJgSFlIOyROtGAx/G hkgO5iNwNJ8tOgGuj4KcWNzbBCaHWWJARb+7mT/y60NorykP4A2eWrwoBF7JwzPFIPP4 a1MLpm1lKqSS//35oIoqYTZoVRaCyy1INwKkNAudh77u18k1LnCZEDRrDWGVG0oSYoIL dQejlyn2CRHtQPO2Bb9ClTiMF8/9Akarc3fzTYs8uEz8GLfzHmT4Jo3f567X9EiFQcfO OVk7RABW0BdbGfQ8bJgaXOck4mKWVT79eYTtvCFQFAAUT5ge/Jb4l3ZU9mjFrr+XKZtq H1Aw== X-Gm-Message-State: AC+VfDyJ0fRMLgjJm1N2HSWHfteg2WR+TTJoa3PMafvi/J/VDgbE3jup UwY3etnadc+ivOW7LycAFTDQcOao1PemI0ucDyGMHVf4GlI= X-Google-Smtp-Source: ACHHUZ6HWOttZ4LOsdCm4nxrJ/jdkpiVh2dlOq9tibJEWmGZbAAWsFlg7xmregcGiSCTYsSXc2WxSp2ikFTLq56R0UY= X-Received: by 2002:a25:bc9:0:b0:bac:6114:7376 with SMTP id 192-20020a250bc9000000b00bac61147376mr636720ybl.15.1684949664566; Wed, 24 May 2023 10:34:24 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> In-Reply-To: From: Mario Marietto Date: Wed, 24 May 2023 19:33:48 +0200 Message-ID: Subject: Re: BHYVE SNAPSHOT image format proposal To: Vitaliy Gusev Cc: Tomek CEDRO , virtualization@freebsd.org, freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="00000000000038930005fc73ea5e" X-Spamd-Result: default: False [-2.44 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.935]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org,freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b2f:from]; RCPT_COUNT_THREE(0.00)[4]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4QRJG946Pcz4FFD X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --00000000000038930005fc73ea5e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable @gusev.vitaliy@gmail.com : Do you want to explain to me how to test the new "snapshot" feature ? I'm interested to test and stress it on my system. Is it ready to be used ? On Wed, May 24, 2023 at 5:11=E2=80=AFPM Vitaliy Gusev wrote: > Hi Tomek, > > Try to answer to the all questions below, please let me know if I miss > some important. > > > On 23 May 2023, at 21:58, Tomek CEDRO wrote: > > On Tue, May 23, 2023 at 6:06=E2=80=AFPM Vitaliy Gusev wrote: > > Hi, > Here is a proposal for bhyve snapshot/checkpoint image format improvement= s. > It implies moving snapshot code to nvlist engine. > > > Hey there Vitaliy :-) bhyve getting more and more traction, I am new > user of bhyve and no expert, but new and missing features are welcome > I guess.. there was a discussion on the mailing lists recently on > better snapshots mechanism :-) > > > Current snapshot implementation has disadvantages: > 3 files per snapshot: .meta, .kern, vram > > > No problem, unless new single file will be protected against > corruption (filesystem, transfer, application crash) and possible to > be easily and cheaply modified in place? > > > Current snapshot implementation doesn=E2=80=99t have it. I would say more= , current > pkg implementation doesn=E2=80=99t track/notify if some of files are chan= ged. > Binary files on a > system can be changed, for example ELF files, without any notification. > > Tar doesn=E2=80=99t have protection for keeping data. Some filesystems l= ike ZFS > guarantee that data is not modified by underlying disks. > > Protecting requires more efforts and it should be clearly defined: what i= s > purpose. If > purpose is having checksum with 99.9% reliability, NVLIST HEADER can be > widen > to have =E2=80=9Cchecksum=E2=80=9D key/value for a Section. > > If purpose is having crypto verification - I believe sha256 program shoul= d > be your choice. > > > Binary Stream format of data. > > > This is small and fast? Will new format too? > > > Small is not so perfect. As the first attempt snapshot code is good. But > if you want to get > values related to some specific device, for example, for NIC or HPET, you > cannot get it easily. Please > try :) > > Stream doesn=E2=80=99t have flexibility. It is good for well specified a= nd long > long time discussed protocols > like XDR (NFS), when it has RFC and each position in the stream is > described. Example: RFC1813. > > New format with NVLIST has flexibility and is fast enough. Note, ZFS uses > nvlist for keeping attributes > and more another things. > > > Adding optional variable - breaks resume > Removing variable - breaks resume > Changing saved order of variables - breaks resume > > > Obviously need improvement :-) > > Hard to get information about what is saved and decode. > Hard to debug if somethings goes wrong > > > Additional tools missing? Will new format allow text editor interaction? > > > Why do you need modify snapshot image ? Could you describe more? Do you > modify current 3 snapshot files? > > > No versions. If change code, resume of an old images can be > passed, but with UB. > > > Is new format future proof and provides backward compatibility? > > > Intention of moving to the new format - to have backward compatibility if > some code > is changed: > > > - Adding optional variable > - Removing variable that is not used anymore > - Change order of saving variables > - =E2=80=9CHot Fixes=E2=80=9D. > > > If changes are critical and are incompatible, restore stage should have > clear information about > incompatibility and break resume. Ideally it should be able to get > informed even before starting > restore process. For this purpose, the new format introduce versions. > > > > New nvlist implementation should solve all things above. The first step - > improve snapshot/checkpoint saving format. It eliminates three files usag= e > per a snapshot. > > (..) > > > So this will be new text config based format with variable =3D value and > sections? > > > This is NVLIST approach with key=3Dvalue, where key is string, and value = can > be > Integer, array, string, etc. > > > How much bigger will be the overal file size increase? > > > Not so huge. NVLIST internals is well specified. For example, for my VM > > [kernel] > > kernel.offset =3D 0x11f6 (4598) > > kernel.size =3D 0x19a7 (6567) > > kernel.type =3D =E2=80=9Cnvlist" > > [devices] > > devices.offset =3D 0x2b9d (11165) > > devices.size =3D 0x10145ba (16860602) > > devices.type =3D =E2=80=9Cnvlist=E2=80=9D > > So packed size for *kernel* is 6567 bytes, for *devices* is 16860602 > including > framebuffer 16MB. If remove fbuf, packed nvlist devices Section has size > 83386 bytes. > > > > How much longer it will take do decode/encode/process files? > > > It is fast, just several milliseconds. NVLIST is very fast format. It is > already integrated > into bhyve as Config engine. > > > > What is the possibility of format change and backward/foward compatibilit= y? > > > If you are talking about compatibility of a Image format - it should be > compatible in > both directions, at least for not so big format changes. > > If consider overall snapshot/resume compatibility - I believe forward > compatibility > is not case and target. Indeed, why do you need to resume an image > created by > a higher version of a program? > > The most important thing - backward compatibility, i.e. when an image is > created > by an older version of a program, but should be resumed on a new one. > > This is target and and intention of this improvement. > > > Have you considered efficiency comparison of current format, proposed > format, and maybe using SQLITE or JSON storage/parsers? For instance > sqlite would be blazingly fast but hard to migrate. json would be most > versatile but more time/memory consuming? > > > Yes, I know about another formats, like JSON or others. NVLIST is the mos= t > effective and suitable for the current purposes. > > > Maybe EFL approach of storing configuration files for limited > resources embedded system storage that use binary storage data but can > be decompressed in chunks that can be replaced in place? > https://www.enlightenment.org/develop/efl/start > > > There are many things that can be used, but it should be well known, easy= , > stable, > fast and supportable. I believe NVLIST is the best choice. > > > Sorry for asking those questions but there may be already good and > verified solutions out there not to reinvent the wheel? :-) > > > Thank you for your questions. If you would like, you can try to test the > new implementation and give feedback. > > =E2=80=94=E2=80=94=E2=80=94 > Vitaliy Gusev > > --=20 Mario. --00000000000038930005fc73ea5e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
@gusev.vitaliy@gmail= .com : Do you want to explain to me how to test the new "snapshot&= quot; feature ? I'm interested to test and stress it on my system. Is i= t ready to be used ?

On Wed, May 24, 2023 at 5:11=E2=80=AFPM Vitaliy G= usev <gusev= .vitaliy@gmail.com> wrote:
Hi Tomek,

Try to answer to the al= l questions below, please let me know if I miss some important.
<= br>

On 23 May 2023, at 21= :58, Tomek CEDRO <= tomek@cedro.info> wrote:

On Tue, May 23, 2023 at = 6:06=E2=80=AFPM Vitaliy Gusev wrote:
Hi,
He= re is a proposal for bhyve snapshot/checkpoint image format improvements.It implies moving snapshot code to nvlist engine.

Hey= there Vitaliy :-) bhyve getting more and more traction, I am new
user o= f bhyve and no expert, but new and missing features are welcome
I guess.= . there was a discussion on the mailing lists recently on
better snapsho= ts mechanism :-)


Current snapshot impl= ementation has disadvantages:
3 files per snapshot: .meta, .kern, vram

No problem, unless new single file will be protected aga= inst
corruption (filesystem, transfer, application crash) and possible t= o
be easily and cheaply modified in place?
<= div>
Current snapshot implementation doesn=E2=80=99t have it.= I would say more, current
pkg implementation doesn=E2=80=99t tra= ck/notify if some of files are changed.=C2=A0 Binary files on a
s= ystem can be changed, for example ELF files, without any notification.

Tar doesn=E2=80=99t have protection for keeping d= ata.=C2=A0 Some filesystems like ZFS
guarantee that data is not m= odified by underlying disks.

Protecting requ= ires more efforts and it should be clearly defined: what is purpose. If
purpose is having checksum with 99.9%=C2=A0reliability, NVLIST HEADER can be widen=
to have =E2=80=9Cchecksum=E2=80=9D key/value for a Sectio= n.

If purpose=C2=A0is having crypto = verification - I believe sha256 program should be your choice.


Binary Stream format of data.

This = is small and fast? Will new format too?
Small is not so perfect. As the first attempt snapshot code is good= . But if you want to get
values related to some specific device, = for example, for NIC or HPET, you cannot get it easily. Please
tr= y :)

Stream doesn=E2=80=99t have flexibility.= It is good for well specified =C2=A0and long long time discussed protocols=
like XDR (NFS), when it has RFC and each position in the stream = is described. Example: RFC1813.

New format with NV= LIST has flexibility and is fast enough. Note, ZFS uses nvlist for keeping = attributes=C2=A0
and more another things.


A= dding =C2=A0optional variable - breaks resume
Removing variable - breaks= resume
Changing saved order of variables - breaks resume

Obviously need improvement :-)

Hard = to get information about what is saved and decode.
Hard to debug if some= things goes wrong

Additional tools missing? Will new fo= rmat allow text editor interaction?

Why do you need modify snapshot image ? Could you describe more? Do you=
modify current 3 snapshot files?


No versions. If change cod= e, resume of an old images can be
passed, but with UB.
<= br>Is new format future proof and provides backward compatibility?

Intention of moving to the new format - = to have backward compatibility if some code
is changed:
  • Adding optional variable=C2=A0
  • Removing variable t= hat is not used anymore
  • Change order of saving variables
  • = =E2=80=9CHot Fixes=E2=80=9D.

If changes are critical and are incompatible, restore stage should h= ave clear information about
incompatibility and break resume. Ide= ally it should be able to get informed even before starting
resto= re process. For this purpose, the new format introduce versions.
=



New nvlist implementation should solve all things above. The firs= t step -
improve snapshot/checkpoint saving format. It eliminates three = files usage
per a snapshot.

(..)

So this will= be new text config based format with variable =3D value and sections?
<= /div>

This is NVLIST approach with key=3Dv= alue, where key is string, and value can be
Integer, array, strin= g, etc.


How much bigg= er will be the overal file size increase?
=
Not so huge. NVLIST internals is well specified. For example, for= my VM

=C2=A0 [kernel]

=C2= =A0 =C2=A0 =C2=A0 =C2=A0 kernel.offset =3D 0x11f6 (4598)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 kernel.size= =3D 0x19a7 (6567)

= =C2=A0 =C2=A0 =C2=A0 =C2=A0 kernel.type =3D =E2=80=9Cnvlist"

=C2=A0 [devices]

=C2=A0 =C2=A0 =C2=A0 =C2=A0 device= s.offset =3D 0x2b9d (11165)

=C2=A0 =C2=A0 =C2=A0 =C2=A0 devices.size =3D 0x10145ba (16860602)

=C2=A0 =C2=A0 =C2=A0 = =C2=A0 devices.type =3D =E2=80=9Cnvlist=E2=80=9D


S= o packed size for kernel=C2=A0 is 65= 67 bytes, for devices=C2=A0 is=C2=A016860602 including
frameb= uffer 16MB. If remove fbuf, packed nvlist devices Section has size=C2=A083386=C2=A0by= tes.



How much longer it will take do decode/encode/process files?
<= /div>

It is fast, just several millis= econds. NVLIST is very fast format. It is already integrated
into= bhyve as Config engine.


=

What is the possibility of format change and backward/foward = compatibility?

If you are t= alking about compatibility of a Image format - it should be compatible in
both directions, at least for not so big format changes.

If consider overall snapshot/resume compatibility - I = believe =C2=A0forward compatibility
is not case and target. Indee= d, why do you need =C2=A0to resume an image created by
a higher v= ersion of a program?=C2=A0

The most important thin= g - backward compatibility, i.e. when an image is created
by an o= lder version of a program, but should be resumed on a new one.
<= div>
This is target and and intention of this improvement.


Have you consider= ed efficiency comparison of current format, proposed
format, and maybe u= sing SQLITE or JSON storage/parsers?=C2=A0 For instance
sqlite would be = blazingly fast but hard to migrate. json would be most
versatile but mor= e time/memory consuming?

Ye= s, I know about another formats, like JSON or others. NVLIST is the most
effective and suitable for the current purposes.


Maybe EFL approach of storing con= figuration files for limited
resources embedded system storage that use = binary storage data but can
be decompressed in chunks that can be replac= ed in place?
https://www.enlightenment.org/develop/efl/start
<= /div>

There are many things that can = be used, but it should be well known, easy, stable,
fast and supp= ortable. I believe NVLIST is the best choice.


Sorry for asking those questions but there may be alrea= dy good and
verified solutions out there not to reinvent the wheel? :-)<= br>

Thank you for your questions. If= you would like, you can try to test the new implementation and give feedba= ck.

=E2=80=94=E2=80=94=E2=80=94
Vitaliy = Gusev


=
--
Mario.
--00000000000038930005fc73ea5e-- From nobody Wed May 24 17:38:50 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRJMW2zcjz4TK3W; Wed, 24 May 2023 17:39:03 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x133.google.com (mail-lf1-x133.google.com [IPv6:2a00:1450:4864:20::133]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRJMW1228z4Gfn; Wed, 24 May 2023 17:39:03 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x133.google.com with SMTP id 2adb3069b0e04-4eed764a10cso1250619e87.0; Wed, 24 May 2023 10:39:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684949941; x=1687541941; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=eGFUaUP6pqdLyTrDBgb8y5Us42JMSTCQV6+Q03LsqPA=; b=C7PXn+PUZcLBLQV6UOn51DKvrbgJnaUeK5LBmqS6J1WMa/0HE0wMb7ojFZwI2ghenB h1DPrDlVLHkAQPk/Ru6Dvllc0duQaSukQizsmIz/ok43PLVypbpctCGfvjOLOFN6VgfQ Aj8Q/mGuSickPPMMcPfum8SOLSoCtjlbObA5hwkGTK22uDMEZ0PZ1NvA1mCKIDynpzL0 dqhpyHCuuiqzkNFFJ9g3hbE9ufkAEvGsC+wXj/zk4HeT472KP6ztRhmabwzZ2uR/Ele5 Lvc96/mWdG472/QOF+oFGNQPXVaQzCP0ObViNQZxht27eSzU/Vl13Eu6UtfZD6AV3Qr2 1Cnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684949941; x=1687541941; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=eGFUaUP6pqdLyTrDBgb8y5Us42JMSTCQV6+Q03LsqPA=; b=g0YqJBSDzZczaMsKOs/akEmIw+wM7rIaL1Fj2tV/8hSa2P3UzFD1vAJg4iFATY3lQa Lgn74DNoM65zSbXmqegYmdvSyKvhyRugPjVVIv97Fmmdx9oHdJNCJQzmBJEFndBXcSnS 0iM0VXXPPW7DIMV+eBKKFnSU3zjzX/Dtg+UfEUJvjDnGZuCjrUh1rAIfthVx4wH+OBLC vEunCEm9BF18PUt4OcFWrNhuZ1AHoHloO3YeJ2h1Daf+FDd4fA+YGw5i2ly0vljNf5dz R5qJdoPLeIiquOAhz77rtbZugGA1WZKRYp53bhLAMJza3LjnrSg39bBcshqdrcR2qYdK c7Rg== X-Gm-Message-State: AC+VfDz9dAb9tkwDsYuYstBcDNtkImw1dvy8Y19KIVk2O8Yq+wZDPwi1 0Sb9wQBm2jsqDlwCLOpokMmgHFgeltQ= X-Google-Smtp-Source: ACHHUZ4SCY+shpDGBTbmFSKpPC5OSpC7Kpomskqz1p80uGYgtQOUGNFSyVwY81KJbTOKPHmuzEgQgg== X-Received: by 2002:ac2:5544:0:b0:4f3:8260:f18c with SMTP id l4-20020ac25544000000b004f38260f18cmr5396965lfk.57.1684949941218; Wed, 24 May 2023 10:39:01 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id b1-20020a056512024100b004b4cbc942a3sm1811551lfo.127.2023.05.24.10.39.00 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 May 2023 10:39:00 -0700 (PDT) From: Vitaliy Gusev Message-Id: <3054B6FB-8408-492C-9F18-91A61516D0EF@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_3D732BA7-A8EF-4056-A822-0ECD52A24FF2" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE SNAPSHOT image format proposal Date: Wed, 24 May 2023 20:38:50 +0300 In-Reply-To: Cc: Tomek CEDRO , virtualization@freebsd.org, freebsd-hackers@freebsd.org To: Mario Marietto References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4QRJMW1228z4Gfn X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_3D732BA7-A8EF-4056-A822-0ECD52A24FF2 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 It is ready but in internal git repo, and it needs time to port it to = CURRENT version. Probably week or so. When I create review, I will notify you. Thanks, Vitaliy Gusev > On 24 May 2023, at 20:33, Mario Marietto = wrote: >=20 > @gusev.vitaliy@gmail.com : Do you = want to explain to me how to test the new "snapshot" feature ? I'm = interested to test and stress it on my system. Is it ready to be used ? >=20 >>=20 >> Thank you for your questions. If you would like, you can try to test = the new implementation and give feedback. >>=20 >> =E2=80=94=E2=80=94=E2=80=94 >> Vitaliy Gusev >>=20 >=20 >=20 > --=20 > Mario. --Apple-Mail=_3D732BA7-A8EF-4056-A822-0ECD52A24FF2 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 It is ready = but in internal git repo, and it needs time to port it to CURRENT = version. Probably week or so.

When I create review, I = will notify you.

Thanks,
Vitaliy = Gusev


On 24 May = 2023, at 20:33, Mario Marietto <marietto2008@gmail.com> = wrote:

@gusev.vitaliy@gmail.com : Do you want to explain to = me how to test the new "snapshot" feature ? I'm interested to test and = stress it on my system. Is it ready to be used ?


Thank you for your = questions. If you would like, you can try to test the new implementation = and give feedback.

=E2=80=94=E2=80=94=E2=80=94
Vitaliy Gusev



--
Mario.

= --Apple-Mail=_3D732BA7-A8EF-4056-A822-0ECD52A24FF2-- From nobody Wed May 24 17:46:13 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRJWv1YhBz4TKTY; Wed, 24 May 2023 17:46:19 +0000 (UTC) (envelope-from SRS0=9ZGR=BN=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRJWt0Qsbz4JF1; Wed, 24 May 2023 17:46:18 +0000 (UTC) (envelope-from SRS0=9ZGR=BN=quip.cz=000.fbsd@elsa.codelab.cz) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of "SRS0=9ZGR=BN=quip.cz=000.fbsd@elsa.codelab.cz" has no SPF policy when checking 94.124.105.4) smtp.mailfrom="SRS0=9ZGR=BN=quip.cz=000.fbsd@elsa.codelab.cz"; dmarc=none Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id AD7E0D7899; Wed, 24 May 2023 19:46:14 +0200 (CEST) Received: from [192.168.145.49] (ip-89-177-27-225.bb.vodafone.cz [89.177.27.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id E2D06D7891; Wed, 24 May 2023 19:46:13 +0200 (CEST) Message-ID: Date: Wed, 24 May 2023 19:46:13 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 Subject: Re: BHYVE SNAPSHOT image format proposal Content-Language: cs-Cestina To: Vitaliy Gusev Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> From: Miroslav Lachman <000.fbsd@quip.cz> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-1.79 / 15.00]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.990]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=9ZGR=BN=quip.cz=000.fbsd@elsa.codelab.cz]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,virtualization@freebsd.org]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; R_DKIM_NA(0.00)[]; R_SPF_NA(0.00)[no SPF record]; MIME_TRACE(0.00)[0:+]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=9ZGR=BN=quip.cz=000.fbsd@elsa.codelab.cz]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TAGGED_RCPT(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[quip.cz]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4QRJWt0Qsbz4JF1 X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On 24/05/2023 17:10, Vitaliy Gusev wrote: >>> Current snapshot implementation has disadvantages: >>> 3 files per snapshot: .meta, .kern, vram >> >> No problem, unless new single file will be protected against >> corruption (filesystem, transfer, application crash) and possible to >> be easily and cheaply modified in place? > > Current snapshot implementation doesn’t have it. I would say more, current > pkg implementation doesn’t track/notify if some of files are changed. >  Binary files on a > system can be changed, for example ELF files, without any notification. pkg stores checksums for installed files. You can check them with pkg check -s -a or pkg check --checksums -a. Changes are reported by daily periodic script. Kind regards Miroslav Lachman From nobody Wed May 24 17:47:42 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRJYm3jtWz4TKYD; Wed, 24 May 2023 17:47:56 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-yb1-xb34.google.com (mail-yb1-xb34.google.com [IPv6:2607:f8b0:4864:20::b34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRJYl1kLpz4KKK; Wed, 24 May 2023 17:47:55 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=m5SEENiR; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::b34 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-yb1-xb34.google.com with SMTP id 3f1490d57ef6-ba1815e12efso1127614276.3; Wed, 24 May 2023 10:47:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684950474; x=1687542474; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=JchYAiXV43rmVn9K0gGeJrAwEh7reGDM2znr87FB/Pk=; b=m5SEENiRKE6l5SqKSfONZ+VrXE0su4MCFGrF2enh0+qp+YPY83f45ihL308tcy8MjX Zf3kuOrMeNOnBD34+u8RoejiBkFcIst57cYrv/6ZZFDHxvX7s0PYCf9BiXSK8xl76F8R px102xCNF+Zd+PZLUUtVzqQ6GHxw0ukRqKzUANZVov++qe6sRiZAKXNiadkNY6bVeNgO +vxd6YVm3dP1Vjy50m9f/+cSNoVLa9RcsLIpWnRl5KiOZRhcENQ98vXEPcDwUhkMbUAP bfdD6YPueqSOAcpOgroG4NtrxhaSqfsRRppNZDVZT5pr2WstnYTRmpBYoiIbU09t32KP S+gA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684950474; x=1687542474; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JchYAiXV43rmVn9K0gGeJrAwEh7reGDM2znr87FB/Pk=; b=L+Pj51pUKvx046+hhD7BbZ0dLG2RcEauRhp29suKYKaJ9FLNQSpH38urQT2fhsuSAN cechAiHd935jZHrGqd+ll5aXndAW3D1RzmDlxour2ZHPC8A02tEBHnK/4J2LPOP7duEB kh4mMnZonmLLtHBgQipyVc+W1QcA9aqtKgg2BG1ZBR0TRIl65hWXAuZS6uHhxHx5pk73 aOzQrR2F2tTnR5KcaaSkyko4we9yUOMHEfl6erDbMMAU0IRCDqU0Obp3CxRclhUR/ZUA 3D7VjGqUX5PwvXBlp4PEJamvBXjW3FEZGpmBaKgTqDf+iscWVy7f0Ungywdv/uPmw0Cb fw2g== X-Gm-Message-State: AC+VfDxjXVj9cssxcwy7R2oUbDiBhVkFUivfU/dASgvwCygwW87AWnRx 1akEuo6w8OoTiNKwQDjulalVl0ms3POzuax4SOo= X-Google-Smtp-Source: ACHHUZ4+6neEUWFW3hFExorhD9Pj2IVcebu/TRpwTk4PYIRV7R/wVe91Nlf7sXR9tEJ13wzy+aVtv2eRJQS9E96fEdI= X-Received: by 2002:a25:aa2b:0:b0:bac:6d5a:f6af with SMTP id s40-20020a25aa2b000000b00bac6d5af6afmr695372ybi.35.1684950474289; Wed, 24 May 2023 10:47:54 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <3054B6FB-8408-492C-9F18-91A61516D0EF@gmail.com> In-Reply-To: <3054B6FB-8408-492C-9F18-91A61516D0EF@gmail.com> From: Mario Marietto Date: Wed, 24 May 2023 19:47:42 +0200 Message-ID: Subject: Re: BHYVE SNAPSHOT image format proposal To: Vitaliy Gusev Cc: Tomek CEDRO , virtualization@freebsd.org, freebsd-hackers Content-Type: multipart/alternative; boundary="0000000000007bf70505fc741ae9" X-Spamd-Result: default: False [-2.44 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.940]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org,freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::b34:from]; RCPT_COUNT_THREE(0.00)[4]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4QRJYl1kLpz4KKK X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --0000000000007bf70505fc741ae9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Give me the internal git. I want to test it asap :) Il mer 24 mag 2023, 19:39 Vitaliy Gusev ha scritto: > It is ready but in internal git repo, and it needs time to port it to > CURRENT version. Probably week or so. > > When I create review, I will notify you. > > Thanks, > Vitaliy Gusev > > > On 24 May 2023, at 20:33, Mario Marietto wrote: > > @gusev.vitaliy@gmail.com : Do you want to > explain to me how to test the new "snapshot" feature ? I'm interested to > test and stress it on my system. Is it ready to be used ? > > >> Thank you for your questions. If you would like, you can try to test the >> new implementation and give feedback. >> >> =E2=80=94=E2=80=94=E2=80=94 >> Vitaliy Gusev >> >> > > -- > Mario. > > > --0000000000007bf70505fc741ae9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Give me the internal git. I want to test it asap :)
=
Il mer= 24 mag 2023, 19:39 Vitaliy Gusev <gusev.vitaliy@gmail.com> ha scritto:
It is ready but = in internal git repo, and it needs time to port it to CURRENT version. Prob= ably week or so.

When I create review, I will notify you= .

Thanks,
Vitaliy Gusev


On 24 May 2023, at 20:33, Mario Marie= tto <marietto2008@gmail.com> wrote:

@gusev.vitaliy@gmail.com : Do you want to explain to me how= to test the new "snapshot" feature ? I'm interested to test = and stress it on my system. Is it ready to be used ?

=
Thank you for your questions. If you would like, you can try to test th= e new implementation and give feedback.

=E2=80=94= =E2=80=94=E2=80=94
Vitaliy Gusev



--
Mario.

--0000000000007bf70505fc741ae9-- From nobody Wed May 24 18:16:27 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRKBw2ymtz4TLxq; Wed, 24 May 2023 18:16:40 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRKBw0xLyz4MkC; Wed, 24 May 2023 18:16:40 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-4f3b337e842so1278211e87.3; Wed, 24 May 2023 11:16:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684952198; x=1687544198; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=c97IKDb8B7IJ+wq3pHrDHqdGy0JjQu5fvMbgBY5XtaI=; b=k7/X4IUHNBSQqWTA6u73OAQKngCrTtF51iJpVh1gkeYu8XpUw5EPBBAX0Sc8HoXaAl cqLsbGCeXvgDiiWIlkp7pMEa92awqngyLqeZDOjXyaVbgKLZlzz0ecb8ITcO1xmtvWSW C6qHk44wrvFyXGtOPw7ZUdyYRT6sB2rWbi+QjZwVKRmw7jTTTGhFiYfdF1QEYy2JgJTQ ntLPoILxcJbXxq9JCSU2k4hCdMNaa+uM9wb6VuqGTrDqUYoIeVmIxkDFBwaEWo26SlHp v0222ZJ0FuUQ/tGJxDNffWto9Qyx8ox1KO0mEyldHj3BNv2Ry1Z/yQ1s6Pyxb472t0Vk ab/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684952198; x=1687544198; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=c97IKDb8B7IJ+wq3pHrDHqdGy0JjQu5fvMbgBY5XtaI=; b=YoBtffFHx3lb6VMSmEnD0IvewCqMU+kbQQ/9W3jK/Ge6gppxfVtJvv6LMgsV2IzWD9 F5SnS730FH7KLxCUxEYlS4O20IyYyppmQU6VleEeEmawqM0fqeLdVXy+BRehqfsDQOTQ FlVJEA+jH74nsueR8O+qTQ2Ppzz+t90jeMft0LQg9mu3NqlLx0ERoLUXldoTak1PTDO7 TOh96hdvL7gQ8zYcWk2ro27A/b3XGT25KZ7wFMgZentS9BmY0ovsGQThWVzX33yNME// KH2CNgWBZNkgjdiFnHvOTht+URg2oftFp6ZP0hKfGpiniBBHt+L3hoGMPLaRRvoAP8l9 uiAA== X-Gm-Message-State: AC+VfDzxXGn9rB2mGcOu+rmIjajxVvVn8xT9cxjUB/eJEYsV/4oUWd9E FBzM7QOt5trCx9+yhRT+s/U= X-Google-Smtp-Source: ACHHUZ7Bj4CXEDY6e7q6r7inG7OG1Y5EHg0SzQRw/F/OUXrsXBjbZkkSpQT3Ji8hu8KV+IcAlOUDxw== X-Received: by 2002:ac2:5298:0:b0:4ef:f725:ae2f with SMTP id q24-20020ac25298000000b004eff725ae2fmr5625196lfm.37.1684952198299; Wed, 24 May 2023 11:16:38 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id u22-20020ac243d6000000b004f021ce4c68sm1815719lfl.80.2023.05.24.11.16.37 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 May 2023 11:16:37 -0700 (PDT) From: Vitaliy Gusev Message-Id: <91DBA80E-C6DD-4394-B69B-3B6BB63BE726@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_5729E1B5-200B-4960-B4A9-D1B168D9AB80" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE SNAPSHOT image format proposal Date: Wed, 24 May 2023 21:16:27 +0300 In-Reply-To: Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org To: Miroslav Lachman <000.fbsd@quip.cz> References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4QRKBw0xLyz4MkC X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_5729E1B5-200B-4960-B4A9-D1B168D9AB80 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi,=20 > On 24 May 2023, at 20:46, Miroslav Lachman <000.fbsd@quip.cz> wrote: >=20 > On 24/05/2023 17:10, Vitaliy Gusev wrote: >=20 >>>> Current snapshot implementation has disadvantages: >>>> 3 files per snapshot: .meta, .kern, vram >>>=20 >>> No problem, unless new single file will be protected against >>> corruption (filesystem, transfer, application crash) and possible to >>> be easily and cheaply modified in place? >> Current snapshot implementation doesn=E2=80=99t have it. I would say = more, current >> pkg implementation doesn=E2=80=99t track/notify if some of files are = changed. Binary files on a >> system can be changed, for example ELF files, without any = notification. >=20 > pkg stores checksums for installed files. You can check them with pkg = check -s -a or pkg check --checksums -a. Changes are reported by daily = periodic script. Yep, my fault. However, I found it doesn=E2=80=99t track sticky bit = setting: # chmod u+t /usr/local/bin/vim # pkg check -s vim Checking vim: 100% My point was that if snapshot image needs checksum verification it could = be done by another program, because there are many purposes (plain integrity, security, etc) and = having it in place in snapshot image could be doing double of work. And additionally note, that NVLIST Header can be widen to have a = checksum for Section data. Thanks, Vitaliy Gusev > Kind regards > Miroslav Lachman >=20 --Apple-Mail=_5729E1B5-200B-4960-B4A9-D1B168D9AB80 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi, 

On 24 May 2023, at 20:46, Miroslav Lachman = <000.fbsd@quip.cz> wrote:

On 24/05/2023 17:10, = Vitaliy Gusev wrote:

Current snapshot implementation = has disadvantages:
3 files per snapshot: .meta, .kern, = vram

No problem, unless new single file will be = protected against
corruption (filesystem, transfer, application = crash) and possible to
be easily and cheaply modified in = place?
Current snapshot implementation doesn=E2=80=99t = have it. I would say more, current
pkg implementation doesn=E2=80=99t = track/notify if some of files are changed.   Binary files on = a
system can be changed, for example ELF files, without any = notification.

pkg stores checksums for installed = files. You can check them with pkg check -s -a or pkg check --checksums = -a. Changes are reported by daily periodic = script.


Yep, = my fault. However, I found it doesn=E2=80=99t track sticky bit = setting:

# = chmod u+t /usr/local/bin/vim


# pkg = check -s vim

Checking vim: = 100%


My point was that if snapshot = image needs checksum verification it could be done by another = program,
because there are many purposes (plain integrity, = security, etc) and having it in place in snapshot image
could = be doing double of work.

And additionally note, = that NVLIST Header can be widen to have a  checksum for Section = data.

Thanks,
Vitaliy = Gusev

Kind regards
Miroslav = Lachman


= --Apple-Mail=_5729E1B5-200B-4960-B4A9-D1B168D9AB80-- From nobody Wed May 24 18:44:34 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRKqL6pp5z4TNQD; Wed, 24 May 2023 18:44:46 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRKqL6FvXz4Sgr; Wed, 24 May 2023 18:44:46 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12e.google.com with SMTP id 2adb3069b0e04-4effb818c37so1327398e87.3; Wed, 24 May 2023 11:44:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684953885; x=1687545885; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=BhglgsPDhDMkvFy5HHyCJo+/I/zn6luCz3jk0H2OX74=; b=df9l9dGWf2SXuMSJ1dcoRBqHsoWepdV3Ag56M8bijLmpTp7CkvHOqLyxuXz3omEE4r RTokLlsiOtCiROS5RjlTGJywj0efHXpoeq/Be++hSLgTl8W6otVmSt6ZYLWJY2PVds7h vLGcw5wXV0pSEHHeut6ZNwYyqzidnHiT3eCf6hoy+ZtD3+ykTCfNrw5efuIjSI1+X+5D i93D7q87D2Itn/1W9zflbA3n6Kk29kgVXP6cptPTl2gyDsns1Pq/2rJp/6EI5q5fnmzK Aifqs6iEMolCJbLeIkm5DNy30jHUQQ2ppXVIiAErtwA6M+I/LPP81A1NQF9S7LAbpNKm 5qXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684953885; x=1687545885; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BhglgsPDhDMkvFy5HHyCJo+/I/zn6luCz3jk0H2OX74=; b=BDOPH8Yt5Lh811u6K3nZMxlfPjD6x2lWTR5EgObahlNaRuujSJfyKuDm/hDqOyNHSm cOjh/deB+Cjl9bBXsyZEKjRCM/vgW0fqOl7jQR+dXMJ30T2lq2qiLe2EcEfEs+wr3YW9 jPkQuPPJ115uO5hwJYOt3ftsDJ9720I+qtmk5AhWYe0zOKrUgL1tqXvOobFSlZFesL+6 r+nKHRFaiunsKPgyaNz/e7aeEaJMSUDl+kEBpwU89YX2xGeAbcuguBKockk1Py7COUM4 GvGmSYjyJgyoHalc2BXNl7rZp+7De4zJUCsSs4qvd6scVFwHBranocGfI0JvxQgnOWYr rC7w== X-Gm-Message-State: AC+VfDxjDFzywO9RZ+2YvNODksBSyqTC7GHLIic2YgkUiHzHk6nIoD2i G+wXtILYTokQElFWA0mKLgg= X-Google-Smtp-Source: ACHHUZ6oEQDl8OReKOmfoTcDPTo25UwtFVLPGozXlG+B8hlA/DtCx1NZNGBZAwMdtcJP7TmjyRvJQA== X-Received: by 2002:ac2:5217:0:b0:4f1:2f5d:8679 with SMTP id a23-20020ac25217000000b004f12f5d8679mr5691286lfl.22.1684953884961; Wed, 24 May 2023 11:44:44 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id v3-20020ac25603000000b004f387d97dafsm1814640lfd.147.2023.05.24.11.44.44 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 May 2023 11:44:44 -0700 (PDT) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_9FA25D1E-D3F5-4B0F-A1FC-15622586B073" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE SNAPSHOT image format proposal Date: Wed, 24 May 2023 21:44:34 +0300 In-Reply-To: Cc: Tomek CEDRO , virtualization@freebsd.org, freebsd-hackers To: Mario Marietto References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <3054B6FB-8408-492C-9F18-91A61516D0EF@gmail.com> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4QRKqL6FvXz4Sgr X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_9FA25D1E-D3F5-4B0F-A1FC-15622586B073 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 It is impossible. However, as first attempt you can verify that multidev = branch works fine for you. https://github.com/gusev-vitaliy/freebsd/tree/dev/D35590 =E2=80=94=E2=80=94 Vitaliy Gusev > On 24 May 2023, at 20:47, Mario Marietto = wrote: >=20 > Give me the internal git. I want to test it asap :) >=20 > Il mer 24 mag 2023, 19:39 Vitaliy Gusev > ha scritto: >> It is ready but in internal git repo, and it needs time to port it to = CURRENT version. Probably week or so. >>=20 >> When I create review, I will notify you. >>=20 >> Thanks, >> Vitaliy Gusev >>=20 >>=20 >>> On 24 May 2023, at 20:33, Mario Marietto > wrote: >>>=20 >>> @gusev.vitaliy@gmail.com : Do you = want to explain to me how to test the new "snapshot" feature ? I'm = interested to test and stress it on my system. Is it ready to be used ? >>>=20 >>>>=20 >>>> Thank you for your questions. If you would like, you can try to = test the new implementation and give feedback. >>>>=20 >>>> =E2=80=94=E2=80=94=E2=80=94 >>>> Vitaliy Gusev >>>>=20 >>>=20 >>>=20 >>> --=20 >>> Mario. >>=20 --Apple-Mail=_9FA25D1E-D3F5-4B0F-A1FC-15622586B073 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 It is = impossible. However, as first attempt you can verify that multidev = branch works fine for you.


=E2=80=94=E2=80=94
Vitali= y Gusev

On 24 May 2023, at = 20:47, Mario Marietto <marietto2008@gmail.com> wrote:

Give me the = internal git. I want to test it asap :)

Il mer 24 = mag 2023, 19:39 Vitaliy Gusev <gusev.vitaliy@gmail.com> = ha scritto:
It is ready but in internal git = repo, and it needs time to port it to CURRENT version. Probably week or = so.

When I create review, I will notify = you.

Thanks,
Vitaliy = Gusev


On 24 May = 2023, at 20:33, Mario Marietto <marietto2008@gmail.com> = wrote:

@gusev.vitaliy@gmail.com : Do you want to explain = to me how to test the new "snapshot" feature ? I'm interested to test = and stress it on my system. Is it ready to be used ?


Thank you for your = questions. If you would like, you can try to test the new implementation = and give feedback.

=E2=80=94=E2=80=94=E2=80=94
Vitaliy Gusev



--
Mario.


= --Apple-Mail=_9FA25D1E-D3F5-4B0F-A1FC-15622586B073-- From nobody Wed May 24 23:12:18 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRRmF44Lrz4V7Th for ; Wed, 24 May 2023 23:12:29 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from mail.nomadlogic.org (mail.nomadlogic.org [66.165.241.226]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.nomadlogic.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRRmF17k0z3PnN for ; Wed, 24 May 2023 23:12:29 +0000 (UTC) (envelope-from pete@nomadlogic.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nomadlogic.org; s=04242021; t=1684969939; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=YyXVx8K2w3+whqxT0S3E0TU9NyGsMLXiULu0liHU83c=; b=CPYv85XN0Ly+AUAWOt0AtXMVryUYPQbEPNIeZx7ztCZUSIWPYp2fxOhiW4q4TX94WTs6MW C9wec3+BOV6k90J3HzYn1rJxgxlZJ4zirhrTY6AbcCrmcyadlpAbYjQ6Y5UWiXCSxcCSp7 tLr2nAwdrGttmNdgc++JsVqhb12jJ40= Received: from topanga.nomadlogic.org (cpe-24-24-168-214.socal.res.rr.com [24.24.168.214]) by mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id aa15c4d6 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Wed, 24 May 2023 23:12:19 +0000 (UTC) Date: Wed, 24 May 2023 16:12:18 -0700 From: Pete Wright To: "elk-wetznet.de" Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve HEVC/H.265 encoding support in ubuntu vm Message-ID: References: <3c1de1f44be2bb9bc1fab961e8a34722@wetznet.de> List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3c1de1f44be2bb9bc1fab961e8a34722@wetznet.de> X-Rspamd-Queue-Id: 4QRRmF17k0z3PnN X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29802, ipnet:66.165.240.0/22, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Wed, May 24, 2023 at 09:12:15AM +0200, elk-wetznet.de wrote: > > HI, > > I run bhyve at Freebsd on a AMD Pro Ryzen 5 and still dont know a way to get > the encoding support of the GPU inside the VM. > > Any hints ? i suspect you'd need to passthrough a GPU PCI device to the gues VM. this presentation may, or may-not, be helpful: https://papers.freebsd.org/2019/eurobsdcon/chiu-bhyve_guests_with_hardware_accelerated_graphics/ -p -- Pete Wright pete@nomadlogic.org From nobody Thu May 25 00:28:11 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRTSR0QfRz4VBvl for ; Thu, 25 May 2023 00:28:55 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-yw1-x1133.google.com (mail-yw1-x1133.google.com [IPv6:2607:f8b0:4864:20::1133]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRTSK2drKz3mFc for ; Thu, 25 May 2023 00:28:49 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=EdH6FEjQ; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::1133 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-yw1-x1133.google.com with SMTP id 00721157ae682-561a7d96f67so22488037b3.3 for ; Wed, 24 May 2023 17:28:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684974528; x=1687566528; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=ev29CKhCgxC805A8S26dqt30++XHusALPNoyWalEqmA=; b=EdH6FEjQcSOc2wbCAOfSJBncPNuOY/HuqBTyLRGyee+O5OTv67a2xNMgNxrSGB2k+x RvSEp7ZMn0aKj2ppAN2Ce7SztAjbzQRx0/v41TyWtopvvC2P0omPuX6lo24RgOZsMEg5 bNiPoUzWF3ERos5r1swpv2n10ECcwd5AnrSw8WuEVKefC7G9E2mNsKep/GfgCu0sLSe3 PKOh4plag28OFyIZzMMww2EKiNuQ540rQEP/gIRsdTuPmMjof2BkkySx/tDzjULea1JO w/oHSu1I5IfDSVNMU7lwzXeBZageNb97M6pfZjEWVH7v8kb5YAfwX7tcUoJO68Acy7c1 NuWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684974528; x=1687566528; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ev29CKhCgxC805A8S26dqt30++XHusALPNoyWalEqmA=; b=YEfE267fTZ34Q4yWdI/zIGL0hMR9RzBu0bk0uWIRqxFm4mz0hf2MxxVxmUvyKlPgqz ztptsDNJXA2oBaEtYHS9eoF9cQoXhPJoKBqwZxQEYRb1KbXhH4grHtCQYqpUKCXJdFxp q9Teig+0Y4iXnUwpayGqdVMGzzvNzWUGYrohL2E4pU8D7UZOmt3ud4Rpd0G1s6UwRxJL UnE8IIWefkxU5czdJdx5UsMIW8sxKFP31MnapQtWD4Ab+3Vc64sYqSJcLgRn7VKIj5lU 6cuXVgF6ZwtfSm7u1dtgq6F9YZV4pv5qqrZ1f5eKoOEIrplz1HzuuLpDjXJpmBsdJjnf ZmSA== X-Gm-Message-State: AC+VfDxJNzjK/fyvjDaBGstzaWe0f1fTpNl0uPOaDAGzQRU/QwqssLdf 96FEjgcRApyJcA+v8JXkBvnIeJmn5/s561SB49zFotYq0go= X-Google-Smtp-Source: ACHHUZ4ySy18hny89pcjLUkYGYQ8fUsHnf4Nute4eTu1DoEsEikGy2LLAwBEUPPguNmFMiXjdcl56BkORLcu5DHaho8= X-Received: by 2002:a0d:ca58:0:b0:55d:c2c3:fbb8 with SMTP id m85-20020a0dca58000000b0055dc2c3fbb8mr20318780ywd.40.1684974528040; Wed, 24 May 2023 17:28:48 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <20230221065505.nvs22svqr4oecwyx@tilda.center> In-Reply-To: <20230221065505.nvs22svqr4oecwyx@tilda.center> From: Mario Marietto Date: Thu, 25 May 2023 02:28:11 +0200 Message-ID: Subject: Fwd: Why Blender Cycles is not able to detect my GPU(s) and CUDA within the Ubuntu / Linuxulator To: FreeBSD virtualization , meka@tilda.center Content-Type: multipart/mixed; boundary="00000000000035105c05fc79b405" X-Spamd-Result: default: False [-1.97 / 15.00]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.59)[-0.589]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.48)[-0.484]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[multipart/mixed,multipart/alternative,text/plain]; MIME_BASE64_TEXT(0.10)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1133:from]; ARC_NA(0.00)[]; BLOCKLISTDE_FAIL(0.00)[2607:f8b0:4864:20::1133:query timed out]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; HAS_ATTACHMENT(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4QRTSK2drKz3mFc X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N --00000000000035105c05fc79b405 Content-Type: multipart/alternative; boundary="00000000000035105a05fc79b403" --00000000000035105a05fc79b403 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Which user should I add ? the user that I have created within the Linuxulator or the user that I have created on the host (FreeBSD) ? thanks. ---------- Forwarded message --------- From: Goran Meki=C4=87 Date: Tue, Feb 21, 2023 at 7:55=E2=80=AFAM Subject: Re: Why Blender Cycles is not able to detect my GPU(s) and CUDA within the Ubuntu / Linuxulator To: Mario Marietto On Sat, Feb 18, 2023 at 03:59:49PM +0100, Mario Marietto wrote: > Hello to everyone. > > I've just installed Ubuntu 22.10 with the Linuxulator on FreeBSD > 13.1-RELEASE p6 as well as these components inside it : > > > 1. nvidia driver Version: 525.78.01 + CUDA 12 > 2. Blender 3.2.2 > > > The nvidia driver 525.78.1 + CUDA 12 work correctly within the linuxulator : > > https://ibb.co/8Ps8J81 > > and Cycles is already able to detect the nvidia driver + CUDA,but only if > blender runs on FreeBSD. Give a look at this picture : > > https://ibb.co/rwZ7q8Q > > What I want to do is to run Blender and I want to render my projects with > cycles using the CUDA libraries and my GPU(s) within the linux emulation > layer. Is this supposed to work ? The error that Blender gives when I try > to do that are the the following ones : > > root@marietto:/# blender > > Read prefs: /root/.config/blender/3.2/config/userpref.blend > libGL error: glx: failed to create dri2 screen > libGL error: failed to load driver: nouveau > could not get a list of mounted file-systems > /var/run/user/1001/gvfs/ non-existent directory > Saved session recovery to '/tmp/quit.blend' > Blender quit > > > why do I use root ? because as a normal user Blender does not start at all. > > marietto@marietto:~$ blender > Unable to open a display > Aborted > > > I'm very curious to understand the reason(s) of the errors I see below : > > libGL error: glx: failed to create dri2 screen > libGL error: failed to load driver: nouveau > > > My sensation is that they can be fixed. If I do : > > cp -r ./blender-3.2.2-linux-x64/3.2/scripts/addons/cycles/lib > /compat/ubuntu2210/usr/share/blender/scripts/addons/cycles/ > > > I see this additional error : > > CUDA cuInit: Unknown error > > > but if I remove the lib directory : > > rm -r /compat/ubuntu2210/usr/share/blender/scripts/addons/cycles/lib > > > the error "CUDA cuInit: Unknown error" disappears,but the other errors ar= e > still there. > > It seems to me that Blender looks for the nouveau driver and it can't fin= d > it. But it should look like the nVidia driver. Since the nouveau driver > does not support CUDA,maybe it should be "unlinked" from Blender and > Blender should be "linked" to the nvidia driver,in some way. What do you > think ? > > -- > Mario. Are you in "video" group? Regards, meka --=20 Mario. --00000000000035105a05fc79b403 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Which user should I add ? the user that I have created wit= hin the Linuxulator or the user that I have created on the host (FreeBSD) ?= thanks.

---------- Forwarded message ---------
From: Goran Meki=C4=87 <meka= @tilda.center>
Date: Tue, Feb 21, 2023 at 7:55=E2=80=AFAM
S= ubject: Re: Why Blender Cycles is not able to detect my GPU(s) and CUDA wit= hin the Ubuntu / Linuxulator
To: Mario Marietto <marietto2008@gmail.com>


On Sa= t, Feb 18, 2023 at 03:59:49PM +0100, Mario Marietto wrote:
> Hello to everyone.
>
> I've just installed Ubuntu 22.10 with the Linuxulator on FreeBSD > 13.1-RELEASE p6 as well as these components inside it :
>
>
>=C2=A0 =C2=A0 1. nvidia driver Version: 525.78.01 + CUDA 12
>=C2=A0 =C2=A0 2. Blender 3.2.2
>
>
> The nvidia driver 525.78.1 + CUDA 12 work correctly within the linuxul= ator :
>
> https://ibb.co/8Ps8J81
>
> and Cycles is already able to detect the nvidia driver + CUDA,but only= if
> blender runs on FreeBSD. Give a look at this picture :
>
> https://ibb.co/rwZ7q8Q
>
> What I want to do is to run Blender and I want to render my projects w= ith
> cycles using the CUDA libraries and my GPU(s) within the linux emulati= on
> layer. Is this supposed to work ? The error that Blender gives when I = try
> to do that are the the following ones :
>
> root@marietto:/# blender
>
> Read prefs: /root/.config/blender/3.2/config/userpref.blend
> libGL error: glx: failed to create dri2 screen
> libGL error: failed to load driver: nouveau
> could not get a list of mounted file-systems
> /var/run/user/1001/gvfs/ non-existent directory
> Saved session recovery to '/tmp/quit.blend'
> Blender quit
>
>
> why do I use root ? because as a normal user Blender does not start at= all.
>
> marietto@marietto:~$ blender
> Unable to open a display
> Aborted
>
>
> I'm very curious to understand the reason(s) of the errors I see b= elow :
>
> libGL error: glx: failed to create dri2 screen
> libGL error: failed to load driver: nouveau
>
>
> My sensation is that they can be fixed. If I do :
>
> cp -r=C2=A0 ./blender-3.2.2-linux-x64/3.2/scripts/addons/cycles/lib > /compat/ubuntu2210/usr/share/blender/scripts/addons/cycles/
>
>
> I see this additional error :
>
> CUDA cuInit: Unknown error
>
>
> but if I remove the lib directory :
>
> rm -r /compat/ubuntu2210/usr/share/blender/scripts/addons/cycles/lib >
>
> the error "CUDA cuInit: Unknown error" disappears,but the ot= her errors are
> still there.
>
> It seems to me that Blender looks for the nouveau driver and it can= 9;t find
> it. But it should look like the nVidia driver. Since the nouveau drive= r
> does not support CUDA,maybe it should be "unlinked" from Ble= nder and
> Blender should be "linked" to the nvidia driver,in some way.= What do you
> think ?
>
> --
> Mario.


Are you in "video" group?

Regards,
meka


--
Mario.
--00000000000035105a05fc79b403-- --00000000000035105c05fc79b405 Content-Type: text/plain; charset="US-ASCII"; name="signature.asc" Content-Disposition: attachment; filename="signature.asc" Content-Transfer-Encoding: base64 Content-ID: <188504cc0c661a8e03b1> X-Attachment-Id: 188504cc0c661a8e03b1 LS0tLS1CRUdJTiBQR1AgU0lHTkFUVVJFLS0tLS0NCg0KaVFJekJBQUJDQUFkRmlFRTFXSUZrWHky WmVNS2pqS0VXajFUa25vdnJMWUZBbVAwYXNZQUNna1FXajFUa25vdg0KckxaNFBBLytQRmJXTjlX UUx5MjB1NlpLQUk0SVRIQmcvUFk3M2VDYzBSZWFCbURuY2dWaFNVVndSaGVGaDBnNQ0KVVRHUVZx OUtJaGFVUlQrRHE5QmJMeE9nTlh5bjdmSnhxOW5hN1dDR0tLYkVnYW5KWkt2ckErUUNEWlFwN2h4 Vw0KOXMwaE5McklBQnU3ZWFRRzAwdmlvdFl4UnNOc0kyUXROVE9sNFN5a1FxSjFGSjhzcDJuNks4 clFnTUg4dEhuQg0KRm8yVjZFSFN0WWJwdUhIbllqTGlRMXg2WlB6SjNPQyt1aXhWUFl4SlVoMzQ2 UGRJNk9MNXdKNjZkMlpJb3VyUA0KeFg1K2E2YlFWaTJFaHRkWkdmbDZJWlZReUNVL01aMlJrYjNn SXZra3FBdmxKMzRtWFl6UEhWNUNOam1MVWZhaw0Ka0VsQ29tSWN1UnZrSU5BZUlaUVZBUXprQ2Iv ZDRBelFTV3NGZEI4eEZOUmlQblB5S0pNdXpYcHZ1b3pqeFUzVA0Kb3l4dlZFU1daUWhCcU9naW40 SjJ0VFpWdnB3TXFZYUtJZGVMVWg3RWdhZmlncy9UTVZmcGY4a1dPUXhjUkhFaQ0KeXd4MnFkT2JQ dFArTzI3b0pGZ2xmWGpaOFBOdURNVi9TQVUxSUozS0FOMEpXUzJVMFpISk9URFhmSVFZWFg2TA0K VFdoTU1PSXZjaFVYdkp2RDhlTmJlVjRxVU1nTVRyYnhDeHNFOE5MYkMxbHhuYVY4UWozcllNUEVu TWlhS2t1bA0KN3dYMUFJSTJLOVlUSmNJZG5uaGxFQ2tKOWVwRUJKcjlEUVFzT3dxQW5MblEwNU5l TUZ4NWMwSXBKKzhUa3A3Qw0KaGhhR3ZiQlM3RWREZHMvcE8zOTBwaTJIN1ZaajFLV0o5U0U4WTU1 Q2J0NnE5a2EyT1BBPQ0KPTZFTHcNCi0tLS0tRU5EIFBHUCBTSUdOQVRVUkUtLS0tLQ0K --00000000000035105c05fc79b405-- From nobody Thu May 25 01:30:18 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRVqY74yLz4CRVV for ; Thu, 25 May 2023 01:30:33 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yw1-x112d.google.com (mail-yw1-x112d.google.com [IPv6:2607:f8b0:4864:20::112d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRVqX4b5Tz3sm8 for ; Thu, 25 May 2023 01:30:32 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cedro.info header.s=google header.b=K34LfFlX; spf=none (mx1.freebsd.org: domain of tomek@cedro.info has no SPF policy when checking 2607:f8b0:4864:20::112d) smtp.mailfrom=tomek@cedro.info; dmarc=none Received: by mail-yw1-x112d.google.com with SMTP id 00721157ae682-5621a279cbbso7203787b3.1 for ; Wed, 24 May 2023 18:30:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1684978231; x=1687570231; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=RUGK9CXZZGThSQNZt+5FalMrWwsvK+w2j4g94Ic1xao=; b=K34LfFlX9OrVdNkeY7CTCkX3afIIRd2RXPmiSXCyQkkIkVOdQpJRDpcSq7xqfWBL49 i7d4Gox9k0qsvEEw2t4IomU6DpqDRcQjH9N+7TSYcYhNHJxWwyGKKPM2tALoXUch+9YV wICw5/3tbf7d7vCtEgR+dngtAZ8TIuV9CLuw4FlzTHrIFdHHUJJQKNGXQdBSmFhR/o8e PH7xV1BIuN009ChXym54U0j8CtusATbSml32ttEfPYRr9YAfAa0PyEs4/97KDalanlO6 lfsvoZgFgCqOdH0Y4V6lSJJiXWc2w+4qrtcrOoWptYy33nCi0uzMw+WkNZntSgD5+M5e 1kEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684978231; x=1687570231; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RUGK9CXZZGThSQNZt+5FalMrWwsvK+w2j4g94Ic1xao=; b=ZvQmW+WC9hOlH880HON0rK63Riyu4I1Lg4Nggt/lXfWfX4KC9mUJyBlWVLcN6LEr0k boVxQPwvs6q1OE/CfNsvOqXPY4rSv0xdxDhXkMpiz4QyL78u12pNGeq77bI26QnSV8VB RWcxAB1uNQpaNwEPk7M2zHhKU/lez+yvup74X7eTZlggmEqfgLRlagkZBQYMpnkyKTT5 njHScryzUOWQgNtQ3jRIKbm85XIgKTjd3Vp2sLI8CvqWBe5YAY0ubwz7mcTmEzywAEoo JTjWwZpoKvMBqAcUw68292RK5ajJAbOflA1Ny/jfoz6j1Pt0LfSncZv7ZGy2dIYNjVT8 YtFA== X-Gm-Message-State: AC+VfDz/70o6XLviEwqdsaKTExRiC0no1mNB3Bd9c10TNKcMg7HP1Ocm GiH3BFWl9A/wf1pP1jaUJy5+ctH3oZ0ehSZUFyg= X-Google-Smtp-Source: ACHHUZ7mT4Xm+4nD8ZWFyjfYPxJGyB7k9r4WaqOh1QJbQ88NJLa4oQGqSMyeZdYktXeJeJxzYvm2QA== X-Received: by 2002:a81:9b56:0:b0:55a:776e:95f3 with SMTP id s83-20020a819b56000000b0055a776e95f3mr997101ywg.25.1684978231218; Wed, 24 May 2023 18:30:31 -0700 (PDT) Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com. [209.85.128.178]) by smtp.gmail.com with ESMTPSA id g4-20020a0ddd04000000b00545a08184b1sm4124776ywe.65.2023.05.24.18.30.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 24 May 2023 18:30:30 -0700 (PDT) Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-55db055b412so7220137b3.0; Wed, 24 May 2023 18:30:30 -0700 (PDT) X-Received: by 2002:a0d:ea05:0:b0:561:9bcc:6c81 with SMTP id t5-20020a0dea05000000b005619bcc6c81mr1402634ywe.24.1684978230412; Wed, 24 May 2023 18:30:30 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> In-Reply-To: From: Tomek CEDRO Date: Thu, 25 May 2023 03:30:18 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: BHYVE SNAPSHOT image format proposal To: Vitaliy Gusev Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; R_DKIM_ALLOW(-0.20)[cedro.info:s=google]; MIME_GOOD(-0.10)[text/plain]; R_SPF_NA(0.00)[no SPF record]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::112d:from,209.85.128.178:received]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; DKIM_TRACE(0.00)[cedro.info:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TAGGED_RCPT(0.00)[]; DMARC_NA(0.00)[cedro.info]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4QRVqX4b5Tz3sm8 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Wed, May 24, 2023 at 5:11=E2=80=AFPM Vitaliy Gusev wrote: > Protecting requires more efforts and it should be clearly defined: what i= s purpose. If > purpose is having checksum with 99.9% reliability, NVLIST HEADER can be w= iden > to have =E2=80=9Cchecksum=E2=80=9D key/value for a Section. Well, this could be optional but useful to make sure snapshot did not break somehow for instance backup medium error or something like that.. even more maybe a way to fix it.. just a design stage idea :-) > If purpose is having crypto verification - I believe sha256 program shoul= d be your choice. My question was more specific to availability of that feature (integrity + repair) rather than specific format :-) The use case here is having a virtual machine (it was VirtualBox) with a bare os installed, plus some common applications, that is snapshoted at some point in time, then experimented a lot, restored from snapshot, etc. I had a backup of such vm + snapshot backed up that got broken somehow. It would be nice to know that something is broken, what is broken, maybe a way to fix :-) > Small is not so perfect. As the first attempt snapshot code is good. But = if you want to get > values related to some specific device, for example, for NIC or HPET, you= cannot get it easily. Please > try :) > > Stream doesn=E2=80=99t have flexibility. It is good for well specified a= nd long long time discussed protocols > like XDR (NFS), when it has RFC and each position in the stream is descri= bed. Example: RFC1813. > > New format with NVLIST has flexibility and is fast enough. Note, ZFS uses= nvlist for keeping attributes > and more another things. Sorry, I was not really aware of that format!! This looks really solid :-) https://github.com/fudosecurity/nvlist https://man.freebsd.org/cgi/man.cgi?query=3Dnvlist > Why do you need modify snapshot image ? Could you describe more? Do you > modify current 3 snapshot files? Analysis that require ram / nvram modification? Not sure if this is already possible, but may come handy for experimenting with uefi and maybe some OS (features) that will not run with unmodified nvram :-P > If you are talking about compatibility of a Image format - it should be c= ompatible in > both directions, at least for not so big format changes. > > If consider overall snapshot/resume compatibility - I believe forward co= mpatibility > is not case and target. Indeed, why do you need to resume an image creat= ed by > a higher version of a program? This happens quite often. For instance there is a bug in application and I need to revert to (at least) one step older version. Then I am unable to work on a file that I just saved (or was autosaved for me). Firefox profile settings let be the first example. KiCAD file format is another example (sometimes I need to switch to a devel build to evade a nasty blocker bug then anyone else that uses a release is blocked for some months including me myself). > The most important thing - backward compatibility, i.e. when an image is = created > by an older version of a program, but should be resumed on a new one. > > This is target and and intention of this improvement. Thank you, this looks promising :-) Just another design stage idea to keep the formats forward and backward compatible at least for some time and/or have an option to skip unknown feature :-) > Yes, I know about another formats, like JSON or others. NVLIST is the mos= t > effective and suitable for the current purposes. Thank you I know that now :-) I have switched to bhyve recently and for my use I prefer it now over virtualbox :-) Thanks again Vitaliy and good luck with the updates :-) --=20 CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From nobody Thu May 25 10:06:06 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRkGQ6jbPz4TJHj for ; Thu, 25 May 2023 10:06:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRkGQ5hMZz3pkK for ; Thu, 25 May 2023 10:06:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1685009166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=avWprIcM2Rk9fnKXt6b7Pracdqsob138r3s6HMNavyU=; b=H+3tfo79SC3N1UalXt5EO81heZqvE9WyyHq3FxpsZJS+oDwSDHfPHGhrH6n4tpTnRjXTwA gDTgtHMLXJcp8wervNIr8ei3pP9jxIumlqVilkaHh9RmOtEPB6isT7kO30hStUoZsuc6eF YPW/N1XCIopU0MzwMtZDgKBMdPtJ+hQxX59cyHEpEvy+zET5+1uYFe16IpkiIA+T+IZmz6 hMULR7ZMf6LXs+7RAzeavVmiVK906pDnTaA2GJFysOzcIR8ueFFkGZzzmv0031f9PncMHT EDFujtsIadCxXderrDvIv2I4GdEkcMiHCAPZ+9w7LovE2QZOiBXpWXnFud6ksA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1685009166; a=rsa-sha256; cv=none; b=dPDZBz/6AofymKwWUKhzz9vtEFDoIX0+FzzGZblDI1siL25xR5UJ1tY7ZgP6RBYQI0LQYk +83g2v9JCGuZiKodv+HHhwgKbfAE86GPRE4fCythugNnVZpoU9uAyEwIaM2jQSIgKrdIuR k7YkXZ2Tl2hgbLoO2zWSp3SDcVgJBWRJwjU60t3Za9+I3DArMiZdTpOg2TPjU3pf6Mv29w m3Hx3d2sNlok09Ye5iAkhmx3jJsQKFmLYwhkmgo9oX9qI1p5fROirSW6rXF0YADRaJRvbV iiC+Yh5ZwQbc+tguJDDyAZjVAfaQDsJK+uDeN2qp/agadHRxQW+VVURi+2YhTg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QRkGQ4pbzzL9B for ; Thu, 25 May 2023 10:06:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 34PA66XW025842 for ; Thu, 25 May 2023 10:06:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 34PA66hA025840 for virtualization@FreeBSD.org; Thu, 25 May 2023 10:06:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 271578] FreeBSD fails to init SMP and hits panic post install in Azure ARM64 Date: Thu, 25 May 2023 10:06:06 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: Andrew@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: Andrew@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status assigned_to bug_file_loc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271578 Andrew Turner changed: What |Removed |Added ---------------------------------------------------------------------------- Status|New |Open Assignee|virtualization@FreeBSD.org |Andrew@FreeBSD.org URL| |https://reviews.freebsd.org | |/D40263 --- Comment #1 from Andrew Turner --- https://reviews.freebsd.org/D40263 should fix the issue --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Thu May 25 13:40:06 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRq1d73RVz4V1KC; Thu, 25 May 2023 13:40:21 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRq1d6XChz49Wg; Thu, 25 May 2023 13:40:21 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x134.google.com with SMTP id 2adb3069b0e04-4f3baf04f0cso2393521e87.1; Thu, 25 May 2023 06:40:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685022018; x=1687614018; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=SgjAC+kTpUWPKowBxrOZR11GblLebEpUF0Yh32dQoN0=; b=OOrtFCtBF3CUSJz0Oav2DCrXZDn0dVZ3B7FXn4I6tj0HVpfgfe9lGu3yehtTYktO1C rfwVnUHaNxEZihY7Lx7k37Qs730NyTwq4Y8+MZHzsuRW1DC4ouryls0uA6pWACfto+TT apZvRvS/yh/7VxJmq7+bbJ71tgVqNfHk/jZ9pGIZ1k1H7MHbtNzrXc+g5Z5BHARH1piS G8Qpo+qPOsdrjvxF9gd9kunj4cpZcL11cO2TO4t9mZmBPXC5ZMbnce9FEpAiHxvW9WUM r1vcJPagCbTvnkcT90OlsYcRzVxRteQD+nTadmnfXxL77V0yZOnskxZHfH3tMf7TphDF +Tbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685022018; x=1687614018; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SgjAC+kTpUWPKowBxrOZR11GblLebEpUF0Yh32dQoN0=; b=iBydoYMu9B69yETrS7gr0ARop2gBAzKTDT1vz94QoVsEbX38m+RnOgTmSZnzp89x3e oLl+EJLkYiSJdIKwkcnw4hawqzmPuvFGzIWWb/dWyHgoyNsGYJXgEh8+EOm+SkcvRaL4 vXhGGW6YMIWwwX/XsHs7IrE4iTWuyFBgLGAwScxRdpIQJStPetpr7Y9OndnM5Kk1nyCi M3bifPNLIp2Wj/0xQWEP1H2SLAiYAqwCKLkrpqXgmCrRXrVnY/dP5BCKgr0y8gDWuXIc nqRO/Yw18vr3Kvc/TX06a22cathx50mjbIbJogcDsq79Y9PBD8aLVkwc0EkoL5QjuKXa PfJQ== X-Gm-Message-State: AC+VfDxsLBp7WNLhH2RJ5lh5arAMy2o+U4+X+PHJRRmJInqlXs2Vrzjo 3QbPWZT6ERGTRhjdCqM1sQpkrKl+1FnI3A== X-Google-Smtp-Source: ACHHUZ4oqGuVc3USY0dp5vYvS11tt0n5YNEj5pBwJk+SUiLwxMKm5qdw/+Mn1TvBB01vT419klJovw== X-Received: by 2002:ac2:5991:0:b0:4f3:940d:abc with SMTP id w17-20020ac25991000000b004f3940d0abcmr5796153lfn.0.1685022017615; Thu, 25 May 2023 06:40:17 -0700 (PDT) Received: from smtpclient.apple ([78.140.234.98]) by smtp.gmail.com with ESMTPSA id p21-20020ac246d5000000b004f3ba3b948dsm210729lfo.284.2023.05.25.06.40.17 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 25 May 2023 06:40:17 -0700 (PDT) From: Vitaliy Gusev Message-Id: <8FE14143-1AA9-418E-A497-FEFB99BF6B9F@gmail.com> Content-Type: multipart/alternative; boundary="Apple-Mail=_CD459CBE-FE38-45F5-8B0C-D194440D4C9B" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE SNAPSHOT image format proposal Date: Thu, 25 May 2023 16:40:06 +0300 In-Reply-To: Cc: virtualization@freebsd.org, freebsd-hackers@freebsd.org To: Tomek CEDRO References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4QRq1d6XChz49Wg X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_CD459CBE-FE38-45F5-8B0C-D194440D4C9B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 25 May 2023, at 04:30, Tomek CEDRO wrote: >=20 > On Wed, May 24, 2023 at 5:11=E2=80=AFPM Vitaliy Gusev wrote: >> Protecting requires more efforts and it should be clearly defined: = what is purpose. If >> purpose is having checksum with 99.9% reliability, NVLIST HEADER can = be widen >> to have =E2=80=9Cchecksum=E2=80=9D key/value for a Section. >=20 > Well, this could be optional but useful to make sure snapshot did not > break somehow for instance backup medium error or something like > that.. even more maybe a way to fix it.. just a design stage idea :- Yes, new format can have checksum of a Section data if implemented. >=20 >=20 >> If purpose is having crypto verification - I believe sha256 program = should be your choice. >=20 > My question was more specific to availability of that feature > (integrity + repair) rather than specific format :-) >=20 > The use case here is having a virtual machine (it was VirtualBox) with > a bare os installed, plus some common applications, that is snapshoted > at some point in time, then experimented a lot, restored from > snapshot, etc. I had a backup of such vm + snapshot backed up that got > broken somehow. It would be nice to know that something is broken, > what is broken, maybe a way to fix :-) =E2=80=9CIntegrity" is a very broad term. What checksum algorithm is = fine enough? =20 For the instance, ZFS has several options for checksum: checksum=3Don|off|fletcher2|fletcher4|sha256|noparity|sha512|skein|edonr =20 Having checksum for a filesystem is strongly recommended. However, If = consider image format, it doesn=E2=80=99t need to care about consistency in a file itself. As = example (!) - binary files in a system. They don=E2=80=99t have checksum integrated, validation is done by = another program - pkg or another. >=20 >=20 >> Why do you need modify snapshot image ? Could you describe more? Do = you >> modify current 3 snapshot files? >=20 > Analysis that require ram / nvram modification? Not sure if this is > already possible, but may come handy for experimenting with uefi and > maybe some OS (features) that will not run with unmodified nvram :-P Sorry I don=E2=80=99t get, why do you need to modify snapshot image, but = not directly vmem on the running VM? Another question, checksum and modifying image - two mutual exclusive = things.=20 >=20 >=20 >> If you are talking about compatibility of a Image format - it should = be compatible in >> both directions, at least for not so big format changes. >>=20 >> If consider overall snapshot/resume compatibility - I believe = forward compatibility >> is not case and target. Indeed, why do you need to resume an image = created by >> a higher version of a program? >=20 > This happens quite often. For instance there is a bug in application > and I need to revert to (at least) one step older version. Then I am > unable to work on a file that I just saved (or was autosaved for me). > Firefox profile settings let be the first example. KiCAD file format > is another example (sometimes I need to switch to a devel build to > evade a nasty blocker bug then anyone else that uses a release is > blocked for some months including me myself). Any additional thing has a cost of development, testing and support. = Current Implementation doesn=E2=80=99t support compatibility at all. Having = compatibility in both directions can be hard. For example, if some variable is removed in bhyve, backward = compatibility is fine, but forward compatibly is not possible unless that removed variable is = being saved into a snapshot image just for forward compatibility. And of course, it = should be tested and verified as worked. Do you like that approach? I don=E2=80=99t think so. So I guess only = backward compatibility should be supported to make the snapshot code simple and robust. Thanks, Vitaliy Gusev --Apple-Mail=_CD459CBE-FE38-45F5-8B0C-D194440D4C9B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 25 = May 2023, at 04:30, Tomek CEDRO <tomek@cedro.info> wrote:

On Wed, May 24, 2023 at = 5:11=E2=80=AFPM Vitaliy Gusev wrote:
Protecting requires more efforts and it should be clearly = defined: what is purpose. If
purpose is having checksum with 99.9% = reliability, NVLIST HEADER can be widen
to have =E2=80=9Cchecksum=E2=80= =9D key/value for a Section.

Well, this could be = optional but useful to make sure snapshot did not
break somehow for = instance backup medium error or something like
that.. even more maybe = a way to fix it.. just a design stage idea = :-

Yes, new format can have checksum of a = Section data if implemented.



If purpose is = having crypto verification - I believe sha256 program should be your = choice.

My question was more specific to = availability of that feature
(integrity + repair) rather than = specific format :-)

The use case here is having a virtual machine = (it was VirtualBox) with
a bare os installed, plus some common = applications, that is snapshoted
at some point in time, then = experimented a lot, restored from
snapshot, etc. I had a backup of = such vm + snapshot backed up that got
broken somehow. It would be = nice to know that something is broken,
what is broken, maybe a way to = fix = :-)


 =E2= =80=9CIntegrity" is a very broad term. What checksum algorithm is fine = enough?
 
For the instance,  ZFS has = several options for checksum:

checksum=3Don|off|fletcher2|fletcher4= |sha256|noparity|sha512|skein|edonr=

      =  


Having = checksum for a filesystem is strongly recommended. However, If consider = image format,
it  doesn=E2=80=99t need to care about = consistency in a file itself. As example (!)  - binary files in a = system.
They don=E2=80=99t have checksum integrated, = validation is done by another program  - pkg or = another.




Why do you = need modify snapshot image ? Could you describe more? Do you
modify = current 3 snapshot files?

Analysis that require ram = / nvram modification? Not sure if this is
already possible, but may = come handy for experimenting with uefi and
maybe some OS (features) = that will not run with unmodified nvram = :-P


Sorry I = don=E2=80=99t get, why do you need to modify snapshot image, but not = directly vmem on the = running
VM?

Another question, = checksum and modifying image - two mutual exclusive = things. 



If you are = talking about compatibility of a Image format - it should be compatible = in
both directions, at least for not so big format changes.

If = consider overall snapshot/resume compatibility - I believe  forward = compatibility
is not case and target. Indeed, why do you need =  to resume an image created by
a higher version of a = program?

This happens quite often. For instance = there is a bug in application
and I need to revert to (at least) one = step older version. Then I am
unable to work on a file that I just = saved (or was autosaved for me).
Firefox profile settings let be the = first example. KiCAD file format
is another example (sometimes I need = to switch to a devel build to
evade a nasty blocker bug then anyone = else that uses a release is
blocked for some months including me = myself).

Any additional = thing has a cost of development, testing and support. = Current
Implementation doesn=E2=80=99t support compatibility = at all. Having compatibility in both
directions can be = hard.

For example, if some variable is removed = in bhyve, backward compatibility is fine,
but forward = compatibly is not possible unless that removed variable is being = saved
into a snapshot image just for forward compatibility. = And of course, it should be tested
and verified as = worked.

Do you like that approach? I don=E2=80=99= t think so. So I guess only backward compatibility
should be = supported to make the snapshot code simple and = robust.

Thanks,
Vitaliy = Gusev


= --Apple-Mail=_CD459CBE-FE38-45F5-8B0C-D194440D4C9B-- From nobody Thu May 25 16:22:10 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRtd85PzBz4V9qf; Thu, 25 May 2023 16:22:52 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-yw1-x112d.google.com (mail-yw1-x112d.google.com [IPv6:2607:f8b0:4864:20::112d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRtd35nxjz3MQ8; Thu, 25 May 2023 16:22:47 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=JrGIqHKf; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::112d as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-yw1-x112d.google.com with SMTP id 00721157ae682-561e5014336so6763057b3.1; Thu, 25 May 2023 09:22:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685031767; x=1687623767; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=r9YI6+hMcd7OV+oac4bpou6A/Jq22N9AbSDYtZeLido=; b=JrGIqHKffw9TPA6Sj+XsLlfkTTM7xrEKCUW/IzFwghJYqqyMS+0K7QyRMY6nH4diW/ YKKrkFKgoVkUGe7Fu4UFdh6AmrS5p1UZ37f4W2tGR4rOaxgozmdOElQ5OpMupfc46zjv OzNVIhJfZgwpsiZ1kkpwBoFRrJ2K9EYwYbxKzC8KtGIYLC3a+HP6YM3Ar6zFVHxYCX77 S3SK4vwfyZ2LLLAPtHQJSGA1pu4hMu5nH5YDYw0gu+NuwbFsv3/rS+AQL/Y+cLwyyZOJ f234oUdyxfMI33as39TQuJl15P8PsiatZt3OphGbME8bXf3a0KsfVE7tS8rIilukmeKh uKDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685031767; x=1687623767; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=r9YI6+hMcd7OV+oac4bpou6A/Jq22N9AbSDYtZeLido=; b=QGWvkSOpbYT3uYEJxUroy0OW97Ylc84IXmtVJqbcA+VA4gchkh1/y35dbdQDa9B1el aHIpwQF8box+LYB2Z7fNp5oxAN3H2po/ENO5ZkYc0cwvi7LNuzHdwG3xmpbmGSQT+bSq riZKz0d+f8ba7WsWtLtSFVe3CG/XMipDAxbMuUwlOtR2LB8SXpPg5kCOTJiZuVuWqqIz orCIsKTUwEpUUXvE04MMftgmzOWHF4NdYpqf3nVFb7DgADT/A8n5crWhHwvD1sXK7CpY fJQ05+pYuJgdFw88G5yvUOPJdBtwTyHiGbg91chue4NUbAYQxgQd1NOjrOJTpLNmLwLW J3Aw== X-Gm-Message-State: AC+VfDz3GfN/HwfKegkhaJWu6PyDiBkrWHH2v1MxDjh6Qb/Rl5rDFMRL 0h+/Kzsuygj00rcGcBb5Sti/EeD+6Kv52nlB7GdOeCWSyxE= X-Google-Smtp-Source: ACHHUZ5C8uWknnhVdFFjNyRwVoLr5IYLumB1Luk0/Cw1GDXjZpjxG0bsGJ2ihhEHiSC5CfS2Yn+mgy0XQ/lBr4lwh5g= X-Received: by 2002:a25:dcd3:0:b0:ba8:364b:8f3c with SMTP id y202-20020a25dcd3000000b00ba8364b8f3cmr3962142ybe.53.1685031766639; Thu, 25 May 2023 09:22:46 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <8FE14143-1AA9-418E-A497-FEFB99BF6B9F@gmail.com> In-Reply-To: <8FE14143-1AA9-418E-A497-FEFB99BF6B9F@gmail.com> From: Mario Marietto Date: Thu, 25 May 2023 18:22:10 +0200 Message-ID: Subject: Re: BHYVE SNAPSHOT image format proposal To: Vitaliy Gusev Cc: Tomek CEDRO , virtualization@freebsd.org, freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="000000000000e2ca2a05fc8707fc" X-Spamd-Result: default: False [-2.50 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org,freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::112d:from]; RCPT_COUNT_THREE(0.00)[4]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4QRtd35nxjz3MQ8 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --000000000000e2ca2a05fc8707fc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Vitaliy, what happens if I clone your repo as source code on my FreeBSD system. Can I test your code directly or not ? Anyway,I think that,before doing this,I need to follow some kind of tutorial,to understand how the workflow is. Otherwise I will be not able to test and stress it. On Thu, May 25, 2023 at 3:40=E2=80=AFPM Vitaliy Gusev wrote: > > > On 25 May 2023, at 04:30, Tomek CEDRO wrote: > > On Wed, May 24, 2023 at 5:11=E2=80=AFPM Vitaliy Gusev wrote: > > Protecting requires more efforts and it should be clearly defined: what i= s > purpose. If > purpose is having checksum with 99.9% reliability, NVLIST HEADER can be > widen > to have =E2=80=9Cchecksum=E2=80=9D key/value for a Section. > > > Well, this could be optional but useful to make sure snapshot did not > break somehow for instance backup medium error or something like > that.. even more maybe a way to fix it.. just a design stage idea :- > > > Yes, new format can have checksum of a Section data if implemented. > > > > If purpose is having crypto verification - I believe sha256 program shoul= d > be your choice. > > > My question was more specific to availability of that feature > (integrity + repair) rather than specific format :-) > > The use case here is having a virtual machine (it was VirtualBox) with > a bare os installed, plus some common applications, that is snapshoted > at some point in time, then experimented a lot, restored from > snapshot, etc. I had a backup of such vm + snapshot backed up that got > broken somehow. It would be nice to know that something is broken, > what is broken, maybe a way to fix :-) > > > > =E2=80=9CIntegrity" is a very broad term. What checksum algorithm is fin= e enough? > > For the instance, ZFS has several options for checksum: > > *checksum*=3D*on*|*off*|*fletcher2*|*fletcher4*|*sha256*|*noparity*|*sha5= 12* > |*skein*|*edonr* > > > > > Having checksum for a filesystem is strongly recommended. However, If > consider image format, > it doesn=E2=80=99t need to care about consistency in a file itself. As e= xample > (!) - binary files in a system. > They don=E2=80=99t have checksum integrated, validation is done by anothe= r program > - pkg or another. > > > > > Why do you need modify snapshot image ? Could you describe more? Do you > modify current 3 snapshot files? > > > Analysis that require ram / nvram modification? Not sure if this is > already possible, but may come handy for experimenting with uefi and > maybe some OS (features) that will not run with unmodified nvram :-P > > > > Sorry I don=E2=80=99t get, why do you need to modify snapshot image, but = not > directly vmem on the running > VM? > > Another question, checksum and modifying image - two mutual exclusive > things. > > > > If you are talking about compatibility of a Image format - it should be > compatible in > both directions, at least for not so big format changes. > > If consider overall snapshot/resume compatibility - I believe forward > compatibility > is not case and target. Indeed, why do you need to resume an image > created by > a higher version of a program? > > > This happens quite often. For instance there is a bug in application > and I need to revert to (at least) one step older version. Then I am > unable to work on a file that I just saved (or was autosaved for me). > Firefox profile settings let be the first example. KiCAD file format > is another example (sometimes I need to switch to a devel build to > evade a nasty blocker bug then anyone else that uses a release is > blocked for some months including me myself). > > > Any additional thing has a cost of development, testing and support. > Current > Implementation doesn=E2=80=99t support compatibility at all. Having compa= tibility > in both > directions can be hard. > > For example, if some variable is removed in bhyve, backward compatibility > is fine, > but forward compatibly is not possible unless that removed variable is > being saved > into a snapshot image just for forward compatibility. And of course, it > should be tested > and verified as worked. > > Do you like that approach? I don=E2=80=99t think so. So I guess only back= ward > compatibility > should be supported to make the snapshot code simple and robust. > > Thanks, > Vitaliy Gusev > > > --=20 Mario. --000000000000e2ca2a05fc8707fc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Vitaliy,

what happens if I c= lone your repo as source code on my FreeBSD system. Can I test your code di= rectly or not ? Anyway,I think that,before doing this,I need to follow some= kind of tutorial,to understand how the workflow is. Otherwise I will be no= t able to test and stress it.

On Thu, May 25, 2023 at 3:40=E2=80= =AFPM Vitaliy Gusev <gusev.vi= taliy@gmail.com> wrote:


On 25 May 2= 023, at 04:30, Tomek CEDRO <tomek@cedro.info> wrote:

On Wed, May = 24, 2023 at 5:11=E2=80=AFPM Vitaliy Gusev wrote:
Protecting requires more efforts and it should be clearly defined: what = is purpose. If
purpose is having checksum with 99.9% reliability, NVLIST= HEADER can be widen
to have =E2=80=9Cchecksum=E2=80=9D key/value for a = Section.

Well, this could be optional but useful to mak= e sure snapshot did not
break somehow for instance backup medium error o= r something like
that.. even more maybe a way to fix it.. just a design = stage idea :-

Yes, new format can have checksum= of a Section data if implemented.

=


If purpose is having crypto ver= ification - I believe sha256 program should be your choice.

My question was more specific to availability of that feature
(inte= grity + repair) rather than specific format :-)

The use case here is= having a virtual machine (it was VirtualBox) with
a bare os installed, = plus some common applications, that is snapshoted
at some point in time,= then experimented a lot, restored from
snapshot, etc. I had a backup of= such vm + snapshot backed up that got
broken somehow. It would be nice = to know that something is broken,
what is broken, maybe a way to fix :-)=


=C2=A0=E2= =80=9CIntegrity" is a very broad term. What checksum algorithm is fine= enough?
=C2=A0
For the instance, =C2=A0ZFS has several= options for checksum:

checksum=3Don|off|fletcher2|fletcher4|= sha256|noparity|sha512|skein|edonr

=C2= =A0 =C2=A0 =C2=A0 =C2=A0


=
Having checksum for a filesystem is strongly recommended. However, If = consider image format,
it =C2=A0doesn=E2=80=99t need to care abou= t consistency in a file itself. As example (!) =C2=A0- binary files in a sy= stem.
They don=E2=80=99t have checksum integrated, validation is = done by another program =C2=A0- pkg or another.



Why do you need modify snapshot image ? Could you describe more? = Do you
modify current 3 snapshot files?

Analysis tha= t require ram / nvram modification? Not sure if this is
already possible= , but may come handy for experimenting with uefi and
maybe some OS (feat= ures) that will not run with unmodified nvram :-P


Sorry I don=E2=80=99t get, why do you need= to modify snapshot image, but not directly vmem on the running
V= M?

Another question, checksum and modifying image = - two mutual exclusive things.=C2=A0



If you are talking about comp= atibility of a Image format - it should be compatible in
both directions= , at least for not so big format changes.

If consider overall snapsh= ot/resume compatibility - I believe =C2=A0forward compatibility
is not c= ase and target. Indeed, why do you need =C2=A0to resume an image created by=
a higher version of a program?

This happens quite o= ften. For instance there is a bug in application
and I need to revert to= (at least) one step older version. Then I am
unable to work on a file t= hat I just saved (or was autosaved for me).
Firefox profile settings let= be the first example. KiCAD file format
is another example (sometimes I= need to switch to a devel build to
evade a nasty blocker bug then anyon= e else that uses a release is
blocked for some months including me mysel= f).

Any additional thing ha= s a cost of development, testing and support. Current
Implementat= ion doesn=E2=80=99t support compatibility at all. Having compatibility in b= oth
directions can be hard.

For example,= if some variable is removed in bhyve, backward compatibility is fine,
but forward compatibly is not possible unless that removed variable i= s being saved
into a snapshot image just for forward compatibilit= y. And of course, it should be tested
and verified as worked.

Do you like that approach? I don=E2=80=99t think so. = So I guess only backward compatibility
should be supported to mak= e the snapshot code simple and robust.

Thank= s,
Vitaliy Gusev




--=
Mario.
--000000000000e2ca2a05fc8707fc-- From nobody Thu May 25 18:40:04 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QRxgp2khrz4WXgg; Thu, 25 May 2023 18:40:22 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QRxgm5q9Tz414f; Thu, 25 May 2023 18:40:20 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x130.google.com with SMTP id 2adb3069b0e04-4f004cc54f4so2956291e87.3; Thu, 25 May 2023 11:40:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1685040017; x=1687632017; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=/ZvE/4ZJL6lRGsKtZoVnqNP57nQpKxBAIEy13pjO8pU=; b=W2arN12tamDFNBgMSDzm+Ttym9+cAt+M6XINwN1XHfxfV780k2icq1jJK+403PpQQZ YvHvGvWruLoFPKHOLELznL/v0t+tC0bKO/HFMy3mG5XZB+soZKJ6gUPFmoC9FuAT2DyJ tt/33P69vR9Qei9LKgAir+DZU/k9Ynh9iPyXRDkhUVEnVI61tYlNtKs3178JKUcAbbIg R2sB6hQFB4CwVX+0OGhtNabKIESDMM1BbWsT2sU+WuxQ50NnJXCccKn4fqyD83OWo2is eN94WYZjpd4vfK3uG1HbYjtbaVqSSC+gdGaktILbJwI8rE/21JphYu6XEK1DQxxTc6lr anHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685040017; x=1687632017; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/ZvE/4ZJL6lRGsKtZoVnqNP57nQpKxBAIEy13pjO8pU=; b=fD2kgkejtgjJRlMLOnehYuTVemzHPnQJmru8DU8RkfxV01Y+t3rY3LFRJsYkH/Uu6C Kxcn8Wrh2Pkh2MAJSIdav2Tps6npUp/ieo9qNA9EVo3Uyh/XbRLRwwE/KsXyZBe7mQkp CZKKc/x3Fkw4NOCSrBnsFyGcZKohV3E6wvmPWXV85ijaUd45Cl/EJzQc1qUyrflpFEFw u8R8+vtHO1pMY5hfzKEFnTpxRmPY7n1dkBcP7NPLsZG6A4pv9dZm/32rlNTp1xed7Hv4 W0cC+p4nsTuokSA20oEoh0mFbVNaqjF0NNV+jjUAMDlnBtkgi2R/8StGgb0C+pZtZra7 kiuQ== X-Gm-Message-State: AC+VfDxh+e8sdDTdy/N/hnf4vOKyiomeI/FcZZ/ZcczhvWBSiUlsjS0o N2sf8CKFpBjJvQQ2XAM5hDzdG4MDI7fWxw== X-Google-Smtp-Source: ACHHUZ6rCYLeKZS+fTF+f8d/Q7atXAwYPBKoF0pLOw/NgqwRmN8pzJB2BV7pEwprrL+x8IHR+BWUpw== X-Received: by 2002:ac2:5d24:0:b0:4eb:104b:bf61 with SMTP id i4-20020ac25d24000000b004eb104bbf61mr6497924lfb.58.1685040016541; Thu, 25 May 2023 11:40:16 -0700 (PDT) Received: from smtpclient.apple ([78.140.234.98]) by smtp.gmail.com with ESMTPSA id p21-20020ac246d5000000b004f3ba3b948dsm293112lfo.284.2023.05.25.11.40.15 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 25 May 2023 11:40:15 -0700 (PDT) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_9EC57A81-2637-440C-A2F4-5E1697C8811A" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE SNAPSHOT image format proposal Date: Thu, 25 May 2023 21:40:04 +0300 In-Reply-To: Cc: Tomek CEDRO , virtualization@freebsd.org, freebsd-hackers@freebsd.org To: Mario Marietto References: <67FDC8A8-86A6-4AE4-85F0-FF7BEF9F2F06@gmail.com> <8FE14143-1AA9-418E-A497-FEFB99BF6B9F@gmail.com> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4QRxgm5q9Tz414f X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_9EC57A81-2637-440C-A2F4-5E1697C8811A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 25 May 2023, at 19:22, Mario Marietto = wrote: >=20 > Vitaliy, >=20 > what happens if I clone your repo as source code on my FreeBSD system. = Can I test your code directly or not ? Anyway,I think that,before doing = this,I need to follow some kind of tutorial,to understand how the = workflow is. Otherwise I will be not able to test and stress it.=20 You should build kernel and tools, install it. Then you can run bhyve, = bhyvectl to deal with suspend/resume. Please follow=20 9.5. Building and Installing a Custom Kernel https://docs.freebsd.org/en/books/handbook/book/#kernelconfig-building Make sure that BHYVE_SNAPSHOT is enabled. Also look at build(7): https://man.freebsd.org/cgi/man.cgi?build(7) >=20 > On Thu, May 25, 2023 at 3:40=E2=80=AFPM Vitaliy Gusev = > wrote: >>=20 >>=20 >>> On 25 May 2023, at 04:30, Tomek CEDRO > wrote: >>>=20 >>> On Wed, May 24, 2023 at 5:11=E2=80=AFPM Vitaliy Gusev wrote: >>>> Protecting requires more efforts and it should be clearly defined: = what is purpose. If >>>> purpose is having checksum with 99.9% reliability, NVLIST HEADER = can be widen >>>> to have =E2=80=9Cchecksum=E2=80=9D key/value for a Section. >>>=20 >>> Well, this could be optional but useful to make sure snapshot did = not >>> break somehow for instance backup medium error or something like >>> that.. even more maybe a way to fix it.. just a design stage idea :- >>=20 >> Yes, new format can have checksum of a Section data if implemented. >>=20 >>>=20 >>>=20 >>>> If purpose is having crypto verification - I believe sha256 program = should be your choice. >>>=20 >>> My question was more specific to availability of that feature >>> (integrity + repair) rather than specific format :-) >>>=20 >>> The use case here is having a virtual machine (it was VirtualBox) = with >>> a bare os installed, plus some common applications, that is = snapshoted >>> at some point in time, then experimented a lot, restored from >>> snapshot, etc. I had a backup of such vm + snapshot backed up that = got >>> broken somehow. It would be nice to know that something is broken, >>> what is broken, maybe a way to fix :-) >>=20 >>=20 >> =E2=80=9CIntegrity" is a very broad term. What checksum algorithm is = fine enough? >> =20 >> For the instance, ZFS has several options for checksum: >>=20 >> = checksum=3Don|off|fletcher2|fletcher4|sha256|noparity|sha512|skein|edonr >> =20 >>=20 >> Having checksum for a filesystem is strongly recommended. However, If = consider image format, >> it doesn=E2=80=99t need to care about consistency in a file itself. = As example (!) - binary files in a system. >> They don=E2=80=99t have checksum integrated, validation is done by = another program - pkg or another. >>=20 >>=20 >>>=20 >>>=20 >>>> Why do you need modify snapshot image ? Could you describe more? Do = you >>>> modify current 3 snapshot files? >>>=20 >>> Analysis that require ram / nvram modification? Not sure if this is >>> already possible, but may come handy for experimenting with uefi and >>> maybe some OS (features) that will not run with unmodified nvram :-P >>=20 >>=20 >> Sorry I don=E2=80=99t get, why do you need to modify snapshot image, = but not directly vmem on the running >> VM? >>=20 >> Another question, checksum and modifying image - two mutual exclusive = things.=20 >>=20 >>>=20 >>>=20 >>>> If you are talking about compatibility of a Image format - it = should be compatible in >>>> both directions, at least for not so big format changes. >>>>=20 >>>> If consider overall snapshot/resume compatibility - I believe = forward compatibility >>>> is not case and target. Indeed, why do you need to resume an image = created by >>>> a higher version of a program? >>>=20 >>> This happens quite often. For instance there is a bug in application >>> and I need to revert to (at least) one step older version. Then I am >>> unable to work on a file that I just saved (or was autosaved for = me). >>> Firefox profile settings let be the first example. KiCAD file format >>> is another example (sometimes I need to switch to a devel build to >>> evade a nasty blocker bug then anyone else that uses a release is >>> blocked for some months including me myself). >>=20 >> Any additional thing has a cost of development, testing and support. = Current >> Implementation doesn=E2=80=99t support compatibility at all. Having = compatibility in both >> directions can be hard. >>=20 >> For example, if some variable is removed in bhyve, backward = compatibility is fine, >> but forward compatibly is not possible unless that removed variable = is being saved >> into a snapshot image just for forward compatibility. And of course, = it should be tested >> and verified as worked. >>=20 >> Do you like that approach? I don=E2=80=99t think so. So I guess only = backward compatibility >> should be supported to make the snapshot code simple and robust. >>=20 >> Thanks, >> Vitaliy Gusev >>=20 >>=20 >=20 >=20 > --=20 > Mario. --Apple-Mail=_9EC57A81-2637-440C-A2F4-5E1697C8811A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 25 = May 2023, at 19:22, Mario Marietto <marietto2008@gmail.com> = wrote:

Vitaliy,

what happens if I = clone your repo as source code on my FreeBSD system. Can I test your = code directly or not ? Anyway,I think that,before doing this,I need to = follow some kind of tutorial,to understand how the workflow is. = Otherwise I will be not able to test and stress it. =


You = should build kernel and tools, install it. Then you can run bhyve, = bhyvectl to deal with suspend/resume.

Please = follow 

 9.5. Building and Installing a Custom = Kernel



Make sure that = BHYVE_SNAPSHOT is enabled.

Also look at = build(7):




On Thu, May 25, 2023 at 3:40=E2=80=AFPM Vitaliy = Gusev <gusev.vitaliy@gmail.com> = wrote:


On 25 May 2023, at 04:30, Tomek CEDRO <tomek@cedro.info> wrote:

On = Wed, May 24, 2023 at 5:11=E2=80=AFPM Vitaliy Gusev wrote:
Protecting requires more efforts and it should be clearly = defined: what is purpose. If
purpose is having checksum with 99.9% = reliability, NVLIST HEADER can be widen
to have =E2=80=9Cchecksum=E2=80= =9D key/value for a Section.

Well, this could be = optional but useful to make sure snapshot did not
break somehow for = instance backup medium error or something like
that.. even more maybe = a way to fix it.. just a design stage idea = :-

Yes, new format can have checksum of a = Section data if implemented.



If purpose is = having crypto verification - I believe sha256 program should be your = choice.

My question was more specific to = availability of that feature
(integrity + repair) rather than = specific format :-)

The use case here is having a virtual machine = (it was VirtualBox) with
a bare os installed, plus some common = applications, that is snapshoted
at some point in time, then = experimented a lot, restored from
snapshot, etc. I had a backup of = such vm + snapshot backed up that got
broken somehow. It would be = nice to know that something is broken,
what is broken, maybe a way to = fix = :-)


 =E2= =80=9CIntegrity" is a very broad term. What checksum algorithm is fine = enough?
 
For the instance,  ZFS has = several options for checksum:

checksum=3Don|off|fletcher2|fletcher4|sha256|noparity|sha5= 12|skein|edonr

    =   =  


Having = checksum for a filesystem is strongly recommended. However, If consider = image format,
it  doesn=E2=80=99t need to care about = consistency in a file itself. As example (!)  - binary files in a = system.
They don=E2=80=99t have checksum integrated, = validation is done by another program  - pkg or = another.




Why do you = need modify snapshot image ? Could you describe more? Do you
modify = current 3 snapshot files?

Analysis that require ram = / nvram modification? Not sure if this is
already possible, but may = come handy for experimenting with uefi and
maybe some OS (features) = that will not run with unmodified nvram = :-P


Sorry I = don=E2=80=99t get, why do you need to modify snapshot image, but not = directly vmem on the = running
VM?

Another question, = checksum and modifying image - two mutual exclusive = things. 



If you are = talking about compatibility of a Image format - it should be compatible = in
both directions, at least for not so big format changes.

If = consider overall snapshot/resume compatibility - I believe  forward = compatibility
is not case and target. Indeed, why do you need =  to resume an image created by
a higher version of a = program?

This happens quite often. For instance = there is a bug in application
and I need to revert to (at least) one = step older version. Then I am
unable to work on a file that I just = saved (or was autosaved for me).
Firefox profile settings let be the = first example. KiCAD file format
is another example (sometimes I need = to switch to a devel build to
evade a nasty blocker bug then anyone = else that uses a release is
blocked for some months including me = myself).

Any additional = thing has a cost of development, testing and support. = Current
Implementation doesn=E2=80=99t support compatibility = at all. Having compatibility in both
directions can be = hard.

For example, if some variable is removed = in bhyve, backward compatibility is fine,
but forward = compatibly is not possible unless that removed variable is being = saved
into a snapshot image just for forward compatibility. = And of course, it should be tested
and verified as = worked.

Do you like that approach? I don=E2=80=99= t think so. So I guess only backward compatibility
should be = supported to make the snapshot code simple and = robust.

Thanks,
Vitaliy = Gusev




-- =
Mario.

= --Apple-Mail=_9EC57A81-2637-440C-A2F4-5E1697C8811A-- From nobody Sat May 27 18:02:09 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QT8kn4znxz4V6jH for ; Sat, 27 May 2023 18:02:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QT8kn1hLMz41nf for ; Sat, 27 May 2023 18:02:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1685210529; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=phiBGIPI0CV0zQwmAEVp5MuWnRG2rc+/H3vOb3R03XU=; b=g0tdWcGSiwPMgx/RmoQAjDrNNspy8BPxVAKR+XdICYIB/Fbs8CHyM3E8dKUDdlBtPvnnL7 dDPgHBhij5wgJlfW0v+51bncVxFWr7HZ4pS/omHo+rmv0GmIFZTLaFoO41j017jE7T+V1F JIhcROas5N+nF/dgl5VQCk53N9HtPwJw3V1sB1fKYoZR+LSze3iFq7RNkHA+JqSkpQpvOT T5X9CjykmENIfJhb0mLQ73gtr9eCHDZ5R2XaOxWFkBELi8k7q7vsahYp9L32NQwYmACSCf 4rs5Qaev2cAKH4K3EQfAwC+KnAJlW5BkwoQVmrZOvFgdn1i1/KGke4TvpRchFw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1685210529; a=rsa-sha256; cv=none; b=YlG7h264tWdSFQLnWCS1QLBJSGgrvp9kQee+GsGAfotppXb1gtsYMuG1Li3OIZAKYhA5l5 rU6rzgDfgKZtiBk5sigwYoCKEKZ4/b+NVMCSSmLI2KRsS5Y4SnhJJLhGfRPxuDRA3igHPk gi7ta2x9EI+tNOPgjeIahFelP9yHLM1Vf8loHzBc6r/IwXxI9cOnUgwgBFa5sFFD71n61o 9RiBVUKDbcpWp1/cC1lmfQqQZU0Q1koqBVn5X1klvEnQU6PA9CrmW8vwW4tJek1YSeH40h QwwaOn3wH7VwXRtJJHWJfFI7pX43J23DdnQ/WkIokr1QGC6Yr8asGc936bz7fw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4QT8kn0n9Zzxpn for ; Sat, 27 May 2023 18:02:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 34RI29Va066197 for ; Sat, 27 May 2023 18:02:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 34RI29UV066196 for virtualization@FreeBSD.org; Sat, 27 May 2023 18:02:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 271578] FreeBSD fails to init SMP and hits panic post install in Azure ARM64 Date: Sat, 27 May 2023 18:02:09 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: crash X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: grahamperrin@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: Andrew@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: keywords bug_status cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271578 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- Keywords| |crash Status|Open |In Progress CC| |grahamperrin@freebsd.org, | |virtualization@FreeBSD.org --- Comment #5 from Graham Perrin --- ^Triage: make virtualization@, previously the assignee, a CC recipient. --=20 You are receiving this mail because: You are on the CC list for the bug.=