From nobody Sat Apr 29 21:24:10 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q82Xp4W7qz48H6t for ; Sat, 29 Apr 2023 21:24:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q82Xp3Jm3z3pmx for ; Sat, 29 Apr 2023 21:24:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682803450; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UqrX8+m8DtOF0eg1TySX3ZBaZvPQ6KyyV44K4AoSWDQ=; b=TO0ZWILalkT4HqwVDNfJ5u8FMCZoCRdgH1lvVzzp467Yl/zZPTa2Qe2aiaJNvMAyJr3/QD v8UxOBcWCvybY6AYVYLlr7yoeRnNVbCG/UmKIjrlJ+6BEXis/E4JYKlDnxXTX7u3qUI4dk n+AyDN9oLwx9Mbtam51JU3PQfBkOw6/LCnPA14YF76pl82p2eUSpjg9gXUFZDoO/YBfzs+ GMzvVHSpfSmUbIKEwximBpzthxUzEPa3WOf/Oown2OLlR6NAezkUXO3vFWI91j/lQ8DHJf 2U0d504ZgntOY3O/0b0ZkaoDTKU/e2CxwvKSkZkLmHv7h3474Yo+t0/Y7aPv7g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1682803450; a=rsa-sha256; cv=none; b=hh9gk08MuV8sf7b3i3+iLT8KC/CBYyRfrtDSPRh7GQ4xmBjrGSwzL+IFY3aw+iEH32xLhi G6DAAqea8geCD45SC6Q4HEW4tJ5s7YssmyoDITAKJhs0f2KNoT+28D2BqREEV3AZ9G4mNi 3mA6EDyjWyfLmGfZTlkFwJbqOFXPLaa16ZXg5MEdzSm+qP1tqLh6doOrwWylN2XtanLQ5t fKWvFGvTzTIUGq945pmu78DqmEQgrdH+lvW/YQsFG7lHDSiEDxbX1OlqEd645N2OY92Fyc 1hCltGIeLW6kFIQFPQCwAENtLjE0P4pkNokzdpkKFtog+WeIGHdRa7j4pi84sg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Q82Xp2NrCzxDW for ; Sat, 29 Apr 2023 21:24:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 33TLOAkx089649 for ; Sat, 29 Apr 2023 21:24:10 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 33TLOArd089647 for virtualization@FreeBSD.org; Sat, 29 Apr 2023 21:24:10 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 271134] bhyve(8), bhyvectl(8), bhyve(4), bhyve_config(5): manual pages cleanup Date: Sat, 29 Apr 2023 21:24:10 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Documentation X-Bugzilla-Component: Manual Pages X-Bugzilla-Version: Latest X-Bugzilla-Keywords: bhyve X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: grehan@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D271134 Peter Grehan changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |grehan@FreeBSD.org --- Comment #5 from Peter Grehan --- (In reply to Graham Perrin from comment #3) At the time bhyve was written, it was commonplace to have to rebuild an o/s kernel with paravirtualized drivers and/or other components to allow it to = run under a hypervisor - "modified". This isn't really the case anymore so that sentence could be removed. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun Apr 30 16:15:28 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8WfB22pwz480hB for ; Sun, 30 Apr 2023 16:15:30 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8Wf96jL9z3Nhn for ; Sun, 30 Apr 2023 16:15:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682871329; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7TAJtTrdPJ5/fYhPp/EhTPET4BVTWfkeC+TSXIkGP/o=; b=tzqXDH29bNm2u4AnJODTkXw4eSMr8V1ULuewzIg+Ezzj86HsEaQjfvtJik5h7dxvsTxLaq R1xaAeoImmDiA0Uvb7hMlPhF1MZyy7lx3nQki4oBDVIjcOon+hDcIUd33llQxpnGfpFh91 RzsE2rZ10Hndqyjg3ouY2nt6OpQC2H+hRrvR94X4gd759F3MzSne2kvZ7X0XpIHlY8dxta J+INwmMHPhRDB7Q1+sMjGEpHZ8kFOaDlK2YVVMBOJxR7Evvienk5sytpPUZUpVBQDTKweo g1irAsIcyhSddNPQZeHS3mxsyPrCQWETnzYxoRUM1Ria/rr+H3Yt9Vs06EoyBA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1682871329; a=rsa-sha256; cv=none; b=bMseQdKUvYkDGLBA8jfrBfw414zWRtP35RuKzITamemUU4F7vnHmdLsQQRGobykh3Y3Uik hUkbPNdwElN4AzcO86UQERoBKKzkOxMBoWqCpR7c28W9MZbQ0ZtLHTRIDvdIh0S+A9y8sC Zuo9vl8qhTmbFgeNAMgQBItN4Ra60OeHqU46uj0CkeehX/v0480Ik9Zk5qnNtwZvP73h6N GsXDATt71kDZBU2Tkk33uiBtsSTXtvgJI73laYNo5z7PMw7fDafjB0nK5k8755QXZQRkLW hZGZcUOgcdYyLBg9G705MpjcUtu99nkOYpeI8kY7QKTQf7UfZ7S8HeSVlFbpyA== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Q8Wf95j3FzVwK for ; Sun, 30 Apr 2023 16:15:29 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 33UGFT1H021631 for ; Sun, 30 Apr 2023 16:15:29 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 33UGFTSK021630 for virtualization@FreeBSD.org; Sun, 30 Apr 2023 16:15:29 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 259651] bhyve process uses all memory/swap Date: Sun, 30 Apr 2023 16:15:28 +0000 X-Bugzilla-Reason: CC AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 12.2-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: grahamperrin@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Overcome By Events X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: grahamperrin@freebsd.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_status resolution assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D259651 Graham Perrin changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |grahamperrin@freebsd.org, | |virtualization@FreeBSD.org Status|New |Closed Resolution|--- |Overcome By Events Assignee|virtualization@FreeBSD.org |grahamperrin@freebsd.org --- Comment #5 from Graham Perrin --- (In reply to Jeremy Lea from comment #3) > =E2=80=A6 I'd say that whatever was causing it is fixed in 13. Thanks; closing.=20 (In reply to void from comment #2) > vfs.zfs.arc_max Others to consider include:=20 vfs.zfs.arc.sys_free vfs.zfs.arc_free_target --=20 You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.= From nobody Sun Apr 30 21:00:51 2023 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8dzS081gz48JHF for ; Sun, 30 Apr 2023 21:00:52 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8dzR21Yqz47rr for ; Sun, 30 Apr 2023 21:00:51 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682888451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=k8/CszNuDMJMTDvp5v9eXQ8aJnoLttcAEFratgh+cyA=; b=uOg3Ckm46bpirRFpsJ19UCjLRGuHNf75CQNZ/har9UBCuEqE6KLI3VE3PBo0oHH9lqFR1+ MbV63G2fH6bmmMTjhP5hq+GwWlHj53MijvolLTAJE8frgUE8Vf+I6GKDQFDpnxDOH4/Rzl Mg57E5uywG+X0zKyJ6eRMrYUUqC9khHThIfnCQTWukutaQh50POluvakXeYNJn83Yqr1TA YPz4nkLSFVEYzdr3dQy9O4ZGczsEwhUEaU82wvGSUsNsg3W1xKN1wc7HIJhxbcGgLx9U9I vQ/CPYpWMrqX66gtA3VrW/TPQq9kjxq198f867Mfa4Q5RmGMtihDUZ5xl6Ypbg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1682888451; a=rsa-sha256; cv=none; b=jDrdoZ5KuaWBg3IdAkY3XI+nqgXhno7Ug5JMFECy3FdJDwUWKSB3zXQBPaVoAoFmyT+qdV 0PLFS9IYegJRWJ/yamh+dCxKnyLcWUX3MKar69UrhyXpjNGo6TrhN2yWntJfk9SRwi1IVI +xOgXGOn31dAU7XzdkmvBKTwTBGJ4/LqAasWt7myV63ZeQoddcg0EEP0TfBk1J0vyiGrTI mF4pTtNXKmaZb5Mkc4miqdbn83EDuGh7gE9Wuxm1c3Emma4MPbm7qGmhCCKa2i38NR+V7m +NzThqEk0aD2ojliGHKfiTgnMZ92vcict+sFsU00eIXirHOtecblENGcf2owgg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Q8dzR0yJMzdZR for ; Sun, 30 Apr 2023 21:00:51 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 33UL0pKY048516 for ; Sun, 30 Apr 2023 21:00:51 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 33UL0p1n048514 for virtualization@FreeBSD.org; Sun, 30 Apr 2023 21:00:51 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202304302100.33UL0p1n048514@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 30 Apr 2023 21:00:51 +0000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="16828884510.2eD9e724.45297" Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N --16828884510.2eD9e724.45297 Date: Sun, 30 Apr 2023 21:00:51 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 247208 | mpt(4): VMWare virtualized LSI controller panics New | 240945 | [hyper-v] [netvsc] hn network driver incorrectly Open | 244838 | "bectl activate -t" does not honor the -t flag in 3 problems total for which you should take action. --16828884510.2eD9e724.45297 Date: Sun, 30 Apr 2023 21:00:51 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
In Progress |    247208 | mpt(4): VMWare virtualized LSI controller panics 
New         |    240945 | [hyper-v] [netvsc] hn network driver incorrectly 
Open        |    244838 | "bectl activate -t" does not honor the -t flag in

3 problems total for which you should take action.
--16828884510.2eD9e724.45297-- From nobody Sun Apr 30 21:41:36 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8ftk3KNrz48LLQ; Sun, 30 Apr 2023 21:41:50 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [38.97.5.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8ftj1lVYz4GWP; Sun, 30 Apr 2023 21:41:49 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 38.97.5.131 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx1.shrew.net (8.15.2/8.15.2) with ESMTP id 33ULfgOY067025; Sun, 30 Apr 2023 16:41:42 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.32] (unknown [136.49.230.220]) by mail.shrew.net (Postfix) with ESMTPSA id 645C2192C40; Sun, 30 Apr 2023 16:41:37 -0500 (CDT) Message-ID: Date: Sun, 30 Apr 2023 16:41:36 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: BHYVE_SNAPSHOT Content-Language: en-US To: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org References: From: Matthew Grooms Cc: elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.shrew.net [10.24.10.10]); Sun, 30 Apr 2023 16:41:42 -0500 (CDT) X-Spamd-Result: default: False [1.40 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_LONG(-1.00)[-0.998]; NEURAL_SPAM_SHORT(0.86)[0.859]; NEURAL_SPAM_MEDIUM(0.79)[0.789]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:174, ipnet:38.0.0.0/8, country:US]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; FREEMAIL_CC(0.00)[gmail.com]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[shrew.net]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Q8ftj1lVYz4GWP X-Spamd-Bar: + X-ThisMailContainsUnwantedMimeParts: N On 4/29/23 06:17, void wrote: > Hi, > > Where can I read up about the recently introduced BHYVE_SNAPSHOT ? > > I can only find terse information about what it does (in man 5 src.conf) > but not about how to use it and/or why. This began as sponsored student work with the Politehnica University of Bucharest back in 2016 with an eye toward developing Live MIgration as a feature. The initial bhyve save/restore patch was committed back in 2020 ... https://reviews.freebsd.org/rS360648 Student projects related to bhyve continued until last year, but we gave up due to there being such a massive accumulation of patches with no path to getting them committed. We tried reaching out to project maintainers and the FreeBSD foundation alike for support and feedback, but there seems to be little to no interest. There have also been attempts by companies that rely on these features to and get them improved and committed. Unfortunately they don't appear to be having much luck either ... https://reviews.freebsd.org/D38858#885651 Would you like to see support for VM snapshots in the generic kernel? How about support for saving/restore checkpoints using QCOW2, VMDK via libvdsk? How about support for warm or live migration? How about USB device pass-through? There are experimental patches for all these features that were developed by students at UPB. In a lot of cases, there are open reviews that have been waiting on feedback for ages. Here is a a presentation given by Elena from UPB just last month at the FreeBSD devsumit in Tokyo ... https://wiki.freebsd.org/DevSummit/202303?action=AttachFile&do=view&target=Presentation+-+bhyvecon.pdf The case is quite plain. I'm not sure what the solution is to this problem. I'd love to hear feedback from the community about how I've got this completely wrong and how the course could be corrected. That would be something. -Matthew From nobody Mon May 1 00:04:22 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8k3S6DNwz48Tt0 for ; Mon, 1 May 2023 00:04:36 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yw1-x1135.google.com (mail-yw1-x1135.google.com [IPv6:2607:f8b0:4864:20::1135]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8k3S2brXz4TMj for ; Mon, 1 May 2023 00:04:36 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1135.google.com with SMTP id 00721157ae682-55a2691637bso10291587b3.0 for ; Sun, 30 Apr 2023 17:04:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1682899475; x=1685491475; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=WS/+SmDg9E6kfzNV+E23WyBQMGjpljvUh1xy/4KnyDI=; b=h2eawhLNv5F7oRQ3+NSJTcOyP7n0JVkWgTx83uOv0UZ5mXdZY85Y3DMRHUuQUGdF0a qG7aBWbATHaKGjBO6ZT65kIA/+uwkNbqCUmWR/nwcwEHzbDDL3/qedybKjqRUt8+/cOL AdNuW8dEwVeC0GIlruIWN/VaaA2EAtqR+8M9qBevKuFA2IvWlHpYtrdbbjMoYFa4e9Tx cFi/Bgv+V6HnHNUPBTzrOGdIaCQDIEP50IIM2tcGGwWXD2dIxuYSmr3fa43cj1o4nqr1 icisuaAysV/wXYhzUFdrzvbqE0yJlztIiZqQ8tI9ubhrKhHQD0+TXXLj/us0O3DHuFax y23g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682899475; x=1685491475; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WS/+SmDg9E6kfzNV+E23WyBQMGjpljvUh1xy/4KnyDI=; b=UcpvFYbyd///E9XjM6V1CqyC7uErPZgn3nsB2jUpWLvde33+ZGnxdCOhFMy/W1NPw9 AXJVe+Qppj6S3CdUcotgSjugkJEHo7S9UI26k9ClLBP30gQEahf68lK0aorwRNblEITQ ARklHw2CkZyfF6VQi2aeDc6HsI9rqtpkS96o6REtKOoPuJLWmZHPm97abxD6onjqChHi 4AvfILZv7y/PfMcFzijNXdFE4ELC+nTTOhiAkvPo3kq0VJxvHsbym+vu653dD9vgVDnb Yneet1Csi8bzH7o0kGt5AgS+lVFogfmLD2WqcDWnT65AXcWF73aMhNToOcsmBI34BZ6/ Bbbw== X-Gm-Message-State: AC+VfDxtreOKU3ZlKV/ElhLmAkHUHeKycwWC2yBWMJpbQLQ9KykCICbu eEX5miyV/2Le6HopPflsrxAA3g== X-Google-Smtp-Source: ACHHUZ7CP5vDtBRIxvautD8B9t4OVqlfgDANkt/FqORg3gwZ8Ld6bgYtk+zgsla4VgNGuWgWMvw0Aw== X-Received: by 2002:a0d:d8c4:0:b0:559:fb89:e2c4 with SMTP id a187-20020a0dd8c4000000b00559fb89e2c4mr4843838ywe.22.1682899474967; Sun, 30 Apr 2023 17:04:34 -0700 (PDT) Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com. [209.85.128.181]) by smtp.gmail.com with ESMTPSA id d190-20020a0df4c7000000b0054fcbf35b94sm6943683ywf.87.2023.04.30.17.04.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 30 Apr 2023 17:04:34 -0700 (PDT) Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-54fc6949475so27382557b3.3; Sun, 30 Apr 2023 17:04:34 -0700 (PDT) X-Received: by 2002:a0d:d648:0:b0:55a:535c:5c53 with SMTP id y69-20020a0dd648000000b0055a535c5c53mr1756086ywd.8.1682899473993; Sun, 30 Apr 2023 17:04:33 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Tomek CEDRO Date: Mon, 1 May 2023 02:04:22 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Matthew Grooms , Graham Perrin , Ed Maste Cc: FreeBSD Hackers , Virtualisation on FreeBSD , elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: multipart/alternative; boundary="0000000000005719b605fa96918e" X-Rspamd-Queue-Id: 4Q8k3S2brXz4TMj X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000005719b605fa96918e Content-Type: text/plain; charset="UTF-8" o_O -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info On Sun, Apr 30, 2023, 23:42 Matthew Grooms wrote: > On 4/29/23 06:17, void wrote: > > Hi, > > > > Where can I read up about the recently introduced BHYVE_SNAPSHOT ? > > > > I can only find terse information about what it does (in man 5 src.conf) > > but not about how to use it and/or why. > > This began as sponsored student work with the Politehnica University of > Bucharest back in 2016 with an eye toward developing Live MIgration as a > feature. The initial bhyve save/restore patch was committed back in 2020 > ... > > https://reviews.freebsd.org/rS360648 > > Student projects related to bhyve continued until last year, but we gave > up due to there being such a massive accumulation of patches with no > path to getting them committed. We tried reaching out to project > maintainers and the FreeBSD foundation alike for support and feedback, > but there seems to be little to no interest. There have also been > attempts by companies that rely on these features to and get them > improved and committed. Unfortunately they don't appear to be having > much luck either ... > > https://reviews.freebsd.org/D38858#885651 > > Would you like to see support for VM snapshots in the generic kernel? > How about support for saving/restore checkpoints using QCOW2, VMDK via > libvdsk? How about support for warm or live migration? How about USB > device pass-through? There are experimental patches for all these > features that were developed by students at UPB. In a lot of cases, > there are open reviews that have been waiting on feedback for ages. Here > is a a presentation given by Elena from UPB just last month at the > FreeBSD devsumit in Tokyo ... > > > https://wiki.freebsd.org/DevSummit/202303?action=AttachFile&do=view&target=Presentation+-+bhyvecon.pdf > > The case is quite plain. I'm not sure what the solution is to this > problem. I'd love to hear feedback from the community about how I've got > this completely wrong and how the course could be corrected. That would > be something. > > -Matthew > > > > --0000000000005719b605fa96918e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
o_O

--
= CeDeROM, SQ7MHZ, http://www.tomek.c= edro.info

On Sun, Apr 30, 2023, 23:42 Matthew Grooms <mgrooms@shrew.net> wrote:
On 4/29/23 06:17, void wrote:
> Hi,
>
> Where can I read up about the recently introduced BHYVE_SNAPSHOT ?
>
> I can only find terse information about what it does (in man 5 src.con= f)
> but not about how to use it and/or why.

This began as sponsored student work with the Politehnica University of Bucharest back in 2016 with an eye toward developing Live MIgration as a feature. The initial bhyve save/restore patch was committed back in 2020 ..= .

https://reviews.freebsd.org/rS360648

Student projects related to bhyve continued until last year, but we gave up due to there being such a massive accumulation of patches with no
path to getting them committed. We tried reaching out to project
maintainers and the FreeBSD foundation alike for support and feedback,
but there seems to be little to no interest. There have also been
attempts by companies that rely on these features to and get them
improved and committed. Unfortunately they don't appear to be having much luck either ...

https://reviews.freebsd.org/D38858#885651
Would you like to see support for VM snapshots in the generic kernel?
How about support for saving/restore checkpoints using QCOW2, VMDK via
libvdsk? How about support for warm or live migration? How about USB
device pass-through? There are experimental patches for all these
features that were developed by students at UPB. In a lot of cases,
there are open reviews that have been waiting on feedback for ages. Here is a a presentation given by Elena from UPB just last month at the
FreeBSD devsumit in Tokyo ...

https://wiki.freebsd.org/DevSummit/202303?action= =3DAttachFile&do=3Dview&target=3DPresentation+-+bhyvecon.pdf
The case is quite plain. I'm not sure what the solution is to this
problem. I'd love to hear feedback from the community about how I'v= e got
this completely wrong and how the course could be corrected. That would be something.

-Matthew



--0000000000005719b605fa96918e-- From nobody Mon May 1 00:45:56 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8kzN553qz48WpS; Mon, 1 May 2023 00:46:08 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-yw1-x112b.google.com (mail-yw1-x112b.google.com [IPv6:2607:f8b0:4864:20::112b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8kzN3T4jz4ZmM; Mon, 1 May 2023 00:46:08 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x112b.google.com with SMTP id 00721157ae682-55a00da4e53so20829927b3.0; Sun, 30 Apr 2023 17:46:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682901967; x=1685493967; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=OLH7E+tNyMAOlFTmQ3zboPvehIf56Te4NVP8VPyTTDU=; b=DSQCRPiSGgG3tn6Pzypqt3oN7zB4N+46aUv4gN0FSd84ZbpNOmMnzoRHkDdFIWmQOW b+BDZq/2Khj1R/oE3Lj0P4jqyE2FGGQk/3PM5oiJboDb4i3dFV2d/DpsV1WP/EcX1jXj TFR1S+ENGiGHkj1JmM6RV3IRjEh6rdm83U/NxeuYcB9HN4Ev402b86mmZe1W+b0/H4rA NKCxSFeHiZ7qp/XKGDI8ALm1+cwWNgREHmK1JFVOa0mVCSvOOukTyhdivKqXPVyfUL9u Q84XsN4pFqnp1lGkgkrZSqesQr8eVAnTngFscomJWGo7sdm5xnGZ1mwFvfBeMmDa/RZe cM2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682901967; x=1685493967; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OLH7E+tNyMAOlFTmQ3zboPvehIf56Te4NVP8VPyTTDU=; b=hCzdzzduSCCt72ySVTpXao+2jgExHj4mwdU7F/p1OZ/nHJaG7LlDK8OHOZwV06VwdF PE/eOajEwmBcX83/k8ghliB4/8wCSQ9Mz+Lvu4uUgm7Z40k0h6zj5dmh94Nhvd0toCPE 0buUO0NjqqNZjIjTmWkzX1DakB4p9EPiWPVcrTVSs3iiwk/q7Xke98Arrlrl9pcMPJ75 1Sk+oj7WV2D4jW9gwvfzo6OGULcXkEJRde+z5eDMfg1TEPIMRMFjzwpo7XVB9Ljgvrtp BsvkuYFcV5yB4BfFOGDYkc5MZ6EUYrBFebn2lrZJWqLQyLoWGR+/bEaK6lBE2zI48i+f g7Ag== X-Gm-Message-State: AC+VfDwL6WJITxNAGiQ3Nr5AKz7coHGeuYia1D86Re4J66lj2YX+6YSr F4Gwgow9XKrG2ZC9aZaXrMfElfnshcnbNnaTMG0= X-Google-Smtp-Source: ACHHUZ7Qmoatazbvvxk4LXs0v60QS2uLutzQkmU3Mq5dBFA2wX00HG/yo843iLvNCwJJKJOzB6Rk5UppoNlQKrfCok0= X-Received: by 2002:a81:71d6:0:b0:559:f52b:7c5f with SMTP id m205-20020a8171d6000000b00559f52b7c5fmr4941081ywc.17.1682901967597; Sun, 30 Apr 2023 17:46:07 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Mon, 1 May 2023 02:45:56 +0200 Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Tomek CEDRO Cc: Matthew Grooms , Graham Perrin , Ed Maste , FreeBSD Hackers , Virtualisation on FreeBSD , elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: multipart/alternative; boundary="000000000000f8772005fa9725dd" X-Rspamd-Queue-Id: 4Q8kzN3T4jz4ZmM X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_RCPT(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000f8772005fa9725dd Content-Type: text/plain; charset="UTF-8" whats the meaning of o_O ? Il lun 1 mag 2023, 02:04 Tomek CEDRO ha scritto: > o_O > > -- > CeDeROM, SQ7MHZ, http://www.tomek.cedro.info > > On Sun, Apr 30, 2023, 23:42 Matthew Grooms wrote: > >> On 4/29/23 06:17, void wrote: >> > Hi, >> > >> > Where can I read up about the recently introduced BHYVE_SNAPSHOT ? >> > >> > I can only find terse information about what it does (in man 5 src.conf) >> > but not about how to use it and/or why. >> >> This began as sponsored student work with the Politehnica University of >> Bucharest back in 2016 with an eye toward developing Live MIgration as a >> feature. The initial bhyve save/restore patch was committed back in 2020 >> ... >> >> https://reviews.freebsd.org/rS360648 >> >> Student projects related to bhyve continued until last year, but we gave >> up due to there being such a massive accumulation of patches with no >> path to getting them committed. We tried reaching out to project >> maintainers and the FreeBSD foundation alike for support and feedback, >> but there seems to be little to no interest. There have also been >> attempts by companies that rely on these features to and get them >> improved and committed. Unfortunately they don't appear to be having >> much luck either ... >> >> https://reviews.freebsd.org/D38858#885651 >> >> Would you like to see support for VM snapshots in the generic kernel? >> How about support for saving/restore checkpoints using QCOW2, VMDK via >> libvdsk? How about support for warm or live migration? How about USB >> device pass-through? There are experimental patches for all these >> features that were developed by students at UPB. In a lot of cases, >> there are open reviews that have been waiting on feedback for ages. Here >> is a a presentation given by Elena from UPB just last month at the >> FreeBSD devsumit in Tokyo ... >> >> >> https://wiki.freebsd.org/DevSummit/202303?action=AttachFile&do=view&target=Presentation+-+bhyvecon.pdf >> >> The case is quite plain. I'm not sure what the solution is to this >> problem. I'd love to hear feedback from the community about how I've got >> this completely wrong and how the course could be corrected. That would >> be something. >> >> -Matthew >> >> >> >> --000000000000f8772005fa9725dd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
whats the meaning of o_O ?

Il lun 1 mag 2023, 02:04 Tomek = CEDRO <tomek@cedro.info> ha s= critto:
o_O

On Sun, Apr 30, 2023, 23:42 Matthew Grooms &l= t;mgrooms@shrew.net> wrote:
O= n 4/29/23 06:17, void wrote:
> Hi,
>
> Where can I read up about the recently introduced BHYVE_SNAPSHOT ?
>
> I can only find terse information about what it does (in man 5 src.con= f)
> but not about how to use it and/or why.

This began as sponsored student work with the Politehnica University of Bucharest back in 2016 with an eye toward developing Live MIgration as a feature. The initial bhyve save/restore patch was committed back in 2020 ..= .

https://reviews.freebsd.org/rS360648
Student projects related to bhyve continued until last year, but we gave up due to there being such a massive accumulation of patches with no
path to getting them committed. We tried reaching out to project
maintainers and the FreeBSD foundation alike for support and feedback,
but there seems to be little to no interest. There have also been
attempts by companies that rely on these features to and get them
improved and committed. Unfortunately they don't appear to be having much luck either ...

https://reviews.freebsd.org/D38858#88= 5651

Would you like to see support for VM snapshots in the generic kernel?
How about support for saving/restore checkpoints using QCOW2, VMDK via
libvdsk? How about support for warm or live migration? How about USB
device pass-through? There are experimental patches for all these
features that were developed by students at UPB. In a lot of cases,
there are open reviews that have been waiting on feedback for ages. Here is a a presentation given by Elena from UPB just last month at the
FreeBSD devsumit in Tokyo ...

https://wiki.freebsd.org/DevSummit/20= 2303?action=3DAttachFile&do=3Dview&target=3DPresentation+-+bhyvecon= .pdf

The case is quite plain. I'm not sure what the solution is to this
problem. I'd love to hear feedback from the community about how I'v= e got
this completely wrong and how the course could be corrected. That would be something.

-Matthew



--000000000000f8772005fa9725dd-- From nobody Mon May 1 01:13:08 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8lZp2NBfz48YSb for ; Mon, 1 May 2023 01:13:22 +0000 (UTC) (envelope-from tomek@cedro.info) Received: from mail-yb1-xb35.google.com (mail-yb1-xb35.google.com [IPv6:2607:f8b0:4864:20::b35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8lZp0nljz3C1t for ; Mon, 1 May 2023 01:13:21 +0000 (UTC) (envelope-from tomek@cedro.info) Authentication-Results: mx1.freebsd.org; none Received: by mail-yb1-xb35.google.com with SMTP id 3f1490d57ef6-b9d8730fe5aso3109651276.1 for ; Sun, 30 Apr 2023 18:13:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cedro.info; s=google; t=1682903601; x=1685495601; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Aq7V3fWEnjCs/Oidw/9YQ0jLto133q9eo2LE+dD8358=; b=S6EwT0kTjPf2vpOaoq/XKm5HJAenAcTe5VFMkYwIYXVnEV4BO0LnIlAU/mUXhjnHaS edSpI8pAUD0oAvzGDFxLavCkWVkpKbdLXB3cV5DNgzLSQ+eUfF6rZQRMSbBF+SRVLKwL tIn4sylCW51DuZmn42a7KcnbNHsRyN22OpVT3cni3alV+Khj8kjJRcvBCyT/mqpmVabk SR2yr4cNaqU87MKb9Ly0pXrkfRcUTVHQngginYO4bTvK2rWTeSXFEjlKFnZO5xS/qg0d vsC+mtLo/WtUkgAXVRJMFT+IhB2kr2yE+GbejcrEiySuefJvXy/ZtU+KciLIJFgypKl7 bFsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682903601; x=1685495601; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Aq7V3fWEnjCs/Oidw/9YQ0jLto133q9eo2LE+dD8358=; b=gKh2940D30MnB/FoeT7knhxt9Z1f938TbciEzwvM9RK//xtWsl/1fBiAh/Hf5NlPH+ A48gYyp7A3h9t5ex+MyMtun2NXjuPvuxhKaag+Y/txJ3iOEQ5nSjQUHr1hZaW3Ze/bPb tv0HF9rzchEAZG+7a/TulqdtJDd0F7m5XbWitRxgVWuzMLZbbITw+kQRt+BJ9xBYGvLe imRbUHHeoN9J+J24dCAwijQshyDs0DfLEhfpn2eHpDN1J2PTD6hyFw3xpCuNj7JKUUa8 WInG9YDJCe7bXDiKgAMcy0/FFwDbPWm2hfWtJk32bjYrcyPV5+/CITCFhSpYPkS+1DT6 8img== X-Gm-Message-State: AC+VfDwM/xpxh4xZtZgBNbKrhtch4OWSyxJZCA2m4pjoTNMYVGNb3ldr kqSJ2FHFjnJTkjejgNmLD9+RsqZn4i7yAaCIBFk= X-Google-Smtp-Source: ACHHUZ6gFx8D0gifXHYVIHt2KvQcMDmInzsATHw/uR/WbetTGSEp+kL0+Vq31UVCvJsekh2r/VJgSQ== X-Received: by 2002:a25:aaab:0:b0:b9a:791f:284e with SMTP id t40-20020a25aaab000000b00b9a791f284emr10869978ybi.29.1682903601154; Sun, 30 Apr 2023 18:13:21 -0700 (PDT) Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com. [209.85.219.178]) by smtp.gmail.com with ESMTPSA id b185-20020a2567c2000000b00b8f46d74e5dsm1427707ybc.37.2023.04.30.18.13.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 30 Apr 2023 18:13:20 -0700 (PDT) Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-b9a659bd135so3079919276.3; Sun, 30 Apr 2023 18:13:20 -0700 (PDT) X-Received: by 2002:a25:b782:0:b0:b95:2bd5:8f86 with SMTP id n2-20020a25b782000000b00b952bd58f86mr10184115ybh.26.1682903600313; Sun, 30 Apr 2023 18:13:20 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Tomek CEDRO Date: Mon, 1 May 2023 03:13:08 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Mario Marietto Cc: Matthew Grooms , Graham Perrin , Ed Maste , FreeBSD Hackers , Virtualisation on FreeBSD , elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4Q8lZp0nljz3C1t X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mon, May 1, 2023 at 2:46=E2=80=AFAM Mario Marietto wrote: > whats the meaning of o_O ? surprise / shock.. can't believe what i see.. in the context of "code is out there but never used" :-P --=20 CeDeROM, SQ7MHZ, http://www.tomek.cedro.info From nobody Mon May 1 02:31:23 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q8nJj4WDgz48d9l; Mon, 1 May 2023 02:31:17 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q8nJj44XLz3KF7; Mon, 1 May 2023 02:31:17 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-94a342f4c8eso61561466b.0; Sun, 30 Apr 2023 19:31:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682908276; x=1685500276; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=OZb/SU58VWwmR4byX4uYgVx8EtL+8JbEAI9GQuJXP3o=; b=rCaofLwsLkWba8NXiEaA58TiFBtfk+WxQNr5CkEBwVxRJjnX9QFTetqOFMl0cgbvw1 hu5mRchRnsCyOpUv6XBxCyhrtpK56kq8usE3KRPlGQzwCyLHsI7Jizebo5V4vHgNGhyc 69MGQlYg8K6HLQYdtUNpHvRjG/8k4G+36/X5C2N966CBdgAgyXKppvjKUC2/AnTrNCjX YxC2peMaSHaqBc340AKW93LOsyt0iI3awRhSad5Yj72miCku4Zf0z3T7i3s1Ni7hbktx tleFhuRpdpuJ4EC79pJ62ct9moBw6EQOnlnqd8QWGKCDhlel6gNim9Aff4S/0cXeyGRd M14A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682908276; x=1685500276; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OZb/SU58VWwmR4byX4uYgVx8EtL+8JbEAI9GQuJXP3o=; b=l88plTLx/H/iwS0BihqK4k5rSYD0aMJ4ihn15K8mXvzzwYQwm1r5j9ukYYH8W2BRa0 uAN2+LM4K2CqgjcpGziyYCIm7Z+PqiWTevd0OasQuEMTqRq+rkPBeSvQwFOMJ9vfc9Mh Rm86OYie0yMvZx08d3LgNkI+NKjwwpGUl2o1dYslx2q4Y/I6v4cr6T9G08cfwbvT3oAo mc4g/HHao+n+fv8+g8oVq529uKFqlZ5OgEapN2MgHgPIByQKHVo/n3KGbqVvz5NMeu+1 xAOB8qYANb3Q0PCWxCdzKhFlUADWzqxhKy2gHEns0OhCkITYloHvrNQmnGtz8oWPgtqB cKSQ== X-Gm-Message-State: AC+VfDx8bT5e1I2PC3u5Fg8mxPbJLtcBwZWXA9fwuVbzdNcePZYfeVqj kHMwEVnrTGyAZFRmoyB27ywWhQU9q7ShO1i9Y/E= X-Google-Smtp-Source: ACHHUZ4DYbtAiLv8F3woIC1hFX3lqGnicM5Cuqon6VPc7QiMv9E+3jcjeNd0Yms+G10V+hfMHvOM7kef8G6iOegbPR0= X-Received: by 2002:a17:906:7483:b0:94e:8b6c:462c with SMTP id e3-20020a170906748300b0094e8b6c462cmr7948246ejl.2.1682908275818; Sun, 30 Apr 2023 19:31:15 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rob Wing Date: Sun, 30 Apr 2023 18:31:23 -0800 Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Matthew Grooms Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: multipart/alternative; boundary="000000000000f8434305fa989d84" X-Rspamd-Queue-Id: 4Q8nJj44XLz3KF7 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000f8434305fa989d84 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hey Matthew, On Sun, Apr 30, 2023 at 1:41=E2=80=AFPM Matthew Grooms = wrote: > > Would you like to see support for VM snapshots in the generic kernel? > Is there a review open that addresses the limitations described in the commit message that brought the snapshot feature in? https://github.com/freebsd/freebsd-src/commit/483d953a86a2507355f8287c5107d= c827a0ff516 > How about support for warm or live migration? This builds off the snapshot work, right? Seems like it'd make more sense to address the current limitations of the snapshot code before extending the functionality off the top of it. > There are experimental patches for all these features that were developed > by students at UPB. In a lot of cases, there are open reviews that have > been waiting on feedback for ages. > In general, most people don't want to review large experimental patches. > The case is quite plain. I'm not sure what the solution is to this > problem. I'd love to hear feedback from the community about how I've got > this completely wrong and how the course could be corrected. That would > be something. > My perspective is that it would have been better to focus student efforts on completing the snapshot feature. By completing the snapshot feature, I mean getting the code into a state where it's compiled in by default and no longer considered an experimental feature. -Rob --000000000000f8434305fa989d84 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hey Matthew,

=
On Sun, Apr 30, 2023 at 1:41=E2=80=AF= PM Matthew Grooms <mgrooms@shrew.ne= t> wrote:

Would you like to see support for VM snapshots in the generic kernel?

Is there a review open that addresses the li= mitations described in the commit message that brought the snapshot feature= in? https://github.com/freebsd/freebsd-src/commit/= 483d953a86a2507355f8287c5107dc827a0ff516
=C2=A0
How about support for warm or live migration?

This builds off the snapshot work, right? Seems like it'd make more = sense to address the current limitations of the snapshot code before extend= ing the functionality off the top of it.
=C2=A0
There are experimental patches fo= r all these features that were developed by students at UPB. In a lot of ca= ses, there are open reviews that have been waiting on feedback for ages.

In general, most people don't want to= review large experimental patches.
=C2=A0
The case is quite plain. I'm not sure what the solution is to this
problem. I'd love to hear feedback from the community about how I'v= e got
this completely wrong and how the course could be corrected. That would be something.

My perspective is that it= would have been better to focus student efforts on completing the snapshot= feature. By completing the snapshot feature, I mean getting the code into = a state where it's compiled in by default and no longer considered an e= xperimental feature.

-Rob
--000000000000f8434305fa989d84-- From nobody Tue May 2 06:16:15 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9VG52Z9Hz48Xr7; Tue, 2 May 2023 06:16:29 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [38.97.5.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9VG36Nx6z4R1T; Tue, 2 May 2023 06:16:27 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 38.97.5.131 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx1.shrew.net (8.15.2/8.15.2) with ESMTP id 3426GKTM090463; Tue, 2 May 2023 01:16:20 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.32] (unknown [136.49.230.220]) by mail.shrew.net (Postfix) with ESMTPSA id A1D37193926; Tue, 2 May 2023 01:16:15 -0500 (CDT) Content-Type: multipart/alternative; boundary="------------PosPzAVxthavBLdS0gbQcyBE" Message-ID: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> Date: Tue, 2 May 2023 01:16:15 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: BHYVE_SNAPSHOT Content-Language: en-US To: Rob Wing Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com References: From: Matthew Grooms In-Reply-To: X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.shrew.net [10.24.10.10]); Tue, 02 May 2023 01:16:20 -0500 (CDT) X-Spamd-Result: default: False [-0.57 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-0.98)[-0.984]; NEURAL_SPAM_SHORT(0.66)[0.663]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-hackers@freebsd.org]; ASN(0.00)[asn:174, ipnet:38.0.0.0/8, country:US]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[shrew.net]; TAGGED_RCPT(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Q9VG36Nx6z4R1T X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------PosPzAVxthavBLdS0gbQcyBE Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 4/30/23 21:31, Rob Wing wrote: > Hey Matthew, > > On Sun, Apr 30, 2023 at 1:41 PM Matthew Grooms wrote: > > > Would you like to see support for VM snapshots in the generic kernel? > > > Is there a review open that addresses the limitations described in the > commit message that brought the snapshot feature in? > https://github.com/freebsd/freebsd-src/commit/483d953a86a2507355f8287c5107dc827a0ff516 Yes. The next set of project goals where not pulled out of thin air. They were selected specifically to address the limitations that prevented snapshots from being in the mainline kernel after the initial commit. That's why patches for AMD CPU, Multiple Devices ( >1 of the same type ), Capsicum and JSON file format for snapshots were developed. They were identified as the major per-requisite for lifting conditional compilation. > How about support for warm or live migration? > > > This builds off the snapshot work, right? Seems like it'd make more > sense to address the current limitations of the snapshot code before > extending the functionality off the top of it. > Yup. See above. I appreciate your input, but the goal of live migration was set in 2016 with a prototype first demonstrated in 2018. How long do you suggest a developer wait without review feedback before moving forward out of tree? > There are experimental patches for all these features that were > developed by students at UPB. In a lot of cases, there are open > reviews that have been waiting on feedback for ages. > > > In general, most people don't want to review large experimental patches. Yup. That approach was attempted with the Warm Migration patches. From slide 17 in Elena's presentation: First review opened in 2021: https://reviews.freebsd.org/D28270 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 (same feature split in multiple parts) A similar request was made recently to Gusev Vitaliy WRT the multiple device support patch which he took ownership of. Thanks for adding feedback to that review BTW. We'll see how that pans out ... https://reviews.freebsd.org/D35590 > > The case is quite plain. I'm not sure what the solution is to this > problem. I'd love to hear feedback from the community about how > I've got > this completely wrong and how the course could be corrected. That > would > be something. > > > My perspective is that it would have been better to focus student > efforts on completing the snapshot feature. By completing the snapshot > feature, I mean getting the code into a state where it's compiled in > by default and no longer considered an experimental feature. > I'm not sure what more to say hear regarding the snapshot feature or what might have been done in the past. We need a solution for the present. If you have any comments related to the follow up reviews submitted by UPB, I'm sure they'd love to hear them. And lastly: I get that FreeBSD is a non paid volunteer project for most. Without the efforts of folks like Peter, Neel, John and others, there would be no bhyve. I'm not saying that they, as project maintainers, should somehow be doing more. We all have limited time to invest, paid work to do and families to feed. I'm asking if there are other developers that might be willing and able to help with reviews? Is there something the FreeBSD foundation can do help out in situations like these? Thanks, -Matthew --------------PosPzAVxthavBLdS0gbQcyBE Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 4/30/23 21:31, Rob Wing wrote:
Hey Matthew,

On Sun, Apr 30, 2023 at 1:41 PM Matthew Grooms <mgrooms@shrew.net> wrote:

Would you like to see support for VM snapshots in the generic kernel?

Is there a review open that addresses the limitations described in the commit message that brought the snapshot feature in? https://github.com/freebsd/freebsd-src/commit/483d953a86a2507355f8287c5107dc827a0ff516

Yes. The next set of project goals where not pulled out of thin air. They were selected specifically to address the limitations that prevented snapshots from being in the mainline kernel after the initial commit. That's why patches for AMD CPU, Multiple Devices ( >1 of the same type ), Capsicum and JSON file format for snapshots were developed. They were identified as the major per-requisite for lifting conditional compilation.

 
How about support for warm or live migration?

This builds off the snapshot work, right? Seems like it'd make more sense to address the current limitations of the snapshot code before extending the functionality off the top of it.

Yup. See above. I appreciate your input, but the goal of live migration was set in 2016 with a prototype first demonstrated in 2018. How long do you suggest a developer wait without review feedback before moving forward out of tree?

 
There are experimental patches for all these features that were developed by students at UPB. In a lot of cases, there are open reviews that have been waiting on feedback for ages.

In general, most people don't want to review large experimental patches.

Yup. That approach was attempted with the Warm Migration patches. From slide 17 in Elena's presentation:

First review opened in 2021: https://reviews.freebsd.org/D28270
5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 (same feature split in multiple parts)

A similar request was made recently to Gusev Vitaliy WRT the multiple device support patch which he took ownership of. Thanks for adding feedback to that review BTW. We'll see how that pans out ...

https://reviews.freebsd.org/D35590

 
The case is quite plain. I'm not sure what the solution is to this
problem. I'd love to hear feedback from the community about how I've got
this completely wrong and how the course could be corrected. That would
be something.

My perspective is that it would have been better to focus student efforts on completing the snapshot feature. By completing the snapshot feature, I mean getting the code into a state where it's compiled in by default and no longer considered an experimental feature.

I'm not sure what more to say hear regarding the snapshot feature or what might have been done in the past. We need a solution for the present. If you have any comments related to the follow up reviews submitted by UPB, I'm sure they'd love to hear them.

And lastly: I get that FreeBSD is a non paid volunteer project for most. Without the efforts of folks like Peter, Neel, John and others, there would be no bhyve. I'm not saying that they, as project maintainers, should somehow be doing more. We all have limited time to invest, paid work to do and families to feed. I'm asking if there are other developers that might be willing and able to help with reviews? Is there something the FreeBSD foundation can do help out in situations like these?

Thanks,

-Matthew

--------------PosPzAVxthavBLdS0gbQcyBE-- From nobody Tue May 2 07:57:21 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9XVX63k9z48fmX; Tue, 2 May 2023 07:57:24 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9XVX5WlQz4d0Z; Tue, 2 May 2023 07:57:24 +0000 (UTC) (envelope-from corvink@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683014244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=K5i1ZpYw/NE7Tzd2JNdf7HyERA7mNMxRRE0rHYb1CKk=; b=jm/va3Ebu81jahPKXRJelJemSO9EMQtUWvWs8KSN3hyV92BUpa1C9yhwnPo9VDbbBoRsL9 yVaJbGc4odf5B/BuhsIpUrlt5glMGStrkYGYbnqlB5EpoXcKnl7Pa8aNKzUJEiIqkp/JhT VTMOlIfX0swd0uA8QL9ygt/ROW7GUbgynzDM/vA51j+arEe0lNHmr4+yVPGma687UJKmPA 6NICc3lZEJYPQ38G1Q01zvKmonGqBS2hFVxwxqcQyyvPp8ycSQj13MyBjmur8F1MdemOqA ++HyImJWV7/WzpnWbJFcGIC8qm9T6SPcXcfxQHvw4MAwyr1WowNw43Wo/U5Oug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683014244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=K5i1ZpYw/NE7Tzd2JNdf7HyERA7mNMxRRE0rHYb1CKk=; b=fthOHJka70ZF3P4VIB23Bp+nMm7ybcrBoQ031r6JfAKqoYeBPIMVWDAOgwJtjFe6vX+uVN tocaO6zE4s9gTrhL0K2JZZf/abEPu1S0PJGhg6oPrwSSq442VD4yZ1OYmF9jF9CDkIbDM3 Ddlh6fOFOVEhg1lOACl73pEyPvqWySVBVMmlqmr6HuLWnYDkXXIOB4i8iUoDpZXxBh0zDi 99MLJBDnlnyznOH1KiHOrzqK2QSpbz5aqCphJkUH1lCuFEpTU6ibgC113LghnzRgDHn1PX wYLNtCJyp5ngJsBmY33LQmutbCSI31rMgSj+2mpXwCeYYFku+ODfZMI1fU0bNQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1683014244; a=rsa-sha256; cv=none; b=LbXd15SyeyhO0Dexxk8PyLJDQ4EqD1PWRjytpNAGWi0Ay3MQhfdfsjQ+RQcNZa5nuWBo8Z 60IRsY08k6xsi35tS3s6KgRw+JjyKTs56g8WqsGDsBEnk1vxVKK0v/PRtEfEr+WkZbR4s3 moRQXQ8KKafhVSByJEbkoeVd8LDBCcLO9kuxqSzET2c5zrCAWpDrSyTyP0n0t6t9YCMH9e cRHnTFdurZdIb/bx/hyxzBuZGj/jHlbag8MSw1ZRh22y2+vJkXL7x254dyU5LvUHwsftMM Ip6G7WBaZbiNlXlf3K08IoBVvDeTViKmt7Mo6c0T0SxJV9jkx1Wx8UtxWtVf/g== Received: from [IPv6:2001:9e8:da52:5e00:4d86:a2e1:ac9d:59c8] (unknown [IPv6:2001:9e8:da52:5e00:4d86:a2e1:ac9d:59c8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: corvink) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Q9XVW5l7Cz16bc; Tue, 2 May 2023 07:57:23 +0000 (UTC) (envelope-from corvink@FreeBSD.org) Message-ID: <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> Subject: Re: BHYVE_SNAPSHOT From: Corvin =?ISO-8859-1?Q?K=F6hne?= To: Matthew Grooms , Rob Wing Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Date: Tue, 02 May 2023 09:57:21 +0200 In-Reply-To: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> Content-Type: multipart/signed; micalg="pgp-sha256"; protocol="application/pgp-signature"; boundary="=-1ejwBLksvRG43eWs0Fu8" User-Agent: Evolution 3.46.4 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N --=-1ejwBLksvRG43eWs0Fu8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, 2023-05-02 at 01:16 -0500, Matthew Grooms wrote: > On 4/30/23 21:31, Rob Wing wrote: > > =C2=A0Hey Matthew, > >=20 > > On Sun, Apr 30, 2023 at 1:41=E2=80=AFPM Matthew Grooms > > wrote: > > >=20 > > > =C2=A0Would you like to see support for VM snapshots in the generic > > > kernel? > >=20 > > Is there a review open that addresses the limitations described in > > the commit message that brought the snapshot feature in? > > https://github.com/freebsd/freebsd-src/commit/483d953a86a2507355f8287c5= 107dc827a0ff516 >=20 > Yes. The next set of project goals where not pulled out of thin air. > They were selected specifically to address the limitations that > prevented snapshots from being in the mainline kernel after the > initial commit. That's why patches for AMD CPU, Multiple Devices ( >1 > of the same type ), Capsicum and JSON file format for snapshots were > developed. They were identified as the major per-requisite for > lifting conditional compilation. > > >=20 > > > =C2=A0How about support for warm or live migration? > >=20 > > This builds off the snapshot work, right? Seems like it'd make more > > sense to address the current limitations of the snapshot code > > before extending the functionality off the top of it. >=20 > Yup. See above. I appreciate your input, but the goal of live > migration was set in 2016 with a prototype first demonstrated in > 2018. How long do you suggest a developer wait without review > feedback before moving forward out of tree? The snapshot feature isn't compiled in by default. So, it's likely that changes break it and only a few people are testing it. We have to focus on getting this into the tree. > > > There are experimental patches for all these features that were > > > developed by students at UPB. In a lot of cases, there are open > > > reviews that have been waiting on feedback for ages. > >=20 > > In general, most people don't want to review large experimental > > patches. >=20 > Yup. That approach was attempted with the Warm Migration patches. > From slide 17 in Elena's presentation: >=20 > =C2=A0First review opened in 2021: https://reviews.freebsd.org/D28270 > =C2=A05 reviews from 2022 starting with https://reviews.freebsd.org/D3471= 7 > (same feature split in multiple parts) > =C2=A0 > =C2=A0A similar request was made recently to Gusev Vitaliy WRT the > multiple device support patch which he took ownership of. Thanks for > adding feedback to that review BTW. We'll see how that pans out ... >=20 > =C2=A0https://reviews.freebsd.org/D35590 >=20 I've already reviewed Vitaliy's multi device support patch=C2=A0and people had more than enough time to complain about it. I'm going to commit it as soon as he splits his commit. =C2=A0=C2=A0 > > > =C2=A0The case is quite plain. I'm not sure what the solution is to > > > this=20 > > > =C2=A0problem. I'd love to hear feedback from the community about how > > > I've got=20 > > > =C2=A0this completely wrong and how the course could be corrected. > > > That would=20 > > > =C2=A0be something. > > >=20 > >=20 > > My perspective is that it would have been better to focus student > > efforts on completing the snapshot feature. By completing the > > snapshot feature, I mean getting the code into a state where it's > > compiled in by default and no longer considered an experimental > > feature. > >=20 > I'm not sure what more to say hear regarding the snapshot feature or > what might have been done in the past. We need a solution for the > present. If you have any comments related to the follow up reviews > submitted by UPB, I'm sure they'd love to hear them. > And lastly: I get that FreeBSD is a non paid volunteer project for > most. Without the efforts of folks like Peter, Neel, John and others, > there would be no bhyve. I'm not saying that they, as project > maintainers, should somehow be doing more. We all have limited time > to invest, paid work to do and families to feed. I'm asking if there > are other developers that might be willing and able to help with > reviews? Is there something the FreeBSD foundation can do help out in > situations like these? > Thanks, > -Matthew > =C2=A0 UPB has developed some interesting features and I'd like to see those in tree. I can take some time to review the patches. Nevertheless, we really need the snapshot feature compiled in by default. Otherwise, it's wasted time for all of us. --=20 Kind regards, Corvin --=-1ejwBLksvRG43eWs0Fu8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEgvRSla3m2t/H2U9G2FTaVjFeAmoFAmRQwmEACgkQ2FTaVjFe Amq4vxAAumSDeuzxwoIng8rnHPIuaUYlTyQveawd+FEGs32aulOJvBs3hgs0KLNm t8C7uj04JjENwj4USNOVYFh87pfk9L5Qve0pmDbkJHRqI/3nNWj8WGhs3Bk4yWtH /b5yXSsrM8PBXBowKX+nYGOj7LJxayLUof/glSNJgNbCTy0/7v9qtCDLc2BMTfXU BZNVo405iJdaTvktDFTrugApWvpUb8hSETaDknZa8pV6UXmerAcR3VVb9Xfe75fS UWuamynh0s5IEJGA5d6Da30aI16Bv9cTvqNJuC05MlWJ/7Oy/G3YMMjuNW4zK+9U 9ePpgu49ZEl2AZlmGpI8O8zoZMw58Cr3YRVgj1cyEBi0Zv97nbvhCZW9ujvmQ3lQ Ub7GcIxQ6EHYKYik9DXruedH0HPwZB5OPHPRCPYM4YPL/47ayyjUcHkleSCMnc2n nA3Zga+0yr3H0qNHcGs3TzWMhUj4x7wrYf4bOi0y6SlHsFkCvWd1jJybRqbGpPR4 9rY9Dpjp8CrVfVloAe8tLp+Npwxs7ddU/EAplgv1NdaE6yzEXDBYoZ6xAjMynNkr G63mZ/+iq0bxwWtdR2gznJYy6PPCkb5o13tGnIoW6236sMA2iHHQkMrskzq8k75M yGq+0lsSWNt13+9pXsX75dhpMZGs5D4SA/lnTNAAdsFliFdVHMw= =dU7I -----END PGP SIGNATURE----- --=-1ejwBLksvRG43eWs0Fu8-- From nobody Tue May 2 11:00:31 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9cZ52Sn7z48r55; Tue, 2 May 2023 11:00:45 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9cZ51Mrhz3kK9; Tue, 2 May 2023 11:00:45 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x129.google.com with SMTP id 2adb3069b0e04-4f00d3f98deso27498614e87.0; Tue, 02 May 2023 04:00:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683025243; x=1685617243; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=Kedj9pEFkDCLD2y7qQ+eu6dhg5vW4Tr+qQw81KkXqec=; b=SVF0KGKMy9QYiqQWUm1oj2V8JOQOuYNIHKnADEqibjxn4r/qHZfxTSP1pCcmqH47yC fU8VW5T5KCOyy5Kb5SardrWBCL0wJF4CzAN/74Pll89Z51UVxIYpI9TLwr+9lEJMre6y SFgsIsp/nzFRIRnRfQnkHyA4ZQkKKUxGx/pqGKXt4SgY97aLvXA7xzQ3gd0JVUwwFgrt PpOH8vUPHRUsb0gczUfP5PYGc000zTti1K6Fp3q844j25NdB9zrT2v3Hss473xnUy0Ak 9Hs0uc2E0RlPvGaD/jmhBihshXNIEE4llP3Kp1D6G7Op+0ZWqnEpYUp87Z1zbM1T4AI5 FWpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683025243; x=1685617243; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Kedj9pEFkDCLD2y7qQ+eu6dhg5vW4Tr+qQw81KkXqec=; b=LJos1MTYNH/XgImyZUXHOe0OyKEhozR11M5M2Pncz6rwxiz4loI7RflUD4d6P2dnje Sl8g4T3WkjsQy6te0thU6qFCHhi6RnKiltVYZcdL3TLndE0CzsAVJcT0l97B7D0u/vpO COdls5zTvi1z6iOCVe3fhZ16GiZ0AAzbBo7OXMLEHcwWYfYqiU3XhtBg0sKk47zLNAxn hsEgLyE8twBxdL3gOCmVBrgezuNJyhNyqHgwudJQ2muFoUHezeBBCF9/AxL+z8NdjZXh Zzo8q2cCHZ1IQfla1tfSGK8qbfB4uSwUhnDhJDZaJ56sJOp1PGyoMCHUY71E9CrfMNrm 1wOw== X-Gm-Message-State: AC+VfDzlZcbOIntSDSFF/CF1KECLTGT2V8vRCHzG48tjvMc6KWEI87cq An0pHtOPvd3Q9AAwqt6NBKonJu0WoOKXUw== X-Google-Smtp-Source: ACHHUZ77Dtxg7EjDYsHu1GG7saceXpD5njMlp+bVj4xmEGz/FVNqjVBj7rYPIeFFRD7mebMqQm3CTQ== X-Received: by 2002:a05:6512:904:b0:4ef:ef9f:f24f with SMTP id e4-20020a056512090400b004efef9ff24fmr4425476lft.13.1683025243070; Tue, 02 May 2023 04:00:43 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id q3-20020ac25143000000b004ec8ca234cfsm5308457lfd.122.2023.05.02.04.00.42 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 May 2023 04:00:42 -0700 (PDT) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE_SNAPSHOT Date: Tue, 2 May 2023 14:00:31 +0300 In-Reply-To: <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> Cc: Matthew Grooms , Rob Wing , freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, Elena Mihailescu , Mihai Carabas To: =?utf-8?Q?Corvin_K=C3=B6hne?= References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4Q9cZ51Mrhz3kK9 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Just add some plans for me: 1. Describe snapshot file format: One file for snapshot. 2. Implement snapshot/resume via nvlist. nvlist implementation brings: Versioning Easy debugging, getting saved values, etc. Validate restored variables: types, sized, etc. Add optional variables without breaking backward compatibility (resume = can be performed with old snapshots) Remove variables without breaking backward compatibility Use one file for snapshot Improve restore command line: "bhyve -r $snapshot=E2=80=9D, i.e. w/o = additional arguments =E2=80=94=E2=80=94 Vitaliy Gusev >>> before extending the functionality off the top of it. >>=20 >> Yup. See above. I appreciate your input, but the goal of live >> migration was set in 2016 with a prototype first demonstrated in >> 2018. How long do you suggest a developer wait without review >> feedback before moving forward out of tree? >=20 > The snapshot feature isn't compiled in by default. So, it's likely = that > changes break it and only a few people are testing it. >=20 > We have to focus on getting this into the tree. >=20 >>>> There are experimental patches for all these features that were >>>> developed by students at UPB. In a lot of cases, there are open >>>> reviews that have been waiting on feedback for ages. >>>=20 >>> In general, most people don't want to review large experimental >>> patches. >>=20 >> Yup. That approach was attempted with the Warm Migration patches. >> =46rom slide 17 in Elena's presentation: >>=20 >> First review opened in 2021: https://reviews.freebsd.org/D28270 >> 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 >> (same feature split in multiple parts) >> =20 >> A similar request was made recently to Gusev Vitaliy WRT the >> multiple device support patch which he took ownership of. Thanks for >> adding feedback to that review BTW. We'll see how that pans out ... >>=20 >> https://reviews.freebsd.org/D35590 >>=20 >=20 > I've already reviewed Vitaliy's multi device support patch and people > had more than enough time to complain about it. I'm going to commit it > as soon as he splits his commit. > =20 >>>> The case is quite plain. I'm not sure what the solution is to >>>> this=20 >>>> problem. I'd love to hear feedback from the community about how >>>> I've got=20 >>>> this completely wrong and how the course could be corrected. >>>> That would=20 >>>> be something. >>>>=20 >>>=20 >>> My perspective is that it would have been better to focus student >>> efforts on completing the snapshot feature. By completing the >>> snapshot feature, I mean getting the code into a state where it's >>> compiled in by default and no longer considered an experimental >>> feature. >>>=20 >> I'm not sure what more to say hear regarding the snapshot feature or >> what might have been done in the past. We need a solution for the >> present. If you have any comments related to the follow up reviews >> submitted by UPB, I'm sure they'd love to hear them. >> And lastly: I get that FreeBSD is a non paid volunteer project for >> most. Without the efforts of folks like Peter, Neel, John and others, >> there would be no bhyve. I'm not saying that they, as project >> maintainers, should somehow be doing more. We all have limited time >> to invest, paid work to do and families to feed. I'm asking if there >> are other developers that might be willing and able to help with >> reviews? Is there something the FreeBSD foundation can do help out in >> situations like these? >> Thanks, >> -Matthew >> =20 >=20 > UPB has developed some interesting features and I'd like to see those > in tree. I can take some time to review the patches. Nevertheless, we > really need the snapshot feature compiled in by default. Otherwise, > it's wasted time for all of us. >=20 >=20 > --=20 > Kind regards, > Corvin --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Just add some = plans for me:

 1. Describe snapshot file format: = One file for snapshot.

 2. Implement = snapshot/resume via nvlist.

  =  nvlist implementation brings:

  • Versioning
  • Easy debugging, getting = saved values, etc.
  • Validate restored variables: types, sized, = etc.
  • Add optional variables without breaking backward = compatibility (resume can be performed with old = snapshots)
  • Remove variables without breaking backward = compatibility
  • Use one file for snapshot
  • Improve restore = command line:  "bhyve -r $snapshot=E2=80=9D, i.e. w/o additional = arguments

=E2=80=94=E2=80=94
= Vitaliy Gusev

before extending the functionality off = the top of it.

Yup. See above. I appreciate your = input, but the goal of live
migration was set in 2016 with a = prototype first demonstrated in
2018. How long do you suggest a = developer wait without review
feedback before moving forward out of = tree?

The = snapshot feature isn't compiled in by default. So, it's likely = that
changes break it and = only a few people are testing it.

We have = to focus on getting this into the tree.

There are experimental patches for all these features that = were
developed by students at UPB. In a lot of cases, there are = open
reviews that have been waiting on feedback for = ages.

In general, most people don't want to review = large experimental
patches.

Yup. That approach = was attempted with the Warm Migration patches.
=46rom slide 17 in = Elena's presentation:

 First review opened in 2021: = https://reviews.freebsd.org/D28270
 5 reviews from 2022 starting = with https://reviews.freebsd.org/D34717
(same feature split in = multiple parts)
 
 A similar request was made recently = to Gusev Vitaliy WRT the
multiple device support patch which he took = ownership of. Thanks for
adding feedback to that review BTW. We'll = see how that pans out = ...

 https://reviews.freebsd.org/D35590

I've already reviewed Vitaliy's multi = device support patch and people
had = more than enough time to complain about it. I'm going to commit = it
as soon as he splits his = commit.
  
 The = case is quite plain. I'm not sure what the solution is to
this 
 problem. I'd love = to hear feedback from the community about how
I've got 
 this completely = wrong and how the course could be corrected.
That would 
 be = something.


My perspective is that it would have = been better to focus student
efforts on completing the snapshot = feature. By completing the
snapshot feature, I mean getting the code = into a state where it's
compiled in by default and no longer = considered an experimental
feature.

I'm not sure = what more to say hear regarding the snapshot feature or
what might = have been done in the past. We need a solution for the
present. If = you have any comments related to the follow up reviews
submitted by = UPB, I'm sure they'd love to hear them.
And lastly: I get that = FreeBSD is a non paid volunteer project for
most. Without the efforts = of folks like Peter, Neel, John and others,
there would be no bhyve. = I'm not saying that they, as project
maintainers, should somehow be = doing more. We all have limited time
to invest, paid work to do and = families to feed. I'm asking if there
are other developers that might = be willing and able to help with
reviews? Is there something the = FreeBSD foundation can do help out in
situations like = these?
Thanks,
-Matthew
 

UPB has developed some interesting features = and I'd like to see those
in = tree. I can take some time to review the patches. Nevertheless, = we
really need the snapshot = feature compiled in by default. Otherwise,
it's = wasted time for all of us.


-- 
Kind regards,
Corvin

= = --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D-- From nobody Tue May 2 11:00:31 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9cZ65lysz48rCs; Tue, 2 May 2023 11:00:46 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9cZ51gWMz3kDK; Tue, 2 May 2023 11:00:45 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-lf1-x12a.google.com with SMTP id 2adb3069b0e04-4f00d41df22so27491739e87.1; Tue, 02 May 2023 04:00:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683025243; x=1685617243; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=Kedj9pEFkDCLD2y7qQ+eu6dhg5vW4Tr+qQw81KkXqec=; b=SVF0KGKMy9QYiqQWUm1oj2V8JOQOuYNIHKnADEqibjxn4r/qHZfxTSP1pCcmqH47yC fU8VW5T5KCOyy5Kb5SardrWBCL0wJF4CzAN/74Pll89Z51UVxIYpI9TLwr+9lEJMre6y SFgsIsp/nzFRIRnRfQnkHyA4ZQkKKUxGx/pqGKXt4SgY97aLvXA7xzQ3gd0JVUwwFgrt PpOH8vUPHRUsb0gczUfP5PYGc000zTti1K6Fp3q844j25NdB9zrT2v3Hss473xnUy0Ak 9Hs0uc2E0RlPvGaD/jmhBihshXNIEE4llP3Kp1D6G7Op+0ZWqnEpYUp87Z1zbM1T4AI5 FWpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683025243; x=1685617243; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Kedj9pEFkDCLD2y7qQ+eu6dhg5vW4Tr+qQw81KkXqec=; b=FR0WRM8/jEImOpkevmvuhPc7x1Jwk6ZhpNln3AQY7nPp1sXkmit1+h6ZUjONt94ktq wCxaUiMUL8JrDaaPtzEdcVAsI3wjgZ9MydRjq3crh8pnHXbG7Nbl5JqcjwNaYs9sUMnl Rxr78LgG4zNl7u+Yhr7Ma7BZOdy3ZRPczdaJSR6t3Mrv9pguq1v50z/GDPC801ZxPF9l rKSjHkRYYIEhjMT97jWbhGBazTySOhXWFoQ/l885WMoAr/ubnlnMAjzY4vYvirGOCcge Xb12WosYmcpBI5ZnH0fgJ/TtY43WL7cPuZlevszotfD0GqBPKS3Hx5YXmPemMIOl9CXx FkOw== X-Gm-Message-State: AC+VfDwjA0byoTAabFas82PbtWHE5Aa9RWv4LqMfY0qiDtWGdbmdWbXZ LP4SydBoivubNfYKxEuU75sJt/PFKCyoJQ== X-Google-Smtp-Source: ACHHUZ5aRGdwhHaiaoV06D14L0shtXjRpbL3GkvbDJnm0KNZgFbF+E9NP/Pt94QwBi7EH8DBb0XkZA== X-Received: by 2002:a05:651c:231b:b0:2a8:bc78:2e3f with SMTP id bi27-20020a05651c231b00b002a8bc782e3fmr6329301ljb.0.1683025242982; Tue, 02 May 2023 04:00:42 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id y14-20020a2e320e000000b00295da33c42dsm5242725ljy.15.2023.05.02.04.00.41 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 May 2023 04:00:42 -0700 (PDT) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE_SNAPSHOT Date: Tue, 2 May 2023 14:00:31 +0300 In-Reply-To: <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> Cc: Matthew Grooms , Rob Wing , freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, Elena Mihailescu , Mihai Carabas To: =?utf-8?Q?Corvin_K=C3=B6hne?= References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4Q9cZ51gWMz3kDK X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Just add some plans for me: 1. Describe snapshot file format: One file for snapshot. 2. Implement snapshot/resume via nvlist. nvlist implementation brings: Versioning Easy debugging, getting saved values, etc. Validate restored variables: types, sized, etc. Add optional variables without breaking backward compatibility (resume = can be performed with old snapshots) Remove variables without breaking backward compatibility Use one file for snapshot Improve restore command line: "bhyve -r $snapshot=E2=80=9D, i.e. w/o = additional arguments =E2=80=94=E2=80=94 Vitaliy Gusev >>> before extending the functionality off the top of it. >>=20 >> Yup. See above. I appreciate your input, but the goal of live >> migration was set in 2016 with a prototype first demonstrated in >> 2018. How long do you suggest a developer wait without review >> feedback before moving forward out of tree? >=20 > The snapshot feature isn't compiled in by default. So, it's likely = that > changes break it and only a few people are testing it. >=20 > We have to focus on getting this into the tree. >=20 >>>> There are experimental patches for all these features that were >>>> developed by students at UPB. In a lot of cases, there are open >>>> reviews that have been waiting on feedback for ages. >>>=20 >>> In general, most people don't want to review large experimental >>> patches. >>=20 >> Yup. That approach was attempted with the Warm Migration patches. >> =46rom slide 17 in Elena's presentation: >>=20 >> First review opened in 2021: https://reviews.freebsd.org/D28270 >> 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 >> (same feature split in multiple parts) >> =20 >> A similar request was made recently to Gusev Vitaliy WRT the >> multiple device support patch which he took ownership of. Thanks for >> adding feedback to that review BTW. We'll see how that pans out ... >>=20 >> https://reviews.freebsd.org/D35590 >>=20 >=20 > I've already reviewed Vitaliy's multi device support patch and people > had more than enough time to complain about it. I'm going to commit it > as soon as he splits his commit. > =20 >>>> The case is quite plain. I'm not sure what the solution is to >>>> this=20 >>>> problem. I'd love to hear feedback from the community about how >>>> I've got=20 >>>> this completely wrong and how the course could be corrected. >>>> That would=20 >>>> be something. >>>>=20 >>>=20 >>> My perspective is that it would have been better to focus student >>> efforts on completing the snapshot feature. By completing the >>> snapshot feature, I mean getting the code into a state where it's >>> compiled in by default and no longer considered an experimental >>> feature. >>>=20 >> I'm not sure what more to say hear regarding the snapshot feature or >> what might have been done in the past. We need a solution for the >> present. If you have any comments related to the follow up reviews >> submitted by UPB, I'm sure they'd love to hear them. >> And lastly: I get that FreeBSD is a non paid volunteer project for >> most. Without the efforts of folks like Peter, Neel, John and others, >> there would be no bhyve. I'm not saying that they, as project >> maintainers, should somehow be doing more. We all have limited time >> to invest, paid work to do and families to feed. I'm asking if there >> are other developers that might be willing and able to help with >> reviews? Is there something the FreeBSD foundation can do help out in >> situations like these? >> Thanks, >> -Matthew >> =20 >=20 > UPB has developed some interesting features and I'd like to see those > in tree. I can take some time to review the patches. Nevertheless, we > really need the snapshot feature compiled in by default. Otherwise, > it's wasted time for all of us. >=20 >=20 > --=20 > Kind regards, > Corvin --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Just add some = plans for me:

 1. Describe snapshot file format: = One file for snapshot.

 2. Implement = snapshot/resume via nvlist.

  =  nvlist implementation brings:

  • Versioning
  • Easy debugging, getting = saved values, etc.
  • Validate restored variables: types, sized, = etc.
  • Add optional variables without breaking backward = compatibility (resume can be performed with old = snapshots)
  • Remove variables without breaking backward = compatibility
  • Use one file for snapshot
  • Improve restore = command line:  "bhyve -r $snapshot=E2=80=9D, i.e. w/o additional = arguments

=E2=80=94=E2=80=94
= Vitaliy Gusev

before extending the functionality off = the top of it.

Yup. See above. I appreciate your = input, but the goal of live
migration was set in 2016 with a = prototype first demonstrated in
2018. How long do you suggest a = developer wait without review
feedback before moving forward out of = tree?

The = snapshot feature isn't compiled in by default. So, it's likely = that
changes break it and = only a few people are testing it.

We have = to focus on getting this into the tree.

There are experimental patches for all these features that = were
developed by students at UPB. In a lot of cases, there are = open
reviews that have been waiting on feedback for = ages.

In general, most people don't want to review = large experimental
patches.

Yup. That approach = was attempted with the Warm Migration patches.
=46rom slide 17 in = Elena's presentation:

 First review opened in 2021: = https://reviews.freebsd.org/D28270
 5 reviews from 2022 starting = with https://reviews.freebsd.org/D34717
(same feature split in = multiple parts)
 
 A similar request was made recently = to Gusev Vitaliy WRT the
multiple device support patch which he took = ownership of. Thanks for
adding feedback to that review BTW. We'll = see how that pans out = ...

 https://reviews.freebsd.org/D35590

I've already reviewed Vitaliy's multi = device support patch and people
had = more than enough time to complain about it. I'm going to commit = it
as soon as he splits his = commit.
  
 The = case is quite plain. I'm not sure what the solution is to
this 
 problem. I'd love = to hear feedback from the community about how
I've got 
 this completely = wrong and how the course could be corrected.
That would 
 be = something.


My perspective is that it would have = been better to focus student
efforts on completing the snapshot = feature. By completing the
snapshot feature, I mean getting the code = into a state where it's
compiled in by default and no longer = considered an experimental
feature.

I'm not sure = what more to say hear regarding the snapshot feature or
what might = have been done in the past. We need a solution for the
present. If = you have any comments related to the follow up reviews
submitted by = UPB, I'm sure they'd love to hear them.
And lastly: I get that = FreeBSD is a non paid volunteer project for
most. Without the efforts = of folks like Peter, Neel, John and others,
there would be no bhyve. = I'm not saying that they, as project
maintainers, should somehow be = doing more. We all have limited time
to invest, paid work to do and = families to feed. I'm asking if there
are other developers that might = be willing and able to help with
reviews? Is there something the = FreeBSD foundation can do help out in
situations like = these?
Thanks,
-Matthew
 

UPB has developed some interesting features = and I'd like to see those
in = tree. I can take some time to review the patches. Nevertheless, = we
really need the snapshot = feature compiled in by default. Otherwise,
it's = wasted time for all of us.


-- 
Kind regards,
Corvin

= = --Apple-Mail=_B895AB4D-5F14-4A57-B65C-D1DB37F70C9D-- From nobody Tue May 2 15:38:15 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9kk82gNjz495rq; Tue, 2 May 2023 15:38:08 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9kk73lkfz4HFB; Tue, 2 May 2023 15:38:07 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=HgATklFY; spf=pass (mx1.freebsd.org: domain of rob.fx907@gmail.com designates 2a00:1450:4864:20::534 as permitted sender) smtp.mailfrom=rob.fx907@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x534.google.com with SMTP id 4fb4d7f45d1cf-504d149839bso826512a12.1; Tue, 02 May 2023 08:38:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683041886; x=1685633886; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=errR5P0dAWMC6MBSES4ce2BPg3tEKzLkevUy/ewpve8=; b=HgATklFY731IyqEuM6nnqsL7XrIQec9by8fB6fMto4i8CZp6B8Vh7lW3EYXu8HqkAA roc04CL0pl6KdryWIsH5gDnOGwrcFt4pApYKEGmsKOGYyEXUFYLwY0upgyCzLa8ke6+/ vbf0tZ+H41aV1shzQ5E42ltVQZspHK5izp3eRnZ9eTPHToutqSNYR2jsAcJmW6WeK9um HMemv51B6VDC5lBdJjooONx46y2wzmSB2vmwlSh+y32Tbm3M3CIccAvXMuUlf+3oG2fY 8HKbcgbDG+3Wx6PlLJXhnkFkHnGeSoq6670sXvw4NZJ2BgzjPODbCnDwxonztQJTPOHh L4Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683041886; x=1685633886; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=errR5P0dAWMC6MBSES4ce2BPg3tEKzLkevUy/ewpve8=; b=Lra+FUjfruB19cQNp7etQoWmvpAP1BWBk636ge0UqKiXE6e9sAMfdqAIKrIDdd1+q3 RpnPbfrx1zqkPGEKvjxr3/7cpb7nHnFQX5dRfoe9JBJaXc7F+H/+rFvb6IFbXWJy0hIp qCQ5y/ha6AIOPW3kLT4YLk+ZGEIS5Xobj1b1c31D2d3E+yEJutRcYD1dwhVwTcYrvgdi ELpNP5X5qb1hJ5qhOmMonU8IAgoMfrksSRhiugjIHf2lvLpmwNusrdv1cPOSAMJQ86hB z5ZLVSKJINje7Z3IYZByry8LRUShu+kKyy60WYTDXu/zZENkrnhjOUJaeaXHojUefht/ TY6Q== X-Gm-Message-State: AC+VfDwC6NjqVyHFQeg2P5iRLnhxkTXHhtZLiWPTWudLkM6MH566kkIV fF9scA/uq/onXo98464qtRDYemjnKlQkenyJr4gQOgDNgyc= X-Google-Smtp-Source: ACHHUZ7vLi5nj6s+ZRe+hwO2SBVskbyMbKh/VNjgmUpyBWYkyWpFXx21oJg896p71yaIrj+ZptldzoiTGE1VtDcWW2w= X-Received: by 2002:a17:906:7a49:b0:95f:db5f:73b7 with SMTP id i9-20020a1709067a4900b0095fdb5f73b7mr2775421ejo.0.1683041886032; Tue, 02 May 2023 08:38:06 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> In-Reply-To: From: Rob Wing Date: Tue, 2 May 2023 07:38:15 -0800 Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Vitaliy Gusev Cc: =?UTF-8?Q?Corvin_K=C3=B6hne?= , Matthew Grooms , freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, Elena Mihailescu , Mihai Carabas Content-Type: multipart/alternative; boundary="000000000000c258dc05fab7b9bf" X-Spamd-Result: default: False [0.12 / 15.00]; SUSPICIOUS_RECIPS(1.50)[]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_SPAM_SHORT(0.57)[0.568]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-virtualization@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::534:from]; ARC_NA(0.00)[]; TAGGED_FROM(0.00)[]; FREEMAIL_CC(0.00)[freebsd.org,shrew.net,gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_SEVEN(0.00)[7]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_DN_SOME(0.00)[] X-Rspamd-Queue-Id: 4Q9kk73lkfz4HFB X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N --000000000000c258dc05fab7b9bf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, May 2, 2023 at 3:00=E2=80=AFAM Vitaliy Gusev wrote: > Just add some plans for me: > > 1. Describe snapshot file format: One file for snapshot. > > 2. Implement snapshot/resume via nvlist. > > *nvlist* implementation brings: > > > - Versioning > - Easy debugging, getting saved values, etc. > - Validate restored variables: types, sized, etc. > - Add optional variables without breaking backward compatibility > (resume can be performed with old snapshots) > - Remove variables without breaking backward compatibility > - Use one file for snapshot > - Improve restore command line: "bhyve -r $snapshot=E2=80=9D, i.e. w/= o > additional arguments > > Do you plan on saving the guest memory into the nvlist? If I have VM with 8 gigs of memory, will the nvlist implementation allocate 8 gigs of memory for the nvlist then write it out to disk? Or..? All the bullet points look good to me. -Rob --000000000000c258dc05fab7b9bf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Tue, May 2, 2023 at 3:00=E2=80=AFA= M Vitaliy Gusev <gusev.vitali= y@gmail.com> wrote:
Just add some plans for me:

=C2=A01. Des= cribe snapshot file format: One file for snapshot.

=C2=A02. Implement snapshot/resume via nvlist.

= =C2=A0 =C2=A0nvlist implementation brings:

=
  • Versioning
  • Easy debugging, getting saved values, etc.
    • <= li>Validate restored variables: types, sized, etc.
  • Add optional var= iables without breaking backward compatibility (resume can be performed wit= h old snapshots)
  • Remove variables without breaking backward compati= bility
  • Use one file for snapshot
  • Improve restore command li= ne: =C2=A0"bhyve -r $snapshot=E2=80=9D, i.e. w/o additional arguments<= /li>

Do you plan on saving= the guest memory into the nvlist? If I have VM with 8 gigs of memory, will= the nvlist implementation allocate 8 gigs of memory for the nvlist then wr= ite it out to disk? Or..?

All the bullet points lo= ok good to me.

-Rob
--000000000000c258dc05fab7b9bf-- From nobody Tue May 2 16:04:02 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9lJK5rZdz496fl; Tue, 2 May 2023 16:04:17 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9lJJ6PW3z4LP8; Tue, 2 May 2023 16:04:16 +0000 (UTC) (envelope-from gusev.vitaliy@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=OcmOsu+W; spf=pass (mx1.freebsd.org: domain of gusev.vitaliy@gmail.com designates 2a00:1450:4864:20::22b as permitted sender) smtp.mailfrom=gusev.vitaliy@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-lj1-x22b.google.com with SMTP id 38308e7fff4ca-2aa39ce5d26so40035301fa.1; Tue, 02 May 2023 09:04:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683043454; x=1685635454; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=QU8tYAy4WZ9pbgxX9JzfnrW1kukUjmYcxtPRSGSst5I=; b=OcmOsu+WBcaN6sJt5MPJWkCvXdQnYfaKKZMtdOwFEuitSQivYBliqWYFt0xL7YwPNO rfs+nCnMI6lvHb8v0bGPCDpNzQigv29EWubYdbaWrPILVSKTLH24jlVZ9Tsu8AW6OrlD w2XNK445Gl+DpDHpnEZLILu9RwDNxahtL96X2VZcFV/t4z9obUfgB0let5FBx10qdnAA RMrTJKKnSAC0IJnFO8us2mdVzP62r9BwhzdaN8JcLakefeDn0VSTdOKcYEwLqpBhbLpU YDIhrA3WpmMznkSQpc8tAZ9RoZNdnP+DYGsubijHddtcbuqse6crVppJOklN8e433Kyt joPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683043454; x=1685635454; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QU8tYAy4WZ9pbgxX9JzfnrW1kukUjmYcxtPRSGSst5I=; b=DUGskCo+1Uz6EUgXlUgpV68jKjOVOZ9VJmYbnzqL3+u57aB1YzLKxGhZhFAnvtCPDT o0dPwIDmtJUcdupS0sgAISHqu5khgq6XYOSfHKo1VuNpdF/bdQ71CvFWiKVGcn8HizMO OuZaXtb7GlKCsg+Q7i2XaD/t7V6RR/xu3xfWNvKkIgkk1cOwad3cTlowfuXQbr47qc1U qitUNuaXbtGbtlMs0+nAbMlLbo8vh4pJOdi+v+HbXf7b/IRnn2kmz9fZ48ndUQ3347wi ulzLbHnpI/wTDWX9k8a9SRTvfHzktTd9UpfhWFISdLn8HbfZC3Aat6Ox4+6YJqMtnoDj ZpkA== X-Gm-Message-State: AC+VfDxLa5MRlRcNtG2cFSjX2Pc3J4LjY+R5yUYfQ7zQISEYgxfjTxVZ Y5ehXkcql1JmwUESsNKj210= X-Google-Smtp-Source: ACHHUZ6CfOOyQ4QngnjYymTo/zwri0JCT4wG0UEAwWeNTXtVmnU7rHrHeuhlbi2HNO0Xa08UZF96rw== X-Received: by 2002:a2e:98c5:0:b0:2a8:e6f8:301e with SMTP id s5-20020a2e98c5000000b002a8e6f8301emr4595615ljj.28.1683043454199; Tue, 02 May 2023 09:04:14 -0700 (PDT) Received: from smtpclient.apple ([188.187.60.230]) by smtp.gmail.com with ESMTPSA id w4-20020ac25d44000000b004eb0c51780bsm5443260lfd.29.2023.05.02.09.04.13 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 May 2023 09:04:13 -0700 (PDT) From: Vitaliy Gusev Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_07C1A3F1-22E1-41CC-8145-2FE0D588B957" List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: BHYVE_SNAPSHOT Date: Tue, 2 May 2023 19:04:02 +0300 In-Reply-To: Cc: =?utf-8?Q?Corvin_K=C3=B6hne?= , Matthew Grooms , freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, Elena Mihailescu , Mihai Carabas To: Rob Wing References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> <3ab4e6d94fb0153fb6ff4a53ac6f53b2eaae0cf7.camel@FreeBSD.org> X-Mailer: Apple Mail (2.3731.500.231) X-Spamd-Result: default: False [-2.44 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.990]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::22b:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-virtualization@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TAGGED_FROM(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_CC(0.00)[freebsd.org,shrew.net,gmail.com] X-Rspamd-Queue-Id: 4Q9lJJ6PW3z4LP8 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_07C1A3F1-22E1-41CC-8145-2FE0D588B957 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 2 May 2023, at 18:38, Rob Wing wrote: >=20 > Do you plan on saving the guest memory into the nvlist? If I have VM = with 8 gigs of memory, will the nvlist implementation allocate 8 gigs of = memory for the nvlist then write it out to disk? Or..? >=20 > All the bullet points look good to me. >=20 > -Rob Of course no. Guest memory should be saved as is. As possible = improvement - only dirty memory/pages.=20 I was taking about saving with nvlist: device=E2=80=99s variables, = registers, internal data, etc.=20 Overall description will be provided shortly. =E2=80=94=E2=80=94=E2=80=94 Vitaliy Gusev --Apple-Mail=_07C1A3F1-22E1-41CC-8145-2FE0D588B957 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On 2 May = 2023, at 18:38, Rob Wing <rob.fx907@gmail.com> = wrote:

Do you plan on saving the = guest memory into the nvlist? If I have VM with 8 gigs of memory, will = the nvlist implementation allocate 8 gigs of memory for the nvlist then = write it out to disk? Or..?

All the bullet points look good to = me.

-Rob


Of course no. Guest memory should be saved as is. As possible = improvement  -  only dirty = memory/pages. 

I was taking about saving = with nvlist: device=E2=80=99s variables, registers, internal data, = etc. 

Overall description will be provided = shortly.

=E2=80=94=E2=80=94=E2=80=94
Vi= taliy Gusev


= --Apple-Mail=_07C1A3F1-22E1-41CC-8145-2FE0D588B957-- From nobody Tue May 2 19:25:12 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9qm16pKyz49KCp; Tue, 2 May 2023 19:25:05 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9qm11hsFz44sC; Tue, 2 May 2023 19:25:05 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62b.google.com with SMTP id a640c23a62f3a-9594916df23so130493866b.1; Tue, 02 May 2023 12:25:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1683055503; x=1685647503; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=LiOIjWmt1NbUmkN9IZ2lHw/4SCRlFTaji52oJE1sx90=; b=MDpEZB2P791/Y+7EPZF+FcImymz5gvGq5hQPaXTGOvD2KrlmFnGAiYyuaVzkiwwpCc OFiwedgo3l4oA3PVAXWPirsAbJtdzwymSrN2ZgR89dNxMDp0xseHNjyRmk33G82y4TrN 5vYsCwisrI1b9j8s4Pb7z9iq0HTov8DpCoezJ9hCmQx3hsIOKiHGjOkVJTBoSFEuo79y if+ffpbW7awlNVjdEdXwR9zu1UJgHciD8SxQC1aPgFRvgv+W1v+eIWwUNPfc1zr+WsZb 5CS2bVSD/Y9v7zMXAAjWVChsV5/BMZOkzb5J9s/MCeL+FA/ll/MBcHXDzkvOCcp7VcdR Jdww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683055503; x=1685647503; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LiOIjWmt1NbUmkN9IZ2lHw/4SCRlFTaji52oJE1sx90=; b=Ux1lEikvmsMv3xX+yFR+0wxKocy/ur0taVbNk/tF1T/ozlMF6/YDp++xmJXC67znBs LpTP3yDfXNJucrmBNPugT/9SfqNtxNAeOwppHctGoY5DJ3lTgweLTJzgz6eqG2FjiLJo xajW7aBhmD1BVQuWhhKaGxZA/Ru93tVj8hoGjFPugCX/aeRBwd+iI6hHTmK2T5z88kEb MixJHCdTTshOjVoQNgxr8C6uB5zMFVhWWeFe5WPUVgZzjgjQN1r5O0hU6TJ+anDk3YZZ hUxXhodADL9ERIlyf4kvApTtnd3KJxbwlb5nKt8jT0fKiy4NnGr8jQfeXLjmovH+vGhf nOoA== X-Gm-Message-State: AC+VfDxx1RrWsA/7lMNHV7lZqNQ7FxRsC0S5IYRknLvJP/Wq+I+O9dbB og/7mBewDfqwoC85Z8GMJMzfZdKrUmQFhtKDwyEy0MJFqHo= X-Google-Smtp-Source: ACHHUZ7AEZ/Ks403S5obdFn8aj7AsOxHE/74/xdMWWVcwHjtYRzFUWGXczp/1bRnaO6yM6mrbzdsqYETFJBRyr6zYY0= X-Received: by 2002:a17:906:7a01:b0:94f:66af:b1f7 with SMTP id d1-20020a1709067a0100b0094f66afb1f7mr2938797ejo.1.1683055503301; Tue, 02 May 2023 12:25:03 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> In-Reply-To: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> From: Rob Wing Date: Tue, 2 May 2023 11:25:12 -0800 Message-ID: Subject: Re: BHYVE_SNAPSHOT To: Matthew Grooms Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com Content-Type: multipart/alternative; boundary="0000000000006964a805fabae503" X-Rspamd-Queue-Id: 4Q9qm11hsFz44sC X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_RCPT(0.00)[]; TAGGED_FROM(0.00)[] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000006964a805fabae503 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, May 1, 2023 at 10:16=E2=80=AFPM Matthew Grooms = wrote: > Yup. See above. I appreciate your input, but the goal of live migration > was set in 2016 with a prototype first demonstrated in 2018. How long do > you suggest a developer wait without review feedback before moving forwar= d > out of tree? > Yea..I'd say a submitter should receive feedback within a week (or so) and when that passes, try rattling the cage of relevant developers to ack a timeline. My comments for the live/warm migration review is to rebase it after the snapshot feature is compiled in by default. > Yup. That approach was attempted with the Warm Migration patches. From > slide 17 in Elena's presentation: > > First review opened in 2021: https://reviews.freebsd.org/D28270 > 5 reviews from 2022 starting with https://reviews.freebsd.org/D34717 > (same feature split in multiple parts) > I understand that, as a submitter, this is frustrating - to spend a non-trivial amount of time preparing/submitting a review and then receive no feedback, not even a "changes too big" or "nope". For future reference, if UPB does another project - might be worth trying to seek out a FreeBSD mentor that could guide the students and be willing to usher the changes in, not unlike a GSOC project. --0000000000006964a805fabae503 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Mon, May 1,= 2023 at 10:16=E2=80=AFPM Matthew Grooms <mgrooms@shrew.net> wrote:

= Yup. See above. I appreciate your input, but the goal of live migration was set in 2016 with a prototype first demonstrated in 2018. How long do you suggest a developer wait without review feedback before moving forward out of tree?


Yea..I'd say a submitter should receive feedback w= ithin a week (or so) and when that passes, try rattling the cage of relevan= t developers to ack a timeline. My comments for the live/warm migration rev= iew is to rebase it after the snapshot feature is compiled in by default.
=C2=A0
=

=20

Yup. That approach was attempted with the Warm Migration patches. From slide 17 in Elena's presentation:

First review opened in 2021: https://reviews.freebsd.org/D28270
5 reviews from 2022 starting with http= s://reviews.freebsd.org/D34717 (same feature split in multiple parts)


<= /div>
I understand that, as a submitter, this is frustrating - to spend= a non-trivial amount of time preparing/submitting a review and then receiv= e no feedback, not even a "changes too big" or "nope".<= /div>

For future reference, if UPB does another project = - might be worth trying to seek out a FreeBSD mentor that could guide the s= tudents and be willing to usher the changes in, not unlike a GSOC project.<= br>
--0000000000006964a805fabae503-- From nobody Tue May 2 21:16:48 2023 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q9tFB0X0Lz48BMY; Tue, 2 May 2023 21:17:02 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [38.97.5.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q9tF919MVz4N1s; Tue, 2 May 2023 21:17:00 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 38.97.5.132 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx2.shrew.net (8.15.2/8.15.2) with ESMTP id 342LGsRT089307; Tue, 2 May 2023 16:16:54 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.22.200.32] (unknown [136.49.230.220]) by mail.shrew.net (Postfix) with ESMTPSA id 1D8EA193A82; Tue, 2 May 2023 16:16:49 -0500 (CDT) Content-Type: multipart/alternative; boundary="------------YGulZ2VgxDlosOBeXKDZgfz0" Message-ID: <7d649a69-8c68-9b15-7999-2102873c1833@shrew.net> Date: Tue, 2 May 2023 16:16:48 -0500 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: BHYVE_SNAPSHOT Content-Language: en-US To: Rob Wing Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org, elenamihailescu22@gmail.com, Mihai Carabas , gusev.vitaliy@gmail.com References: <89c84bd7-a925-02f4-acbe-12c3000e7007@shrew.net> From: Matthew Grooms In-Reply-To: X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx2.shrew.net [10.24.10.11]); Tue, 02 May 2023 16:16:54 -0500 (CDT) X-Spamd-Result: default: False [-0.32 / 15.00]; SUBJ_ALL_CAPS(1.05)[14]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.44)[-0.443]; NEURAL_SPAM_MEDIUM(0.37)[0.370]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-hackers@freebsd.org]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:174, ipnet:38.0.0.0/8, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; FREEMAIL_CC(0.00)[freebsd.org,gmail.com]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[shrew.net]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4Q9tF919MVz4N1s X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --------------YGulZ2VgxDlosOBeXKDZgfz0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 5/2/23 14:25, Rob Wing wrote: > > For future reference, if UPB does another project - might be worth > trying to seek out a FreeBSD mentor that could guide the students and > be willing to usher the changes in, not unlike a GSOC project. Good idea. That would be wonderful. -Matthew --------------YGulZ2VgxDlosOBeXKDZgfz0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit
On 5/2/23 14:25, Rob Wing wrote:

For future reference, if UPB does another project - might be worth trying to seek out a FreeBSD mentor that could guide the students and be willing to usher the changes in, not unlike a GSOC project.

Good idea. That would be wonderful.

-Matthew

--------------YGulZ2VgxDlosOBeXKDZgfz0--