From owner-freebsd-virtualization@freebsd.org Mon Jan 25 06:21:23 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0D8BF4EADB0 for ; Mon, 25 Jan 2021 06:21:23 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPKWP5xFLz57nS for ; Mon, 25 Jan 2021 06:21:21 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 10P6LDqj067489 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 24 Jan 2021 22:21:13 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 10P6LDJM067487; Sun, 24 Jan 2021 22:21:13 -0800 (PST) (envelope-from jmg) Date: Sun, 24 Jan 2021 22:21:13 -0800 From: John-Mark Gurney To: Matt Churchyard Cc: Elena Mihailescu , "freebsd-virtualization@freebsd.org" Subject: Re: Warm Migration feature for bhyve - review on Phabricator Message-ID: <20210125062113.GL31099@funkthat.com> Mail-Followup-To: Matt Churchyard , Elena Mihailescu , "freebsd-virtualization@freebsd.org" References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Sun, 24 Jan 2021 22:21:13 -0800 (PST) X-Rspamd-Queue-Id: 4DPKWP5xFLz57nS X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [-1.80 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[208.87.223.18:from]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; R_DKIM_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[jmg]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; AUTH_NA(1.00)[]; SPAMHAUS_ZRD(0.00)[208.87.223.18:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 06:21:23 -0000 Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000: > > Hello, all, > > > We have recently opened a review on Phabricator for the warm migration code for > bhyve [1]. Please take a look and let us know if it is anything we can improve. > > > [1] https://reviews.freebsd.org/D28270 > > > Thank you, > > Elena > > I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") > > The current process (from a mile high overview) effectively seems to be the following - > > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 > > What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - > > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 > > Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? There's also hastd which can aid with this... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."