From owner-freebsd-virtualization@freebsd.org Fri Jan 22 10:10:11 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 55F1F4EFFD7 for ; Fri, 22 Jan 2021 10:10:11 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.userve.net", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DMZkp1HqTz4fg3 for ; Fri, 22 Jan 2021 10:10:09 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id 2CFBC2385E4; Fri, 22 Jan 2021 10:10:02 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=uk1; t=1611310202; bh=KzIALorVfSyx3i4skv4Ti4Pn9l3vMHvBZ+HTQ16K6+8=; h=From:To:Subject:Date:References:In-Reply-To; b=HXiB38apkK7Ethgi1RD7gWgYwret2I/Sv2pTfOsAPPrfhBbao1LE4B1tIHT4mPosK fuoNHshg0ristlm3Ffu6H4xozkNxa+9/by8q88w7YvMmIcFx4KITjmU/J2swy9ap4E mpZ+2gsS/QqMuumtp/KmbHHAhaaOclYkGjZlxx2g= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Fri, 22 Jan 2021 10:10:00 +0000 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Fri, 22 Jan 2021 10:10:00 +0000 From: Matt Churchyard To: Elena Mihailescu , "freebsd-virtualization@freebsd.org" Subject: RE: Warm Migration feature for bhyve - review on Phabricator Thread-Topic: Warm Migration feature for bhyve - review on Phabricator Thread-Index: AQHW8BfDZfixYSjLVkKO3L++W2nfGaozZNBw Date: Fri, 22 Jan 2021 10:09:59 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Rspamd-Queue-Id: 4DMZkp1HqTz4fg3 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=userve.net header.s=uk1 header.b=HXiB38ap; dmarc=none; spf=pass (mx1.freebsd.org: domain of matt.churchyard@userve.net designates 217.196.1.22 as permitted sender) smtp.mailfrom=matt.churchyard@userve.net X-Spamd-Result: default: False [-1.50 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[217.196.1.22:from]; R_DKIM_ALLOW(-0.20)[userve.net:s=uk1]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.196.1.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[userve.net]; ARC_NA(0.00)[]; NEURAL_SPAM_SHORT(1.00)[1.000]; SPAMHAUS_ZRD(0.00)[217.196.1.22:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[userve.net:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:20652, ipnet:217.196.0.0/20, country:GB]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2021 10:10:11 -0000 > Hello, all, > We have recently opened a review on Phabricator for the warm migration co= de for > bhyve [1]. Please take a look and let us know if it is anything we= can improve. > [1] https://reviews.freebsd.org/D28270 > Thank you, > Elena I appreciate that this isn't really related to the current review, and comm= end the work being put into bhyve - it's an invaluable addition to FreeBSD.= I'm just wondering if any thought has been put into the future possibility= for transfer of disk data during migration (i.e. the equivalent of "storag= e vmotion") The current process (from a mile high overview) effectively seems to be the= following - * Start guest on host2 pointing at a shared disk, and halt any execution * use bhyvectl to pause and begin migration on host1 * Start the guest on host2 What would be the feasibility of being able to run a process such as the fo= llowing? Obviously it would likely need to be orchestrated by an external t= ool, but to me it seems the main requirement is really just to be able to p= rovide separate control over the pause and migrate steps on host1 - * send a ZFS snapshot of the running machine to host2 * start the guest in migrate recv mode on host2 * pause the guest on host1 * send a new snapshot * initiate the migration of memory/device data * start guest on host2 Are there any major complications here I'm not aware of other than the requ= irement to pause the guest and kick off the state migration as two separate= calls? Shared storage is great but becomes complex and expensive if you want high = performance and reliability (seeing as the storage quickly becomes a major = single point of failure without enterprise active/active kit). I suspect th= e most common deployment of bhyve is independent hosts with local ZFS pools= as this is easy/cheap and gives great performance. Most hypervisors only h= ad "shared storage" migration for a long time but the big ones now also sup= port transferring disk data live. It would be great to be able to do this o= ut of the gate. I did have a poor mans version of this in vm-bhyve, but obviously relied on= stopping and restarting the guest. I always had problems with the nc comma= nds not exiting cleanly though and hanging the process so never made it off= icial. Regards, Matt _______________________________________________ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/m= ailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebs= d.org"