From owner-freebsd-virtualization@freebsd.org Mon Jan 25 10:46:47 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DD8504DBDC9 for ; Mon, 25 Jan 2021 10:46:47 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.userve.net", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPRPg02f0z3l8t for ; Mon, 25 Jan 2021 10:46:46 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id 4F7442384B3; Mon, 25 Jan 2021 10:46:39 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=uk1; t=1611571599; bh=EUbH7g/IhvWGNlpX3HpBF5M7zjt97KH7GB1FbpouFfs=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=Xxf7ZNQ2TlAFLcEe/EVKZl6wH00mKbaV4B7PbVEbgo9ro4Bg55FBD9G33MCFM1bU2 hy8aq03g+qN9sRxKRDy4cj8eH8aL9nEEILLkjxa3zMLOf1zicc+V0dm+P7iVfQLG7p FIoqYbB/UrGrNUX4VFJTiHuhpNq/PFzCsW+VXWVw= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Mon, 25 Jan 2021 10:46:38 +0000 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Mon, 25 Jan 2021 10:46:38 +0000 From: Matt Churchyard To: John-Mark Gurney CC: "freebsd-virtualization@freebsd.org" Subject: RE: Warm Migration feature for bhyve - review on Phabricator Thread-Topic: Warm Migration feature for bhyve - review on Phabricator Thread-Index: AQHW8BfDZfixYSjLVkKO3L++W2nfGaozZNBwgAR/j4CAADybsA== Date: Mon, 25 Jan 2021 10:46:38 +0000 Message-ID: <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> References: <20210125062113.GL31099@funkthat.com> In-Reply-To: <20210125062113.GL31099@funkthat.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Rspamd-Queue-Id: 4DPRPg02f0z3l8t X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=userve.net header.s=uk1 header.b=Xxf7ZNQ2; dmarc=none; spf=pass (mx1.freebsd.org: domain of matt.churchyard@userve.net designates 217.196.1.22 as permitted sender) smtp.mailfrom=matt.churchyard@userve.net X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[217.196.1.22:from]; R_DKIM_ALLOW(-0.20)[userve.net:s=uk1]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.196.1.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[userve.net]; ARC_NA(0.00)[]; SPAMHAUS_ZRD(0.00)[217.196.1.22:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[userve.net:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:20652, ipnet:217.196.0.0/20, country:GB]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-Mailman-Approved-At: Mon, 25 Jan 2021 11:26:49 +0000 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 10:46:47 -0000 -----Original Message----- From: John-Mark Gurney =20 Sent: 25 January 2021 06:21 To: Matt Churchyard Cc: Elena Mihailescu ; freebsd-virtualization@= freebsd.org Subject: Re: Warm Migration feature for bhyve - review on Phabricator Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000: > > Hello, all, >=20 > > We have recently opened a review on Phabricator for the warm migration = code for > bhyve [1]. Please take a look and let us know if it is anything = we can improve. >=20 > > [1] https://reviews.freebsd.org/D28270 >=20 > > Thank you, > > Elena >=20 > I appreciate that this isn't really related to the current review, and co= mmend the work being put into bhyve - it's an invaluable addition to FreeBS= D. I'm just wondering if any thought has been put into the future possibili= ty for transfer of disk data during migration (i.e. the equivalent of "stor= age vmotion") >=20 > The current process (from a mile high overview) effectively seems to be t= he following - >=20 > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 >=20 > What would be the feasibility of being able to run a process such as the = following? Obviously it would likely need to be orchestrated by an external= tool, but to me it seems the main requirement is really just to be able to= provide separate control over the pause and migrate steps on host1 - >=20 > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 >=20 > Are there any major complications here I'm not aware of other than the re= quirement to pause the guest and kick off the state migration as two separa= te calls? > There's also hastd which can aid with this... Thanks for the reply. I've always been wary of the additional complexity of= HAST and ZFS, as it doesn't seem to have widespread usage or support, and = things get ugly fast when storage is involved. However, the idea of using HAST on top of zvols to provide network mirrored= storage for a guest is interesting. It adds a lot of extra complexity, and= probably performance impact though if it's just for the ability to move a = guest between systems that may only happen every now and then. I'm also not= sure it would help (or would at least be even more complex) if I have 4 ho= sts and want to be able to move guests anywhere. The main reason for the email was to explore my theory that, by leveraging = ZFS, any bhyve user could roll their own storage migration ability with a f= ew commands as long as the following two (seemingly minor) abilities were p= resent 1) the ability to suspend a guest without it being done as part of the migr= ate call. (I assume suspend/resume support via bhyvectl is planned anyway, = if not already in place at this point) 2) modification of the migrate call to skip suspend if the guest is already= suspended. The main thing I'm not sure of is whether the migrate call has any specific= reliance on doing the suspend itself (e.g. if it needs to do anything befo= re the suspend, which will obviously be problematic if suspend & migrate ar= e called separately). Or if there's something else I missed that means this= is not feasible. I'm not really after massive changes to the current revie= w to implement disk migration in bhyve itself. Matt --=20 John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."