Date: Mon, 25 Jan 2021 10:46:38 +0000 From: Matt Churchyard <matt.churchyard@userve.net> To: John-Mark Gurney <jmg@funkthat.com> Cc: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: RE: Warm Migration feature for bhyve - review on Phabricator Message-ID: <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> In-Reply-To: <20210125062113.GL31099@funkthat.com> References: <CAGOCPLhP3i4hPcKHX=RrKFbQiSenjtJrvzTx%2BARnP8=QN%2BzmqQ@mail.gmail.com> <e9a65e1ad4e0491083820fbb03d873e8@SERVER.ad.usd-group.com> <20210125062113.GL31099@funkthat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
-----Original Message----- From: John-Mark Gurney <jmg@funkthat.com>=20 Sent: 25 January 2021 06:21 To: Matt Churchyard <matt.churchyard@userve.net> Cc: Elena Mihailescu <elenamihailescu22@gmail.com>; freebsd-virtualization@= freebsd.org Subject: Re: Warm Migration feature for bhyve - review on Phabricator Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000: > > Hello, all, >=20 > > We have recently opened a review on Phabricator for the warm migration = code for > bhyve [1]. Please take a look and let us know if it is anything = we can improve. >=20 > > [1] https://reviews.freebsd.org/D28270 >=20 > > Thank you, > > Elena >=20 > I appreciate that this isn't really related to the current review, and co= mmend the work being put into bhyve - it's an invaluable addition to FreeBS= D. I'm just wondering if any thought has been put into the future possibili= ty for transfer of disk data during migration (i.e. the equivalent of "stor= age vmotion") >=20 > The current process (from a mile high overview) effectively seems to be t= he following - >=20 > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 >=20 > What would be the feasibility of being able to run a process such as the = following? Obviously it would likely need to be orchestrated by an external= tool, but to me it seems the main requirement is really just to be able to= provide separate control over the pause and migrate steps on host1 - >=20 > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 >=20 > Are there any major complications here I'm not aware of other than the re= quirement to pause the guest and kick off the state migration as two separa= te calls? > There's also hastd which can aid with this... Thanks for the reply. I've always been wary of the additional complexity of= HAST and ZFS, as it doesn't seem to have widespread usage or support, and = things get ugly fast when storage is involved. However, the idea of using HAST on top of zvols to provide network mirrored= storage for a guest is interesting. It adds a lot of extra complexity, and= probably performance impact though if it's just for the ability to move a = guest between systems that may only happen every now and then. I'm also not= sure it would help (or would at least be even more complex) if I have 4 ho= sts and want to be able to move guests anywhere. The main reason for the email was to explore my theory that, by leveraging = ZFS, any bhyve user could roll their own storage migration ability with a f= ew commands as long as the following two (seemingly minor) abilities were p= resent 1) the ability to suspend a guest without it being done as part of the migr= ate call. (I assume suspend/resume support via bhyvectl is planned anyway, = if not already in place at this point) 2) modification of the migrate call to skip suspend if the guest is already= suspended. The main thing I'm not sure of is whether the migrate call has any specific= reliance on doing the suspend itself (e.g. if it needs to do anything befo= re the suspend, which will obviously be problematic if suspend & migrate ar= e called separately). Or if there's something else I missed that means this= is not feasible. I'm not really after massive changes to the current revie= w to implement disk migration in bhyve itself. Matt --=20 John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7769d16522fa49558049f4a6e936982e>