From owner-freebsd-virtualization@freebsd.org Mon Jan 25 14:25:52 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7ED1C4E3D70 for ; Mon, 25 Jan 2021 14:25:52 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPXGR5gR4z4W0X for ; Mon, 25 Jan 2021 14:25:51 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: by mail-ej1-x633.google.com with SMTP id g3so18278092ejb.6 for ; Mon, 25 Jan 2021 06:25:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=+D18jmZDAUiZO3X2pX9KV+g34pLVOqcMa7pvVbQADkg=; b=X3uX5OWVI+dywV09nUhtfJXRt21CQwBt/QRv+SZZYqOZiyYB9DkErRbZEwYEXjGf3U EyolJX5vxGlGIVzqMLmBf5FHyFi60DbTbphRhYiRLcVAMYlBmX6dCzmBSMw/TogmNXxz 5/tNHN3HtvCJCBxNMGaGXXm9B9Mah0ls88U4R3MDeR2Y86r9c5Wx5Prlg3koJ3Ni4Db2 oiV/1Rf17LhVPOjsQR0P9QbLX111ky9SwlK51RnIytmSr/gJW0fxw5KaGctNgxNqzeDI dH5Gkt5zAi2IutogKEyWrCYbQ2AqULgQ0oR1zNWk9ZKN73btA3lqxPTOHbfn3+AnAmZQ 5rHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=+D18jmZDAUiZO3X2pX9KV+g34pLVOqcMa7pvVbQADkg=; b=K1MPY9TCAqJhJ+XDWeDkjVgy5U762Z6gmA/XAaEQAqmdoUIz4kmOgiqZBSh9nIfgdM OcAHG6QqD6PmpgVsVdXymz/FUEER8Nmc7gXF034rpbFMUk5SzzYr6WHmjzqEa8wovCWL z4q8qYxdWnPE4qJ7O28YK31GUJu8XDtg9KvMnOTstshVYu6TDqG4mo1zfDJZww9LBPZK hnpHrKjtORHmckCOx3CRhfoxPz40dMHK5FOeOKFKCyC5g0ChyWayfYmCM6Gg13C0kygW FcngQyukFXqJ6r4ILp1xprSP4jJlVeu2ccZ393fPaG9AP19sxUsSTuwHI3xbo5m2sbDJ ZxRw== X-Gm-Message-State: AOAM531nBsLEiuiBDjJTjQq+1e6AzWJgzppgLSYiazB9fKYL8a81YItE pUpQmia4Ra2k7S92IcvBEf/WK+XNM/8nrsRmFvzokMTaBSU= X-Google-Smtp-Source: ABdhPJwJbVFeV7/TMN/T4X8GwDE0JQuKOMubNsyGW8hxhuJ9p95ipFIO0o9X9KrMDtLpAsDuHCIGojWCeSOmRtSzhJI= X-Received: by 2002:a17:906:33c5:: with SMTP id w5mr529589eja.319.1611584750466; Mon, 25 Jan 2021 06:25:50 -0800 (PST) MIME-Version: 1.0 References: <20210125062113.GL31099@funkthat.com> <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> In-Reply-To: <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> From: Elena Mihailescu Date: Mon, 25 Jan 2021 16:25:24 +0200 Message-ID: Subject: Re: Warm Migration feature for bhyve - review on Phabricator To: Matt Churchyard Cc: John-Mark Gurney , "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4DPXGR5gR4z4W0X X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=X3uX5OWV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of elenamihailescu22@gmail.com designates 2a00:1450:4864:20::633 as permitted sender) smtp.mailfrom=elenamihailescu22@gmail.com X-Spamd-Result: default: False [-3.80 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.80)[-0.804]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::633:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::633:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::633:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 14:25:52 -0000 On Mon, 25 Jan 2021 at 13:26, Matt Churchyard wrote: > > -----Original Message----- > From: John-Mark Gurney > Sent: 25 January 2021 06:21 > To: Matt Churchyard > Cc: Elena Mihailescu ; freebsd-virtualizatio= n@freebsd.org > Subject: Re: Warm Migration feature for bhyve - review on Phabricator > > Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000: > > > Hello, all, > > > > > We have recently opened a review on Phabricator for the warm migratio= n code for > bhyve [1]. Please take a look and let us know if it is anythin= g we can improve. > > > > > [1] https://reviews.freebsd.org/D28270 > > > > > Thank you, > > > Elena > > > > I appreciate that this isn't really related to the current review, and = commend the work being put into bhyve - it's an invaluable addition to Free= BSD. I'm just wondering if any thought has been put into the future possibi= lity for transfer of disk data during migration (i.e. the equivalent of "st= orage vmotion") > > > > The current process (from a mile high overview) effectively seems to be= the following - > > > > * Start guest on host2 pointing at a shared disk, and halt any executio= n > > * use bhyvectl to pause and begin migration on host1 > > * Start the guest on host2 > > > > What would be the feasibility of being able to run a process such as th= e following? Obviously it would likely need to be orchestrated by an extern= al tool, but to me it seems the main requirement is really just to be able = to provide separate control over the pause and migrate steps on host1 - > > > > * send a ZFS snapshot of the running machine to host2 > > * start the guest in migrate recv mode on host2 > > * pause the guest on host1 > > * send a new snapshot > > * initiate the migration of memory/device data > > * start guest on host2 > > > > Are there any major complications here I'm not aware of other than the = requirement to pause the guest and kick off the state migration as two sepa= rate calls? > > > There's also hastd which can aid with this... > > Thanks for the reply. I've always been wary of the additional complexity = of HAST and ZFS, as it doesn't seem to have widespread usage or support, an= d things get ugly fast when storage is involved. > > However, the idea of using HAST on top of zvols to provide network mirror= ed storage for a guest is interesting. It adds a lot of extra complexity, a= nd probably performance impact though if it's just for the ability to move = a guest between systems that may only happen every now and then. I'm also n= ot sure it would help (or would at least be even more complex) if I have 4 = hosts and want to be able to move guests anywhere. > > The main reason for the email was to explore my theory that, by leveragin= g ZFS, any bhyve user could roll their own storage migration ability with a= few commands as long as the following two (seemingly minor) abilities were= present > > 1) the ability to suspend a guest without it being done as part of the mi= grate call. (I assume suspend/resume support via bhyvectl is planned anyway= , if not already in place at this point) > 2) modification of the migrate call to skip suspend if the guest is alrea= dy suspended. > > The main thing I'm not sure of is whether the migrate call has any specif= ic reliance on doing the suspend itself (e.g. if it needs to do anything be= fore the suspend, which will obviously be problematic if suspend & migrate = are called separately). Or if there's something else I missed that means th= is is not feasible. I'm not really after massive changes to the current rev= iew to implement disk migration in bhyve itself. Thank you for your input related to the disk migration. We were (still are, actually) focusing on the live migration feature for bhyve and did not take into consideration the disk migration. As Mihai said, the patches for the two (guest's state and guest's disk migration) are or will be quite big by themselves and we want the review process to go as smoothly as possible. After we have a pretty clear view of the live migration implementation (a patch on this feature will be uploaded on Phabricator in a couple of weeks) and how we should improve it, we will look into the disk migration feature and how it can be implemented for bhyve. As I said, our objective is to add the live migration functionality to bhyve which is based on the warm migration implementation which in turn is based on the suspend option in bhyve (the snapshotting/suspend option was added in the upstream last year, the FreeBSD code must be compiled with the BHYVE_SNAPSHOT option for it to work). Elena > > Matt > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@free= bsd.org"