From owner-freebsd-virtualization@freebsd.org Fri Jan 22 13:54:03 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5A6A94F5DB4 for ; Fri, 22 Jan 2021 13:54:03 +0000 (UTC) (envelope-from mihai.carabas@gmail.com) Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DMgj62zHJz3DPL for ; Fri, 22 Jan 2021 13:54:02 +0000 (UTC) (envelope-from mihai.carabas@gmail.com) Received: by mail-lf1-x12e.google.com with SMTP id v67so7647677lfa.0 for ; Fri, 22 Jan 2021 05:54:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=oXoKL6G0lCqZfJZQXrFJ2xSD+rvQyn38jZaGMTF/e7c=; b=NpbKzenP7huU/uJ1FJUKY5fqSBJKud29crxS4CTOxzvjvdGUnjXgk3Z01DpX7rEOGr DRzJbW0u5bxmDhmFATRMTPsXqOOgLoKV5j/uNC932IywPJv5XKryaN/DV4EdcHyuFnQC HVtrArMrCK2rU+9nUcM5s8QdyCrjc0H8Y8iQE7TlKyUQGw329dAUqZKJYWbxmvWP8tQO yt5tYeeWZQeB2vACNS4MyoywVLug6CBPbLJi61+6scCipRD5QQOcNxlB+Ne8ohJOlbsT EQM1eOmHyAoMyq4k47iSPTss4ElvqN2RTAvgHPuwiy7ZjB/KlQSXaRXYuhY6XiNO4BWA kkcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=oXoKL6G0lCqZfJZQXrFJ2xSD+rvQyn38jZaGMTF/e7c=; b=K1zMKJP+qdzqt0JsNoL2TRaBi+rkFJNjOu50gSPa8Mq3/WSC/fo+1u46v/h3/wlwVx bDdnXyrwClyXJ4PN8aUg4TxzoP/3w8vkIaHnQmTyrSc/eqU1S3e891PeNqvnUedEe5Oe ZTcEtaeR7k6ZldIGvTxC65spX3YvWSMVN2RTNRJJvmktaSIl8Z9U7t0spHXlUPs+OFsO hCTktLEuGD/5RGi7vCcGPA4e5ZRQlaOwRmV3NKI/OXMlhRG9zE2iODT4Bhn/ZppFq4n5 GFbEBp4PlYd6kwhDBkBossSRB4DDI8dsNdr/7mxAXSkHiTZu/LTMDWcui+UOflVi2eey Swug== X-Gm-Message-State: AOAM532vwApqXGaYnviqEN0629lv0rhLheIRjlLC8WZLtH2RE9nB0q/h tHFnYkGHctvlDlGLAFg4Jf9E9mSjutJsKYWQEIC9HElE X-Google-Smtp-Source: ABdhPJxqXAoW+9+zgq98FbV0xOR4KCk+I+wOm1UzDpHkNu+h9FqpfaCsAVKKMS6VdKQg9iByh6vwy3Ba3P26wMZhlYY= X-Received: by 2002:ac2:48a5:: with SMTP id u5mr2175628lfg.97.1611323640237; Fri, 22 Jan 2021 05:54:00 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Mihai Carabas Date: Fri, 22 Jan 2021 15:53:50 +0200 Message-ID: Subject: Re: Warm Migration feature for bhyve - review on Phabricator To: Matt Churchyard Cc: Elena Mihailescu , "freebsd-virtualization@freebsd.org" X-Rspamd-Queue-Id: 4DMgj62zHJz3DPL X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=NpbKzenP; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mihaicarabas@gmail.com designates 2a00:1450:4864:20::12e as permitted sender) smtp.mailfrom=mihaicarabas@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::12e:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::12e:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::12e:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2021 13:54:03 -0000 Hello Matt, Thank you for your input. The Warm/Live Migration feature is designated for cloud and cluster environments. Storage Live Migration is a feature that would be nice to have, but will be tackled after we get in warm migration and live migration using a shared storage. The code base is big enough as it is. Adding storage migration will make it hard to get in. Also this is the track that was followed by all hypervisors until now (KVM, Hyper-V, VMware). Thanks, Mihai On Fri, Jan 22, 2021 at 12:10 PM Matt Churchyard wrote: > > Hello, all, > > > We have recently opened a review on Phabricator for the warm migration > code for > bhyve [1]. Please take a look and let us know if it is anything > we can improve. > > > [1] https://reviews.freebsd.org/D28270 > > > Thank you, > > Elena > > I appreciate that this isn't really related to the current review, and > commend the work being put into bhyve - it's an invaluable addition to > FreeBSD. I'm just wondering if any thought has been put into the future > possibility for transfer of disk data during migration (i.e. the equivalent > of "storage vmotion") > > The current process (from a mile high overview) effectively seems to be > the following - > > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 > > What would be the feasibility of being able to run a process such as the > following? Obviously it would likely need to be orchestrated by an external > tool, but to me it seems the main requirement is really just to be able to > provide separate control over the pause and migrate steps on host1 - > > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 > > Are there any major complications here I'm not aware of other than the > requirement to pause the guest and kick off the state migration as two > separate calls? > > Shared storage is great but becomes complex and expensive if you want high > performance and reliability (seeing as the storage quickly becomes a major > single point of failure without enterprise active/active kit). I suspect > the most common deployment of bhyve is independent hosts with local ZFS > pools as this is easy/cheap and gives great performance. Most hypervisors > only had "shared storage" migration for a long time but the big ones now > also support transferring disk data live. It would be great to be able to > do this out of the gate. > > I did have a poor mans version of this in vm-bhyve, but obviously relied > on stopping and restarting the guest. I always had problems with the nc > commands not exiting cleanly though and hanging the process so never made > it official. > > Regards, > Matt > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" >