From owner-freebsd-virtualization@freebsd.org Thu Jan 28 00:33:45 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 073054EED61 for ; Thu, 28 Jan 2021 00:33:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router10G.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DR1fv3BFpz3ql4 for ; Thu, 28 Jan 2021 00:33:42 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router10g.digiware.nl (localhost.digiware.nl [127.0.0.1]) by router10G.digiware.nl (Postfix) with ESMTP id 3664A35A02; Thu, 28 Jan 2021 01:33:35 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from router10G.digiware.nl ([127.0.0.1]) by router10g.digiware.nl (router10g.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 863U2h6diqJr; Thu, 28 Jan 2021 01:33:34 +0100 (CET) Received: from [192.168.11.254] (WJW-LAPTOP.digiware.nl [192.168.11.254]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by router10G.digiware.nl (Postfix) with ESMTPSA id F420935839; Thu, 28 Jan 2021 01:33:33 +0100 (CET) Subject: Re: Warm Migration feature for bhyve - review on Phabricator To: Matt Churchyard , Elena Mihailescu , "freebsd-virtualization@freebsd.org" References: From: Willem Jan Withagen Message-ID: <1c3ca65f-ac65-a710-7775-e0865abfe954@digiware.nl> Date: Thu, 28 Jan 2021 01:33:40 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.12.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4DR1fv3BFpz3ql4 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-2.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; HFILTER_HELO_IP_A(1.00)[router10g.digiware.nl]; HFILTER_HELO_NORES_A_OR_MX(0.30)[router10g.digiware.nl]; RCVD_COUNT_THREE(0.00)[4]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[userve.net,gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[176.74.240.9:from]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[digiware.nl]; NEURAL_HAM_LONG(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[176.74.240.9:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jan 2021 00:33:45 -0000 On 22-1-2021 11:09, Matt Churchyard wrote: >> Hello, all, >> We have recently opened a review on Phabricator for the warm migration code for > bhyve [1]. Please take a look and let us know if it is anything we can improve. >> [1] https://reviews.freebsd.org/D28270 >> Thank you, >> Elena > I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") > > The current process (from a mile high overview) effectively seems to be the following - > > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 > > What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - > > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 > > Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? > > Shared storage is great but becomes complex and expensive if you want high performance and reliability (seeing as the storage quickly becomes a major single point of failure without enterprise active/active kit). I suspect the most common deployment of bhyve is independent hosts with local ZFS pools as this is easy/cheap and gives great performance. Most hypervisors only had "shared storage" migration for a long time but the big ones now also support transferring disk data live. It would be great to be able to do this out of the gate. > Great work to see this landing. This is actually one of the reasons I started porting Ceph and build a bhyve backend that interfaces to Ceph RBD. So you do not need to migrate the disk data. You are right that Ceph does not come cheap in hardware, but the designers try to jump through hoops to make it reliable. And it will also take some performance, but I'm very happy running my FreeBSD-vm's on our Openstack/Ceph cluster. Warm migration has been on the horizon for quite some time, so I very happy to see it getting there. Unfortunately I have limited time ATM to actually see if warm migration could be made to work with Ceph a diskstorage. --WjW