From owner-freebsd-virtualization@freebsd.org Fri Jan 29 00:13:23 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1765F4F221C for ; Fri, 29 Jan 2021 00:13:23 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DRd8y093dz4ttT for ; Fri, 29 Jan 2021 00:13:21 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 10T0Bq3g000826 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 28 Jan 2021 16:11:53 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 10T0BoSY000822; Thu, 28 Jan 2021 16:11:50 -0800 (PST) (envelope-from jmg) Date: Thu, 28 Jan 2021 16:11:50 -0800 From: John-Mark Gurney To: Willem Jan Withagen Cc: Matt Churchyard , "freebsd-virtualization@freebsd.org" Subject: Re: Warm Migration feature for bhyve - review on Phabricator Message-ID: <20210129001150.GR31099@funkthat.com> Mail-Followup-To: Willem Jan Withagen , Matt Churchyard , "freebsd-virtualization@freebsd.org" References: <20210125062113.GL31099@funkthat.com> <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> <20210125184204.GM31099@funkthat.com> <5c6ff016-e15d-afad-e855-8cfed32276f2@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5c6ff016-e15d-afad-e855-8cfed32276f2@digiware.nl> X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Thu, 28 Jan 2021 16:11:53 -0800 (PST) X-Rspamd-Queue-Id: 4DRd8y093dz4ttT X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [-1.80 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[jmg]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[208.87.223.18:from]; AUTH_NA(1.00)[]; MID_RHS_MATCH_FROM(0.00)[]; SPAMHAUS_ZRD(0.00)[208.87.223.18:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2021 00:13:23 -0000 Willem Jan Withagen wrote this message on Thu, Jan 28, 2021 at 01:42 +0100: > On 25-1-2021 19:42, John-Mark Gurney wrote: > > Matt Churchyard wrote this message on Mon, Jan 25, 2021 at 10:46 +0000: > >> -----Original Message----- > >> From: John-Mark Gurney > >> Sent: 25 January 2021 06:21 > >> To: Matt Churchyard > >> Cc: Elena Mihailescu ; freebsd-virtualization@freebsd.org > >> Subject: Re: Warm Migration feature for bhyve - review on Phabricator > >> > >>> There's also hastd which can aid with this... > >> Thanks for the reply. I've always been wary of the additional complexity of HAST and ZFS, as it doesn't seem to have widespread usage or support, and things get ugly fast when storage is involved. > > Totally agree... > > > >> However, the idea of using HAST on top of zvols to provide network mirrored storage for a guest is interesting. It adds a lot of extra complexity, and probably performance impact though if it's just for the ability to move a guest between systems that may only happen every now and then. I'm also not sure it would help (or would at least be even more complex) if I have 4 hosts and want to be able to move guests anywhere. > > gmirror + ggate is another option as well... > > I did try all kinds of these solutions that are based on a 2 system > backend storage, and it was relatively easy for me to get the state of > the storage in a split-brain situation which is not really repairable > without a lot of manual user intervention. > So I gave up on HAST and gmirror over ggate. Can you explain more about the split brain w/ gmirror+ggate? If writes didn't happen to one of the members of gmirror, then that should be investigated.. I guess I could see a situation where ggate got behind on writes (connection drop), and gmirror didn't kick out the ggate member, or failed to mark it as stale, but those are bugs that need to be identified and fixed... I'm partly currious about the ggate, because I've recently done some work w/ ggate (performance improvements and working of sftp), and it's simple enough that issues are a bit surprising.. I'm not at all familar w/ HAST to advice on any issues w/ that... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."