From owner-freebsd-virtualization@freebsd.org Sun Jan 24 15:57:22 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 230D64D0554 for ; Sun, 24 Jan 2021 15:57:22 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DNyLS75XCz3jgD for ; Sun, 24 Jan 2021 15:57:20 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: by mail-ed1-x52e.google.com with SMTP id b21so12280406edy.6 for ; Sun, 24 Jan 2021 07:57:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli-net.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=pTcGVVx6qtfoQESAi4KvG6agl4eITKU7V2xGp9Tk6C8=; b=cbSMbDoxmQIdxP023EO9RTLBSwtC3R5Z1ZW0TLFWDv75xQLULLjs/yOayt/ogjMa+/ Y8GvuV2EgZ5zjzrY+oLo5alY31Pxv0lo5OJE8toK0zxbfaQFry/9u1ITByGbpPWsKyZW 00lrrf3M1rQ79gi4EuaghPocjHj6REqVp6lJFHIDthySRI8QDmgyc7WayZO0UTmJVuPa hvZ+yLCa/LiOTo7vbnamTHgUEcLNFwC8kW7JFvY+GaCaOdFRx47NV+QrIAgI2xmdyKmP JzLq9L8C/ZS6up6ggIhsadOqRG85mxQStKW9csMzwfFCGYmLDmnFHmFTvqe309WCRoFR t24g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=pTcGVVx6qtfoQESAi4KvG6agl4eITKU7V2xGp9Tk6C8=; b=sVv62+0c2OotBfYSiREsIh/UyYSNf3f1ZZcw34BBtMVV3qKwnwYpU12+RUzOrNPr3x miKki2ljzzQ91ne94KJojwh2pYy9tr0cMCoIYcOeXZ4KDAx5+QOwOpTGVD+FR5RofiCj 7bAzcrgPsvl2vwo5tXgRZxHZNq7FsOCdH4j0Pu335Nnvn8MgLT463qM3tetnYd13Ny3G WU5qnotLqRKnhK1wlP4lxnZsc4VpxHlg5rBRpK0J6Poqpq5sKOe++m2ZUd8HCy55xeHs DD43nvscQeqNnkw7JPandxUIkMmyMJNzEcBXn1f4iIxQwA9gR5Gbnsdyu6lueHvSuo9P 7gMw== X-Gm-Message-State: AOAM532F6g8NkHqqBcqrHvhCsQ5o1KL4hwifSV4kRWCrJwRMz8gWlygr 3p/UXUkz93R6NCPt+4eD1v4RLhEL/PCvnP/Xtvq8/6Hc+TcbeQIy X-Google-Smtp-Source: ABdhPJyHD+hN8F3UnU3M7nzIwCvu/6NeKNmznDSgKK+0jMzSs4NsESv6wvZhJ/3s+Qox+q/9WrQDFQAz2QV/fbSP4Os= X-Received: by 2002:aa7:c813:: with SMTP id a19mr726044edt.136.1611503838306; Sun, 24 Jan 2021 07:57:18 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Chuck Tuffli Date: Sun, 24 Jan 2021 07:57:07 -0800 Message-ID: Subject: Re: RHEL virtualization To: John Kennedy Cc: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DNyLS75XCz3jgD X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tuffli-net.20150623.gappssmtp.com header.s=20150623 header.b=cbSMbDox; dmarc=none; spf=permerror (mx1.freebsd.org: domain of chuck@tuffli.net uses mechanism not recognized by this client) smtp.mailfrom=chuck@tuffli.net X-Spamd-Result: default: False [-3.30 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[tuffli-net.20150623.gappssmtp.com:s=20150623]; FREEFALL_USER(0.00)[chuck]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; DMARC_NA(0.00)[tuffli.net]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::52e:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[empty SPF record]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52e:from]; DKIM_TRACE(0.00)[tuffli-net.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.997]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::52e:from]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jan 2021 15:57:22 -0000 On Sat, Jan 23, 2021 at 3:16 PM John Kennedy wrote: ... > That being said, RHEL on bhyve has been a pain to figure out. The best I've > done so far is using sysutils/grub2-bhyve to set up the boot CD, using > BHYVE_UEFI.fd as UEFI firmware (sysutils/bhyve-firmware I think) and then > getting at the console via net/tigervnc-viewer. > > Currently I'm fighting grub-bhyve's issue finding the kernel to load (if I'm > finding the right problem reports, it doesn't seem to like modern XFS or > ext4 partitions). I couldn't get net/ipxe to PXE boot anything, and I din't > manage to get very far with sysutils/uefi-edk2-bhyve. And of course some > of these are flagged with python2.7 isses. Hi I ran into a similar situation (RHELL VM's in bhyve) and created a version of grub-bhyve that knows how to find the grub.cfg and/or syslinux.conf files in a VM's disk image. Because of this, you don't have to maintain a copy of the grub configuration just for grub-bhyve. I implemented this on the latest version of grub which includes XFS and other goodies. The fine print: - The syntax is slightly different. To load an image use grub-bhyve --disk=/path/to/vm/disk.img --vm=vm_name note that the root, device-map, and directory options are no longer necessary - The project is at the "it works for me" stage and hasn't received much usage aside from myself. That said, feedback is welcome. - It builds on FreeBSD 12.x but 13.x has an issue Code is here: https://gitlab.com/ctuffli/grub To build: $ env PYTHON=python3.7 ./bootstrap $ env MAKE=gmake ./configure --with-platform=bhyve $ gmake The resulting binary, `grub-bhyve`, will be in the `grub-core/` directory. If you have success or troubles with it, please let me know. --chuck From owner-freebsd-virtualization@freebsd.org Sun Jan 24 21:00:19 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ED51A4DA016 for ; Sun, 24 Jan 2021 21:00:19 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DP5434gd2z4ZT8 for ; Sun, 24 Jan 2021 21:00:19 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 92B134D9F24; Sun, 24 Jan 2021 21:00:19 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 926184D9F23 for ; Sun, 24 Jan 2021 21:00:19 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DP5432hnpz4Zmy for ; Sun, 24 Jan 2021 21:00:19 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3CF6721B8 for ; Sun, 24 Jan 2021 21:00:19 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10OL0JN3029545 for ; Sun, 24 Jan 2021 21:00:19 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10OL0Jkk029544 for virtualization@FreeBSD.org; Sun, 24 Jan 2021 21:00:19 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202101242100.10OL0Jkk029544@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 24 Jan 2021 21:00:19 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jan 2021 21:00:20 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 247208 | mpt(4): VMWare virtualized LSI controller panics New | 240945 | [hyper-v] [netvsc] hn network driver incorrectly 2 problems total for which you should take action. From owner-freebsd-virtualization@freebsd.org Mon Jan 25 06:21:23 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0D8BF4EADB0 for ; Mon, 25 Jan 2021 06:21:23 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPKWP5xFLz57nS for ; Mon, 25 Jan 2021 06:21:21 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 10P6LDqj067489 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 24 Jan 2021 22:21:13 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 10P6LDJM067487; Sun, 24 Jan 2021 22:21:13 -0800 (PST) (envelope-from jmg) Date: Sun, 24 Jan 2021 22:21:13 -0800 From: John-Mark Gurney To: Matt Churchyard Cc: Elena Mihailescu , "freebsd-virtualization@freebsd.org" Subject: Re: Warm Migration feature for bhyve - review on Phabricator Message-ID: <20210125062113.GL31099@funkthat.com> Mail-Followup-To: Matt Churchyard , Elena Mihailescu , "freebsd-virtualization@freebsd.org" References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Sun, 24 Jan 2021 22:21:13 -0800 (PST) X-Rspamd-Queue-Id: 4DPKWP5xFLz57nS X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [-1.80 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[208.87.223.18:from]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; R_DKIM_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[jmg]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; AUTH_NA(1.00)[]; SPAMHAUS_ZRD(0.00)[208.87.223.18:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 06:21:23 -0000 Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000: > > Hello, all, > > > We have recently opened a review on Phabricator for the warm migration code for > bhyve [1]. Please take a look and let us know if it is anything we can improve. > > > [1] https://reviews.freebsd.org/D28270 > > > Thank you, > > Elena > > I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") > > The current process (from a mile high overview) effectively seems to be the following - > > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 > > What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - > > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 > > Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? There's also hastd which can aid with this... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-virtualization@freebsd.org Mon Jan 25 10:46:47 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DD8504DBDC9 for ; Mon, 25 Jan 2021 10:46:47 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.userve.net", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPRPg02f0z3l8t for ; Mon, 25 Jan 2021 10:46:46 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id 4F7442384B3; Mon, 25 Jan 2021 10:46:39 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=uk1; t=1611571599; bh=EUbH7g/IhvWGNlpX3HpBF5M7zjt97KH7GB1FbpouFfs=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=Xxf7ZNQ2TlAFLcEe/EVKZl6wH00mKbaV4B7PbVEbgo9ro4Bg55FBD9G33MCFM1bU2 hy8aq03g+qN9sRxKRDy4cj8eH8aL9nEEILLkjxa3zMLOf1zicc+V0dm+P7iVfQLG7p FIoqYbB/UrGrNUX4VFJTiHuhpNq/PFzCsW+VXWVw= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Mon, 25 Jan 2021 10:46:38 +0000 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Mon, 25 Jan 2021 10:46:38 +0000 From: Matt Churchyard To: John-Mark Gurney CC: "freebsd-virtualization@freebsd.org" Subject: RE: Warm Migration feature for bhyve - review on Phabricator Thread-Topic: Warm Migration feature for bhyve - review on Phabricator Thread-Index: AQHW8BfDZfixYSjLVkKO3L++W2nfGaozZNBwgAR/j4CAADybsA== Date: Mon, 25 Jan 2021 10:46:38 +0000 Message-ID: <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> References: <20210125062113.GL31099@funkthat.com> In-Reply-To: <20210125062113.GL31099@funkthat.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Rspamd-Queue-Id: 4DPRPg02f0z3l8t X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=userve.net header.s=uk1 header.b=Xxf7ZNQ2; dmarc=none; spf=pass (mx1.freebsd.org: domain of matt.churchyard@userve.net designates 217.196.1.22 as permitted sender) smtp.mailfrom=matt.churchyard@userve.net X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[217.196.1.22:from]; R_DKIM_ALLOW(-0.20)[userve.net:s=uk1]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.196.1.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[userve.net]; ARC_NA(0.00)[]; SPAMHAUS_ZRD(0.00)[217.196.1.22:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[userve.net:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:20652, ipnet:217.196.0.0/20, country:GB]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-Mailman-Approved-At: Mon, 25 Jan 2021 11:26:49 +0000 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 10:46:47 -0000 -----Original Message----- From: John-Mark Gurney =20 Sent: 25 January 2021 06:21 To: Matt Churchyard Cc: Elena Mihailescu ; freebsd-virtualization@= freebsd.org Subject: Re: Warm Migration feature for bhyve - review on Phabricator Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000: > > Hello, all, >=20 > > We have recently opened a review on Phabricator for the warm migration = code for > bhyve [1]. Please take a look and let us know if it is anything = we can improve. >=20 > > [1] https://reviews.freebsd.org/D28270 >=20 > > Thank you, > > Elena >=20 > I appreciate that this isn't really related to the current review, and co= mmend the work being put into bhyve - it's an invaluable addition to FreeBS= D. I'm just wondering if any thought has been put into the future possibili= ty for transfer of disk data during migration (i.e. the equivalent of "stor= age vmotion") >=20 > The current process (from a mile high overview) effectively seems to be t= he following - >=20 > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 >=20 > What would be the feasibility of being able to run a process such as the = following? Obviously it would likely need to be orchestrated by an external= tool, but to me it seems the main requirement is really just to be able to= provide separate control over the pause and migrate steps on host1 - >=20 > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 >=20 > Are there any major complications here I'm not aware of other than the re= quirement to pause the guest and kick off the state migration as two separa= te calls? > There's also hastd which can aid with this... Thanks for the reply. I've always been wary of the additional complexity of= HAST and ZFS, as it doesn't seem to have widespread usage or support, and = things get ugly fast when storage is involved. However, the idea of using HAST on top of zvols to provide network mirrored= storage for a guest is interesting. It adds a lot of extra complexity, and= probably performance impact though if it's just for the ability to move a = guest between systems that may only happen every now and then. I'm also not= sure it would help (or would at least be even more complex) if I have 4 ho= sts and want to be able to move guests anywhere. The main reason for the email was to explore my theory that, by leveraging = ZFS, any bhyve user could roll their own storage migration ability with a f= ew commands as long as the following two (seemingly minor) abilities were p= resent 1) the ability to suspend a guest without it being done as part of the migr= ate call. (I assume suspend/resume support via bhyvectl is planned anyway, = if not already in place at this point) 2) modification of the migrate call to skip suspend if the guest is already= suspended. The main thing I'm not sure of is whether the migrate call has any specific= reliance on doing the suspend itself (e.g. if it needs to do anything befo= re the suspend, which will obviously be problematic if suspend & migrate ar= e called separately). Or if there's something else I missed that means this= is not feasible. I'm not really after massive changes to the current revie= w to implement disk migration in bhyve itself. Matt --=20 John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-virtualization@freebsd.org Mon Jan 25 11:03:40 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7F9B64DCD30 for ; Mon, 25 Jan 2021 11:03:40 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.userve.net", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPRn73STJz3mYJ for ; Mon, 25 Jan 2021 11:03:39 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id 38DD72384A3; Mon, 25 Jan 2021 11:03:38 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=uk1; t=1611572618; bh=fBKPQGnVc8osc/n++9CAG4Ht8kXwjhw5U0X/g9XVW4w=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=E7jOSOYTenmnIKiIkDo06lyX0Y8ktTCrXjtWUTYPWU9giKRRK4+VE6OLXqbV09jtL 0ztZ9uOCEGBZIhA8cDbfIlJizwgTyMctdJMVCJggeN9eYLc9s+d0hIHqvZhFJD32Rx t3VbphRYhYsZ53GzHJKTGwLH2H9VaFb8o+amUpNs= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Mon, 25 Jan 2021 11:03:37 +0000 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Mon, 25 Jan 2021 11:03:37 +0000 From: Matt Churchyard To: Miroslav Lachman <000.fbsd@quip.cz> CC: "freebsd-virtualization@freebsd.org" Subject: RE: Warm Migration feature for bhyve - review on Phabricator Thread-Topic: Warm Migration feature for bhyve - review on Phabricator Thread-Index: AQHW8BfDZfixYSjLVkKO3L++W2nfGaozZNBwgATG/ACAAAP88A== Date: Mon, 25 Jan 2021 11:03:37 +0000 Message-ID: <41d46930f91b46bd85cea7e07d0cc014@SERVER.ad.usd-group.com> References: <82ad55fb-b203-2c69-82bc-a709586da8fc@quip.cz> In-Reply-To: <82ad55fb-b203-2c69-82bc-a709586da8fc@quip.cz> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Rspamd-Queue-Id: 4DPRn73STJz3mYJ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=userve.net header.s=uk1 header.b=E7jOSOYT; dmarc=none; spf=pass (mx1.freebsd.org: domain of matt.churchyard@userve.net designates 217.196.1.22 as permitted sender) smtp.mailfrom=matt.churchyard@userve.net X-Spamd-Result: default: False [-2.40 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[userve.net:s=uk1]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.196.1.0/24]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[userve.net]; RBL_DBL_DONT_QUERY_IPS(0.00)[217.196.1.22:from]; MIME_BASE64_TEXT_BOGUS(1.00)[]; SPAMHAUS_ZRD(0.00)[217.196.1.22:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[userve.net:+]; MIME_BASE64_TEXT(0.10)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:20652, ipnet:217.196.0.0/20, country:GB]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-Mailman-Approved-At: Mon, 25 Jan 2021 11:26:58 +0000 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 11:03:40 -0000 LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IE1pcm9zbGF2IExhY2htYW4gPDAwMC5m YnNkQHF1aXAuY3o+IA0KU2VudDogMjUgSmFudWFyeSAyMDIxIDEwOjM3DQpUbzogTWF0dCBDaHVy Y2h5YXJkIDxtYXR0LmNodXJjaHlhcmRAdXNlcnZlLm5ldD4NClN1YmplY3Q6IFJlOiBXYXJtIE1p Z3JhdGlvbiBmZWF0dXJlIGZvciBiaHl2ZSAtIHJldmlldyBvbiBQaGFicmljYXRvcg0KDQpPbiAy Mi8wMS8yMDIxIDExOjA5LCBNYXR0IENodXJjaHlhcmQgd3JvdGU6DQoNClsuLi5dDQoNCj4gU2hh cmVkIHN0b3JhZ2UgaXMgZ3JlYXQgYnV0IGJlY29tZXMgY29tcGxleCBhbmQgZXhwZW5zaXZlIGlm IHlvdSB3YW50IGhpZ2ggcGVyZm9ybWFuY2UgYW5kIHJlbGlhYmlsaXR5IChzZWVpbmcgYXMgdGhl IHN0b3JhZ2UgcXVpY2tseSBiZWNvbWVzIGEgbWFqb3Igc2luZ2xlIHBvaW50IG9mIGZhaWx1cmUg d2l0aG91dCBlbnRlcnByaXNlIGFjdGl2ZS9hY3RpdmUga2l0KS4gSSBzdXNwZWN0IHRoZSBtb3N0 IGNvbW1vbiBkZXBsb3ltZW50IG9mIGJoeXZlIGlzIGluZGVwZW5kZW50IGhvc3RzIHdpdGggbG9j YWwgWkZTIHBvb2xzIGFzIHRoaXMgaXMgZWFzeS9jaGVhcCBhbmQgZ2l2ZXMgZ3JlYXQgcGVyZm9y bWFuY2UuIE1vc3QgaHlwZXJ2aXNvcnMgb25seSBoYWQgInNoYXJlZCBzdG9yYWdlIiBtaWdyYXRp b24gZm9yIGEgbG9uZyB0aW1lIGJ1dCB0aGUgYmlnIG9uZXMgbm93IGFsc28gc3VwcG9ydCB0cmFu c2ZlcnJpbmcgZGlzayBkYXRhIGxpdmUuIEl0IHdvdWxkIGJlIGdyZWF0IHRvIGJlIGFibGUgdG8g ZG8gdGhpcyBvdXQgb2YgdGhlIGdhdGUuDQo+IA0KPiBJIGRpZCBoYXZlIGEgcG9vciBtYW5zIHZl cnNpb24gb2YgdGhpcyBpbiB2bS1iaHl2ZSwgYnV0IG9idmlvdXNseSByZWxpZWQgb24gc3RvcHBp bmcgYW5kIHJlc3RhcnRpbmcgdGhlIGd1ZXN0LiBJIGFsd2F5cyBoYWQgcHJvYmxlbXMgd2l0aCB0 aGUgbmMgY29tbWFuZHMgbm90IGV4aXRpbmcgY2xlYW5seSB0aG91Z2ggYW5kIGhhbmdpbmcgdGhl IHByb2Nlc3Mgc28gbmV2ZXIgbWFkZSBpdCBvZmZpY2lhbC4NCg0KPkkgZG9uJ3Qga25vdyBtdWNo IGRldGFpbHMgYWJvdXQgeW91ciBzZXR1cCBhbmQgeW91ciBwcm9ibGVtcyB3aXRoIG5jDQo+IChu ZXRjYXQpIGJ1dCBJIGd1ZXNzIGl0IGlzIHVzZWQgZm9yIHpmcyBzZW5kICYgcmVjZWl2ZS4gWW91 IGNhbiB0cnkgdG8gdXNlIG1idWZmZXIgZnJvbSBwb3J0cyBpbnN0ZWFkIG9mIG5jOg0KPiBodHRw czovL2V2ZXJ5Y2l0eS5jby51ay9hbGFzZGFpci8yMDEwLzA3L3VzaW5nLW1idWZmZXItdG8tc3Bl ZWQtdXAtc2xvdy16ZnMtc2VuZC16ZnMtcmVjZWl2ZS8NCg0KPiBJdCB3aWxsIGFkZCBzb21lIGRl cGVuZGVuY3kgb24gb3RoZXIgcG9ydCBidXQgaWYgc29tZWJvZHkgd2FudHMgc3RvcmFnZSBtaWdy YXRpb24gd2l0aCB5b3VyIGdyZWF0ZSB0b29sIHZtLWJoeXZlIEkgdGhpbmsgaXQgaXMgc21hbGwg cHJpY2UgdG8gcGF5LiAoYW5vdGhlciB3YXkgaXMgdG8gcHJvcGVybHkgc2V0dXAgU1NIIGFuZCA+ IGRvIG5vdCB1c2UgbmMgbm9yIG1idWZmZXIpDQoNClRoYW5rcyBmb3IgdGhlIHJlcGx5LiBZZXMs IEkgd2FzIHVzaW5nIGl0IHRvIHNlbmQgemZzIHNuYXBzaG90cyBiZXR3ZWVuIGhvc3RzLiBJIHRo aW5rIEkgYWN0dWFsbHkgaGF2ZSB0aGUgc2FtZSBpc3N1ZSB3aGVuIGRvaW5nIGl0IGJ5IGhhbmQu IFRoZSBuYyBjYWxsIHNlZW1zIGxpa2UgaXQgaGFuZ3Mgb25jZSBjb21wbGV0ZSBidXQgd2lsbCBp bnN0YW50bHkgcmV0dXJuIHRvIHRoZSBwcm9tcHQgd2hlbiBwcmVzc2luZyBlbnRlci4NCg0KQXMg eW91IHNheSwgYSBiZXR0ZXIgc29sdXRpb24gd291bGQgYmUgdG8gdXNlIHNzaC4gSXQgbWF5IGV2 ZW4gYmUgcG9zc2libGUgdG8gdHJpZ2dlciB0aGUgcmVjZWl2ZSBjb21tYW5kIGRpcmVjdGx5IGZy b20gdGhlIHNlbmRpbmcgaG9zdCBpZiBJIGdldCBhIGJpdCBoYWNreSBhbmQgc3BsaXQgb2ZmIGEg c2Vjb25kIGJhY2tncm91bmQgdGFzay4gSSBtYXkgaGF2ZSBhbm90aGVyIGxvb2sgYXQgaXQuIEl0 IHdvcmtlZCBwcmV0dHkgd2VsbCBhbGwgaW4gYWxsIGV2ZW4gaWYgaXQgZGlkIGhhdmUgdG8gc3Rv cC9zdGFydCB0aGUgZ3Vlc3QgYnJpZWZseS4NCg0KTWF0dA0KDQo+IEtpbmQgcmVnYXJkcw0KPiBN aXJvc2xhdiBMYWNobWFuDQo= From owner-freebsd-virtualization@freebsd.org Mon Jan 25 14:25:52 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7ED1C4E3D70 for ; Mon, 25 Jan 2021 14:25:52 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPXGR5gR4z4W0X for ; Mon, 25 Jan 2021 14:25:51 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: by mail-ej1-x633.google.com with SMTP id g3so18278092ejb.6 for ; Mon, 25 Jan 2021 06:25:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=+D18jmZDAUiZO3X2pX9KV+g34pLVOqcMa7pvVbQADkg=; b=X3uX5OWVI+dywV09nUhtfJXRt21CQwBt/QRv+SZZYqOZiyYB9DkErRbZEwYEXjGf3U EyolJX5vxGlGIVzqMLmBf5FHyFi60DbTbphRhYiRLcVAMYlBmX6dCzmBSMw/TogmNXxz 5/tNHN3HtvCJCBxNMGaGXXm9B9Mah0ls88U4R3MDeR2Y86r9c5Wx5Prlg3koJ3Ni4Db2 oiV/1Rf17LhVPOjsQR0P9QbLX111ky9SwlK51RnIytmSr/gJW0fxw5KaGctNgxNqzeDI dH5Gkt5zAi2IutogKEyWrCYbQ2AqULgQ0oR1zNWk9ZKN73btA3lqxPTOHbfn3+AnAmZQ 5rHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=+D18jmZDAUiZO3X2pX9KV+g34pLVOqcMa7pvVbQADkg=; b=K1MPY9TCAqJhJ+XDWeDkjVgy5U762Z6gmA/XAaEQAqmdoUIz4kmOgiqZBSh9nIfgdM OcAHG6QqD6PmpgVsVdXymz/FUEER8Nmc7gXF034rpbFMUk5SzzYr6WHmjzqEa8wovCWL z4q8qYxdWnPE4qJ7O28YK31GUJu8XDtg9KvMnOTstshVYu6TDqG4mo1zfDJZww9LBPZK hnpHrKjtORHmckCOx3CRhfoxPz40dMHK5FOeOKFKCyC5g0ChyWayfYmCM6Gg13C0kygW FcngQyukFXqJ6r4ILp1xprSP4jJlVeu2ccZ393fPaG9AP19sxUsSTuwHI3xbo5m2sbDJ ZxRw== X-Gm-Message-State: AOAM531nBsLEiuiBDjJTjQq+1e6AzWJgzppgLSYiazB9fKYL8a81YItE pUpQmia4Ra2k7S92IcvBEf/WK+XNM/8nrsRmFvzokMTaBSU= X-Google-Smtp-Source: ABdhPJwJbVFeV7/TMN/T4X8GwDE0JQuKOMubNsyGW8hxhuJ9p95ipFIO0o9X9KrMDtLpAsDuHCIGojWCeSOmRtSzhJI= X-Received: by 2002:a17:906:33c5:: with SMTP id w5mr529589eja.319.1611584750466; Mon, 25 Jan 2021 06:25:50 -0800 (PST) MIME-Version: 1.0 References: <20210125062113.GL31099@funkthat.com> <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> In-Reply-To: <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> From: Elena Mihailescu Date: Mon, 25 Jan 2021 16:25:24 +0200 Message-ID: Subject: Re: Warm Migration feature for bhyve - review on Phabricator To: Matt Churchyard Cc: John-Mark Gurney , "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4DPXGR5gR4z4W0X X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=X3uX5OWV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of elenamihailescu22@gmail.com designates 2a00:1450:4864:20::633 as permitted sender) smtp.mailfrom=elenamihailescu22@gmail.com X-Spamd-Result: default: False [-3.80 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.80)[-0.804]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::633:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::633:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::633:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 14:25:52 -0000 On Mon, 25 Jan 2021 at 13:26, Matt Churchyard wrote: > > -----Original Message----- > From: John-Mark Gurney > Sent: 25 January 2021 06:21 > To: Matt Churchyard > Cc: Elena Mihailescu ; freebsd-virtualizatio= n@freebsd.org > Subject: Re: Warm Migration feature for bhyve - review on Phabricator > > Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000: > > > Hello, all, > > > > > We have recently opened a review on Phabricator for the warm migratio= n code for > bhyve [1]. Please take a look and let us know if it is anythin= g we can improve. > > > > > [1] https://reviews.freebsd.org/D28270 > > > > > Thank you, > > > Elena > > > > I appreciate that this isn't really related to the current review, and = commend the work being put into bhyve - it's an invaluable addition to Free= BSD. I'm just wondering if any thought has been put into the future possibi= lity for transfer of disk data during migration (i.e. the equivalent of "st= orage vmotion") > > > > The current process (from a mile high overview) effectively seems to be= the following - > > > > * Start guest on host2 pointing at a shared disk, and halt any executio= n > > * use bhyvectl to pause and begin migration on host1 > > * Start the guest on host2 > > > > What would be the feasibility of being able to run a process such as th= e following? Obviously it would likely need to be orchestrated by an extern= al tool, but to me it seems the main requirement is really just to be able = to provide separate control over the pause and migrate steps on host1 - > > > > * send a ZFS snapshot of the running machine to host2 > > * start the guest in migrate recv mode on host2 > > * pause the guest on host1 > > * send a new snapshot > > * initiate the migration of memory/device data > > * start guest on host2 > > > > Are there any major complications here I'm not aware of other than the = requirement to pause the guest and kick off the state migration as two sepa= rate calls? > > > There's also hastd which can aid with this... > > Thanks for the reply. I've always been wary of the additional complexity = of HAST and ZFS, as it doesn't seem to have widespread usage or support, an= d things get ugly fast when storage is involved. > > However, the idea of using HAST on top of zvols to provide network mirror= ed storage for a guest is interesting. It adds a lot of extra complexity, a= nd probably performance impact though if it's just for the ability to move = a guest between systems that may only happen every now and then. I'm also n= ot sure it would help (or would at least be even more complex) if I have 4 = hosts and want to be able to move guests anywhere. > > The main reason for the email was to explore my theory that, by leveragin= g ZFS, any bhyve user could roll their own storage migration ability with a= few commands as long as the following two (seemingly minor) abilities were= present > > 1) the ability to suspend a guest without it being done as part of the mi= grate call. (I assume suspend/resume support via bhyvectl is planned anyway= , if not already in place at this point) > 2) modification of the migrate call to skip suspend if the guest is alrea= dy suspended. > > The main thing I'm not sure of is whether the migrate call has any specif= ic reliance on doing the suspend itself (e.g. if it needs to do anything be= fore the suspend, which will obviously be problematic if suspend & migrate = are called separately). Or if there's something else I missed that means th= is is not feasible. I'm not really after massive changes to the current rev= iew to implement disk migration in bhyve itself. Thank you for your input related to the disk migration. We were (still are, actually) focusing on the live migration feature for bhyve and did not take into consideration the disk migration. As Mihai said, the patches for the two (guest's state and guest's disk migration) are or will be quite big by themselves and we want the review process to go as smoothly as possible. After we have a pretty clear view of the live migration implementation (a patch on this feature will be uploaded on Phabricator in a couple of weeks) and how we should improve it, we will look into the disk migration feature and how it can be implemented for bhyve. As I said, our objective is to add the live migration functionality to bhyve which is based on the warm migration implementation which in turn is based on the suspend option in bhyve (the snapshotting/suspend option was added in the upstream last year, the FreeBSD code must be compiled with the BHYVE_SNAPSHOT option for it to work). Elena > > Matt > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@free= bsd.org" From owner-freebsd-virtualization@freebsd.org Mon Jan 25 17:10:54 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 46C3C4E9732 for ; Mon, 25 Jan 2021 17:10:54 +0000 (UTC) (envelope-from warlock@phouka1.phouka.net) Received: from phouka1.phouka.net (phouka1.phouka.net [107.170.196.116]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "phouka.net", Issuer "Go Daddy Secure Certificate Authority - G2" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPbwr5fr3z4j8B for ; Mon, 25 Jan 2021 17:10:52 +0000 (UTC) (envelope-from warlock@phouka1.phouka.net) Received: from phouka1.phouka.net (localhost [127.0.0.1]) by phouka1.phouka.net (8.16.1/8.16.1) with ESMTPS id 10PH9YDc043015 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Mon, 25 Jan 2021 09:09:34 -0800 (PST) (envelope-from warlock@phouka1.phouka.net) Received: (from warlock@localhost) by phouka1.phouka.net (8.16.1/8.16.1/Submit) id 10PH9Xrm043012 for freebsd-virtualization@freebsd.org; Mon, 25 Jan 2021 09:09:34 -0800 (PST) (envelope-from warlock) Date: Mon, 25 Jan 2021 09:09:33 -0800 From: John Kennedy To: freebsd-virtualization@freebsd.org Subject: Re: RHEL virtualization Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4DPbwr5fr3z4j8B X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of warlock@phouka1.phouka.net has no SPF policy when checking 107.170.196.116) smtp.mailfrom=warlock@phouka1.phouka.net X-Spamd-Result: default: False [-1.74 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[107.170.196.116:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[107.170.196.116:from:127.0.2.255]; DMARC_NA(0.00)[phouka.net]; NEURAL_HAM_SHORT(-0.94)[-0.936]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[warlock@phouka.net,warlock@phouka1.phouka.net]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:14061, ipnet:107.170.192.0/18, country:US]; FROM_NEQ_ENVFROM(0.00)[warlock@phouka.net,warlock@phouka1.phouka.net]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 17:10:54 -0000 On Sat, Jan 23, 2021 at 03:14:53PM -0800, John Kennedy wrote: > At work, we have RHEL (-ish; some RHEL, some CentOS, some OEL). Mostly v7, > some v8. Since I'm doing the Covid work-from-home telecommute, I'm trying to > recreate some of my work infrastructure while trying to plan a bit towards > the future (migrating a lot of VMs to Azure). > > What I'd like to recreate is my existing kickstart infrastructure, where I > PXE boot the system, feed it anaconda goodness which dovetails into puppet > and I can generate a clean system from a template. Works great for VMWare > and HyperV, not so much for Azure but if I can generate a snapshot disk > image Azure can ingest, I'll be happy on that score. > > I've been very happy with bhyve for FreeBSD. I messed with VirtualBox for > a while (a long time ago), but with my tendency to track stable (think: > kernel modules) and keep very current on ports-from-source (frequent > package updates, upon which VirtualBox has MANY dependencies) made that a > poorer experience than I had with it on Windows. I've been very happy with > bhyve since it's basically baked right in. Let me restate some of this in a different way to maybe get some more thinking. Using the BHYVE_UEFI.fd from uefi-edk2-bhyve, I can boot my OEL8 (RHEL8 clone). That currently worries me because it has the big python-2.7 warning on it (as does uefi-edk2-bhyve-csm). On physical boxes, I've been able to grab a PXEBOOT ISO when the firmware lacks PXE booting, but I haven't got that to work yet for these. Those python worries are basically what is driving me to look elsewhere (like fighting with grub-bhyve and away from the only UEFI booting that I know about). I personally like PXE-booting a new system (and possibly making a gold image from that, depending on what I'm doing) because it basically answers that little auditor-voice in the back of my head that, in the event of some possible security problem, how do I know that my backups haven't been compromised. In all of those gigabytes, after all of the toxic recursive mindless non-logic, how do you *know*? My happy answer to myself is: "here is a configuration file that I can review, all the binaries are on the vendor's site or re-downloaded, here are the puppet customization rules, blam! done! 10 minutes later I have a clean system." In any case, that is why I'm chasing PXE booting, although I'd be interested in the way other people solve that problem. That really doesn't work that way in Azure, thus the gold images approach I'll probably have to take with them in the future. From owner-freebsd-virtualization@freebsd.org Mon Jan 25 18:42:08 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CA9A14EC620 for ; Mon, 25 Jan 2021 18:42:08 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPdy74vGYz4qgq for ; Mon, 25 Jan 2021 18:42:07 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 10PIg4rb097643 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 25 Jan 2021 10:42:04 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 10PIg4cX097642; Mon, 25 Jan 2021 10:42:04 -0800 (PST) (envelope-from jmg) Date: Mon, 25 Jan 2021 10:42:04 -0800 From: John-Mark Gurney To: Matt Churchyard Cc: "freebsd-virtualization@freebsd.org" Subject: Re: Warm Migration feature for bhyve - review on Phabricator Message-ID: <20210125184204.GM31099@funkthat.com> Mail-Followup-To: Matt Churchyard , "freebsd-virtualization@freebsd.org" References: <20210125062113.GL31099@funkthat.com> <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Mon, 25 Jan 2021 10:42:04 -0800 (PST) X-Rspamd-Queue-Id: 4DPdy74vGYz4qgq X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [-1.80 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[208.87.223.18:from]; FREEFALL_USER(0.00)[jmg]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; AUTH_NA(1.00)[]; MID_RHS_MATCH_FROM(0.00)[]; SPAMHAUS_ZRD(0.00)[208.87.223.18:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 18:42:08 -0000 Matt Churchyard wrote this message on Mon, Jan 25, 2021 at 10:46 +0000: > -----Original Message----- > From: John-Mark Gurney > Sent: 25 January 2021 06:21 > To: Matt Churchyard > Cc: Elena Mihailescu ; freebsd-virtualization@freebsd.org > Subject: Re: Warm Migration feature for bhyve - review on Phabricator > > Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000: > > > Hello, all, > > > > > We have recently opened a review on Phabricator for the warm migration code for > bhyve [1]. Please take a look and let us know if it is anything we can improve. > > > > > [1] https://reviews.freebsd.org/D28270 > > > > > Thank you, > > > Elena > > > > I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") > > > > The current process (from a mile high overview) effectively seems to be the following - > > > > * Start guest on host2 pointing at a shared disk, and halt any execution > > * use bhyvectl to pause and begin migration on host1 > > * Start the guest on host2 > > > > What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - > > > > * send a ZFS snapshot of the running machine to host2 > > * start the guest in migrate recv mode on host2 > > * pause the guest on host1 > > * send a new snapshot > > * initiate the migration of memory/device data > > * start guest on host2 > > > > Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? > > > There's also hastd which can aid with this... > > Thanks for the reply. I've always been wary of the additional complexity of HAST and ZFS, as it doesn't seem to have widespread usage or support, and things get ugly fast when storage is involved. Totally agree... > However, the idea of using HAST on top of zvols to provide network mirrored storage for a guest is interesting. It adds a lot of extra complexity, and probably performance impact though if it's just for the ability to move a guest between systems that may only happen every now and then. I'm also not sure it would help (or would at least be even more complex) if I have 4 hosts and want to be able to move guests anywhere. gmirror + ggate is another option as well... > The main reason for the email was to explore my theory that, by leveraging ZFS, any bhyve user could roll their own storage migration ability with a few commands as long as the following two (seemingly minor) abilities were present > > 1) the ability to suspend a guest without it being done as part of the migrate call. (I assume suspend/resume support via bhyvectl is planned anyway, if not already in place at this point) Yeah, I'd hope that there would be ability to insert custom commands in between the suspend on one host and the resumption on the new host.. But it is good explicitly state this.. > 2) modification of the migrate call to skip suspend if the guest is already suspended. > > The main thing I'm not sure of is whether the migrate call has any specific reliance on doing the suspend itself (e.g. if it needs to do anything before the suspend, which will obviously be problematic if suspend & migrate are called separately). Or if there's something else I missed that means this is not feasible. I'm not really after massive changes to the current review to implement disk migration in bhyve itself. +1 -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-virtualization@freebsd.org Mon Jan 25 19:18:17 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 92CAB4ED076 for ; Mon, 25 Jan 2021 19:18:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DPfls3ZDkz4skt for ; Mon, 25 Jan 2021 19:18:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 78D114ECEFC; Mon, 25 Jan 2021 19:18:17 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7895D4ED39E for ; Mon, 25 Jan 2021 19:18:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPfls2wg0z4swN for ; Mon, 25 Jan 2021 19:18:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 537BC1BE87 for ; Mon, 25 Jan 2021 19:18:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10PJIHSa052997 for ; Mon, 25 Jan 2021 19:18:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10PJIHnc052996 for virtualization@FreeBSD.org; Mon, 25 Jan 2021 19:18:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 253004] Win 10 guest fails to restart unless I --destroy Date: Mon, 25 Jan 2021 19:18:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: bergerkos@yahoo.co.uk X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 19:18:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253004 Bug ID: 253004 Summary: Win 10 guest fails to restart unless I --destroy Product: Base System Version: 13.0-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: bergerkos@yahoo.co.uk In 13.0-ALPHA2 I still have this problem: Running Win 10 guest with 2 passthrough devices (USB controller + NIC) I ca= n't start guest after shutdown. Get this error: Assertion failed: (error =3D=3D 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 501. .twm/programs/./bhyve-hda.sh: line 13: 90467 Abo= rt trap bhyve -S -c sockets=3D1,cores=3D2,threads=3D2 -m 6G -H -w= -s 0,hostbridge -s 3,passthru,7/0/1 -s 4,virtio-blk,/bhyve/win-alt.img,sectorsize=3D512/4096 -s 6,hda,rec=3D/dev/dsp,play=3D/dev/dsp -s 8,passthru,12/0/0 -s 29,fbuf,tcp=3D0.0.0.0:5900,w=3D1600,h=3D900,wait -s 30,xhci,tablet -s 31,lp= c -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd windows-= 10 Interestingly, this error particular error was reported and addressed by a patch, so it doesn't happen again. But it comes out when I restar guest aft= er shutdown. It was mentioned here, including also the patch: http://freebsd.1045724.x6.nabble.com/Windows-10-guests-fail-to-boot-when-at= tempting-to-passthrough-network-card-td6330452.html Solution: the only one that works is to run: #bhyvectl --vm=3D$my_vm_name --destroy --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Jan 25 19:27:17 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9985F4ED8F8 for ; Mon, 25 Jan 2021 19:27:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DPfyF3mcYz4tPS for ; Mon, 25 Jan 2021 19:27:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 812974EDBAC; Mon, 25 Jan 2021 19:27:17 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 80EF84EDA19 for ; Mon, 25 Jan 2021 19:27:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPfyF34qpz4tXQ for ; Mon, 25 Jan 2021 19:27:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5D57E1BF15 for ; Mon, 25 Jan 2021 19:27:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10PJRHin057593 for ; Mon, 25 Jan 2021 19:27:17 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10PJRHHb057592 for virtualization@FreeBSD.org; Mon, 25 Jan 2021 19:27:17 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 253004] Win 10 guest fails to restart unless I --destroy Date: Mon, 25 Jan 2021 19:27:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: bergerkos@yahoo.co.uk X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 19:27:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253004 --- Comment #1 from bergerkos@yahoo.co.uk --- Yes, the bug mentioned above (and patched successfully more than a year ago= , I believe) happened on first boot and prevented bhyve from booting at all with certain PCI cards. This thing, however, only happens when trying to restart the bhyve guest (W= in 10 in my case). --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Jan 25 15:01:02 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3D9C14E5BBA for ; Mon, 25 Jan 2021 15:01:02 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-a.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.userve.net", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPY2s5hYXz4XtX for ; Mon, 25 Jan 2021 15:00:53 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp-a.userve.net (Postfix) with ESMTPS id 66E752384A5; Mon, 25 Jan 2021 15:00:51 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=uk1; t=1611586851; bh=EUJzpYvKMPzls40kZNXN+Zf3oJ107xrmtiDoPY9rHMY=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=qAGpkMfoVcEzyoNl3fXaUTebovz7YEVa5X+/nMRLtzwZ8M+knaItOjgnrvSACYEpQ Y1ceCcroNEJ2TO5YGbYINf0OcYpnB32TpkSPnUS9cV2hzM6PifIkPo0RbSSe7kDbCb MaDumtieiOxFjKEQ2z7Bh569TeVfq1G8fJ5V8k38= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Mon, 25 Jan 2021 15:00:50 +0000 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Mon, 25 Jan 2021 15:00:50 +0000 From: Matt Churchyard To: Elena Mihailescu CC: John-Mark Gurney , "freebsd-virtualization@freebsd.org" Subject: RE: Warm Migration feature for bhyve - review on Phabricator Thread-Topic: Warm Migration feature for bhyve - review on Phabricator Thread-Index: AQHW8BfDZfixYSjLVkKO3L++W2nfGaozZNBwgAR/j4CAADybsIAASqwAgAABNWA= Date: Mon, 25 Jan 2021 15:00:50 +0000 Message-ID: References: <20210125062113.GL31099@funkthat.com> <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Rspamd-Queue-Id: 4DPY2s5hYXz4XtX X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=userve.net header.s=uk1 header.b=qAGpkMfo; dmarc=none; spf=pass (mx1.freebsd.org: domain of matt.churchyard@userve.net designates 217.196.1.22 as permitted sender) smtp.mailfrom=matt.churchyard@userve.net X-Spamd-Result: default: False [-2.40 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[userve.net:s=uk1]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.196.1.0/24]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[userve.net]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_BASE64_TEXT_BOGUS(1.00)[]; SPAMHAUS_ZRD(0.00)[217.196.1.22:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[userve.net:+]; MIME_BASE64_TEXT(0.10)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[217.196.1.22:from]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-Mailman-Approved-At: Mon, 25 Jan 2021 19:43:09 +0000 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 15:01:02 -0000 LS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCkZyb206IEVsZW5hIE1paGFpbGVzY3UgPGVsZW5h bWloYWlsZXNjdTIyQGdtYWlsLmNvbT4gDQpTZW50OiAyNSBKYW51YXJ5IDIwMjEgMTQ6MjUNClRv OiBNYXR0IENodXJjaHlhcmQgPG1hdHQuY2h1cmNoeWFyZEB1c2VydmUubmV0Pg0KQ2M6IEpvaG4t TWFyayBHdXJuZXkgPGptZ0BmdW5rdGhhdC5jb20+OyBmcmVlYnNkLXZpcnR1YWxpemF0aW9uQGZy ZWVic2Qub3JnDQpTdWJqZWN0OiBSZTogV2FybSBNaWdyYXRpb24gZmVhdHVyZSBmb3IgYmh5dmUg LSByZXZpZXcgb24gUGhhYnJpY2F0b3INCg0KT24gTW9uLCAyNSBKYW4gMjAyMSBhdCAxMzoyNiwg TWF0dCBDaHVyY2h5YXJkIDxtYXR0LmNodXJjaHlhcmRAdXNlcnZlLm5ldD4gd3JvdGU6DQo+DQo+ IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IEpvaG4tTWFyayBHdXJuZXkgPGpt Z0BmdW5rdGhhdC5jb20+DQo+IFNlbnQ6IDI1IEphbnVhcnkgMjAyMSAwNjoyMQ0KPiBUbzogTWF0 dCBDaHVyY2h5YXJkIDxtYXR0LmNodXJjaHlhcmRAdXNlcnZlLm5ldD4NCj4gQ2M6IEVsZW5hIE1p aGFpbGVzY3UgPGVsZW5hbWloYWlsZXNjdTIyQGdtYWlsLmNvbT47IA0KPiBmcmVlYnNkLXZpcnR1 YWxpemF0aW9uQGZyZWVic2Qub3JnDQo+IFN1YmplY3Q6IFJlOiBXYXJtIE1pZ3JhdGlvbiBmZWF0 dXJlIGZvciBiaHl2ZSAtIHJldmlldyBvbiBQaGFicmljYXRvcg0KPg0KPiBNYXR0IENodXJjaHlh cmQgd3JvdGUgdGhpcyBtZXNzYWdlIG9uIEZyaSwgSmFuIDIyLCAyMDIxIGF0IDEwOjA5ICswMDAw Og0KPiA+ID4gSGVsbG8sIGFsbCwNCj4gPg0KPiA+ID4gV2UgaGF2ZSByZWNlbnRseSBvcGVuZWQg YSByZXZpZXcgb24gUGhhYnJpY2F0b3IgZm9yIHRoZSB3YXJtIG1pZ3JhdGlvbiBjb2RlIGZvciA+ IGJoeXZlIFsxXS4gUGxlYXNlIHRha2UgYSBsb29rIGFuZCBsZXQgdXMga25vdyBpZiBpdCBpcyBh bnl0aGluZyB3ZSBjYW4gaW1wcm92ZS4NCj4gPg0KPiA+ID4gWzFdIGh0dHBzOi8vcmV2aWV3cy5m cmVlYnNkLm9yZy9EMjgyNzANCj4gPg0KPiA+ID4gVGhhbmsgeW91LA0KPiA+ID4gRWxlbmENCj4g Pg0KPiA+IEkgYXBwcmVjaWF0ZSB0aGF0IHRoaXMgaXNuJ3QgcmVhbGx5IHJlbGF0ZWQgdG8gdGhl IGN1cnJlbnQgcmV2aWV3LCANCj4gPiBhbmQgY29tbWVuZCB0aGUgd29yayBiZWluZyBwdXQgaW50 byBiaHl2ZSAtIGl0J3MgYW4gaW52YWx1YWJsZSANCj4gPiBhZGRpdGlvbiB0byBGcmVlQlNELiBJ J20ganVzdCB3b25kZXJpbmcgaWYgYW55IHRob3VnaHQgaGFzIGJlZW4gcHV0IA0KPiA+IGludG8g dGhlIGZ1dHVyZSBwb3NzaWJpbGl0eSBmb3IgdHJhbnNmZXIgb2YgZGlzayBkYXRhIGR1cmluZyAN Cj4gPiBtaWdyYXRpb24gKGkuZS4gdGhlIGVxdWl2YWxlbnQgb2YgInN0b3JhZ2Ugdm1vdGlvbiIp DQo+ID4NCj4gPiBUaGUgY3VycmVudCBwcm9jZXNzIChmcm9tIGEgbWlsZSBoaWdoIG92ZXJ2aWV3 KSBlZmZlY3RpdmVseSBzZWVtcyB0byANCj4gPiBiZSB0aGUgZm9sbG93aW5nIC0NCj4gPg0KPiA+ ICogU3RhcnQgZ3Vlc3Qgb24gaG9zdDIgcG9pbnRpbmcgYXQgYSBzaGFyZWQgZGlzaywgYW5kIGhh bHQgYW55IA0KPiA+IGV4ZWN1dGlvbg0KPiA+ICogdXNlIGJoeXZlY3RsIHRvIHBhdXNlIGFuZCBi ZWdpbiBtaWdyYXRpb24gb24gaG9zdDENCj4gPiAqIFN0YXJ0IHRoZSBndWVzdCBvbiBob3N0Mg0K PiA+DQo+ID4gV2hhdCB3b3VsZCBiZSB0aGUgZmVhc2liaWxpdHkgb2YgYmVpbmcgYWJsZSB0byBy dW4gYSBwcm9jZXNzIHN1Y2ggYXMgDQo+ID4gdGhlIGZvbGxvd2luZz8gT2J2aW91c2x5IGl0IHdv dWxkIGxpa2VseSBuZWVkIHRvIGJlIG9yY2hlc3RyYXRlZCBieSANCj4gPiBhbiBleHRlcm5hbCB0 b29sLCBidXQgdG8gbWUgaXQgc2VlbXMgdGhlIG1haW4gcmVxdWlyZW1lbnQgaXMgcmVhbGx5IA0K PiA+IGp1c3QgdG8gYmUgYWJsZSB0byBwcm92aWRlIHNlcGFyYXRlIGNvbnRyb2wgb3ZlciB0aGUg cGF1c2UgYW5kIA0KPiA+IG1pZ3JhdGUgc3RlcHMgb24gaG9zdDEgLQ0KPiA+DQo+ID4gKiBzZW5k IGEgWkZTIHNuYXBzaG90IG9mIHRoZSBydW5uaW5nIG1hY2hpbmUgdG8gaG9zdDINCj4gPiAqIHN0 YXJ0IHRoZSBndWVzdCBpbiBtaWdyYXRlIHJlY3YgbW9kZSBvbiBob3N0Mg0KPiA+ICogcGF1c2Ug dGhlIGd1ZXN0IG9uIGhvc3QxDQo+ID4gKiBzZW5kIGEgbmV3IHNuYXBzaG90DQo+ID4gKiBpbml0 aWF0ZSB0aGUgbWlncmF0aW9uIG9mIG1lbW9yeS9kZXZpY2UgZGF0YQ0KPiA+ICogc3RhcnQgZ3Vl c3Qgb24gaG9zdDINCj4gPg0KPiA+IEFyZSB0aGVyZSBhbnkgbWFqb3IgY29tcGxpY2F0aW9ucyBo ZXJlIEknbSBub3QgYXdhcmUgb2Ygb3RoZXIgdGhhbiB0aGUgcmVxdWlyZW1lbnQgdG8gcGF1c2Ug dGhlIGd1ZXN0IGFuZCBraWNrIG9mZiB0aGUgc3RhdGUgbWlncmF0aW9uIGFzIHR3byBzZXBhcmF0 ZSBjYWxscz8NCj4NCj4gPiBUaGVyZSdzIGFsc28gaGFzdGQgd2hpY2ggY2FuIGFpZCB3aXRoIHRo aXMuLi4NCj4NCj4gVGhhbmtzIGZvciB0aGUgcmVwbHkuIEkndmUgYWx3YXlzIGJlZW4gd2FyeSBv ZiB0aGUgYWRkaXRpb25hbCBjb21wbGV4aXR5IG9mIEhBU1QgYW5kIFpGUywgYXMgaXQgZG9lc24n dCBzZWVtIHRvIGhhdmUgd2lkZXNwcmVhZCB1c2FnZSBvciBzdXBwb3J0LCBhbmQgdGhpbmdzIGdl dCB1Z2x5IGZhc3Qgd2hlbiBzdG9yYWdlIGlzIGludm9sdmVkLg0KPg0KPiBIb3dldmVyLCB0aGUg aWRlYSBvZiB1c2luZyBIQVNUIG9uIHRvcCBvZiB6dm9scyB0byBwcm92aWRlIG5ldHdvcmsgbWly cm9yZWQgc3RvcmFnZSBmb3IgYSBndWVzdCBpcyBpbnRlcmVzdGluZy4gSXQgYWRkcyBhIGxvdCBv ZiBleHRyYSBjb21wbGV4aXR5LCBhbmQgcHJvYmFibHkgcGVyZm9ybWFuY2UgaW1wYWN0IHRob3Vn aCBpZiBpdCdzIGp1c3QgZm9yIHRoZSBhYmlsaXR5IHRvIG1vdmUgYSBndWVzdCBiZXR3ZWVuIHN5 c3RlbXMgdGhhdCBtYXkgb25seSBoYXBwZW4gZXZlcnkgbm93IGFuZCB0aGVuLiBJJ20gYWxzbyBu b3Qgc3VyZSBpdCB3b3VsZCBoZWxwIChvciB3b3VsZCBhdCBsZWFzdCBiZSBldmVuIG1vcmUgY29t cGxleCkgaWYgSSBoYXZlIDQgaG9zdHMgYW5kIHdhbnQgdG8gYmUgYWJsZSB0byBtb3ZlIGd1ZXN0 cyBhbnl3aGVyZS4NCj4NCj4gVGhlIG1haW4gcmVhc29uIGZvciB0aGUgZW1haWwgd2FzIHRvIGV4 cGxvcmUgbXkgdGhlb3J5IHRoYXQsIGJ5IA0KPiBsZXZlcmFnaW5nIFpGUywgYW55IGJoeXZlIHVz ZXIgY291bGQgcm9sbCB0aGVpciBvd24gc3RvcmFnZSBtaWdyYXRpb24gDQo+IGFiaWxpdHkgd2l0 aCBhIGZldyBjb21tYW5kcyBhcyBsb25nIGFzIHRoZSBmb2xsb3dpbmcgdHdvIChzZWVtaW5nbHkg DQo+IG1pbm9yKSBhYmlsaXRpZXMgd2VyZSBwcmVzZW50DQo+DQo+IDEpIHRoZSBhYmlsaXR5IHRv IHN1c3BlbmQgYSBndWVzdCB3aXRob3V0IGl0IGJlaW5nIGRvbmUgYXMgcGFydCBvZiB0aGUgDQo+ IG1pZ3JhdGUgY2FsbC4gKEkgYXNzdW1lIHN1c3BlbmQvcmVzdW1lIHN1cHBvcnQgdmlhIGJoeXZl Y3RsIGlzIHBsYW5uZWQgDQo+IGFueXdheSwgaWYgbm90IGFscmVhZHkgaW4gcGxhY2UgYXQgdGhp cyBwb2ludCkNCj4gMikgbW9kaWZpY2F0aW9uIG9mIHRoZSBtaWdyYXRlIGNhbGwgdG8gc2tpcCBz dXNwZW5kIGlmIHRoZSBndWVzdCBpcyBhbHJlYWR5IHN1c3BlbmRlZC4NCj4NCj4gVGhlIG1haW4g dGhpbmcgSSdtIG5vdCBzdXJlIG9mIGlzIHdoZXRoZXIgdGhlIG1pZ3JhdGUgY2FsbCBoYXMgYW55 IHNwZWNpZmljIHJlbGlhbmNlIG9uIGRvaW5nIHRoZSBzdXNwZW5kIGl0c2VsZiAoZS5nLiBpZiBp dCBuZWVkcyB0byBkbyBhbnl0aGluZyBiZWZvcmUgdGhlIHN1c3BlbmQsIHdoaWNoIHdpbGwgb2J2 aW91c2x5IGJlIHByb2JsZW1hdGljIGlmIHN1c3BlbmQgJiBtaWdyYXRlIGFyZSBjYWxsZWQgc2Vw YXJhdGVseSkuIE9yIGlmIHRoZXJlJ3Mgc29tZXRoaW5nIGVsc2UgSSBtaXNzZWQgdGhhdCBtZWFu cyB0aGlzIGlzIG5vdCBmZWFzaWJsZS4gSSdtIG5vdCByZWFsbHkgYWZ0ZXIgbWFzc2l2ZSBjaGFu Z2VzIHRvIHRoZSBjdXJyZW50IHJldmlldyB0byBpbXBsZW1lbnQgZGlzayBtaWdyYXRpb24gaW4g Ymh5dmUgaXRzZWxmLg0KDQo+IFRoYW5rIHlvdSBmb3IgeW91ciBpbnB1dCByZWxhdGVkIHRvIHRo ZSBkaXNrIG1pZ3JhdGlvbi4gV2Ugd2VyZSAoc3RpbGwgYXJlLCBhY3R1YWxseSkgZm9jdXNpbmcg b24gdGhlIGxpdmUgbWlncmF0aW9uIGZlYXR1cmUgZm9yIGJoeXZlIGFuZCBkaWQgbm90IHRha2Ug aW50byBjb25zaWRlcmF0aW9uIHRoZSBkaXNrIG1pZ3JhdGlvbi4gQXMgPiBNaWhhaSBzYWlkLCB0 aGUgcGF0Y2hlcyBmb3IgdGhlIHR3byAoZ3Vlc3QncyBzdGF0ZSBhbmQgZ3Vlc3QncyBkaXNrIG1p Z3JhdGlvbikgYXJlIG9yIHdpbGwgYmUgcXVpdGUgYmlnIGJ5IHRoZW1zZWx2ZXMgYW5kIHdlIHdh bnQgdGhlIHJldmlldyBwcm9jZXNzIHRvIGdvIGFzIHNtb290aGx5IGFzIHBvc3NpYmxlLg0KDQo+ IEFmdGVyIHdlIGhhdmUgYSBwcmV0dHkgY2xlYXIgdmlldyBvZiB0aGUgbGl2ZSBtaWdyYXRpb24g aW1wbGVtZW50YXRpb24gKGEgcGF0Y2ggb24gdGhpcyBmZWF0dXJlIHdpbGwgYmUgdXBsb2FkZWQg b24gUGhhYnJpY2F0b3IgaW4gYSBjb3VwbGUgb2Ygd2Vla3MpIGFuZCBob3cgd2Ugc2hvdWxkIGlt cHJvdmUgaXQsIHdlIHdpbGwgIGxvb2sgaW50byB0aGUgZGlzayBtaWdyYXRpb24gZmVhdHVyZSBh bmQgaG93IGl0IGNhbiBiZSBpbXBsZW1lbnRlZCBmb3IgYmh5dmUuDQoNCj4gQXMgSSBzYWlkLCBv dXIgb2JqZWN0aXZlIGlzIHRvIGFkZCB0aGUgbGl2ZSBtaWdyYXRpb24gZnVuY3Rpb25hbGl0eSB0 byBiaHl2ZSB3aGljaCBpcyBiYXNlZCBvbiB0aGUgd2FybSBtaWdyYXRpb24gaW1wbGVtZW50YXRp b24gd2hpY2ggaW4gdHVybiBpcyBiYXNlZCBvbiB0aGUgc3VzcGVuZCBvcHRpb24gaW4gYmh5dmUg KHRoZSBzbmFwc2hvdHRpbmcvc3VzcGVuZCBvcHRpb24gd2FzIGFkZGVkIGluIHRoZSB1cHN0cmVh bSBsYXN0IHllYXIsIHRoZSBGcmVlQlNEIGNvZGUgbXVzdCBiZSBjb21waWxlZCB3aXRoIHRoZSBC SFlWRV9TTkFQU0hPVCBvcHRpb24gZm9yIGl0IHRvIHdvcmspLg0KDQo+IEVsZW5hDQoNClRoYW5r cyBmb3IgdGhlIGFkZGl0aW9uYWwgcmVzcG9uc2UuIEkgd2FzIGFjdHVhbGx5IGluIHRoZSBtaWRk bGUgb2Ygd3JpdGluZyBhIGZvbGxvd3VwIG1lc3NhZ2UgdG8gdGhlIG1haWxpbmcgbGlzdCBhcyBJ IG1heSBoYXZlIGNvbmZ1c2VkIHRoaW5ncyBieSB1c2luZyB0aGUgdGVybSAic3RvcmFnZSB2bW90 aW9uIi4gSXQgYXBwZWFycyBpbiB2bXdhcmUgbGFuZCB0aGF0IHJlZmVycyB0byBtb3ZpbmcgZGlz ayBkYXRhIGZvciBhIHJ1bm5pbmcgdm0sIGFuZCBlZmZlY3RpdmVseSBnZXR0aW5nIHRoZSBleGlz dGluZyBydW5uaW5nIHByb2Nlc3MgdG8gc3dpdGNoIHRvIHRoZSBuZXcgcGF0aCwgd2hpY2ggaXMg YSBzbGlnaHRseSBkaWZmZXJlbnQgcHJvY2VzcyBhbmQgb2J2aW91c2x5IGFuIGFkZGl0aW9uYWwg ZmVhdHVyZSBvZiB0aGUgaHlwZXJ2aXNvci4gKGVzcGVjaWFsbHkgaW4gdm13YXJlIHNlZWluZyBh cyBpdCBkb2Vzbid0IHJlbHkgb24gYW55IGJhY2tlbmQgZmlsZSBzeXN0ZW0gbWFuYWdlciBhbmQg aGFuZGxlcyBzZW5kaW5nIGRpc2sgZGF0YSBhbmQgY2hhbmdlZCBibG9ja3MgaXRzZWxmKQ0KDQpJ IHdhcyBzaW1wbHkgaW50cmlndWVkIGJ5IHRoZSBwb3NzaWJpbGl0eSBvZiB1c2luZyB0aGUgY29k ZSB0aGF0IGFscmVhZHkgZXhpc3RzIGZ1bGx5IGluIHRoaXMgcmV2aWV3LCBidXQgd2hldGhlciBp dCB3b3VsZCBiZSBwb3NzaWJsZSB0byBhY3R1YWxseSBtb3ZlIHRoZSBkYXRhIGF0IHRoZSBzYW1l IHRpbWUgYnkgc3F1ZWV6aW5nIGluIGEgemZzIHNlbmQgYmVmb3JlICYgYmV0d2VlbiB0aGUgc3Vz cGVuZC9taWdyYXRlIHN0ZXBzLiAoQWdhaW4gb2J2aW91c2x5IHZpYSBhbiBleHRlcm5hbCB0b29s LCBJIHdvdWxkbid0IGV4cGVjdCBiaHl2ZS9iaHl2ZWN0bCB0byBnZXQgaW52b2x2ZWQgd2l0aCB6 ZnMgY29tbWFuZHMpDQoNCk1hdHQNCg0KPg0KPiBNYXR0DQo+DQo+IC0tDQo+ICAgSm9obi1NYXJr IEd1cm5leSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFZvaWNlOiArMSA0MTUgMjI1IDU1 NzkNCj4NCj4gICAgICAiQWxsIHRoYXQgSSB3aWxsIGRvLCBoYXMgYmVlbiBkb25lLCBBbGwgdGhh dCBJIGhhdmUsIGhhcyBub3QuIg0KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXw0KPiBmcmVlYnNkLXZpcnR1YWxpemF0aW9uQGZyZWVic2Qub3JnIG1haWxp bmcgbGlzdCANCj4gaHR0cHM6Ly9saXN0cy5mcmVlYnNkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2Zy ZWVic2QtdmlydHVhbGl6YXRpb24NCj4gVG8gdW5zdWJzY3JpYmUsIHNlbmQgYW55IG1haWwgdG8g ImZyZWVic2QtdmlydHVhbGl6YXRpb24tdW5zdWJzY3JpYmVAZnJlZWJzZC5vcmciDQo= From owner-freebsd-virtualization@freebsd.org Mon Jan 25 20:03:26 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B10644F01D0 for ; Mon, 25 Jan 2021 20:03:26 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Received: from mail-ot1-x32d.google.com (mail-ot1-x32d.google.com [IPv6:2607:f8b0:4864:20::32d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPgly0b5Qz3GZQ for ; Mon, 25 Jan 2021 20:03:25 +0000 (UTC) (envelope-from rob.fx907@gmail.com) Received: by mail-ot1-x32d.google.com with SMTP id d7so6276127otf.3 for ; Mon, 25 Jan 2021 12:03:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=3gdn7mFoWhigZDV8Nx5hbQCh1UrYQCX67mj7U7mm7M4=; b=BYUEAVnef/VWQUOEHHm5CJrsX28zXcazDIBPlFC0CsAjmY9WL2zFjVh0Dz/D9hysU8 ZRYNVuJfE2LyJtutdkUgDq3/v/EzgpmDHerOp1F8hgtwmxeVBys9iXHHDg5BOtCd215i 28o2UsDcqa81MYqA6JKOLe+EJQSJQzbeOarPcR8jAmX5+ZOYoUMhGzdKE8Dbp38SSr4j iZ2sU6jdfHTS8P3jjCZyG3Ay0oE61J20OeZZbYwWMokHpC4WyKb374R6c1J0dt0cbdLx EuniQmFTPNWw16kS04SQ1Op2E4PMjvxzYOZgkTYjgUGjnK0+RKw1OGAz/Gm08ZRsZWwK D2Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3gdn7mFoWhigZDV8Nx5hbQCh1UrYQCX67mj7U7mm7M4=; b=A/ox3/4oinblEQZPVKkw4FOff09+lGYER5c2QJ7A3StqZlha3A5ZBHtq2mFUgVTW3M k4BZyjX9Evb/ChXDCcv5W2SzfJUzu+KktMr8un5BwxjhIGkvosdHJ1F/kGBEPUaQKEXg eSXUiJISBWvTH2vTO/DKuhbxZ6IhP9wsuotFORqP1O1/GXM5J7UsOCfjB+oNe8kSxojZ 1m6pPuZUEag8Sxp95Ik3Ebj2Y2OMHTmGh753n0vABa+M2CLgy06fFRf6w+P1jKEhJt34 Vg9/YTAhlxSp1hO23xlThEl6ynNYsEicuDA76GMRGlZN3BzLbAzaEkgNZmLdigT509o+ oZsQ== X-Gm-Message-State: AOAM530JaRVtyrpDJ8e6kR1IwgL5ziCh6EU6l0JKaoqQSjG9H4j9tq5X s0YDf0oNJst4T8vzsod2N6vk+V+LhJmclEG0WRSxS8cFGEM= X-Google-Smtp-Source: ABdhPJyi/ce37QPQi/bngYMlwV0m4SbDQbCNPsoo6AymOxNlukdwRgIMB4doyqgsLz5fFzFpdqOj/XzmWT50VFB61Do= X-Received: by 2002:a9d:1425:: with SMTP id h34mr1526156oth.230.1611605004442; Mon, 25 Jan 2021 12:03:24 -0800 (PST) MIME-Version: 1.0 References: <20210125062113.GL31099@funkthat.com> <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> In-Reply-To: From: Rob Wing Date: Mon, 25 Jan 2021 11:02:37 -0900 Message-ID: Subject: Re: Warm Migration feature for bhyve - review on Phabricator To: Matt Churchyard Cc: Elena Mihailescu , "freebsd-virtualization@freebsd.org" X-Rspamd-Queue-Id: 4DPgly0b5Qz3GZQ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=BYUEAVne; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of robfx907@gmail.com designates 2607:f8b0:4864:20::32d as permitted sender) smtp.mailfrom=robfx907@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::32d:from]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::32d:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32d:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 20:03:26 -0000 The experimental suspend/resume feature for bhyve exists, but isn't turned on by default. Also, in my testing, it only works with intel gear. I understand the migration feature is a wrapper around the suspend/resume functionality. A question, is the migration feature dependent upon the underlying file format of suspend/resume? If so, something to consider is that there will be additional work to fix the migration feature after the suspend/resume file format is dialed in. My concern is that building experimental features on top of experimental features could become problematic when the foundation of the experimental features is changed. When suspend/resume was brought in, the commit log made it very explicit that the file format needs to be fleshed out a bit more before being enabled by default. -Rob p.s. part of me thinks some of this should happen at the process level (i.e. suspend the bhyve process, migrate it, and resume it elsewhere) - easier said than done. On Mon, Jan 25, 2021 at 10:43 AM Matt Churchyard wrote: > -----Original Message----- > From: Elena Mihailescu > Sent: 25 January 2021 14:25 > To: Matt Churchyard > Cc: John-Mark Gurney ; > freebsd-virtualization@freebsd.org > Subject: Re: Warm Migration feature for bhyve - review on Phabricator > > On Mon, 25 Jan 2021 at 13:26, Matt Churchyard > wrote: > > > > -----Original Message----- > > From: John-Mark Gurney > > Sent: 25 January 2021 06:21 > > To: Matt Churchyard > > Cc: Elena Mihailescu ; > > freebsd-virtualization@freebsd.org > > Subject: Re: Warm Migration feature for bhyve - review on Phabricator > > > > Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000: > > > > Hello, all, > > > > > > > We have recently opened a review on Phabricator for the warm > migration code for > bhyve [1]. Please take a look and let us know if it is > anything we can improve. > > > > > > > [1] https://reviews.freebsd.org/D28270 > > > > > > > Thank you, > > > > Elena > > > > > > I appreciate that this isn't really related to the current review, > > > and commend the work being put into bhyve - it's an invaluable > > > addition to FreeBSD. I'm just wondering if any thought has been put > > > into the future possibility for transfer of disk data during > > > migration (i.e. the equivalent of "storage vmotion") > > > > > > The current process (from a mile high overview) effectively seems to > > > be the following - > > > > > > * Start guest on host2 pointing at a shared disk, and halt any > > > execution > > > * use bhyvectl to pause and begin migration on host1 > > > * Start the guest on host2 > > > > > > What would be the feasibility of being able to run a process such as > > > the following? Obviously it would likely need to be orchestrated by > > > an external tool, but to me it seems the main requirement is really > > > just to be able to provide separate control over the pause and > > > migrate steps on host1 - > > > > > > * send a ZFS snapshot of the running machine to host2 > > > * start the guest in migrate recv mode on host2 > > > * pause the guest on host1 > > > * send a new snapshot > > > * initiate the migration of memory/device data > > > * start guest on host2 > > > > > > Are there any major complications here I'm not aware of other than the > requirement to pause the guest and kick off the state migration as two > separate calls? > > > > > There's also hastd which can aid with this... > > > > Thanks for the reply. I've always been wary of the additional complexity > of HAST and ZFS, as it doesn't seem to have widespread usage or support, > and things get ugly fast when storage is involved. > > > > However, the idea of using HAST on top of zvols to provide network > mirrored storage for a guest is interesting. It adds a lot of extra > complexity, and probably performance impact though if it's just for the > ability to move a guest between systems that may only happen every now and > then. I'm also not sure it would help (or would at least be even more > complex) if I have 4 hosts and want to be able to move guests anywhere. > > > > The main reason for the email was to explore my theory that, by > > leveraging ZFS, any bhyve user could roll their own storage migration > > ability with a few commands as long as the following two (seemingly > > minor) abilities were present > > > > 1) the ability to suspend a guest without it being done as part of the > > migrate call. (I assume suspend/resume support via bhyvectl is planned > > anyway, if not already in place at this point) > > 2) modification of the migrate call to skip suspend if the guest is > already suspended. > > > > The main thing I'm not sure of is whether the migrate call has any > specific reliance on doing the suspend itself (e.g. if it needs to do > anything before the suspend, which will obviously be problematic if suspend > & migrate are called separately). Or if there's something else I missed > that means this is not feasible. I'm not really after massive changes to > the current review to implement disk migration in bhyve itself. > > > Thank you for your input related to the disk migration. We were (still > are, actually) focusing on the live migration feature for bhyve and did not > take into consideration the disk migration. As > Mihai said, the patches > for the two (guest's state and guest's disk migration) are or will be quite > big by themselves and we want the review process to go as smoothly as > possible. > > > After we have a pretty clear view of the live migration implementation > (a patch on this feature will be uploaded on Phabricator in a couple of > weeks) and how we should improve it, we will look into the disk migration > feature and how it can be implemented for bhyve. > > > As I said, our objective is to add the live migration functionality to > bhyve which is based on the warm migration implementation which in turn is > based on the suspend option in bhyve (the snapshotting/suspend option was > added in the upstream last year, the FreeBSD code must be compiled with the > BHYVE_SNAPSHOT option for it to work). > > > Elena > > Thanks for the additional response. I was actually in the middle of > writing a followup message to the mailing list as I may have confused > things by using the term "storage vmotion". It appears in vmware land that > refers to moving disk data for a running vm, and effectively getting the > existing running process to switch to the new path, which is a slightly > different process and obviously an additional feature of the hypervisor. > (especially in vmware seeing as it doesn't rely on any backend file system > manager and handles sending disk data and changed blocks itself) > > I was simply intrigued by the possibility of using the code that already > exists fully in this review, but whether it would be possible to actually > move the data at the same time by squeezing in a zfs send before & between > the suspend/migrate steps. (Again obviously via an external tool, I > wouldn't expect bhyve/bhyvectl to get involved with zfs commands) > > Matt > > > > > Matt > > > > -- > > John-Mark Gurney Voice: +1 415 225 5579 > > > > "All that I will do, has been done, All that I have, has not." > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > From owner-freebsd-virtualization@freebsd.org Mon Jan 25 20:32:17 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 007E54F0DE4 for ; Mon, 25 Jan 2021 20:32:17 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPhPD2W9Dz3JZx for ; Mon, 25 Jan 2021 20:32:16 +0000 (UTC) (envelope-from elenamihailescu22@gmail.com) Received: by mail-ed1-x52b.google.com with SMTP id f1so17082243edr.12 for ; Mon, 25 Jan 2021 12:32:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=H1/4szci1U0EnIoVRD8qWxKdthoKcVx1qM1Xx5rcj54=; b=bSdjhpc6vcAC+wtzEr2VxonHqsMddBP/17zqqBSvpCrgM/pkTvh0YP746jrdzC7Zmx jL6WM9MDJyXaA+xcnFjR1pk1WoYHLLINFzkkb3G44nig3CWO2vEpOHBS+FZ1NCDeC+Sd IVj5hIBZG+FbHuAf/KFZDxJObZs/s+Ohq5WevhQvS7lmK76873hGXb5Dvilif4SIbIJJ 1zNuqzr/wp9ioiDAGQiUztrqIXEHwbqccxLlaemu6QptiBr2onpAOFK0qHGb77hhugT9 EsI0ov8pCXjYd6SrjRjtGAIhtoEZJDQen8kRzlMF7BVib+YcwtAoPVSrICP87LyNtsU7 X3/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=H1/4szci1U0EnIoVRD8qWxKdthoKcVx1qM1Xx5rcj54=; b=NP+pCjtZt6m3wXgga2hQm/CI6/NuJ3G/ZhG+awUS1LOIDGj/JwpkyDzY3VVNtB76DR 2X011IYOmAQcS3IWrkTZk8EM0evlEughtlZl/ab8H/yPc1MCQ1oefHj2BJbIQk4E2+Vi btBzreSBR5OfFakYUin0QxQ+SUhjxbkvSsA46WYItH5SQPd5EZfxQL4PWmicM075oBxt bqzTZkB+Qcp4Vc6wASiUlv095C6RI4aUqO2zO9NCe56se9M/ixaa3GN5ihjcx3oYpTZa S0tvDiwuchiPWCsY4yzsnAtkFmw25/GLt511eXooRE8fQ/Vi04H1Jq2MWfaDUiTCMUo5 +4FQ== X-Gm-Message-State: AOAM532bCP658r+2JiQDvCp1nusikj7dw550haxwbsq2I+WDX8jEvVxf 0C9LlSpneZUGMy18z0+emzUaeiZYAKphIZcYSCoAL/BDecU= X-Google-Smtp-Source: ABdhPJxnjyzJmTlhwaVy7y5Ti+EJw1TanRXeVIpnW1VvXyFxgx1gxLEyG7maOCIOZe6qQw31J+vopkAoc0uGytOivgc= X-Received: by 2002:a50:9ee9:: with SMTP id a96mr1934769edf.343.1611606733981; Mon, 25 Jan 2021 12:32:13 -0800 (PST) MIME-Version: 1.0 References: <20210125062113.GL31099@funkthat.com> <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> In-Reply-To: From: Elena Mihailescu Date: Mon, 25 Jan 2021 22:31:47 +0200 Message-ID: Subject: Re: Warm Migration feature for bhyve - review on Phabricator To: Rob Wing Cc: Matt Churchyard , "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4DPhPD2W9Dz3JZx X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=bSdjhpc6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of elenamihailescu22@gmail.com designates 2a00:1450:4864:20::52b as permitted sender) smtp.mailfrom=elenamihailescu22@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::52b:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::52b:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52b:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 20:32:17 -0000 Indeed, the save/restore functionality works only for Intel CPUs and it's experimental. We are working to extend the save/restore functionality so it can be enabled by default in the future. We are working on the file format, too. The warm migration is a wrapper on the save/restore functionality, indeed. However, it doesn't use files and sends the guest's state (memory, kernel structures, emulated devices) over the network, using a socket. Basically, the work does the following on the source host: pause the guest state, use the save mechanism to extract the guest's state, send the state though the network. On the destination host, we have the reverse process: receive the guest's state, use the restore mechanism to restore the guest's state, resume the guest. Beside the guest's state, we check whether the source and destination hosts are compatible (e.g. they have to have the same CPU, otherwise, the resume on the destination host will fail). Elena On Mon, 25 Jan 2021 at 22:03, Rob Wing wrote: > > The experimental suspend/resume feature for bhyve exists, but isn't turne= d on by default. Also, in my testing, it only works with intel gear. > > I understand the migration feature is a wrapper around the suspend/resume= functionality. > > A question, is the migration feature dependent upon the underlying file f= ormat of suspend/resume? If so, something to consider is that there will be= additional work to fix the migration feature after the suspend/resume file= format is dialed in. > > My concern is that building experimental features on top of experimental = features could become problematic when the foundation > of the experimental features is changed. > > When suspend/resume was brought in, the commit log made it very explicit = that the file format needs to be fleshed out a bit more before being enable= d by default. > > -Rob > > p.s. > > part of me thinks some of this should happen at the process level (i.e. s= uspend the bhyve process, migrate it, and resume it elsewhere) - easier sai= d than done. > > On Mon, Jan 25, 2021 at 10:43 AM Matt Churchyard wrote: >> >> -----Original Message----- >> From: Elena Mihailescu >> Sent: 25 January 2021 14:25 >> To: Matt Churchyard >> Cc: John-Mark Gurney ; freebsd-virtualization@freebsd.= org >> Subject: Re: Warm Migration feature for bhyve - review on Phabricator >> >> On Mon, 25 Jan 2021 at 13:26, Matt Churchyard wrote: >> > >> > -----Original Message----- >> > From: John-Mark Gurney >> > Sent: 25 January 2021 06:21 >> > To: Matt Churchyard >> > Cc: Elena Mihailescu ; >> > freebsd-virtualization@freebsd.org >> > Subject: Re: Warm Migration feature for bhyve - review on Phabricator >> > >> > Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000= : >> > > > Hello, all, >> > > >> > > > We have recently opened a review on Phabricator for the warm migra= tion code for > bhyve [1]. Please take a look and let us know if it is anyt= hing we can improve. >> > > >> > > > [1] https://reviews.freebsd.org/D28270 >> > > >> > > > Thank you, >> > > > Elena >> > > >> > > I appreciate that this isn't really related to the current review, >> > > and commend the work being put into bhyve - it's an invaluable >> > > addition to FreeBSD. I'm just wondering if any thought has been put >> > > into the future possibility for transfer of disk data during >> > > migration (i.e. the equivalent of "storage vmotion") >> > > >> > > The current process (from a mile high overview) effectively seems to >> > > be the following - >> > > >> > > * Start guest on host2 pointing at a shared disk, and halt any >> > > execution >> > > * use bhyvectl to pause and begin migration on host1 >> > > * Start the guest on host2 >> > > >> > > What would be the feasibility of being able to run a process such as >> > > the following? Obviously it would likely need to be orchestrated by >> > > an external tool, but to me it seems the main requirement is really >> > > just to be able to provide separate control over the pause and >> > > migrate steps on host1 - >> > > >> > > * send a ZFS snapshot of the running machine to host2 >> > > * start the guest in migrate recv mode on host2 >> > > * pause the guest on host1 >> > > * send a new snapshot >> > > * initiate the migration of memory/device data >> > > * start guest on host2 >> > > >> > > Are there any major complications here I'm not aware of other than t= he requirement to pause the guest and kick off the state migration as two s= eparate calls? >> > >> > > There's also hastd which can aid with this... >> > >> > Thanks for the reply. I've always been wary of the additional complexi= ty of HAST and ZFS, as it doesn't seem to have widespread usage or support,= and things get ugly fast when storage is involved. >> > >> > However, the idea of using HAST on top of zvols to provide network mir= rored storage for a guest is interesting. It adds a lot of extra complexity= , and probably performance impact though if it's just for the ability to mo= ve a guest between systems that may only happen every now and then. I'm als= o not sure it would help (or would at least be even more complex) if I have= 4 hosts and want to be able to move guests anywhere. >> > >> > The main reason for the email was to explore my theory that, by >> > leveraging ZFS, any bhyve user could roll their own storage migration >> > ability with a few commands as long as the following two (seemingly >> > minor) abilities were present >> > >> > 1) the ability to suspend a guest without it being done as part of the >> > migrate call. (I assume suspend/resume support via bhyvectl is planned >> > anyway, if not already in place at this point) >> > 2) modification of the migrate call to skip suspend if the guest is al= ready suspended. >> > >> > The main thing I'm not sure of is whether the migrate call has any spe= cific reliance on doing the suspend itself (e.g. if it needs to do anything= before the suspend, which will obviously be problematic if suspend & migra= te are called separately). Or if there's something else I missed that means= this is not feasible. I'm not really after massive changes to the current = review to implement disk migration in bhyve itself. >> >> > Thank you for your input related to the disk migration. We were (still= are, actually) focusing on the live migration feature for bhyve and did no= t take into consideration the disk migration. As > Mihai said, the patches = for the two (guest's state and guest's disk migration) are or will be quite= big by themselves and we want the review process to go as smoothly as poss= ible. >> >> > After we have a pretty clear view of the live migration implementation= (a patch on this feature will be uploaded on Phabricator in a couple of we= eks) and how we should improve it, we will look into the disk migration fe= ature and how it can be implemented for bhyve. >> >> > As I said, our objective is to add the live migration functionality to= bhyve which is based on the warm migration implementation which in turn is= based on the suspend option in bhyve (the snapshotting/suspend option was = added in the upstream last year, the FreeBSD code must be compiled with the= BHYVE_SNAPSHOT option for it to work). >> >> > Elena >> >> Thanks for the additional response. I was actually in the middle of writ= ing a followup message to the mailing list as I may have confused things by= using the term "storage vmotion". It appears in vmware land that refers to= moving disk data for a running vm, and effectively getting the existing ru= nning process to switch to the new path, which is a slightly different proc= ess and obviously an additional feature of the hypervisor. (especially in v= mware seeing as it doesn't rely on any backend file system manager and hand= les sending disk data and changed blocks itself) >> >> I was simply intrigued by the possibility of using the code that already= exists fully in this review, but whether it would be possible to actually = move the data at the same time by squeezing in a zfs send before & between = the suspend/migrate steps. (Again obviously via an external tool, I wouldn'= t expect bhyve/bhyvectl to get involved with zfs commands) >> >> Matt >> >> > >> > Matt >> > >> > -- >> > John-Mark Gurney Voice: +1 415 225 5579 >> > >> > "All that I will do, has been done, All that I have, has not." >> > _______________________________________________ >> > freebsd-virtualization@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@f= reebsd.org" >> _______________________________________________ >> freebsd-virtualization@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fre= ebsd.org" From owner-freebsd-virtualization@freebsd.org Tue Jan 26 15:09:42 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BD6EA4F25C8 for ; Tue, 26 Jan 2021 15:09:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DQ9BZ4pJNz4WKW for ; Tue, 26 Jan 2021 15:09:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id A4D644F25C7; Tue, 26 Jan 2021 15:09:42 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A4A004F278C for ; Tue, 26 Jan 2021 15:09:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQ9BZ4990z4WQF for ; Tue, 26 Jan 2021 15:09:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 81EF237AF for ; Tue, 26 Jan 2021 15:09:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10QF9gFJ057481 for ; Tue, 26 Jan 2021 15:09:42 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10QF9gq4057480 for virtualization@FreeBSD.org; Tue, 26 Jan 2021 15:09:42 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 253004] Win 10 guest fails to restart unless I --destroy Date: Tue, 26 Jan 2021 15:09:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: bergerkos@yahoo.co.uk X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jan 2021 15:09:42 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253004 --- Comment #2 from bergerkos@yahoo.co.uk --- Can't tell whether or not this is a duplicate of bug #242666. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Jan 27 03:59:03 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D93B44EBB4E for ; Wed, 27 Jan 2021 03:59:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DQVGH5cm1z4tnJ for ; Wed, 27 Jan 2021 03:59:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id C0E4F4EBB4D; Wed, 27 Jan 2021 03:59:03 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C0AC24EBDBA for ; Wed, 27 Jan 2021 03:59:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQVGH51Myz4tvB for ; Wed, 27 Jan 2021 03:59:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9F45D161A4 for ; Wed, 27 Jan 2021 03:59:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10R3x3kv080653 for ; Wed, 27 Jan 2021 03:59:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10R3x3CK080652 for virtualization@FreeBSD.org; Wed, 27 Jan 2021 03:59:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 168298] VirtualBox using AIO on a zvol crashes Date: Wed, 27 Jan 2021 03:58:59 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: hjf@mailinator.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Works As Intended X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2021 03:59:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D168298 hjf changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hjf@mailinator.com --- Comment #39 from hjf --- (In reply to ml from comment #38) Are you able to build this port on 12.2? I recently upgraded and can't roll back to the 5.2.34 version I was using. I tried portdowngrade to revert to r549922 but I get all sorts of error compiling. I think this is the problem I'm having. the VM works but it reports so many disk errors it's simply unable to boot. Back in 12.1-RELEASE I installed 5.2.34_4 pkg's cache, and it worked. But I can't do the same with 12.2-RELE= ASE --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Jan 27 07:30:21 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7248A4EF6EB for ; Wed, 27 Jan 2021 07:30:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DQZy52ZdMz3LGW for ; Wed, 27 Jan 2021 07:30:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 581B94EF6EA; Wed, 27 Jan 2021 07:30:21 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 57E384EFBBC for ; Wed, 27 Jan 2021 07:30:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQZy520JKz3LLr for ; Wed, 27 Jan 2021 07:30:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3716718778 for ; Wed, 27 Jan 2021 07:30:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10R7ULZN093440 for ; Wed, 27 Jan 2021 07:30:21 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10R7ULXA093439 for virtualization@FreeBSD.org; Wed, 27 Jan 2021 07:30:21 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 168298] VirtualBox using AIO on a zvol crashes Date: Wed, 27 Jan 2021 07:30:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ml@netfence.it X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Works As Intended X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2021 07:30:21 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D168298 --- Comment #40 from ml@netfence.it --- (In reply to hjf from comment #39) Yes, I'm able to build it (using Poudriere), with or without disabling AIO. However, as I said in comment #38, the latest official version does not work for me. Reenabling AIO is much better (I occasionally get AHCI timeouts in a FreeBSD guest, but that's fine as I don't use vbox for anything too serious, at lea= st now; also, it probably did before, I'm not sure). If you are having the problem I had (not sure, you don't tell), instead of reverting to the previous version, you might just try the latest without disabling AIO: use the following: # svn diff Index: files/patch-src-VBox-Runtime-Makefile.kmk =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- files/patch-src-VBox-Runtime-Makefile.kmk (revision 563032) +++ files/patch-src-VBox-Runtime-Makefile.kmk (working copy) @@ -12,12 +12,3 @@ # Unicode Specification reader used to regenerate unidata-*.cpp. # uniread_TEMPLATE =3D VBoxBldProg -@@ -1632,7 +1637,7 @@ VBoxRT_SOURCES.solaris +=3D \ - VBoxRT_SOURCES.darwin +=3D \ - r3/posix/fileaio-posix.cpp - VBoxRT_SOURCES.freebsd +=3D \ -- r3/freebsd/fileaio-freebsd.cpp -+ r3/posix/fileaio-posix.cpp - VBoxRT_INCS :=3D $(RuntimeR3_INCS) - VBoxRT_INCS.$(KBUILD_TARGET) :=3D $(RuntimeR3_INCS.$(KBUILD_TARGET)) - VBoxRT_INCS.$(KBUILD_TARGET).$(KBUILD_TARGET_ARCH) :=3D $(RuntimeR3_INCS.$(KBUILD_TARGET).$(KBUILD_TARGET_ARCH)) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Jan 27 13:54:26 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 990764F9DE5 for ; Wed, 27 Jan 2021 13:54:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DQlTG3lYKz4XxS for ; Wed, 27 Jan 2021 13:54:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 809FD4F9CEA; Wed, 27 Jan 2021 13:54:26 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8068B4FA1FD for ; Wed, 27 Jan 2021 13:54:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQlTG38Zrz4Xb3 for ; Wed, 27 Jan 2021 13:54:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 5F54D1DC23 for ; Wed, 27 Jan 2021 13:54:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10RDsQT1096752 for ; Wed, 27 Jan 2021 13:54:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10RDsQ1M096751 for virtualization@FreeBSD.org; Wed, 27 Jan 2021 13:54:26 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 168298] VirtualBox using AIO on a zvol crashes Date: Wed, 27 Jan 2021 13:54:26 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: hjf@mailinator.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Works As Intended X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2021 13:54:26 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D168298 --- Comment #41 from hjf --- (In reply to ml from comment #40) Thank you! This patch worked for me. I can use my VMs again For more detail: The Linux VMs were backed on a ZFS ZVOL. Since the upgrade to .44, I got re= ad errors. The sort of error Linux gives you on a bad disk. ATA timeouts, etc.= The errors were similar to: failed command: READ DMA EXT, status DRDY ERR , err= or IDNF. After applying the patch you provided, everything is working again. The aio kernel tuning options discussed in this thread did nothing for my c= ase. Can we reopen this bug? This port is obviously broken for people using zvol= s. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Jan 27 14:00:56 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7D5224F9FEF for ; Wed, 27 Jan 2021 14:00:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DQlcm2xrzz4YB6 for ; Wed, 27 Jan 2021 14:00:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 62FB94FA21A; Wed, 27 Jan 2021 14:00:56 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 62BDA4FA066 for ; Wed, 27 Jan 2021 14:00:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQlcm2FwRz4Xtg for ; Wed, 27 Jan 2021 14:00:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 40DB51DC9E for ; Wed, 27 Jan 2021 14:00:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10RE0uv7098032 for ; Wed, 27 Jan 2021 14:00:56 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10RE0uHk098031 for virtualization@FreeBSD.org; Wed, 27 Jan 2021 14:00:56 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 168298] VirtualBox using AIO on a zvol crashes Date: Wed, 27 Jan 2021 14:00:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pete@twisted.org.uk X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Works As Intended X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2021 14:00:56 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D168298 --- Comment #42 from Pete French --- !!! I actually saw this effect when I upgraded. My Windows 10 VM booted fine after the 12.2 upgrade, but my OpenSUSE install just crashed. I ended up deleting it and installing Ubuntu instead, using the setting which worked f= or the Windows VM, as I assumed it was my inexperience with Linux, not a reoccurrence of a bug in the port. Damn, wish I had said something at the t= ime now. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Jan 27 15:51:25 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0320C4FCBA6 for ; Wed, 27 Jan 2021 15:51:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DQp4D6VJ8z4gT8 for ; Wed, 27 Jan 2021 15:51:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id DEE404FCA99; Wed, 27 Jan 2021 15:51:24 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DEAFC4FC910 for ; Wed, 27 Jan 2021 15:51:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQp4D5vPhz4gHy for ; Wed, 27 Jan 2021 15:51:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BDE9B1F510 for ; Wed, 27 Jan 2021 15:51:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10RFpOaX057704 for ; Wed, 27 Jan 2021 15:51:24 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10RFpO7h057703 for virtualization@FreeBSD.org; Wed, 27 Jan 2021 15:51:24 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 168298] VirtualBox using AIO on a zvol crashes Date: Wed, 27 Jan 2021 15:51:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: hjf@mailinator.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Works As Intended X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2021 15:51:25 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D168298 --- Comment #43 from hjf --- (In reply to Pete French from comment #42) Are you also using a zvol as the backing storage for your VM? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Jan 27 16:02:14 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E7E4A4FCC40 for ; Wed, 27 Jan 2021 16:02:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DQpJk63GDz4hdT for ; Wed, 27 Jan 2021 16:02:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id CF9A14FCE23; Wed, 27 Jan 2021 16:02:14 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CF5834FC97B for ; Wed, 27 Jan 2021 16:02:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQpJk5Rkkz4hpS for ; Wed, 27 Jan 2021 16:02:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AE1251F5D3 for ; Wed, 27 Jan 2021 16:02:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10RG2Exs062261 for ; Wed, 27 Jan 2021 16:02:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10RG2EtR062260 for virtualization@FreeBSD.org; Wed, 27 Jan 2021 16:02:14 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 168298] VirtualBox using AIO on a zvol crashes Date: Wed, 27 Jan 2021 16:02:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: pete@twisted.org.uk X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Works As Intended X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2021 16:02:15 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D168298 --- Comment #44 from Pete French --- (In reply to hjf from comment #43) Yes - indeed, I am the original reporter of this bug ;-) --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Jan 27 16:37:19 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 163974FDA14 for ; Wed, 27 Jan 2021 16:37:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DQq5B725rz4kC0 for ; Wed, 27 Jan 2021 16:37:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id EF96E4FDA13; Wed, 27 Jan 2021 16:37:18 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF5C04FD560 for ; Wed, 27 Jan 2021 16:37:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQq5B6JJFz4kBy for ; Wed, 27 Jan 2021 16:37:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C9C891FBDE for ; Wed, 27 Jan 2021 16:37:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10RGbIFI083133 for ; Wed, 27 Jan 2021 16:37:18 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10RGbIOt083132 for virtualization@FreeBSD.org; Wed, 27 Jan 2021 16:37:18 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 168298] VirtualBox using AIO on a zvol crashes Date: Wed, 27 Jan 2021 16:37:17 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: ml@netfence.it X-Bugzilla-Status: Closed X-Bugzilla-Resolution: Works As Intended X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2021 16:37:19 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D168298 --- Comment #45 from ml@netfence.it --- (In reply to hjf from comment #41) Notice I'm not using zvols, so it's not "obviously broken" only "for people using zvols". --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Jan 28 00:33:45 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 073054EED61 for ; Thu, 28 Jan 2021 00:33:45 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router10G.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DR1fv3BFpz3ql4 for ; Thu, 28 Jan 2021 00:33:42 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router10g.digiware.nl (localhost.digiware.nl [127.0.0.1]) by router10G.digiware.nl (Postfix) with ESMTP id 3664A35A02; Thu, 28 Jan 2021 01:33:35 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from router10G.digiware.nl ([127.0.0.1]) by router10g.digiware.nl (router10g.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 863U2h6diqJr; Thu, 28 Jan 2021 01:33:34 +0100 (CET) Received: from [192.168.11.254] (WJW-LAPTOP.digiware.nl [192.168.11.254]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by router10G.digiware.nl (Postfix) with ESMTPSA id F420935839; Thu, 28 Jan 2021 01:33:33 +0100 (CET) Subject: Re: Warm Migration feature for bhyve - review on Phabricator To: Matt Churchyard , Elena Mihailescu , "freebsd-virtualization@freebsd.org" References: From: Willem Jan Withagen Message-ID: <1c3ca65f-ac65-a710-7775-e0865abfe954@digiware.nl> Date: Thu, 28 Jan 2021 01:33:40 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.12.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4DR1fv3BFpz3ql4 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-2.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; HFILTER_HELO_IP_A(1.00)[router10g.digiware.nl]; HFILTER_HELO_NORES_A_OR_MX(0.30)[router10g.digiware.nl]; RCVD_COUNT_THREE(0.00)[4]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[userve.net,gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[176.74.240.9:from]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[digiware.nl]; NEURAL_HAM_LONG(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[176.74.240.9:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jan 2021 00:33:45 -0000 On 22-1-2021 11:09, Matt Churchyard wrote: >> Hello, all, >> We have recently opened a review on Phabricator for the warm migration code for > bhyve [1]. Please take a look and let us know if it is anything we can improve. >> [1] https://reviews.freebsd.org/D28270 >> Thank you, >> Elena > I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") > > The current process (from a mile high overview) effectively seems to be the following - > > * Start guest on host2 pointing at a shared disk, and halt any execution > * use bhyvectl to pause and begin migration on host1 > * Start the guest on host2 > > What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - > > * send a ZFS snapshot of the running machine to host2 > * start the guest in migrate recv mode on host2 > * pause the guest on host1 > * send a new snapshot > * initiate the migration of memory/device data > * start guest on host2 > > Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? > > Shared storage is great but becomes complex and expensive if you want high performance and reliability (seeing as the storage quickly becomes a major single point of failure without enterprise active/active kit). I suspect the most common deployment of bhyve is independent hosts with local ZFS pools as this is easy/cheap and gives great performance. Most hypervisors only had "shared storage" migration for a long time but the big ones now also support transferring disk data live. It would be great to be able to do this out of the gate. > Great work to see this landing. This is actually one of the reasons I started porting Ceph and build a bhyve backend that interfaces to Ceph RBD. So you do not need to migrate the disk data. You are right that Ceph does not come cheap in hardware, but the designers try to jump through hoops to make it reliable. And it will also take some performance, but I'm very happy running my FreeBSD-vm's on our Openstack/Ceph cluster. Warm migration has been on the horizon for quite some time, so I very happy to see it getting there. Unfortunately I have limited time ATM to actually see if warm migration could be made to work with Ceph a diskstorage. --WjW From owner-freebsd-virtualization@freebsd.org Thu Jan 28 00:41:57 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 599324EEDA5 for ; Thu, 28 Jan 2021 00:41:57 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router10G.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DR1rN2xyvz3r2b for ; Thu, 28 Jan 2021 00:41:56 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router10g.digiware.nl (localhost.digiware.nl [127.0.0.1]) by router10G.digiware.nl (Postfix) with ESMTP id 27BDD35915; Thu, 28 Jan 2021 01:41:55 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from router10G.digiware.nl ([127.0.0.1]) by router10g.digiware.nl (router10g.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VJjiJ-O9Hlrx; Thu, 28 Jan 2021 01:41:54 +0100 (CET) Received: from [192.168.11.254] (WJW-LAPTOP.digiware.nl [192.168.11.254]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by router10G.digiware.nl (Postfix) with ESMTPSA id 1D3BF35765; Thu, 28 Jan 2021 01:41:54 +0100 (CET) Subject: Re: Warm Migration feature for bhyve - review on Phabricator To: Matt Churchyard , "freebsd-virtualization@freebsd.org" References: <20210125062113.GL31099@funkthat.com> <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> <20210125184204.GM31099@funkthat.com> From: Willem Jan Withagen Message-ID: <5c6ff016-e15d-afad-e855-8cfed32276f2@digiware.nl> Date: Thu, 28 Jan 2021 01:42:00 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0 Thunderbird/68.12.1 MIME-Version: 1.0 In-Reply-To: <20210125184204.GM31099@funkthat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4DR1rN2xyvz3r2b X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of wjw@digiware.nl designates 176.74.240.9 as permitted sender) smtp.mailfrom=wjw@digiware.nl X-Spamd-Result: default: False [-2.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; HFILTER_HELO_IP_A(1.00)[router10g.digiware.nl]; RBL_DBL_DONT_QUERY_IPS(0.00)[176.74.240.9:from]; HFILTER_HELO_NORES_A_OR_MX(0.30)[router10g.digiware.nl]; SPAMHAUS_ZRD(0.00)[176.74.240.9:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[digiware.nl]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:28878, ipnet:176.74.224.0/19, country:NL]; RCVD_TLS_LAST(0.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jan 2021 00:41:57 -0000 On 25-1-2021 19:42, John-Mark Gurney wrote: > Matt Churchyard wrote this message on Mon, Jan 25, 2021 at 10:46 +0000: >> -----Original Message----- >> From: John-Mark Gurney >> Sent: 25 January 2021 06:21 >> To: Matt Churchyard >> Cc: Elena Mihailescu ; freebsd-virtualization@freebsd.org >> Subject: Re: Warm Migration feature for bhyve - review on Phabricator >> >> Matt Churchyard wrote this message on Fri, Jan 22, 2021 at 10:09 +0000: >>>> Hello, all, >>>> We have recently opened a review on Phabricator for the warm migration code for > bhyve [1]. Please take a look and let us know if it is anything we can improve. >>>> [1] https://reviews.freebsd.org/D28270 >>>> Thank you, >>>> Elena >>> I appreciate that this isn't really related to the current review, and commend the work being put into bhyve - it's an invaluable addition to FreeBSD. I'm just wondering if any thought has been put into the future possibility for transfer of disk data during migration (i.e. the equivalent of "storage vmotion") >>> >>> The current process (from a mile high overview) effectively seems to be the following - >>> >>> * Start guest on host2 pointing at a shared disk, and halt any execution >>> * use bhyvectl to pause and begin migration on host1 >>> * Start the guest on host2 >>> >>> What would be the feasibility of being able to run a process such as the following? Obviously it would likely need to be orchestrated by an external tool, but to me it seems the main requirement is really just to be able to provide separate control over the pause and migrate steps on host1 - >>> >>> * send a ZFS snapshot of the running machine to host2 >>> * start the guest in migrate recv mode on host2 >>> * pause the guest on host1 >>> * send a new snapshot >>> * initiate the migration of memory/device data >>> * start guest on host2 >>> >>> Are there any major complications here I'm not aware of other than the requirement to pause the guest and kick off the state migration as two separate calls? >>> There's also hastd which can aid with this... >> Thanks for the reply. I've always been wary of the additional complexity of HAST and ZFS, as it doesn't seem to have widespread usage or support, and things get ugly fast when storage is involved. > Totally agree... > >> However, the idea of using HAST on top of zvols to provide network mirrored storage for a guest is interesting. It adds a lot of extra complexity, and probably performance impact though if it's just for the ability to move a guest between systems that may only happen every now and then. I'm also not sure it would help (or would at least be even more complex) if I have 4 hosts and want to be able to move guests anywhere. > gmirror + ggate is another option as well... I did try all kinds of these solutions that are based on a 2 system backend storage, and it was relatively easy for me to get the state of the storage in a split-brain situation which is not really repairable without a lot of manual user intervention. So I gave up on HAST and gmirror over ggate. --WjW From owner-freebsd-virtualization@freebsd.org Fri Jan 29 00:13:23 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1765F4F221C for ; Fri, 29 Jan 2021 00:13:23 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DRd8y093dz4ttT for ; Fri, 29 Jan 2021 00:13:21 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 10T0Bq3g000826 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 28 Jan 2021 16:11:53 -0800 (PST) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 10T0BoSY000822; Thu, 28 Jan 2021 16:11:50 -0800 (PST) (envelope-from jmg) Date: Thu, 28 Jan 2021 16:11:50 -0800 From: John-Mark Gurney To: Willem Jan Withagen Cc: Matt Churchyard , "freebsd-virtualization@freebsd.org" Subject: Re: Warm Migration feature for bhyve - review on Phabricator Message-ID: <20210129001150.GR31099@funkthat.com> Mail-Followup-To: Willem Jan Withagen , Matt Churchyard , "freebsd-virtualization@freebsd.org" References: <20210125062113.GL31099@funkthat.com> <7769d16522fa49558049f4a6e936982e@SERVER.ad.usd-group.com> <20210125184204.GM31099@funkthat.com> <5c6ff016-e15d-afad-e855-8cfed32276f2@digiware.nl> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5c6ff016-e15d-afad-e855-8cfed32276f2@digiware.nl> X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Thu, 28 Jan 2021 16:11:53 -0800 (PST) X-Rspamd-Queue-Id: 4DRd8y093dz4ttT X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [-1.80 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[jmg]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[funkthat.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[208.87.223.18:from]; AUTH_NA(1.00)[]; MID_RHS_MATCH_FROM(0.00)[]; SPAMHAUS_ZRD(0.00)[208.87.223.18:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2021 00:13:23 -0000 Willem Jan Withagen wrote this message on Thu, Jan 28, 2021 at 01:42 +0100: > On 25-1-2021 19:42, John-Mark Gurney wrote: > > Matt Churchyard wrote this message on Mon, Jan 25, 2021 at 10:46 +0000: > >> -----Original Message----- > >> From: John-Mark Gurney > >> Sent: 25 January 2021 06:21 > >> To: Matt Churchyard > >> Cc: Elena Mihailescu ; freebsd-virtualization@freebsd.org > >> Subject: Re: Warm Migration feature for bhyve - review on Phabricator > >> > >>> There's also hastd which can aid with this... > >> Thanks for the reply. I've always been wary of the additional complexity of HAST and ZFS, as it doesn't seem to have widespread usage or support, and things get ugly fast when storage is involved. > > Totally agree... > > > >> However, the idea of using HAST on top of zvols to provide network mirrored storage for a guest is interesting. It adds a lot of extra complexity, and probably performance impact though if it's just for the ability to move a guest between systems that may only happen every now and then. I'm also not sure it would help (or would at least be even more complex) if I have 4 hosts and want to be able to move guests anywhere. > > gmirror + ggate is another option as well... > > I did try all kinds of these solutions that are based on a 2 system > backend storage, and it was relatively easy for me to get the state of > the storage in a split-brain situation which is not really repairable > without a lot of manual user intervention. > So I gave up on HAST and gmirror over ggate. Can you explain more about the split brain w/ gmirror+ggate? If writes didn't happen to one of the members of gmirror, then that should be investigated.. I guess I could see a situation where ggate got behind on writes (connection drop), and gmirror didn't kick out the ggate member, or failed to mark it as stale, but those are bugs that need to be identified and fixed... I'm partly currious about the ggate, because I've recently done some work w/ ggate (performance improvements and working of sftp), and it's simple enough that issues are a bit surprising.. I'm not at all familar w/ HAST to advice on any issues w/ that... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-virtualization@freebsd.org Fri Jan 29 23:37:13 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0DB514FC888 for ; Fri, 29 Jan 2021 23:37:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DSDJm2c6nz4ZdS for ; Fri, 29 Jan 2021 23:37:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 42D664FC5D2; Fri, 29 Jan 2021 23:37:12 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 410F94FC3D0 for ; Fri, 29 Jan 2021 23:37:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DSDJk4l1Jz4ZdJ for ; Fri, 29 Jan 2021 23:37:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 3D6CB3428 for ; Fri, 29 Jan 2021 23:37:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10TNb82s005952 for ; Fri, 29 Jan 2021 23:37:08 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10TNb80I005951 for virtualization@FreeBSD.org; Fri, 29 Jan 2021 23:37:08 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 253087] Timeout of bufdaemon happens at shutdown time with -CURRENT amd64 and VirtualBox VM Date: Fri, 29 Jan 2021 23:37:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: short_desc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Jan 2021 23:37:13 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253087 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|Timeout of bufdaemon |Timeout of bufdaemon |happens at shutdown time |happens at shutdown time |with -CURRENT amd64 and |with -CURRENT amd64 and |VirtulaBox VM |VirtualBox VM Assignee|bugs@FreeBSD.org |virtualization@FreeBSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sat Jan 30 02:08:40 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 522A55213FA for ; Sat, 30 Jan 2021 02:08:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DSHgX1h8Sz4kdw for ; Sat, 30 Jan 2021 02:08:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 380235210D6; Sat, 30 Jan 2021 02:08:40 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 37C975215BB for ; Sat, 30 Jan 2021 02:08:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DSHgX0zNcz4kTH for ; Sat, 30 Jan 2021 02:08:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 147AC578F for ; Sat, 30 Jan 2021 02:08:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10U28eIx086572 for ; Sat, 30 Jan 2021 02:08:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10U28etU086571 for virtualization@FreeBSD.org; Sat, 30 Jan 2021 02:08:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 253088] bhyveload: /boot/userboot.so: Undefined symbol "gfx_fb_bezier" Date: Sat, 30 Jan 2021 02:08:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: warlock@phouka.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jan 2021 02:08:40 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253088 Bug ID: 253088 Summary: bhyveload: /boot/userboot.so: Undefined symbol "gfx_fb_bezier" Product: Base System Version: 13.0-STABLE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: warlock@phouka.net FreeBSD 13.0-ALPHA3 stable/13-c256224-ga455b23cf348 I haven't been kicking the tires on my FreeBSD BHYVEs lately, but I tried to start one today and ran across this: (pulling out bit that I need from /usr/share/examples/bhyve/vmrun.sh) /usr/sbin/bhyveload -c stdio -m 12G -d /zroot/stash/iso/FreeBSD-13.0-CURRENT-amd64-20200806-r363935-disc1.iso bsd= 13 /boot/userboot.so: Undefined symbol "gfx_fb_bezier" If I truss it, the final bits are: open("/boot/userboot.so",O_RDONLY|O_CLOEXEC|O_VERIFY,00) =3D 4 (0x4) fstat(4,{ mode=3D-r--r--r-- ,inode=3D14806,size=3D845368,blksize=3D131072 }= ) =3D 0 (0x0) mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,4,0x0) =3D 34376941568 (0x801068000) mmap(0x0,67932160,PROT_NONE,MAP_GUARD,-1,0x0) =3D 48358227968 (0xb426000= 00) mmap(0xb42600000,151552,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREF= AULT_READ,4,0x0) =3D 48358227968 (0xb42600000) mmap(0xb42625000,626688,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCOR= E|MAP_PREFAULT_READ,4,0x24000) =3D 48358379520 (0xb42625000) mmap(0xb426be000,16384,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFA= ULT_READ,4,0xbc000) =3D 48359006208 (0xb426be000) mmap(0xb426c2000,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFA= ULT_READ,4,0xbf000) =3D 48359022592 (0xb426c2000) mmap(0xb426c5000,67125248,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_AN= ON,-1,0x0) =3D 48359034880 (0xb426c5000) munmap(0x801068000,4096) =3D 0 (0x0) close(4) =3D 0 (0x0) munmap(0x801065000,12288) =3D 0 (0x0) mmap(0x0,163840,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) =3D 34381= 344768 (0x80149b000) munmap(0xb42600000,67932160) =3D 0 (0x0) fstat(1,{ mode=3Dcrw--w---- ,inode=3D225,size=3D0,blksize=3D4096 }) =3D 0 (= 0x0) ioctl(1,TIOCGETA,0x7fffffffe6b0) =3D 0 (0x0) /boot/userboot.so: Undefined symbol "gfx_fb_bezier" write(1,"/boot/userboot.so: Undefined sym"...,52) =3D 52 (0x34) exit(0x1) process exit, rval =3D 1 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sat Jan 30 14:43:07 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 55BD64EED58 for ; Sat, 30 Jan 2021 14:43:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DScQ31nwCz4cmr for ; Sat, 30 Jan 2021 14:43:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 3D95D4EED57; Sat, 30 Jan 2021 14:43:07 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3D5F14EEBC8 for ; Sat, 30 Jan 2021 14:43:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DScQ31Crrz4cqb for ; Sat, 30 Jan 2021 14:43:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1CC2A17780 for ; Sat, 30 Jan 2021 14:43:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10UEh7c0094569 for ; Sat, 30 Jan 2021 14:43:07 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10UEh7Ai094568 for virtualization@FreeBSD.org; Sat, 30 Jan 2021 14:43:07 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 253088] bhyveload: /boot/userboot.so: Undefined symbol "gfx_fb_bezier" Date: Sat, 30 Jan 2021 14:43:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: warlock@phouka.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: attachments.created Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jan 2021 14:43:07 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253088 --- Comment #1 from John Kennedy --- Created attachment 222021 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D222021&action= =3Dedit src.conf In sleeping on it, as nobody else has noticed it yet, chances are probably = good that having WITH_BIND_NOW in /etc/src.conf has a lot to do with that. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sat Jan 30 14:46:57 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 930E54EF12F for ; Sat, 30 Jan 2021 14:46:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 4DScVT3Zg2z4dNt for ; Sat, 30 Jan 2021 14:46:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 7AC934EF1A9; Sat, 30 Jan 2021 14:46:57 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7A8CC4EF282 for ; Sat, 30 Jan 2021 14:46:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DScVT2zKKz4dFc for ; Sat, 30 Jan 2021 14:46:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 55098170ED for ; Sat, 30 Jan 2021 14:46:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10UEkvTj095207 for ; Sat, 30 Jan 2021 14:46:57 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10UEkvaG095206 for virtualization@FreeBSD.org; Sat, 30 Jan 2021 14:46:57 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 253088] bhyveload: /boot/userboot.so: Undefined symbol "gfx_fb_bezier" Date: Sat, 30 Jan 2021 14:46:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: kevans@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jan 2021 14:46:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253088 Kyle Evans changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kevans@freebsd.org, | |tsoome@freebsd.org --- Comment #2 from Kyle Evans --- +tsoome --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sat Jan 30 19:16:10 2021 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 73DCE4F64B3 for ; Sat, 30 Jan 2021 19:16:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 4DSkT62cvJz3CYR for ; Sat, 30 Jan 2021 19:16:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 580174F64B2; Sat, 30 Jan 2021 19:16:10 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 57C374F6627 for ; Sat, 30 Jan 2021 19:16:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DSkT61nzRz3CPd for ; Sat, 30 Jan 2021 19:16:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2C9D51AE3B for ; Sat, 30 Jan 2021 19:16:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 10UJGAJb041779 for ; Sat, 30 Jan 2021 19:16:10 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 10UJGA3R041778 for virtualization@FreeBSD.org; Sat, 30 Jan 2021 19:16:10 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 253088] bhyveload: /boot/userboot.so: Undefined symbol "gfx_fb_bezier" Date: Sat, 30 Jan 2021 19:16:10 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: tsoome@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: tsoome@freebsd.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Jan 2021 19:16:10 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D253088 Toomas Soome changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|virtualization@FreeBSD.org |tsoome@freebsd.org --=20 You are receiving this mail because: You are the assignee for the bug.=