From owner-freebsd-virtualization@FreeBSD.ORG Sun Oct 12 14:54:56 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B2CC79B; Sun, 12 Oct 2014 14:54:56 +0000 (UTC) Received: from mail-la0-x230.google.com (mail-la0-x230.google.com [IPv6:2a00:1450:4010:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A39FDFDC; Sun, 12 Oct 2014 14:54:55 +0000 (UTC) Received: by mail-la0-f48.google.com with SMTP id gi9so5432817lab.21 for ; Sun, 12 Oct 2014 07:54:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=fKttOQ+LwRQsP7ca0NpjhlR2B6R/m7xxk+rgXAbfmW8=; b=wEqkwB8GE6vaZfLzNlqzsb4e9PHckwTM7EjoW945UhNfv1ZRy+c2std05JAfg5I0sD TO+3AODKxHZTSK87zR4ptof47hpgqn1Slvc6Ea7HsNRYpqprgb3acxS7tfLQx6HBwqNU p8AivQGGo0uRsafWdMf1G21nVySueAxlGKSNnNzpjhLToNxU7gOob0IisF2RFI24bKuN YnrXjWXR79kd9Uw4paNbHx8XG9pwPVMNcwACP1tufws8qfVMDucyo+9Gdb8RJvNfvei2 8ixw7dnK4C/4is5O94pDG4SWnai8k2cIT9qlPKyOkGLhIVlsRyrymm9RHEcMUa3CXYUC C2hw== X-Received: by 10.112.219.34 with SMTP id pl2mr17977945lbc.75.1413125693625; Sun, 12 Oct 2014 07:54:53 -0700 (PDT) Received: from kloomba ([95.104.130.152]) by mx.google.com with ESMTPSA id ld7sm3673037lac.23.2014.10.12.07.54.52 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 12 Oct 2014 07:54:52 -0700 (PDT) Sender: Roman Bogorodskiy Date: Sun, 12 Oct 2014 18:51:54 +0400 From: Roman Bogorodskiy To: Craig Rodrigues Subject: Re: libvirt improvements for bhyve Message-ID: <20141012145153.GB91663@kloomba> References: <20141011180528.GA26301@kloomba> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="1LKvkjL3sHcu1TtY" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: "freebsd-virtualization@freebsd.org" , Neel Natu X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Oct 2014 14:54:56 -0000 --1LKvkjL3sHcu1TtY Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Craig Rodrigues wrote: > On Sat, Oct 11, 2014 at 11:05 AM, Roman Bogorodskiy > wrote: >=20 > > > > Frankly speaking, it's a hard topic for me and currently I don't have a > > complete idea how to design that. The thing I'm worried about is that as > > far as I understand that exit codes are sort of temporary solution and > > will be dropped in the future. Also, it seems the behaviour is changing > > already, e.g. IIRC on -CURRENT the bhyve proccess doesn't exit on > > reboots (unlike in 10-R). My goal is to support 10-R and I'm not sure > > how to probe if exit codes are available or not (and how to probe if > > e.g. if new exit code will be added in some future version). > > >=20 > Any reason why you are stuck on 10-R? > Unlike other parts of FreeBSD, bhyve is not mature, and is evolving over > time. > Serious users of bhyve are following CURRENT or 10-STABLE. > 10-STABLE has the same error status codes as CURRENT, and 10.1-R is > coming out in a few weeks. > libvirt for bhyve is even less mature than bhyve itself, so tracking > bhyve beyond the release should be OK, but it's up to you. Well, it's not an ultimate goal to support 10-R, but while it's not very painful, I think it's a nice thing to do. Also, it doesn't mean that I'm not adding features available only in -STABLE, I just want user to have a possibility to have a meaningful error messages if certain feature is not support. Also, I forgot to mention one more important issue with the exit code approach. Libvirt doesn't serve as a parent process for the bhyve processes. The reason for that is to allow libvirt restart without having to interrupt the VMs (that's useful for upgrades or e.g. libvirtd crash). I'm not sure how I could track exit codes in such case. IMHO, the good way to solve this problem is to introduce some sort of control socket on the bhyve side that could be used to query VM status and push events about its change. I've been wanting to implement a PoC for that for quite some time, but, unfortunately, haven't got to it yet. > In its current form, my experience has been that bhyve and libvirt are > unusable for serious work, based > on the issues which I've described before for starting/stopping VM's. > bhyve + custom scripting is the only way to go. >=20 > I tried to use libvirt + bhyve for jenkins.freebsd.org, but found it > unusable, even though Jenkins has libvirt > support for starting VM's on demand. I backed away from libvirt, and end= ed > up writing my > own scripts. I am trying to convince some people to go with libvirt + > bhyve for a simulation/testing system, > but am having a hard time because of all the rough edges, and Linux + KVM= + > libvirt looks a lot more > mature and easier to use. >=20 > It's too bad, because I think if you can incorporate a few minor > enhancements to libvirt for bhyve, > it can definitely be a replacement for Linux + KVM + libvirt >=20 >=20 > Neel, Peter, can you comment on the status exit codes from bhyve? > Are those temporary, or will you keep them? There is no other way > for external scripts to detect reset/poweroff. It looks like the confusi= on > over this issue > is blocking Roman from moving forward with libvirt. >=20 > Thanks. > -- > Craig Roman Bogorodskiy --1LKvkjL3sHcu1TtY Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (FreeBSD) iQEcBAEBAgAGBQJUOpWJAAoJEMltX/4IwiJqF+IH/2G0EHnU6uPY1mjrcIKd+D5o Z/kMBBfyhqJs3t7qaJ8tGzW662FBbQWLy33Ejy76cKNLsJgcChj4Ii/AaYSdLZoR 1GASRLd6VWh0pzDqADV3lswO2LXmYdHBinci6vJ+1IdzZ5fE/uijjnq1HAZ8WSfO I0yu+Puuhd3AoYAGwzqfmVZLljY6smkLO+zJB9iQ459uf6hjbCbZJOvh+iw+SSpw BY1ouYpJ4R4KYuarCvmvpDdIjKA2Q79gsW+oUqKQ1ItKGE8j42dWhIwLCNEm7tR6 5IHZZ0NvByvnbwU3XVu31b5jp03Xel6PjeccUaRd7qfO76pLved1w1yzm/qXmHA= =PeTW -----END PGP SIGNATURE----- --1LKvkjL3sHcu1TtY--