From owner-freebsd-virtualization@FreeBSD.ORG Sun Feb 9 22:34:24 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 133F2E2D; Sun, 9 Feb 2014 22:34:24 +0000 (UTC) Received: from mail-pd0-x22e.google.com (mail-pd0-x22e.google.com [IPv6:2607:f8b0:400e:c02::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D45191E24; Sun, 9 Feb 2014 22:34:23 +0000 (UTC) Received: by mail-pd0-f174.google.com with SMTP id z10so5292630pdj.19 for ; Sun, 09 Feb 2014 14:34:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=5Aa9IafV0IthK8jIZMhWAm9CpW2I/mbiG77nm4lqLcc=; b=Oy7Ijf6JY69sjyLUrcFJI8l66mjOAyS8Dy1CcCycLEyiRTYpcDG7bFMixGh21HSszP VCj1Yl+11eGALI++OXm7qicVeOPUkQI+YkF7Emg3rFFjU/qqtPA5Q3HPYQn5zX5AUIqd RBeAgi1TUM3dlgUVZCWN89V+PIoUtkPcbgljyfKy6MOedO0GznEWWyMcnFRpO6mg1clO Q4Of/yS6dqQt3h2Vacn0wmtK31ueL+YmWB4aMODRaw6gf+g+hHp6Fu++rjXiXffzSCqb IsJstSnuxy5hgD7v57sQz2VqUh6wcKucErWG2yrGHCzIGlq4kaU4lS3YxUx1L1M2mJcY G8qA== MIME-Version: 1.0 X-Received: by 10.68.255.101 with SMTP id ap5mr33781623pbd.41.1391985263248; Sun, 09 Feb 2014 14:34:23 -0800 (PST) Received: by 10.68.155.38 with HTTP; Sun, 9 Feb 2014 14:34:23 -0800 (PST) In-Reply-To: References: Date: Sun, 9 Feb 2014 17:34:23 -0500 Message-ID: Subject: Re: Detect of BHyve VM was powered off or rebooted? From: Aryeh Friedman To: Craig Rodrigues Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Feb 2014 22:34:24 -0000 On Sun, Feb 9, 2014 at 5:18 PM, Craig Rodrigues wrote: > > > > On Sun, Feb 9, 2014 at 1:32 PM, Aryeh Friedman wrote: > >> >> >> >> On Sun, Feb 9, 2014 at 4:11 PM, Craig Rodrigues wrote: >> >>> Hi, >>> >>> I posted some rc.d scripts that I am using to boot a BHyve VM >>> and send the output to a serial console using the /dev/nmdm >>> driver: >>> >>> >>> http://lists.freebsd.org/pipermail/freebsd-virtualization/2014-January/002040.html >>> >>> It works quite well. There is some things I would like to improve, >>> and would like some advice on the best way to do it. >>> >>> (1) If the VM was destroyed with bhyvectl --destroy --vm ${VM_NAME}, >>> then I do not want to automatically restart the VM in the script. >>> User should manually: service bhyvevm start >>> >>> (2) If the VM was powered down, via shutdown -p, or halt -p, >>> then in my script I do not want to restart the VM in the script. >>> User should manually: service bhyvevm start >>> >>> (3) If the VM was rebooted via "reboot" or "shutdown -r", >>> then I *do* want the script to restart the VM. >>> >>> I think if I change my start_vm.sh script to do something like: >>> >>> >>> >>> ( >>> while [ -e /dev/vmm/${VM} ]; do >>> /usr/sbin/bhyve -c 16 -m 8G -A -H -P -g 0 -s 0:0,hostbridge -s >>> 1:0,lpc >>> -s 2:0,virtio-net,${TAP} -s 3:0,virtio-blk,${IMG} -l com1,${CONS_A} >>> ${VM}" >>> done >>> >>> ) & >>> >>> >>> then this might cover cases (1) and (3), but what will cover >>> case (2)? >>> >>> Thanks for any advice. >>> >> >> The options you gave (which are really the only ones available) do not >> distinguish between the reason for the termination of the instance of bhyve >> pointing to /dev/vmm/XXX (it just does a normal termination). >> > > Did you play with any of these flags to bhyvectl? > > [--get-vmcs-exit-ctls] > [--get-vmcs-exit-reason] > [--get-vmcs-exit-qualification] > [--get-vmcs-exit-interruption-info] > > If I do: > bhyvectl --get-vmcs-exit-reason --vm vm1 > > I get: > vmcs_exit_reason[0] 0x000000000000001e > > If I look at: > > > http://svnweb.freebsd.org/base/head/sys/amd64/vmm/intel/vmcs.h?view=markup#l310 > > we have: > #define EXIT_REASON_INOUT 30 > You will likely need to detect the following ones also (all for different reasons that may or may not have anything to do with an issue with the VM instead of some host [OS or hardware] issue): #define EXIT_REASON_EXT_INTR 1 #define EXIT_REASON_INIT 3 #define EXIT_REASON_HLT 12 #define EXIT_REASON_CPUID 10 #define EXIT_REASON_VMLAUNCH 20 > Linux has this stuff also: > http://lxr.free-electrons.com/source/arch/x86/include/asm/vmx.h?v=3.1#L250 > > so I guess these values are defined in some Intel manual. > > > >> In my playing with this for almost a month the only solution seems to be >> put a trigger on instance shutdown via whatever method to leave a "cookie" >> file behind on the host (note I have not nor plan to do this in any of my >> own work). >> > > Yes, I can see why this might need to be done. I might need to do that > myself. However, if bhyvectl can be used to provide adequate status > information as to why the VM exited, I might be able to avoid this. > > It would be nicer though if /usr/sbin/bhyve returned > the EXIT_REASON in its status code. > > > >> >> An other question is it wise to auto-reboot regardless of reason of the >> hypervisor termination? >> > > I think the answer is "it depends". If someone > does "reboot" or "shutdown -r", I think it is reasonable to restart the > VM. If someone does "halt -p" or "shutdown -p", I think it is reasonable > to not restart the VM. For any other error condition, I think it is > reasonable to not restart the VM. > > I think that there is no single answer for all users and all applications. > In my case, I am running BHyve VM's as "headless" which > I access via the /dev/nmdm driver. If the VM reboots normally, I want it > to restart. > It seems almost any boot failure would be host side for example if there is no SSH access (which if the instance is never logged into then it is likely a host issue) then it is highly unlikely the VM is in bootable form (this is after a manual reboot of it)... an other reason (just learned the hardway) is the user uses a mismatch between backing file formats (my case using raw images with ahci-hd) -- Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org