From owner-freebsd-virtualization@freebsd.org Tue Apr 19 18:02:29 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEB0CB155B7 for ; Tue, 19 Apr 2016 18:02:29 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-lf0-x22d.google.com (mail-lf0-x22d.google.com [IPv6:2a00:1450:4010:c07::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5489D1DE5 for ; Tue, 19 Apr 2016 18:02:29 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by mail-lf0-x22d.google.com with SMTP id c126so25698378lfb.2 for ; Tue, 19 Apr 2016 11:02:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=YqS4zxR2wUrR4+ahTdSl3MdKoLHi2o9Syi86NoEzC3E=; b=PuuLbQS8xqZTTiwgLpW7o+/f7W+f1LvebsgV5PZi/aoxx4kzSak8xDzSsxM1U8dbO2 DxJtGRBRgaCobYx0bQ4ZHpWTCJnijsXeBMmq/Fwv2r1pHpSMGc0sVSntaSzrxioS+fz3 YeuF2x6SeCP0NRor/swpX6oxqbqiOypIIa0XiMQR8XR3GWoCw7ujsX9MHdg1i0WCXOas 8CVT6MvT7ajW77pxkXDjE5nu4fFkFU9phqETy69tdSzB4Pq4DeO3XST0RnE1aZcMZNxl 4w+pfjc6pij6IiloyQf3lWsgo3u49zgCLvsZw7OLHRMhofAu6Kxs5YurinzE08I5ylVk Ws0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=YqS4zxR2wUrR4+ahTdSl3MdKoLHi2o9Syi86NoEzC3E=; b=h+Jj76MC0vls6YtCbwYtVclZdPesvbxUkS5cWPMUmleLVwOMw5m/lexEhceXyDx1gt Mugl4U+XMTRfjzyXRxlM120aEK6xVeZXN044uCgpngCjw9YB4OHsvJH1FxMUDH4Z2/Tl xYGb3uCLsJrCvdqqZmMoDMmb5Z9pbaBOT8xnH8a5W/GPpK3hx5Sas8PHP8/zEa5N699g BCjpxaNELH26vjIf5X5bayiRk2iKC93m2k6wjS1sHOa1T1EP5eWJw0AU4AFoGcevpv4L 8XzLGR4COyrwSZZhNLGQVAm1tyyRMFwCGlrTxzIhup3oXOKRyJo8TOxhKL0IHlGwAzzz s8HA== X-Gm-Message-State: AOPr4FUGbQSDMecdnuLlct4lGPGtkcZbrD/ngWDitgXxmQT/iHG2mO/UbYK7sxQKTaQFJiHxTaqTb+X9REfAsg== MIME-Version: 1.0 X-Received: by 10.25.165.20 with SMTP id o20mr1836227lfe.105.1461088947602; Tue, 19 Apr 2016 11:02:27 -0700 (PDT) Received: by 10.114.4.169 with HTTP; Tue, 19 Apr 2016 11:02:27 -0700 (PDT) In-Reply-To: <20160413105520.GB84953@dev.san.ru> References: <20160413105520.GB84953@dev.san.ru> Date: Tue, 19 Apr 2016 11:02:27 -0700 Message-ID: Subject: Re: Understanding Bhyve shutdown From: Neel Natu To: Roman Bogorodskiy Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Apr 2016 18:02:29 -0000 Hi Roman, On Wed, Apr 13, 2016 at 3:55 AM, Roman Bogorodskiy wrote: > Hi, > > I was trying to get better understanding of how to properly shutdown VMs > in bhyve, but unfortunately the documentation does not provide much > details on that. > > Specifically, handbook [I] suggests to reboot a machine and then run > bhyvectl --destroy on it. > > The bhyvectl(8) manpage mentions the '--force-reset' and > '--force-poweroff' switches, but does not give details on those. > > I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html > > I tried all the options I know and wrote down the results. I also have > some questions, hopefully you'll be able to answer some of them. > > 1. bhyvectl --vm=$name --destroy > > * looks like hard poweroff in the guest > * the corresponding bhyve(8) process goes away > * /dev/vmm/ entry goes away > > In my experience, it's a dangerous way to shutdown a VM because > sometimes it appears it damages the image and VM fails to boot with > something like this: > > --- > Starting devd. > mode = 0100600, inum = 170269, fs = / > panic: ffs_valloc: dup alloc > cpuid = 0 > KDB: stack backtrace: > #0 0xffffffff80984e30 at kdb_backtrace+0x60 > #1 0xffffffff809489e6 at vpanic+0x126 > #2 0xffffffff809488b3 at panic+0x43 > #3 0xffffffff80b74a6e at ffs_valloc+0x84e > #4 0xffffffff80bb60ad at ufs_makeinode+0x7d > #5 0xffffffff80bb24fd at ufs_create+0x2d > #6 0xffffffff80e71841 at VOP_CREATE_APV+0xa1 > #7 0xffffffff809cd9e6 at uipc_bindat+0x346 > #8 0xffffffff809c5488 at kern_bindat+0x108 > #9 0xffffffff809c52a7 at sys_bind+0x77 > #10 0xffffffff80d4b3f7 at amd64_syscall+0x357 > #11 0xffffffff80d30adb at Xfast_syscall+0xfb > Uptime: 3s > > Dump failed. Partition too small. > --- > Yup, this is biggest hammer you could use to shutdown a virtual machine. As you noticed, this is usually a bad thing because it does not give the guest OS an opportunity to shutdown cleanly. > 2. kill -SIGTERM $bhyve_pid > > If guest supports ACPI shutdown: > > * guest shuts down cleanly > * the corresponding bhyve(8) process terminates > * /dev/vmm entry is still here, need bhyvectl --destroy for complete > cleanup > > If guest does not support ACPI shutdown (such as doing sysctl > hw.acpi.power_button_state=NONE): > > * Nothing happens > > Q1: Is there a way to know if a guest reacted to power button but > waiting for the bhyve process to terminate? Not really, except to wait for some amount of time to give the guest a chance to shutdown. > Q2: Why it's not done via bhyvectl (it seems that it's easier for users > + don't have to overload a useful SIGTERM signal) > It seems natural to overload SIGTERM to do this. For e.g. when the host is shutting down it will send a SIGTERM to all running processes and translating this into an ACPI poweroff event for the guest allows it to shutdown cleanly. > 3. bhyvectl --vm=$name --force-poweroff > > * looks like hard poweroff in the guest > * the corresponding bhyve(8) process goes away > * /dev/vmm entry is still here, need bhyvectl --destroy for complete > cleanup > > Q: what's the practical difference with just doing --destroy right away? > 'force-poweroff' is equivalent to a hard power off on real hardware. The only practical difference between '--force-poweroff' and '--destroy' is that destroy will also remove the device node in /dev/vmm and release any memory allocated for the guest back to the host. > 4. bhyvectl --vm=$name --force-reset > > Looks very similar to item #3 with just different exit code (reboot > appears to be using 0, while shutdown and halt use 1 and 2). > > Q: what's the practical use of it? > The exit code can be used by scripts on top of 'bhyve' to decide whether to restart execution of the guest (reset) or stop completely (poweroff). > Would greatly appreciate if somebody could provide more details on that. > I guess we'll need to update Handbook with this information as well > because it needs to mention SIGTERM for ACPI shutdown at least. > Hope that helps. best Neel > Roman Bogorodskiy > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org"