From owner-freebsd-virtualization@freebsd.org Sun Apr 17 04:03:47 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B4E5AEDC7F for ; Sun, 17 Apr 2016 04:03:47 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-lf0-x243.google.com (mail-lf0-x243.google.com [IPv6:2a00:1450:4010:c07::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B349F1692; Sun, 17 Apr 2016 04:03:46 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: by mail-lf0-x243.google.com with SMTP id e190so21686068lfe.1; Sat, 16 Apr 2016 21:03:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=KUNxs619jafCZnOx2wYyZxnmz8MkSVnYgchQC4VLkZc=; b=nIAQZhOuCKV8IFDzvR3aBuZF87dO4hARu6ISZPnepsPss6ilA2rdFYMUhk/zxodr2E kDDUKnN1i2PeUOApClJu//BF/6BWQQJzwTiuHPWZ/4FQOumI6w6nCtJg9nrllh/910XG Vo5WrnKJeahXwbfIDAfYINwWrUaJ7FKswJofYTdSIte9o08qInFT4ik2xjLhf28Tiejb th2MX+x69T+8QMGrX2N9fGoAWH9b/4UAfyrw/wnuMjGnGn1epsB8Of7EwUGC6j+uRv27 Bgu73Pb4apNoOM6u8T4TAvIF4QlwhdqBX8kUOwHrRVXWnOkAGzyjTcCjOijJLjYtv0Wg uouA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=KUNxs619jafCZnOx2wYyZxnmz8MkSVnYgchQC4VLkZc=; b=GQvRTzSstqVT0Cr0AodqG3gQbsXE4p4Xw0jOV94qx7Hesr1MG91Gc/oRWI+Med42Ga nzkV7AORjZQvlRS8s/Umv7x1SxdLeyZ/dbCCR9RtqVCxiOZ23Ue2Me3K4xolrgiRNu2V kwQwvGZ/slTwl/OJxr8jxnBYOUHuNe5iarkvk5V2WXXdP+4htwCUz0F0uVrrOZDF7KxD n+9DsKAtsqrzQZOmcasrAZHX5aP/Udy/XvRouvEXSdMAdxORFo8t9VFnlmwXYQC+KC9O WYFfCkJuHZGeA9adXXHYdoS2pxS8/8X0AeX6sffeaPg9KCtscrGfHbAfV5Y3Fnqxglz2 L+lQ== X-Gm-Message-State: AOPr4FVYHVGQN1tnUOpjaZcaVGE9ch1OLcMuFWlz5kN51yahc+vwlBq/2ProP9wllhfO3A== X-Received: by 10.112.171.161 with SMTP id av1mr12146740lbc.82.1460865825005; Sat, 16 Apr 2016 21:03:45 -0700 (PDT) Received: from kloomba ([213.147.222.31]) by smtp.gmail.com with ESMTPSA id bd5sm7582608lbc.12.2016.04.16.21.03.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 16 Apr 2016 21:03:44 -0700 (PDT) Sender: Roman Bogorodskiy Date: Sun, 17 Apr 2016 07:03:39 +0300 From: Roman Bogorodskiy To: Matt Churchyard Cc: Roman Bogorodskiy , FreeBSD virtualization , Peter Grehan , Neel Natu Subject: Re: Understanding Bhyve shutdown Message-ID: <20160417040337.GA2106@kloomba> References: <20160413105520.GB84953@dev.san.ru> <548d6c81c4784447b91f6f1459863eed@SERVER.ad.usd-group.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="k1lZvvs/B4yU6o8G" Content-Disposition: inline In-Reply-To: <548d6c81c4784447b91f6f1459863eed@SERVER.ad.usd-group.com> User-Agent: Mutt/1.6.0 (2016-04-01) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Apr 2016 04:03:47 -0000 --k1lZvvs/B4yU6o8G Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Matt Churchyard wrote: > As I understand it >=20 > 1) shutdown from guest > 2) 'kill ' -> pressing the power button once. > 3) --force-poweroff -> holding power button in > 4) --force-reset -> pressing the reset button > 5) bhyvectl --destroy -> same as 3? (although vmm is destroyed as well) >=20 > 1 or 2 are obviously preferred. I've found however that some guests don't= respond that well to the apci event. CentOS 6 seems to need apci installin= g and enabling?!, and my Win2012R2 machine will only shutdown if I send the= signal twice. >=20 > The rest are all hard shutdowns that should not be seen as a way to stop = the guest, just a last resort if it can't be stopped correctly. I don't kno= w the practical difference between poweroff&destroy vs just destroy. I CCed Peter and Neel, probably they'll shed some light on 'bhyvectl --force-poweroff && bhyvectl --destroy' vs just 'bhyvectl --destroy'. > I see no reason why the documentation suggests reboot rather than shutdow= n. Bhyve exits either way and the only difference is the exit code. When us= ing one of the bhyve management tools that support reboots (such as vmrun.s= h/vm-bhyve/iohyve) however, a "restart" exit code (0) will cause the guest = to restart automatically; If a guest is locked up for example, a --force-re= set will cause it to immediately reboot. >=20 > From a host, the only safe choice is 'kill ' (possibly multiple time= s) and wait for bhyve process to (hopefully) exit. If that doesn't work eit= her the acpi issue needs fixing or ideally the guest needs to be stopped us= ing its built-in shutdown function. >=20 > The devs will have to confirm why they're implemented the way they are. M= y instinct is that bhyvectl works on the vmm device, and reset/poweroff are= just functions that are possible via that interface, and so have been expo= sed. The ACPI shutdown event likely needs to be initiated by the bhyve proc= ess itself, hence using a signal to it. /end-speculation >=20 > I think most users will want to use a bhyve tool so the raw specifics of = the bhyve/bhyvectl commands are glossed over, although that doesn't mean th= e handbook documentation of the base commands shouldn't be as complete/corr= ect as possible of course. FWIW, I've created a patch: https://reviews.freebsd.org/D5982 which at least documents exit codes and the SIGTERM thing. Roman Bogorodskiy --k1lZvvs/B4yU6o8G Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXEwsZAAoJEMltX/4IwiJqqU8IAKTGTwtWrbHxRWxXp43rrVoS OkZh9HfBF6JX2H4UkTyLc3+jaFVFo1avQlp3jpAPus5nmlkw+vFZePbYyYdz6boW YzVntRzPcu1GbnlXJsX9XHrGJnyMjzIofP4rKp89s9sLQ1tisajf7ukzcVOv7ZQg 7LOlOad0zh+xAfwE012M7KIrp80GW32BRKVCl7f5lu97kUppafBfw7DFsvGTA611 Z7C6NdfUoVn/jubfgjO5ProtF/u/bsWNSslL87wBCQU5h4VOaiH8Ufjm84qfp/ER VIozyAYgOQ/Gaq+FadW1FPpoCt3/KJRCKpY540aFgVP6PTXjdekv9bhBqb1RI3I= =/Qtd -----END PGP SIGNATURE----- --k1lZvvs/B4yU6o8G-- From owner-freebsd-virtualization@freebsd.org Sun Apr 17 21:00:17 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6D1EB12042 for ; Sun, 17 Apr 2016 21:00:17 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B3B111512 for ; Sun, 17 Apr 2016 21:00:17 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u3HL01JH043394 for ; Sun, 17 Apr 2016 21:00:17 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201604172100.u3HL01JH043394@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-virtualization@FreeBSD.org Subject: Problem reports for freebsd-virtualization@FreeBSD.org that need special attention Date: Sun, 17 Apr 2016 21:00:17 +0000 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Apr 2016 21:00:18 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 202321 | [bhyve,patch] More verbose error reporting in bhy New | 202322 | [bhyve,patch] add option to have bhyve write its 2 problems total for which you should take action. From owner-freebsd-virtualization@freebsd.org Tue Apr 19 18:02:29 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEB0CB155B7 for ; Tue, 19 Apr 2016 18:02:29 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-lf0-x22d.google.com (mail-lf0-x22d.google.com [IPv6:2a00:1450:4010:c07::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5489D1DE5 for ; Tue, 19 Apr 2016 18:02:29 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by mail-lf0-x22d.google.com with SMTP id c126so25698378lfb.2 for ; Tue, 19 Apr 2016 11:02:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=YqS4zxR2wUrR4+ahTdSl3MdKoLHi2o9Syi86NoEzC3E=; b=PuuLbQS8xqZTTiwgLpW7o+/f7W+f1LvebsgV5PZi/aoxx4kzSak8xDzSsxM1U8dbO2 DxJtGRBRgaCobYx0bQ4ZHpWTCJnijsXeBMmq/Fwv2r1pHpSMGc0sVSntaSzrxioS+fz3 YeuF2x6SeCP0NRor/swpX6oxqbqiOypIIa0XiMQR8XR3GWoCw7ujsX9MHdg1i0WCXOas 8CVT6MvT7ajW77pxkXDjE5nu4fFkFU9phqETy69tdSzB4Pq4DeO3XST0RnE1aZcMZNxl 4w+pfjc6pij6IiloyQf3lWsgo3u49zgCLvsZw7OLHRMhofAu6Kxs5YurinzE08I5ylVk Ws0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=YqS4zxR2wUrR4+ahTdSl3MdKoLHi2o9Syi86NoEzC3E=; b=h+Jj76MC0vls6YtCbwYtVclZdPesvbxUkS5cWPMUmleLVwOMw5m/lexEhceXyDx1gt Mugl4U+XMTRfjzyXRxlM120aEK6xVeZXN044uCgpngCjw9YB4OHsvJH1FxMUDH4Z2/Tl xYGb3uCLsJrCvdqqZmMoDMmb5Z9pbaBOT8xnH8a5W/GPpK3hx5Sas8PHP8/zEa5N699g BCjpxaNELH26vjIf5X5bayiRk2iKC93m2k6wjS1sHOa1T1EP5eWJw0AU4AFoGcevpv4L 8XzLGR4COyrwSZZhNLGQVAm1tyyRMFwCGlrTxzIhup3oXOKRyJo8TOxhKL0IHlGwAzzz s8HA== X-Gm-Message-State: AOPr4FUGbQSDMecdnuLlct4lGPGtkcZbrD/ngWDitgXxmQT/iHG2mO/UbYK7sxQKTaQFJiHxTaqTb+X9REfAsg== MIME-Version: 1.0 X-Received: by 10.25.165.20 with SMTP id o20mr1836227lfe.105.1461088947602; Tue, 19 Apr 2016 11:02:27 -0700 (PDT) Received: by 10.114.4.169 with HTTP; Tue, 19 Apr 2016 11:02:27 -0700 (PDT) In-Reply-To: <20160413105520.GB84953@dev.san.ru> References: <20160413105520.GB84953@dev.san.ru> Date: Tue, 19 Apr 2016 11:02:27 -0700 Message-ID: Subject: Re: Understanding Bhyve shutdown From: Neel Natu To: Roman Bogorodskiy Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Apr 2016 18:02:29 -0000 Hi Roman, On Wed, Apr 13, 2016 at 3:55 AM, Roman Bogorodskiy wrote: > Hi, > > I was trying to get better understanding of how to properly shutdown VMs > in bhyve, but unfortunately the documentation does not provide much > details on that. > > Specifically, handbook [I] suggests to reboot a machine and then run > bhyvectl --destroy on it. > > The bhyvectl(8) manpage mentions the '--force-reset' and > '--force-poweroff' switches, but does not give details on those. > > I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html > > I tried all the options I know and wrote down the results. I also have > some questions, hopefully you'll be able to answer some of them. > > 1. bhyvectl --vm=$name --destroy > > * looks like hard poweroff in the guest > * the corresponding bhyve(8) process goes away > * /dev/vmm/ entry goes away > > In my experience, it's a dangerous way to shutdown a VM because > sometimes it appears it damages the image and VM fails to boot with > something like this: > > --- > Starting devd. > mode = 0100600, inum = 170269, fs = / > panic: ffs_valloc: dup alloc > cpuid = 0 > KDB: stack backtrace: > #0 0xffffffff80984e30 at kdb_backtrace+0x60 > #1 0xffffffff809489e6 at vpanic+0x126 > #2 0xffffffff809488b3 at panic+0x43 > #3 0xffffffff80b74a6e at ffs_valloc+0x84e > #4 0xffffffff80bb60ad at ufs_makeinode+0x7d > #5 0xffffffff80bb24fd at ufs_create+0x2d > #6 0xffffffff80e71841 at VOP_CREATE_APV+0xa1 > #7 0xffffffff809cd9e6 at uipc_bindat+0x346 > #8 0xffffffff809c5488 at kern_bindat+0x108 > #9 0xffffffff809c52a7 at sys_bind+0x77 > #10 0xffffffff80d4b3f7 at amd64_syscall+0x357 > #11 0xffffffff80d30adb at Xfast_syscall+0xfb > Uptime: 3s > > Dump failed. Partition too small. > --- > Yup, this is biggest hammer you could use to shutdown a virtual machine. As you noticed, this is usually a bad thing because it does not give the guest OS an opportunity to shutdown cleanly. > 2. kill -SIGTERM $bhyve_pid > > If guest supports ACPI shutdown: > > * guest shuts down cleanly > * the corresponding bhyve(8) process terminates > * /dev/vmm entry is still here, need bhyvectl --destroy for complete > cleanup > > If guest does not support ACPI shutdown (such as doing sysctl > hw.acpi.power_button_state=NONE): > > * Nothing happens > > Q1: Is there a way to know if a guest reacted to power button but > waiting for the bhyve process to terminate? Not really, except to wait for some amount of time to give the guest a chance to shutdown. > Q2: Why it's not done via bhyvectl (it seems that it's easier for users > + don't have to overload a useful SIGTERM signal) > It seems natural to overload SIGTERM to do this. For e.g. when the host is shutting down it will send a SIGTERM to all running processes and translating this into an ACPI poweroff event for the guest allows it to shutdown cleanly. > 3. bhyvectl --vm=$name --force-poweroff > > * looks like hard poweroff in the guest > * the corresponding bhyve(8) process goes away > * /dev/vmm entry is still here, need bhyvectl --destroy for complete > cleanup > > Q: what's the practical difference with just doing --destroy right away? > 'force-poweroff' is equivalent to a hard power off on real hardware. The only practical difference between '--force-poweroff' and '--destroy' is that destroy will also remove the device node in /dev/vmm and release any memory allocated for the guest back to the host. > 4. bhyvectl --vm=$name --force-reset > > Looks very similar to item #3 with just different exit code (reboot > appears to be using 0, while shutdown and halt use 1 and 2). > > Q: what's the practical use of it? > The exit code can be used by scripts on top of 'bhyve' to decide whether to restart execution of the guest (reset) or stop completely (poweroff). > Would greatly appreciate if somebody could provide more details on that. > I guess we'll need to update Handbook with this information as well > because it needs to mention SIGTERM for ACPI shutdown at least. > Hope that helps. best Neel > Roman Bogorodskiy > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@freebsd.org Wed Apr 20 16:14:26 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C5C9B16382 for ; Wed, 20 Apr 2016 16:14:26 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-lb0-x241.google.com (mail-lb0-x241.google.com [IPv6:2a00:1450:4010:c04::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A4EA71107 for ; Wed, 20 Apr 2016 16:14:25 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: by mail-lb0-x241.google.com with SMTP id mx9so1481633lbb.2 for ; Wed, 20 Apr 2016 09:14:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=g75FcyJzN0yW4FbnPECHBTbRrpjJyyUQ45DTXjh5OS0=; b=kKKY45amuX+JYBHHQDYF0aps6O9zIxa1ju2mzumc+S/9LcXwEdYelkbvbz4ES/QcLK UMGKom/NSAxbaZq2kxiL8pVZbnPnzWXS+OiDePIxqRxfuiRETNd1u7SJC+c7q8XiSRoh VPoRA0UqxhaqfIm0j/D/xASvUaU64dMeNDav0LdEsHg1N5pjxU/PEWb+pmmQoiTozgZk BJuOFn04sVwEvY0W4+8wDGFEmh7y1FHWXDYB/j7C4PCUeoFDftLS8ObteGU/BdgOdQYx RFA7f6IR1HNwMFNywh6Q0vveBsXOODzDznfEi6MRTYL5xAJmSE9rvmCHKDRCXkuK9+Ux VU4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=g75FcyJzN0yW4FbnPECHBTbRrpjJyyUQ45DTXjh5OS0=; b=OQMQDrki1a1KqbNlYTYqxC/dEExap3PIpnmg8eWEcjREYTJY5hzE+VTiDNwPsBvQ1h swl1CTSA+HjaykErxx0MHVl24RlK/NyZ2W8GOTdhr805OJU49qhDNHEvxH9fbXW58yNM H+8RUNnEawcDyVxnpzs5WTBqceqY/fmC0gI/Ta+HHD25LrDsWabONVoRwjmqArSJMsFC x4XrwEwHMdYPtZzfv1xGemLLR50ZFHa0qwFR4lyvCgnhr16nwJZJY3vrs/nw+3aho1Xc N7lbn15HGb1A6z+NNejownNxwYICvEZZEvj/lAlfpXMla7uLC+HAHMn4CXkjyEiFSdHL 7vzw== X-Gm-Message-State: AOPr4FUY44iwJaEWBVxQ9ecCEb+mk68uyZ8xGjFLselIyyyZpM+Txbjpl3c+d9dEjS68iw== X-Received: by 10.112.85.43 with SMTP id e11mr4294012lbz.80.1461168863469; Wed, 20 Apr 2016 09:14:23 -0700 (PDT) Received: from kloomba ([31.29.237.95]) by smtp.gmail.com with ESMTPSA id 28sm1203394lfx.43.2016.04.20.09.14.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Apr 2016 09:14:22 -0700 (PDT) Sender: Roman Bogorodskiy Date: Wed, 20 Apr 2016 19:14:16 +0300 From: Roman Bogorodskiy To: Neel Natu Cc: Roman Bogorodskiy , "freebsd-virtualization@freebsd.org" Subject: Re: Understanding Bhyve shutdown Message-ID: <20160420161414.GA87187@kloomba> References: <20160413105520.GB84953@dev.san.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="uAKRQypu60I7Lcqm" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.0 (2016-04-01) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Apr 2016 16:14:26 -0000 --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Neel Natu wrote: > Hi Roman, >=20 > On Wed, Apr 13, 2016 at 3:55 AM, Roman Bogorodskiy > wrote: > > Hi, > > > > I was trying to get better understanding of how to properly shutdown VMs > > in bhyve, but unfortunately the documentation does not provide much > > details on that. > > > > Specifically, handbook [I] suggests to reboot a machine and then run > > bhyvectl --destroy on it. > > > > The bhyvectl(8) manpage mentions the '--force-reset' and > > '--force-poweroff' switches, but does not give details on those. > > > > I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html > > > > I tried all the options I know and wrote down the results. I also have > > some questions, hopefully you'll be able to answer some of them. > > > > 1. bhyvectl --vm=3D$name --destroy > > > > * looks like hard poweroff in the guest > > * the corresponding bhyve(8) process goes away > > * /dev/vmm/ entry goes away > > > > In my experience, it's a dangerous way to shutdown a VM because > > sometimes it appears it damages the image and VM fails to boot with > > something like this: > > > > --- > > Starting devd. > > mode =3D 0100600, inum =3D 170269, fs =3D / > > panic: ffs_valloc: dup alloc > > cpuid =3D 0 > > KDB: stack backtrace: > > #0 0xffffffff80984e30 at kdb_backtrace+0x60 > > #1 0xffffffff809489e6 at vpanic+0x126 > > #2 0xffffffff809488b3 at panic+0x43 > > #3 0xffffffff80b74a6e at ffs_valloc+0x84e > > #4 0xffffffff80bb60ad at ufs_makeinode+0x7d > > #5 0xffffffff80bb24fd at ufs_create+0x2d > > #6 0xffffffff80e71841 at VOP_CREATE_APV+0xa1 > > #7 0xffffffff809cd9e6 at uipc_bindat+0x346 > > #8 0xffffffff809c5488 at kern_bindat+0x108 > > #9 0xffffffff809c52a7 at sys_bind+0x77 > > #10 0xffffffff80d4b3f7 at amd64_syscall+0x357 > > #11 0xffffffff80d30adb at Xfast_syscall+0xfb > > Uptime: 3s > > > > Dump failed. Partition too small. > > --- > > >=20 > Yup, this is biggest hammer you could use to shutdown a virtual > machine. As you noticed, this is usually a bad thing because it does > not give the guest OS an opportunity to shutdown cleanly. >=20 > > 2. kill -SIGTERM $bhyve_pid > > > > If guest supports ACPI shutdown: > > > > * guest shuts down cleanly > > * the corresponding bhyve(8) process terminates > > * /dev/vmm entry is still here, need bhyvectl --destroy for complete > > cleanup > > > > If guest does not support ACPI shutdown (such as doing sysctl > > hw.acpi.power_button_state=3DNONE): > > > > * Nothing happens > > > > Q1: Is there a way to know if a guest reacted to power button but > > waiting for the bhyve process to terminate? >=20 > Not really, except to wait for some amount of time to give the guest a > chance to shutdown. >=20 > > Q2: Why it's not done via bhyvectl (it seems that it's easier for users > > + don't have to overload a useful SIGTERM signal) > > >=20 > It seems natural to overload SIGTERM to do this. For e.g. when the > host is shutting down it will send a SIGTERM to all running processes > and translating this into an ACPI poweroff event for the guest allows > it to shutdown cleanly. >=20 > > 3. bhyvectl --vm=3D$name --force-poweroff > > > > * looks like hard poweroff in the guest > > * the corresponding bhyve(8) process goes away > > * /dev/vmm entry is still here, need bhyvectl --destroy for complete > > cleanup > > > > Q: what's the practical difference with just doing --destroy right away? > > >=20 > 'force-poweroff' is equivalent to a hard power off on real hardware. >=20 > The only practical difference between '--force-poweroff' and > '--destroy' is that destroy will also remove the device node in > /dev/vmm and release any memory allocated for the guest back to the > host. >=20 > > 4. bhyvectl --vm=3D$name --force-reset > > > > Looks very similar to item #3 with just different exit code (reboot > > appears to be using 0, while shutdown and halt use 1 and 2). > > > > Q: what's the practical use of it? > > >=20 > The exit code can be used by scripts on top of 'bhyve' to decide > whether to restart execution of the guest (reset) or stop completely > (poweroff). >=20 > > Would greatly appreciate if somebody could provide more details on that. > > I guess we'll need to update Handbook with this information as well > > because it needs to mention SIGTERM for ACPI shutdown at least. > > >=20 > Hope that helps. >=20 > best > Neel Neel, thanks for the response, that's a kind of an answer I was looking for! One question regarding that '--force-poweroff' vs '--destroy' thing: what is the point of not releasing guest resources by doing '--force-powerof'? Initially I thought it could be useful to speed up the next boot by not requiring running 'bhyveload' (or other loader) for the guest again, but, apparently, it's still required to run '--destroy' and then bhyveload again, otherwise I get something like this: bhyve: could not activate CPU 0: Device busy I'm wondering if that's just for debugging purposes or there is something else? Sorry for being extra curious. :-) Roman Bogorodskiy --uAKRQypu60I7Lcqm Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXF6rWAAoJEMltX/4IwiJq1+IH/iAbACAob+uicIyhcKPfh6zv ahRnSoxYB3kNkqAxGqF05RWxJEqs8v/n/vjbbJzgIejo5zCoKm3Rs2s31xPbi1X7 /DCvQ91j2/QmqHZSLeNNvKUp4o8y0Azm5rgJ/uLZTVN2cg8GixVU+OshSmhad60T MqXzJJLUV26i9JMaa+5xnFxHgo+CorHSO0JQhvFn3DY4D4AZmxO6OfAGdcHuUGJ6 IlzuYhEG5aD4U5nigDtUoU784FJLLM9b+MLW4OPI0I2YVsKUG9jF80lsugWy8vdo j0mUfYiIB5lLA8PrLn/1lzto+f5zQBr7pajrRx6laFclcnw1hei5mCs70alzjTE= =NGYJ -----END PGP SIGNATURE----- --uAKRQypu60I7Lcqm-- From owner-freebsd-virtualization@freebsd.org Sat Apr 23 22:40:24 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81F71B1A3E8 for ; Sat, 23 Apr 2016 22:40:24 +0000 (UTC) (envelope-from saper@saper.info) Received: from m.saper.info (m.saper.info [IPv6:2a01:4f8:a0:7383::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "m.saper.info", Issuer "Marcin Cieslak 2016" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 158051491 for ; Sat, 23 Apr 2016 22:40:23 +0000 (UTC) (envelope-from saper@saper.info) Received: from m.saper.info (saper@m.saper.info [IPv6:2a01:4f8:a0:7383::]) by m.saper.info (8.15.2/8.15.2) with ESMTPS id u3NMeLJi053183 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Sat, 23 Apr 2016 22:40:21 GMT (envelope-from saper@saper.info) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=saper.info; s=Sep2014; t=1461451221; bh=fECJv5kbSx5vePjCV0kB0tnQ4YATLQX5voclO3+RNpA=; h=Date:From:To:Subject; b=oriTPg4FdO64aYLO/awOFEPLvzI8SrZPOI65hhNniyqvwpkiAhxZR44+cl3xel0fm 1gzdFJ+t2fYOFA1eBVLwf/V2dh11/UFFbSMIPsxWLPTLPf6M7MEn7V3IYz/FNmjljw +Nb6SS666TjM26tq6gwmMdnJuFrA9nDPcJj+9CT0= Received: from localhost (saper@localhost) by m.saper.info (8.15.2/8.15.2/Submit) with ESMTP id u3NMeKDd053180 for ; Sat, 23 Apr 2016 22:40:21 GMT (envelope-from saper@saper.info) X-Authentication-Warning: m.saper.info: saper owned process doing -bs Date: Sat, 23 Apr 2016 22:40:20 +0000 From: Marcin Cieslak To: freebsd-virtualization@freebsd.org Subject: Booting r298488 as Xen Dom0 may break ZFS pool? Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Apr 2016 22:40:24 -0000 On a freshly installed (via upgrade from 10.3 from source) -CURRENT on this machine: FreeBSD 11.0-CURRENT #0 r298488: Sat Apr 23 11:10:01 UTC 2016 root@o.saper.info:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) WARNING: WITNESS option enabled, expect reduced performance. VT(vga): resolution 640x480 CPU: Intel(R) Xeon(R) CPU E31245 @ 3.30GHz (3300.09-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206a7 Family=0x6 Model=0x2a Stepping=7 Features=0xbfebfbff Features2=0x1fbae3ff AMD Features=0x28100800 AMD Features2=0x1 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 17179869184 (16384 MB) avail memory = 16475140096 (15711 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads random: unblocking device. ioapic0 irqs 0-23 on motherboard ... i am trying to boot the system as Dom0 under Xen (4.5.2_2 installed via pkg install). The kernel boots in one of four cases, mostly though I don't get a block cursor after Xen messages and the machine sits and waits. After trying Xen suddenly the system no longer boots again: Booting from local disk... PXE-M0F: Existing Intel Boot Agent. ZFS: i/o error - all block copies unavailable ZFS: i/o error - all block copies unavailable ZFS: i/o error - all block copies unavailable Can't find /boot/zfsloader FreeBSD/x86 boot Default: zroot:/boot/kernel/kernel boot: ZFS: i/o error - all block copies unavailable (...) The zpool imports without problems when booting from the rescue mfsbsd (10.3): pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p3 ONLINE 0 0 0 ada1p3 ONLINE 0 0 0 errors: No known data errors gpart layout: => 34 5860533101 ada0 GPT (2.7T) 34 1024 1 freebsd-boot (512K) 1058 4194304 2 freebsd-swap (2.0G) 4195362 5856337773 3 freebsd-zfs (2.7T) => 34 5860533101 ada1 GPT (2.7T) 34 1024 1 freebsd-boot (512K) 1058 4194304 2 freebsd-swap (2.0G) 4195362 5856337773 3 freebsd-zfs (2.7T) I have managed to make zpool boot again by doing voodoo similar to this one: [root@rescue ~]# zpool import -R /mnt zroot [root@rescue ~]# mount -t devfs devfs /mnt/dev [root@rescue ~]# chroot /mnt /bin/tcsh (... Running make install in /usr/src/sys/boot ...) root@rescue:/ # gpart bootcode -p /boot/gptzfsboot -i 1 ada0 partcode written to ada0p1 root@rescue:/ # gpart bootcode -p /boot/gptzfsboot -i 1 ada1 partcode written to ada1p1 root@rescue:/ # exit [root@rescue ~]# umount /mnt/dev [root@rescue ~]# zpool export zroot [root@rescue ~]# reboot Why zpool metadata get corrupted? -- Marcin Xen commandline from /boot/loader.conf: xen_kernel="/boot/xen" xen_cmdline="dom0_mem=2048M dom0_max_vcpus=4 dom0pvh=1 com1=115200,8n1 guest_loglvl=all loglvl=all" dmesg when started under Xen successfully: Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.0-CURRENT #0 r298488: Sat Apr 23 11:10:01 UTC 2016 root@o.saper.info:/usr/obj/usr/src/sys/GENERIC amd64 FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0) WARNING: WITNESS option enabled, expect reduced performance. VT(vga): text 80x25 CPU: Intel(R) Xeon(R) CPU E31245 @ 3.30GHz (3300.02-MHz K8-class CPU) Origin="GenuineIntel" Id=0x206a7 Family=0x6 Model=0x2a Stepping=7 Features=0x1fc3ebff Features2=0x9fb82283 AMD Features=0x20100800 AMD Features2=0x1 XSAVE Features=0x1 TSC: P-state invariant, performance statistics Hypervisor: Origin = "XenVMMXenVMM" real memory = 2152079360 (2052 MB) avail memory = 2025705472 (1931 MB) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s)