From owner-freebsd-virtualization@freebsd.org Wed Apr 20 16:14:26 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C5C9B16382 for ; Wed, 20 Apr 2016 16:14:26 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-lb0-x241.google.com (mail-lb0-x241.google.com [IPv6:2a00:1450:4010:c04::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A4EA71107 for ; Wed, 20 Apr 2016 16:14:25 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: by mail-lb0-x241.google.com with SMTP id mx9so1481633lbb.2 for ; Wed, 20 Apr 2016 09:14:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=g75FcyJzN0yW4FbnPECHBTbRrpjJyyUQ45DTXjh5OS0=; b=kKKY45amuX+JYBHHQDYF0aps6O9zIxa1ju2mzumc+S/9LcXwEdYelkbvbz4ES/QcLK UMGKom/NSAxbaZq2kxiL8pVZbnPnzWXS+OiDePIxqRxfuiRETNd1u7SJC+c7q8XiSRoh VPoRA0UqxhaqfIm0j/D/xASvUaU64dMeNDav0LdEsHg1N5pjxU/PEWb+pmmQoiTozgZk BJuOFn04sVwEvY0W4+8wDGFEmh7y1FHWXDYB/j7C4PCUeoFDftLS8ObteGU/BdgOdQYx RFA7f6IR1HNwMFNywh6Q0vveBsXOODzDznfEi6MRTYL5xAJmSE9rvmCHKDRCXkuK9+Ux VU4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=g75FcyJzN0yW4FbnPECHBTbRrpjJyyUQ45DTXjh5OS0=; b=OQMQDrki1a1KqbNlYTYqxC/dEExap3PIpnmg8eWEcjREYTJY5hzE+VTiDNwPsBvQ1h swl1CTSA+HjaykErxx0MHVl24RlK/NyZ2W8GOTdhr805OJU49qhDNHEvxH9fbXW58yNM H+8RUNnEawcDyVxnpzs5WTBqceqY/fmC0gI/Ta+HHD25LrDsWabONVoRwjmqArSJMsFC x4XrwEwHMdYPtZzfv1xGemLLR50ZFHa0qwFR4lyvCgnhr16nwJZJY3vrs/nw+3aho1Xc N7lbn15HGb1A6z+NNejownNxwYICvEZZEvj/lAlfpXMla7uLC+HAHMn4CXkjyEiFSdHL 7vzw== X-Gm-Message-State: AOPr4FUY44iwJaEWBVxQ9ecCEb+mk68uyZ8xGjFLselIyyyZpM+Txbjpl3c+d9dEjS68iw== X-Received: by 10.112.85.43 with SMTP id e11mr4294012lbz.80.1461168863469; Wed, 20 Apr 2016 09:14:23 -0700 (PDT) Received: from kloomba ([31.29.237.95]) by smtp.gmail.com with ESMTPSA id 28sm1203394lfx.43.2016.04.20.09.14.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 20 Apr 2016 09:14:22 -0700 (PDT) Sender: Roman Bogorodskiy Date: Wed, 20 Apr 2016 19:14:16 +0300 From: Roman Bogorodskiy To: Neel Natu Cc: Roman Bogorodskiy , "freebsd-virtualization@freebsd.org" Subject: Re: Understanding Bhyve shutdown Message-ID: <20160420161414.GA87187@kloomba> References: <20160413105520.GB84953@dev.san.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="uAKRQypu60I7Lcqm" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.0 (2016-04-01) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Apr 2016 16:14:26 -0000 --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Neel Natu wrote: > Hi Roman, >=20 > On Wed, Apr 13, 2016 at 3:55 AM, Roman Bogorodskiy > wrote: > > Hi, > > > > I was trying to get better understanding of how to properly shutdown VMs > > in bhyve, but unfortunately the documentation does not provide much > > details on that. > > > > Specifically, handbook [I] suggests to reboot a machine and then run > > bhyvectl --destroy on it. > > > > The bhyvectl(8) manpage mentions the '--force-reset' and > > '--force-poweroff' switches, but does not give details on those. > > > > I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html > > > > I tried all the options I know and wrote down the results. I also have > > some questions, hopefully you'll be able to answer some of them. > > > > 1. bhyvectl --vm=3D$name --destroy > > > > * looks like hard poweroff in the guest > > * the corresponding bhyve(8) process goes away > > * /dev/vmm/ entry goes away > > > > In my experience, it's a dangerous way to shutdown a VM because > > sometimes it appears it damages the image and VM fails to boot with > > something like this: > > > > --- > > Starting devd. > > mode =3D 0100600, inum =3D 170269, fs =3D / > > panic: ffs_valloc: dup alloc > > cpuid =3D 0 > > KDB: stack backtrace: > > #0 0xffffffff80984e30 at kdb_backtrace+0x60 > > #1 0xffffffff809489e6 at vpanic+0x126 > > #2 0xffffffff809488b3 at panic+0x43 > > #3 0xffffffff80b74a6e at ffs_valloc+0x84e > > #4 0xffffffff80bb60ad at ufs_makeinode+0x7d > > #5 0xffffffff80bb24fd at ufs_create+0x2d > > #6 0xffffffff80e71841 at VOP_CREATE_APV+0xa1 > > #7 0xffffffff809cd9e6 at uipc_bindat+0x346 > > #8 0xffffffff809c5488 at kern_bindat+0x108 > > #9 0xffffffff809c52a7 at sys_bind+0x77 > > #10 0xffffffff80d4b3f7 at amd64_syscall+0x357 > > #11 0xffffffff80d30adb at Xfast_syscall+0xfb > > Uptime: 3s > > > > Dump failed. Partition too small. > > --- > > >=20 > Yup, this is biggest hammer you could use to shutdown a virtual > machine. As you noticed, this is usually a bad thing because it does > not give the guest OS an opportunity to shutdown cleanly. >=20 > > 2. kill -SIGTERM $bhyve_pid > > > > If guest supports ACPI shutdown: > > > > * guest shuts down cleanly > > * the corresponding bhyve(8) process terminates > > * /dev/vmm entry is still here, need bhyvectl --destroy for complete > > cleanup > > > > If guest does not support ACPI shutdown (such as doing sysctl > > hw.acpi.power_button_state=3DNONE): > > > > * Nothing happens > > > > Q1: Is there a way to know if a guest reacted to power button but > > waiting for the bhyve process to terminate? >=20 > Not really, except to wait for some amount of time to give the guest a > chance to shutdown. >=20 > > Q2: Why it's not done via bhyvectl (it seems that it's easier for users > > + don't have to overload a useful SIGTERM signal) > > >=20 > It seems natural to overload SIGTERM to do this. For e.g. when the > host is shutting down it will send a SIGTERM to all running processes > and translating this into an ACPI poweroff event for the guest allows > it to shutdown cleanly. >=20 > > 3. bhyvectl --vm=3D$name --force-poweroff > > > > * looks like hard poweroff in the guest > > * the corresponding bhyve(8) process goes away > > * /dev/vmm entry is still here, need bhyvectl --destroy for complete > > cleanup > > > > Q: what's the practical difference with just doing --destroy right away? > > >=20 > 'force-poweroff' is equivalent to a hard power off on real hardware. >=20 > The only practical difference between '--force-poweroff' and > '--destroy' is that destroy will also remove the device node in > /dev/vmm and release any memory allocated for the guest back to the > host. >=20 > > 4. bhyvectl --vm=3D$name --force-reset > > > > Looks very similar to item #3 with just different exit code (reboot > > appears to be using 0, while shutdown and halt use 1 and 2). > > > > Q: what's the practical use of it? > > >=20 > The exit code can be used by scripts on top of 'bhyve' to decide > whether to restart execution of the guest (reset) or stop completely > (poweroff). >=20 > > Would greatly appreciate if somebody could provide more details on that. > > I guess we'll need to update Handbook with this information as well > > because it needs to mention SIGTERM for ACPI shutdown at least. > > >=20 > Hope that helps. >=20 > best > Neel Neel, thanks for the response, that's a kind of an answer I was looking for! One question regarding that '--force-poweroff' vs '--destroy' thing: what is the point of not releasing guest resources by doing '--force-powerof'? Initially I thought it could be useful to speed up the next boot by not requiring running 'bhyveload' (or other loader) for the guest again, but, apparently, it's still required to run '--destroy' and then bhyveload again, otherwise I get something like this: bhyve: could not activate CPU 0: Device busy I'm wondering if that's just for debugging purposes or there is something else? Sorry for being extra curious. :-) Roman Bogorodskiy --uAKRQypu60I7Lcqm Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAEBAgAGBQJXF6rWAAoJEMltX/4IwiJq1+IH/iAbACAob+uicIyhcKPfh6zv ahRnSoxYB3kNkqAxGqF05RWxJEqs8v/n/vjbbJzgIejo5zCoKm3Rs2s31xPbi1X7 /DCvQ91j2/QmqHZSLeNNvKUp4o8y0Azm5rgJ/uLZTVN2cg8GixVU+OshSmhad60T MqXzJJLUV26i9JMaa+5xnFxHgo+CorHSO0JQhvFn3DY4D4AZmxO6OfAGdcHuUGJ6 IlzuYhEG5aD4U5nigDtUoU784FJLLM9b+MLW4OPI0I2YVsKUG9jF80lsugWy8vdo j0mUfYiIB5lLA8PrLn/1lzto+f5zQBr7pajrRx6laFclcnw1hei5mCs70alzjTE= =NGYJ -----END PGP SIGNATURE----- --uAKRQypu60I7Lcqm--