From owner-freebsd-virtualization@freebsd.org Wed Apr 13 14:15:33 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 633B0B0EBF5 for ; Wed, 13 Apr 2016 14:15:33 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-outbound.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.userve.net", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1588A1775 for ; Wed, 13 Apr 2016 14:15:32 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) by smtp-outbound.userve.net (8.15.1/8.15.1) with ESMTPS id u3DEFG5h091864 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=FAIL); Wed, 13 Apr 2016 15:15:17 +0100 (BST) (envelope-from matt.churchyard@userve.net) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=201508; t=1460556919; bh=GTJB8VNwGaHpoi9j5aE/ZBGqGB1J/8saNZNuyuz4IXk=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=WBmqzCtdhcICHhLDyA1JUhWiBFW4yU+s+lQQh/69D9JgWnC6oQ/bSNffMMpeE7k88 nqsd/Ne16MslRYMzp7OvNNS+kLmmKdGpvjbDeRkV0ttN+gQWpOojE2ehEkv8adrsIB /8hbdwutBXNx4hyXTOgjxs/A7cbFivZ1WqgCE3zE= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Wed, 13 Apr 2016 15:15:11 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Wed, 13 Apr 2016 15:15:10 +0100 From: Matt Churchyard To: Roman Bogorodskiy CC: FreeBSD virtualization Subject: RE: Understanding Bhyve shutdown Thread-Topic: Understanding Bhyve shutdown Thread-Index: AQHRlXMDYRnGHYTzkkieaIiropecY5+H6fIQ Date: Wed, 13 Apr 2016 14:15:10 +0000 Message-ID: <548d6c81c4784447b91f6f1459863eed@SERVER.ad.usd-group.com> References: <20160413105520.GB84953@dev.san.ru> In-Reply-To: <20160413105520.GB84953@dev.san.ru> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Apr 2016 14:15:33 -0000 As I understand it 1) shutdown from guest 2) 'kill ' -> pressing the power button once. 3) --force-poweroff -> holding power button in 4) --force-reset -> pressing the reset button 5) bhyvectl --destroy -> same as 3? (although vmm is destroyed as well) 1 or 2 are obviously preferred. I've found however that some guests don't r= espond that well to the apci event. CentOS 6 seems to need apci installing = and enabling?!, and my Win2012R2 machine will only shutdown if I send the s= ignal twice. The rest are all hard shutdowns that should not be seen as a way to stop th= e guest, just a last resort if it can't be stopped correctly. I don't know = the practical difference between poweroff&destroy vs just destroy. I see no reason why the documentation suggests reboot rather than shutdown.= Bhyve exits either way and the only difference is the exit code. When usin= g one of the bhyve management tools that support reboots (such as vmrun.sh/= vm-bhyve/iohyve) however, a "restart" exit code (0) will cause the guest to= restart automatically; If a guest is locked up for example, a --force-rese= t will cause it to immediately reboot. >From a host, the only safe choice is 'kill ' (possibly multiple times)= and wait for bhyve process to (hopefully) exit. If that doesn't work eithe= r the acpi issue needs fixing or ideally the guest needs to be stopped usin= g its built-in shutdown function. The devs will have to confirm why they're implemented the way they are. My = instinct is that bhyvectl works on the vmm device, and reset/poweroff are j= ust functions that are possible via that interface, and so have been expose= d. The ACPI shutdown event likely needs to be initiated by the bhyve proces= s itself, hence using a signal to it. /end-speculation I think most users will want to use a bhyve tool so the raw specifics of th= e bhyve/bhyvectl commands are glossed over, although that doesn't mean the = handbook documentation of the base commands shouldn't be as complete/correc= t as possible of course. Matt -----Original Message----- From: owner-freebsd-virtualization@freebsd.org [mailto:owner-freebsd-virtua= lization@freebsd.org] On Behalf Of Roman Bogorodskiy Sent: 13 April 2016 11:55 To: freebsd-virtualization@FreeBSD.org Subject: Understanding Bhyve shutdown Hi, I was trying to get better understanding of how to properly shutdown VMs in= bhyve, but unfortunately the documentation does not provide much details o= n that. Specifically, handbook [I] suggests to reboot a machine and then run bhyvec= tl --destroy on it.=20 The bhyvectl(8) manpage mentions the '--force-reset' and '--force-poweroff'= switches, but does not give details on those.=20 I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html I tried all the options I know and wrote down the results. I also have some= questions, hopefully you'll be able to answer some of them. 1. bhyvectl --vm=3D$name --destroy * looks like hard poweroff in the guest * the corresponding bhyve(8) process goes away * /dev/vmm/ entry goes away In my experience, it's a dangerous way to shutdown a VM because sometimes i= t appears it damages the image and VM fails to boot with something like thi= s: --- Starting devd. mode =3D 0100600, inum =3D 170269, fs =3D / panic: ffs_valloc: dup alloc cpuid =3D 0 KDB: stack backtrace: #0 0xffffffff80984e30 at kdb_backtrace+0x60 #1 0xffffffff809489e6 at vpanic+0x126 #2 0xffffffff809488b3 at panic+0x43 #3 0xffffffff80b74a6e at ffs_valloc+0x84e #4 0xffffffff80bb60ad at ufs_makeinode+0x7d #5 0xffffffff80bb24fd at ufs_create+0x2d #6 0xffffffff80e71841 at VOP_CREATE_APV+0xa1 #7 0xffffffff809cd9e6 at uipc_bindat+0x346 #8 0xffffffff809c5488 at kern_bindat+0x108 #9 0xffffffff809c52a7 at sys_bind+0x77 #10 0xffffffff80d4b3f7 at amd64_syscall+0x357 #11 0xffffffff80d30adb at Xfast_syscall+0xfb Uptime: 3s Dump failed. Partition too small. --- 2. kill -SIGTERM $bhyve_pid If guest supports ACPI shutdown: * guest shuts down cleanly * the corresponding bhyve(8) process terminates * /dev/vmm entry is still here, need bhyvectl --destroy for complete cleanup If guest does not support ACPI shutdown (such as doing sysctl hw.acpi.power_button_state=3DNONE): * Nothing happens Q1: Is there a way to know if a guest reacted to power button but waiting for the bhyve process to terminate? =20 Q2: Why it's not done via bhyvectl (it seems that it's easier for users + don't have to overload a useful SIGTERM signal) 3. bhyvectl --vm=3D$name --force-poweroff * looks like hard poweroff in the guest * the corresponding bhyve(8) process goes away * /dev/vmm entry is still here, need bhyvectl --destroy for complete cleanup Q: what's the practical difference with just doing --destroy right away? 4. bhyvectl --vm=3D$name --force-reset Looks very similar to item #3 with just different exit code (reboot appears to be using 0, while shutdown and halt use 1 and 2). =20 Q: what's the practical use of it? Would greatly appreciate if somebody could provide more details on that. I guess we'll need to update Handbook with this information as well because= it needs to mention SIGTERM for ACPI shutdown at least.=20 Roman Bogorodskiy _______________________________________________ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/m= ailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebs= d.org"