From owner-freebsd-virtualization@freebsd.org Wed Apr 13 10:55:28 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9742BB0FA62 for ; Wed, 13 Apr 2016 10:55:28 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: from mail-lf0-x244.google.com (mail-lf0-x244.google.com [IPv6:2a00:1450:4010:c07::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 28E41180A for ; Wed, 13 Apr 2016 10:55:28 +0000 (UTC) (envelope-from bogorodskiy@gmail.com) Received: by mail-lf0-x244.google.com with SMTP id e190so6658260lfe.1 for ; Wed, 13 Apr 2016 03:55:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:subject:message-id:mime-version:content-disposition :user-agent; bh=wk8KN/7nmGK/0/2Ndx4wVwE/ILtRqJ13VFl+ubbdnFE=; b=BJn0pibyo5tJKlIz3FgCvDTpIOqxiF/2t4wLxrvdVPjnSIEXnav86aICPZK/pbQDsp Dr9Nyd3Na61wQYmEURFHn0fy3IDhl/qCKDQl6CNviAAxYzc1oadUknCIy308PzIMsMxA iyw5cWlb9wVXgILLdOYwJIczkHkwqHjIFVZjYrPJdJ5OW7ZlGU6LUfN3WjJEr7kgmavc m4hC3qrA1R6B7VEDgib5sove3wwVDce2qtzjiU4u9CWM3SZHqDqOLH3Bje6LR3oihLPn ij5/5nRGDEePhnYEDYGXCQoWIGWMeGuLsnmWH0KJ4cpZ/LqJtdomty7sAJtaOtCxwshh Dzhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:subject:message-id:mime-version :content-disposition:user-agent; bh=wk8KN/7nmGK/0/2Ndx4wVwE/ILtRqJ13VFl+ubbdnFE=; b=iyj+nO7y9ZH30BKCWQ3i5uAyZ0LcT7Eh1hJxd42RhcjKJsRyQ31P3IRisR3rRroXOE EDNOwsRVED064UMGRywT63QvN3K8cXzZy45M0SqkHhRzAmXQqAQXzsy1IsnvP2oYo54E h6kfU3Zv5zl//Glz5Sx2rMrMgKCQhh/uGpBjb2v/stvqy52tX0kjNtef1lZJpluPUzWg 9LjaaOGKkggaL05QAbWXkgpcQ+BNoVtXvzF8V1S/FKAheWb/pIJq+Ljv/SYwZSZG0N0e wjDHYUOwj0VT+2fyQTJt8oqZl723LcKZwBKifxGN2jqW3w/4BP875TnLwjU93Q6VHIYx aIpQ== X-Gm-Message-State: AOPr4FUpefyCJUfVIPS3kLNTODs71MH1E0tghp6OszlGJNl1GzN+N3GJw5WseBswfuC41w== X-Received: by 10.25.149.144 with SMTP id x138mr3673735lfd.88.1460544926129; Wed, 13 Apr 2016 03:55:26 -0700 (PDT) Received: from dev.san.ru ([88.147.129.60]) by smtp.gmail.com with ESMTPSA id us9sm1883820lbc.24.2016.04.13.03.55.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 Apr 2016 03:55:24 -0700 (PDT) Date: Wed, 13 Apr 2016 13:55:22 +0300 From: Roman Bogorodskiy To: freebsd-virtualization@FreeBSD.org Subject: Understanding Bhyve shutdown Message-ID: <20160413105520.GB84953@dev.san.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Apr 2016 10:55:28 -0000 Hi, I was trying to get better understanding of how to properly shutdown VMs in bhyve, but unfortunately the documentation does not provide much details on that. Specifically, handbook [I] suggests to reboot a machine and then run bhyvectl --destroy on it. The bhyvectl(8) manpage mentions the '--force-reset' and '--force-poweroff' switches, but does not give details on those. I: https://www.freebsd.org/doc/handbook/virtualization-host-bhyve.html I tried all the options I know and wrote down the results. I also have some questions, hopefully you'll be able to answer some of them. 1. bhyvectl --vm=$name --destroy * looks like hard poweroff in the guest * the corresponding bhyve(8) process goes away * /dev/vmm/ entry goes away In my experience, it's a dangerous way to shutdown a VM because sometimes it appears it damages the image and VM fails to boot with something like this: --- Starting devd. mode = 0100600, inum = 170269, fs = / panic: ffs_valloc: dup alloc cpuid = 0 KDB: stack backtrace: #0 0xffffffff80984e30 at kdb_backtrace+0x60 #1 0xffffffff809489e6 at vpanic+0x126 #2 0xffffffff809488b3 at panic+0x43 #3 0xffffffff80b74a6e at ffs_valloc+0x84e #4 0xffffffff80bb60ad at ufs_makeinode+0x7d #5 0xffffffff80bb24fd at ufs_create+0x2d #6 0xffffffff80e71841 at VOP_CREATE_APV+0xa1 #7 0xffffffff809cd9e6 at uipc_bindat+0x346 #8 0xffffffff809c5488 at kern_bindat+0x108 #9 0xffffffff809c52a7 at sys_bind+0x77 #10 0xffffffff80d4b3f7 at amd64_syscall+0x357 #11 0xffffffff80d30adb at Xfast_syscall+0xfb Uptime: 3s Dump failed. Partition too small. --- 2. kill -SIGTERM $bhyve_pid If guest supports ACPI shutdown: * guest shuts down cleanly * the corresponding bhyve(8) process terminates * /dev/vmm entry is still here, need bhyvectl --destroy for complete cleanup If guest does not support ACPI shutdown (such as doing sysctl hw.acpi.power_button_state=NONE): * Nothing happens Q1: Is there a way to know if a guest reacted to power button but waiting for the bhyve process to terminate? Q2: Why it's not done via bhyvectl (it seems that it's easier for users + don't have to overload a useful SIGTERM signal) 3. bhyvectl --vm=$name --force-poweroff * looks like hard poweroff in the guest * the corresponding bhyve(8) process goes away * /dev/vmm entry is still here, need bhyvectl --destroy for complete cleanup Q: what's the practical difference with just doing --destroy right away? 4. bhyvectl --vm=$name --force-reset Looks very similar to item #3 with just different exit code (reboot appears to be using 0, while shutdown and halt use 1 and 2). Q: what's the practical use of it? Would greatly appreciate if somebody could provide more details on that. I guess we'll need to update Handbook with this information as well because it needs to mention SIGTERM for ACPI shutdown at least. Roman Bogorodskiy