From owner-freebsd-virtualization@FreeBSD.ORG Tue Dec 23 02:21:50 2014 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4487CBE4; Tue, 23 Dec 2014 02:21:50 +0000 (UTC) Received: from mail-wg0-x230.google.com (mail-wg0-x230.google.com [IPv6:2a00:1450:400c:c00::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CB71664097; Tue, 23 Dec 2014 02:21:49 +0000 (UTC) Received: by mail-wg0-f48.google.com with SMTP id y19so8025683wgg.35; Mon, 22 Dec 2014 18:21:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7THP3RhMp4vUXyT+5efkzqQc3niG0Qf7N2VxYniAgWQ=; b=fU8KnBtH98ulZiD+aiQKxd4bRI2G29c+w4w6HPRRwP3Py4DDdpAsSe8Pk+T5dPkv8g aWUGXOl/e2LBokdxkSRFHmaEeB/vCWzVuTf21L2j2r/h/w+1Bm2elD1uebLjde4zRDSR WRaxcm33vErsAuEB511rwxIUwhfoU2EH0nLcDYNw8xedTng4xA9CGXnv/AIxhH7SF99w 3+M1MdAAnxlsuf7kEdSynZok0CK2whZVVsr83wo9AJexlLBb/Vnj3VrQwl/I7NzdHddc KVEyTIJWIn5soGinYF8pkoTnU0IVfn5kqycunEKzCZD9htsU6oUvNZ7qzdLf7gTxFYms lp8g== MIME-Version: 1.0 X-Received: by 10.180.109.46 with SMTP id hp14mr37036371wib.54.1419301307719; Mon, 22 Dec 2014 18:21:47 -0800 (PST) Received: by 10.27.5.207 with HTTP; Mon, 22 Dec 2014 18:21:47 -0800 (PST) In-Reply-To: References: <54773CF8.2020306@freebsd.org> <54789699.7030507@freebsd.org> Date: Mon, 22 Dec 2014 18:21:47 -0800 Message-ID: Subject: Re: bhyve failure From: Neel Natu To: Julian Elischer Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Dec 2014 02:21:50 -0000 Hi Julian, On Sat, Nov 29, 2014 at 6:31 PM, Neel Natu wrote: > Hi Julian, > > On Fri, Nov 28, 2014 at 7:36 AM, Julian Elischer wrote: >> On 11/28/14, 3:12 AM, Neel Natu wrote: >>> >>> Hi Julian, >>> >>> On Thu, Nov 27, 2014 at 7:02 AM, Julian Elischer >>> wrote: >>>> >>>> I added WITNESS to a kernel I've been running and during boot I got this >>>> result: >>>> >>>> vm exit[0] >>>> reason VMX >>>> rip 0xffffffff8078c1a7 >>>> inst_length 7 >>>> status 0 >>>> exit_reason 2 >>> >>> That's a triple fault (from sys/amd64/vmm/intel/vmcs.h) >>> >>> #define EXIT_REASON_TRIPLE_FAULT 2 >>> >>> >>>> qualification 0x0000000000000000 >>>> inst_type 0 >>>> inst_error 0 >>>> Abort trap (core dumped) >>>> >>>> anyone have a decoder ring? >>>> >>> A few questions to narrow this down: >>> >>> Did you add WITNESS to the guest kernel or the host kernel when this >>> happened? >> >> yes I added it to the guest kernel.. I suspect it ay be related to this, >> especially since there was a kernel module loaded that was not recompiled. >> I'm trying to ascertain now whether a withness kernel requires special >> handling for its modules. >> >> >>> >>> Is this readily reproducible? >> >> don't know yet. >> > > Its interesting that the guest went all the way to a triple fault > instead of stopping at the double fault handler. > > I'll be happy to track this down if there is a reliably repro case > (guest binaries would be sufficient). > I just committed a change that should let you trace every exception seen by the guest: https://svnweb.freebsd.org/changeset/base/276098 It will let you trace all the exceptions leading up to the triple fault via ktr(4). best Neel > best > Neel > >>> >>> Can you share the host and guest revision numbers? >> >> >> guest is custom sources based on stable-10 on the same day as 10.0 was >> released.basically whatever svn revision# was on 10.0 but on the stable >> branch. >> host is >> FreeBSD 10.1-BETA1 #0 r271671: Tue Sep 16 09:31:23 PDT 2014 >> >> >> >> >>> >>> best >>> Neel >>> >>>> julian >>>> _______________________________________________ >>>> freebsd-virtualization@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >>>> To unsubscribe, send any mail to >>>> "freebsd-virtualization-unsubscribe@freebsd.org" >> >> From owner-freebsd-virtualization@FreeBSD.ORG Fri Dec 26 14:26:02 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3A1726C6 for ; Fri, 26 Dec 2014 14:26:02 +0000 (UTC) Received: from mail-vc0-x229.google.com (mail-vc0-x229.google.com [IPv6:2607:f8b0:400c:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E66FB2427 for ; Fri, 26 Dec 2014 14:26:01 +0000 (UTC) Received: by mail-vc0-f169.google.com with SMTP id hy10so3958652vcb.14 for ; Fri, 26 Dec 2014 06:26:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=0L0dj4TCCLe/m7FSsv3nk6bwkVe5audzaX1MPDqNLWs=; b=jpBJI5JzeF5oXtAT7IM+MEVJffUYnuI2WGVanrI2DxhLznXP8J5dgou42t6QOzY74o derHsnN9MhwqQsRopZkCDr2zxWO4/V2Z03JhwE2Gnl3RV9Lnwv68zPRJNZb5GIaVLVz1 g0YYn5YdCCAmT2kKON+4Os3n8R0LqHwgQ91Ut0QX/MLp0NLDcy6vT+IHgErREqzdpNvq AhGaJ7QPE0GSHXIqxLjCVAt6ZcFBcUnRvH8eMGXow2vCmBZx9aVPrqzgolPHEAqxtXRn 3QYPc2WUfJC/h5DZzexGOueVrfeNUnHXQxldIv2glo7bd3E3W5/PQx4XtM2PYgsifTAD lFdQ== MIME-Version: 1.0 X-Received: by 10.220.103.136 with SMTP id k8mr20851305vco.14.1419603961013; Fri, 26 Dec 2014 06:26:01 -0800 (PST) Received: by 10.52.35.161 with HTTP; Fri, 26 Dec 2014 06:26:00 -0800 (PST) Date: Fri, 26 Dec 2014 09:26:00 -0500 Message-ID: Subject: FreeBSD bhyve VM Exiting From: Shawn Webb To: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Dec 2014 14:26:02 -0000 Hey All, I'm running a FreeBSD/amd64 11-current (r276204) guest VM in bhyve. It seems that running Poudriere in it causes the VM to exit. This is what bhyve outputs when the VM dies: reason VMX rip 0xffffffff80d68564 inst_length 5 status 0 exit_reason 2 qualification 0x0000000000000000 inst_type 0 inst_error 0 I have no idea what "reason VMX" means or what "exit_reason 2" means. Is there a way to decode that? What kinds of info/data do you guys need to debug this? Thanks, Shawn From owner-freebsd-virtualization@FreeBSD.ORG Fri Dec 26 20:07:46 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6C99F9D7 for ; Fri, 26 Dec 2014 20:07:46 +0000 (UTC) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [IPv6:2a00:1450:400c:c00::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F38FF64B63 for ; Fri, 26 Dec 2014 20:07:45 +0000 (UTC) Received: by mail-wg0-f46.google.com with SMTP id x13so14930408wgg.5 for ; Fri, 26 Dec 2014 12:07:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=+vWqtylCJKuES7OcqowMHQP6nuBWMG1E1qoZvxnDrJQ=; b=fFey19v9v9GgXNMPQO7ayvyRjUE7aG0PEkgV2dEcO0YBm+T8gKpiVjWXco+4Z+VxRf 7AkqvDdv9HaDjp/g5gXGWwBTdZmiS8eR7qMwnofM9RYqn6Ww0Ap6OBafcr13ItxpPSV/ BAZJPvt2W5WnKxldHcP1/ISE8ciN/q7LGjOn1+7Ku03UT2ICYF84cMOW7jXP/KjFEJsO Km5nLzsUi88FwDmVsVNOKMzekWEqlBcmxgMTQWNzMmm44EbDGoHsU/topTMO4CmJzxUt oHqowTkhc2aiSyc5hyfgQYWBOWtkdhL4TQiwMedi3sWDitMBHt4gmbuHt1ZQxPFxbV8I JJLQ== MIME-Version: 1.0 X-Received: by 10.180.75.237 with SMTP id f13mr70646556wiw.69.1419624464268; Fri, 26 Dec 2014 12:07:44 -0800 (PST) Received: by 10.27.5.207 with HTTP; Fri, 26 Dec 2014 12:07:44 -0800 (PST) In-Reply-To: References: Date: Fri, 26 Dec 2014 12:07:44 -0800 Message-ID: Subject: Re: FreeBSD bhyve VM Exiting From: Neel Natu To: Shawn Webb Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Dec 2014 20:07:46 -0000 Hi Shawn, On Fri, Dec 26, 2014 at 6:26 AM, Shawn Webb wrote: > Hey All, > > I'm running a FreeBSD/amd64 11-current (r276204) guest VM in bhyve. It > seems that running Poudriere in it causes the VM to exit. > > This is what bhyve outputs when the VM dies: > > reason VMX > rip 0xffffffff80d68564 > inst_length 5 > status 0 > exit_reason 2 > qualification 0x0000000000000000 > inst_type 0 > inst_error 0 > > I have no idea what "reason VMX" means or what "exit_reason 2" means. Is > there a way to decode that? > Exit reason '2' is a triple fault. > What kinds of info/data do you guys need to debug this? > You'll need a FreeBSD host with svn revision >= 276098 and a custom kernel config to enable KTR. https://svnweb.freebsd.org/base?view=revision&revision=276098 Then you can set the following tunable "hw.vmm.trace_guest_exceptions" to "1" before loading vmm.ko. This will cause bhyve to trace all guest exceptions via KTR. You can verify that the tunable took effect using 'sysctl hw.vmm.trace_guest_exceptions'. When the bhyve process exits you can then dump the KTR entries using something like this: 'sudo ktrdump -cto /tmp/ktrdump.out'. You'll probably want to do this in a script so the ktrdump(8) happens immediately after bhyve(8) exits. I use the following options in my kernel config to enable KTR: options KTR options KTR_MASK=(KTR_GEN) options KTR_ENTRIES=(4*1024*1024) Hope this helps. best Neel > Thanks, > > Shawn > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Sat Dec 27 19:23:03 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 637EF721 for ; Sat, 27 Dec 2014 19:23:03 +0000 (UTC) Received: from mail-qc0-x22e.google.com (mail-qc0-x22e.google.com [IPv6:2607:f8b0:400d:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1456B1339 for ; Sat, 27 Dec 2014 19:23:03 +0000 (UTC) Received: by mail-qc0-f174.google.com with SMTP id c9so8169729qcz.33 for ; Sat, 27 Dec 2014 11:23:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:user-agent:in-reply-to :references:mime-version:content-type; bh=ScixE67LjN7LqxVwEHrC8aD5m2OAHqr3WOVoIYPKnZg=; b=lUBH/O2//dP747EUb6/h8QUAE3B08abc/WzWiEsCTm8sk06tlXe+lQjdlEwipIOmwr DOavol0eklKcMSpGSsH+1qw1CLIkWpgnqc9jqlQvthSELZLHIISYgUlH1tFHZnbcVviV iI65T2rt7F65J7M5F0Pw0P2jLt/Aq20/b0GoSYn4JLmhWO4iM/rwgJ6/kI5ojXlSbOPD 6B54y6m8b6e51tO73a+OHRAMcpyeCgBBq7Mgxj/n3P+lvNF/HmU+hTu6gRl2oft+TTY0 ciwbKJ/agtj8qY9U1ygF2blOupryBr08tg5IcScFpzeb5QCshPsUS5iTuK3Aaitd6Cf3 AH/Q== X-Received: by 10.224.171.129 with SMTP id h1mr33064337qaz.74.1419708182082; Sat, 27 Dec 2014 11:23:02 -0800 (PST) Received: from hbsd-dev-local.localnet ([73.173.99.185]) by mx.google.com with ESMTPSA id k102sm8990096qgd.7.2014.12.27.11.23.01 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 27 Dec 2014 11:23:01 -0800 (PST) From: Shawn Webb To: Neel Natu Subject: Re: FreeBSD bhyve VM Exiting Date: Sat, 27 Dec 2014 14:22:57 -0500 Message-ID: <2487562.Nj6jnlQzjX@hbsd-dev-local> User-Agent: KMail/4.14.2 (FreeBSD/11.0-CURRENT; KDE/4.14.2; amd64; ; ) In-Reply-To: References: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1728346.4RnBHVCunU"; micalg="pgp-sha256"; protocol="application/pgp-signature" Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Dec 2014 19:23:03 -0000 --nextPart1728346.4RnBHVCunU Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" On Friday, December 26, 2014 12:07:44 PM Neel Natu wrote: > Hi Shawn, > > On Fri, Dec 26, 2014 at 6:26 AM, Shawn Webb wrote: > > Hey All, > > > > I'm running a FreeBSD/amd64 11-current (r276204) guest VM in bhyve. It > > seems that running Poudriere in it causes the VM to exit. > > > > This is what bhyve outputs when the VM dies: > > reason VMX > > rip 0xffffffff80d68564 > > inst_length 5 > > status 0 > > exit_reason 2 > > qualification 0x0000000000000000 > > inst_type 0 > > inst_error 0 > > > > I have no idea what "reason VMX" means or what "exit_reason 2" means. Is > > there a way to decode that? > > Exit reason '2' is a triple fault. > > > What kinds of info/data do you guys need to debug this? > > You'll need a FreeBSD host with svn revision >= 276098 and a custom > kernel config to enable KTR. > https://svnweb.freebsd.org/base?view=revision&revision=276098 > > Then you can set the following tunable "hw.vmm.trace_guest_exceptions" > to "1" before loading vmm.ko. This will cause bhyve to trace all guest > exceptions via KTR. You can verify that the tunable took effect using > 'sysctl hw.vmm.trace_guest_exceptions'. > > When the bhyve process exits you can then dump the KTR entries using > something like this: 'sudo ktrdump -cto /tmp/ktrdump.out'. You'll > probably want to do this in a script so the ktrdump(8) happens > immediately after bhyve(8) exits. > > I use the following options in my kernel config to enable KTR: > options KTR > options KTR_MASK=(KTR_GEN) > options KTR_ENTRIES=(4*1024*1024) > > Hope this helps. > > best > Neel Hey Neel, Thanks for the helpful suggestions! The good thing is that my host is already on a revision >= 276098 and my kernel is already built with KTR. With 31c3 going on, the soonest I can do that will be Tuesday or Wednesday of next week. I'll update you then. Thanks, Shawn --nextPart1728346.4RnBHVCunU Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJUnwcRAAoJEGqEZY9SRW7udhAP/1T0NtXGwdTLFdlpSg9YNy1f UkmIC4KYgdGLYoy57y13Fp/6doUoPqPLpH6/N2iU32pUvluPXwEhzYg1Zu6e1gaC MzdSrCCMwV0nfGuoxpcHcY1C7xneKfXX6L+vqPi7axIL59SGmx+TI/fPRvjvTtee il8vT8QCNYTNyYEo7IGCQKptDRjvg/JTHn8PCBEdG3K4M20uxa2QU/XfxwCWC5Ui E+JRm6ZFoJtVFUd5ViqNrPik57s0mBJ09SjtjsXaLx/biC7z0VQNqHBRjnm7LNgL YoCFkTgjq93BnQAZucmDBEbKjS1B9Ty1pezmRSfS9ln5+ylrTQ8B05XOlJrT72ra jJ0mom995Zzb75epZ7KeHHBbCvxfy8nhXfBweZiYPBxQlzU7SHbINC+bE1fN1xQs nW0RmBIfCuByy3O/IYXTNsu8dhwknJAVWvru7gXKSV17SWAfKbtl2oJGr68swkgu fk5hspmLN+MK+zMR5LveARjNDN77VxFF1MZob51QBbCTgvHkpC1WnUcbRmQ8sQ/g faaWx1beIy3egjB+ZgAbeCQjRqNWZ8BhDh1ZiNLrOylaAdO2BcQM5FUXhHNqcQhu fxR1dlNCp0Lc5eIrrZQCmR1LrFUnwREcgT42fKslFG9EnNtLBIaeC6eSRSkf0Mqe UzgN/BgP+LnPMZtw9p9w =nEMY -----END PGP SIGNATURE----- --nextPart1728346.4RnBHVCunU--