From owner-freebsd-virtualization@FreeBSD.ORG Thu Dec 19 08:37:25 2013 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 81B01315; Thu, 19 Dec 2013 08:37:25 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9ADCB1908; Thu, 19 Dec 2013 08:37:24 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,512,1384300800"; d="scan'208";a="83823740" Received: from accessns.citrite.net (HELO FTLPEX01CL03.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 19 Dec 2013 08:37:15 +0000 Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.80) with Microsoft SMTP Server id 14.2.342.4; Thu, 19 Dec 2013 03:37:15 -0500 Message-ID: <52B2B03B.7060903@citrix.com> Date: Thu, 19 Dec 2013 09:37:15 +0100 From: =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: John Baldwin Subject: Re: Panic starting a bhyve guest after resume References: <201312121511.38608.jhb@freebsd.org> <52AC13B1.8060402@citrix.com> <201312181502.27806.jhb@freebsd.org> In-Reply-To: <201312181502.27806.jhb@freebsd.org> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 8bit X-DLP: MIA2 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Dec 2013 08:37:25 -0000 On 18/12/13 21:02, John Baldwin wrote: > On Saturday, December 14, 2013 3:15:45 am Roger Pau Monné wrote: >> On 14/12/13 03:28, Neel Natu wrote: >>> Hi John, >>> >>> On Fri, Dec 13, 2013 at 2:09 PM, John Baldwin wrote: >>>> On Thursday, December 12, 2013 4:00:08 pm Neel Natu wrote: >>>>> Hi John, >>>>> >>>>> On Thu, Dec 12, 2013 at 12:11 PM, John Baldwin wrote: >>>>>> If I suspend and resume my laptop and then try to start a guest after > the >>>>>> resume, I get an odd panic. It generates a privileged instruction > fault (in >>>>>> kernel mode) for 'vmclear'. I've checked CR4 and it claims that VMXE > is set. >>>>>> I dont have any other ideas off the top of my head on what I should be > poking >>>>>> at? It looks like we read a bunch of MSRs in vmx_init(), but we don't > write >>>>>> to them, and all vmx_enable() does on each CPU is set VMXE in CR4 from > what I >>>>>> can tell. >>>>>> >>>>> >>>>> It also does a "vmxon" on each logical cpu which may also need to be >>>>> done after a resume. >>>> >>>> Ah, yes it does. That was sufficient both for starting a new guest after >>>> resume and even doing a suspend/resume while a guest was active (and the >>>> guest continued to run fine). I have a hacky patch for this. One, it >>>> includes both a suspend and resume hook for VMM, though for my testing I > only >>>> needed a resume hook to invoke vmxon. Second, the name of vmx_resume2() >>>> is a total hack (because vmx_resume() was already taken. I think for now >>>> if I were to commit this, I'd just add the resme hook and maybe call the >>>> Intel method vmx_reset() or vmx_restore()? >>>> >>>> http://people.freebsd.org/~jhb/patches/bhyve_resume.patch >>>> >>> >>> There seems to be a race after the APs are restarted and before >>> 'vmm_resume_p()' where it would be problematic to execute a VMX >>> instruction. >>> >>> Perhaps we should enable VMX on each cpu before they return to the >>> interrupted code? >> >> Can you use the hook in cpususpend_handler? It's cpu_ops.cpu_resume, and >> gets called on each CPU before returning from the handler. > > That is the right place, yes. However, I'm worried about collisions. Can you > run nested VMM's under Xen? That is, can a xenhvm guest start a bhyve vmm? In theory yes, but AFAIK nested virtualization support in Xen is not really stable (it's still marked as experimental). > If so, then you would need to run both cpu_resume handlers. Also, cpu_resume > isn't run on the CPU that initiates the suspend. For now I will stick with a > dedicated vmm_resume_p hook, but we may want to revisit that at some point. Using something like an event handler would be the best solution, but that doesn't work on interrupt context due to the usage of mutexes.