From owner-freebsd-virtualization@FreeBSD.ORG Sat Dec 14 08:17:12 2013 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1392441F; Sat, 14 Dec 2013 08:17:12 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 146F31124; Sat, 14 Dec 2013 08:17:10 +0000 (UTC) X-IronPort-AV: E=Sophos;i="4.95,484,1384300800"; d="scan'208";a="82241502" Received: from accessns.citrite.net (HELO FTLPEX01CL01.citrite.net) ([10.9.154.239]) by FTLPIPO02.CITRIX.COM with ESMTP; 14 Dec 2013 08:15:45 +0000 Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.2.342.4; Sat, 14 Dec 2013 03:15:44 -0500 Message-ID: <52AC13B1.8060402@citrix.com> Date: Sat, 14 Dec 2013 09:15:45 +0100 From: =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Neel Natu , John Baldwin Subject: Re: Panic starting a bhyve guest after resume References: <201312121511.38608.jhb@freebsd.org> <201312131709.20264.jhb@freebsd.org> In-Reply-To: X-Enigmail-Version: 1.6 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: 7bit X-DLP: MIA1 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Dec 2013 08:17:12 -0000 On 14/12/13 03:28, Neel Natu wrote: > Hi John, > > On Fri, Dec 13, 2013 at 2:09 PM, John Baldwin wrote: >> On Thursday, December 12, 2013 4:00:08 pm Neel Natu wrote: >>> Hi John, >>> >>> On Thu, Dec 12, 2013 at 12:11 PM, John Baldwin wrote: >>>> If I suspend and resume my laptop and then try to start a guest after the >>>> resume, I get an odd panic. It generates a privileged instruction fault (in >>>> kernel mode) for 'vmclear'. I've checked CR4 and it claims that VMXE is set. >>>> I dont have any other ideas off the top of my head on what I should be poking >>>> at? It looks like we read a bunch of MSRs in vmx_init(), but we don't write >>>> to them, and all vmx_enable() does on each CPU is set VMXE in CR4 from what I >>>> can tell. >>>> >>> >>> It also does a "vmxon" on each logical cpu which may also need to be >>> done after a resume. >> >> Ah, yes it does. That was sufficient both for starting a new guest after >> resume and even doing a suspend/resume while a guest was active (and the >> guest continued to run fine). I have a hacky patch for this. One, it >> includes both a suspend and resume hook for VMM, though for my testing I only >> needed a resume hook to invoke vmxon. Second, the name of vmx_resume2() >> is a total hack (because vmx_resume() was already taken. I think for now >> if I were to commit this, I'd just add the resme hook and maybe call the >> Intel method vmx_reset() or vmx_restore()? >> >> http://people.freebsd.org/~jhb/patches/bhyve_resume.patch >> > > There seems to be a race after the APs are restarted and before > 'vmm_resume_p()' where it would be problematic to execute a VMX > instruction. > > Perhaps we should enable VMX on each cpu before they return to the > interrupted code? Can you use the hook in cpususpend_handler? It's cpu_ops.cpu_resume, and gets called on each CPU before returning from the handler.