From owner-freebsd-virtualization@FreeBSD.ORG Mon Dec 23 19:15:28 2013 Return-Path: Delivered-To: virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5A2D5840; Mon, 23 Dec 2013 19:15:28 +0000 (UTC) Received: from mail-qc0-x229.google.com (mail-qc0-x229.google.com [IPv6:2607:f8b0:400d:c01::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 07DA616C1; Mon, 23 Dec 2013 19:15:27 +0000 (UTC) Received: by mail-qc0-f169.google.com with SMTP id r5so5295395qcx.14 for ; Mon, 23 Dec 2013 11:15:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=HPCSO1hEN/pnEozwB5X0Us+BmdESuRWr05PcimvxB3Q=; b=k4SezGXsykf2t6LQmi/Z41JT05LtD0Yb4VMOV2Xvxi8v9GG8LCpqE5AhTmdXFj+rEA h0J0igH4H+t5bYbWbTN6TfQ2HrwyM2s9jKUMjXrAMrIARaICAAx3M4MAAB6Sgn4BxWj+ WYldTf4Pdz/VINEpleg9dA9fTbGn3SsNfSgGd+W+FO9uD3uI5wOV5539fuJhLybggnyu uvENABq8KU7Ra6RirmfpDoNGKdJAdVCvsj++p+yiEK3bowxiWhwP4mmX71UoVBlLprJo rZx7Vjzr08UHMTKpjWenEQ4B4jtP3masKzdwANA0fcuqUZRrV5O8XdgQos58Ui8tWciW CQ+Q== MIME-Version: 1.0 X-Received: by 10.229.194.1 with SMTP id dw1mr45015151qcb.20.1387826127198; Mon, 23 Dec 2013 11:15:27 -0800 (PST) Received: by 10.140.34.17 with HTTP; Mon, 23 Dec 2013 11:15:27 -0800 (PST) In-Reply-To: <201312201723.46978.jhb@freebsd.org> References: <201312121511.38608.jhb@freebsd.org> <201312131709.20264.jhb@freebsd.org> <201312201723.46978.jhb@freebsd.org> Date: Mon, 23 Dec 2013 11:15:27 -0800 Message-ID: Subject: Re: Panic starting a bhyve guest after resume From: Neel Natu To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Dec 2013 19:15:28 -0000 Hi John, On Fri, Dec 20, 2013 at 2:23 PM, John Baldwin wrote: > On Friday, December 13, 2013 9:28:29 pm Neel Natu wrote: >> Hi John, >> >> On Fri, Dec 13, 2013 at 2:09 PM, John Baldwin wrote: >> > On Thursday, December 12, 2013 4:00:08 pm Neel Natu wrote: >> >> Hi John, >> >> >> >> On Thu, Dec 12, 2013 at 12:11 PM, John Baldwin wrote: >> >> > If I suspend and resume my laptop and then try to start a guest after the >> >> > resume, I get an odd panic. It generates a privileged instruction fault (in >> >> > kernel mode) for 'vmclear'. I've checked CR4 and it claims that VMXE is set. >> >> > I dont have any other ideas off the top of my head on what I should be poking >> >> > at? It looks like we read a bunch of MSRs in vmx_init(), but we don't write >> >> > to them, and all vmx_enable() does on each CPU is set VMXE in CR4 from what I >> >> > can tell. >> >> > >> >> >> >> It also does a "vmxon" on each logical cpu which may also need to be >> >> done after a resume. >> > >> > Ah, yes it does. That was sufficient both for starting a new guest after >> > resume and even doing a suspend/resume while a guest was active (and the >> > guest continued to run fine). I have a hacky patch for this. One, it >> > includes both a suspend and resume hook for VMM, though for my testing I only >> > needed a resume hook to invoke vmxon. Second, the name of vmx_resume2() >> > is a total hack (because vmx_resume() was already taken. I think for now >> > if I were to commit this, I'd just add the resme hook and maybe call the >> > Intel method vmx_reset() or vmx_restore()? >> > >> > http://people.freebsd.org/~jhb/patches/bhyve_resume.patch >> > >> >> There seems to be a race after the APs are restarted and before >> 'vmm_resume_p()' where it would be problematic to execute a VMX >> instruction. >> >> Perhaps we should enable VMX on each cpu before they return to the >> interrupted code? > > I've updated the patch at the URL above to do just that. This also works > in my testing. > Looks great! best Neel > -- > John Baldwin