From owner-freebsd-virtualization@FreeBSD.ORG Tue Mar 26 05:04:58 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BB6FC2D8 for ; Tue, 26 Mar 2013 05:04:58 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-ia0-x22d.google.com (mail-ia0-x22d.google.com [IPv6:2607:f8b0:4001:c02::22d]) by mx1.freebsd.org (Postfix) with ESMTP id 9130B9D2 for ; Tue, 26 Mar 2013 05:04:58 +0000 (UTC) Received: by mail-ia0-f173.google.com with SMTP id h37so6159616iak.32 for ; Mon, 25 Mar 2013 22:04:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=0UbwFeVvJM+LztMJpsX18UCYHt5WH0sKIvW84ONMik4=; b=aF1Rg8AWIlFhPBip5FMYYWJK2lsSSesC3X8YiFTxZQrqMbNPxl5M41eH8bs2+cWTFL Toi8WmfGv4XM/5TMdVh3wlwHF85PNuHdDXcWdmUmNEIiew8NOvpTpjH3yrMYqTeEDz8y BTcs9YdPv8UUYleKNySr0dd4NLKDCEueraMqxkE49JxfmIvEgG+Gw0/znLg7RosIWwB7 AQ6h9OaW41xZSyBg6kik2YYTPOXzSFYhswDxTCNCiHOSRKF3SlYPAvkc84jLyXimj1W5 MUCX6ieAY+FgkE7TO7k/UROrkHS27zwkdUrkbkSXocOrWGUTXTuRunm6Dj9osYVWGfWt M03w== MIME-Version: 1.0 X-Received: by 10.43.62.12 with SMTP id wy12mr8429981icb.19.1364274298348; Mon, 25 Mar 2013 22:04:58 -0700 (PDT) Received: by 10.42.23.132 with HTTP; Mon, 25 Mar 2013 22:04:58 -0700 (PDT) In-Reply-To: References: Date: Mon, 25 Mar 2013 22:04:58 -0700 Message-ID: Subject: Re: suspend/resume on BHyVe From: Neel Natu To: Iori YONEJI Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Mar 2013 05:04:58 -0000 Hi Lori, On Mon, Mar 25, 2013 at 11:53 AM, Iori=E3=80=80YONEJI wrote: > Hello. > > I'm thinking of adding suspend/resume features on BHyVe. > Fantastic! > For this, I think those below must be implemented. > > - virtual machine state command interface > - saving registers per CPU > - dumping physical memory > - saving virt-io and other device emulation state > > > To save registers, the sysctl used in bhyvectl (vmmctl command > previously) is helpful, > however, it's interface is no good because getting register value > cause a sysctl call > so to context-switch per one register, and for getting all registers, > it's not efficient. > I think it's more preferable to make a struct to set or unset boolean > fields per register, > to tell which registers kernel should return, and kernel returns those > state with struct vmxctx. > > struct vmxctx is such. > struct vmxctx { > register_t tmpstk[32]; /* vmx_return() stack */ > register_t tmpstktop; > > register_t guest_rdi; /* Guest state */ > register_t guest_rsi; > : > : > > > And, considering memory dump, /dev/vmm/vmname is a file that is a map > of guest memory, > so memory dump doesn't seem hard, just stop vm, write back all guest > cache, and copy > memory file to a regular file. > > Finally, I don't know much about device state, but I think there must > some state to be saved, like > network stack. > > I'm not sure I wrote former, so I appreciate your ideas and suggestions. > I think that you are on the right track. A brute force way of figuring out all the state must be saved is to look at all the initialization functions that are called when a vm and a vcpu are created. So, this would be vm_create() and vcpu_init() in the kernel module. There is also the hardware assist state that is maintained by the processor (VT-x or SVM) and this includes things like guest interruptibility, guest run state etc. I am assuming that it would be sufficient to save the VMCS page after telling the processor to flush any state it may be caching on chip. There is also emulated pci bus, virtio devices and legacy isa device state that would need to be saved by the userspace 'bhyve' process. And finally there is the matter of how to communicate with 'bhyve' process that it needs to suspend the virtual machine and write its state to disk - perhaps a signal would be good enough place to start. This certainly sounds like an interesting and challenging project and we would be happy to help in any way we can. best Neel > Thanks, Iori. > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@free= bsd.org"