From owner-freebsd-virtualization@FreeBSD.ORG  Mon Mar 25 18:54:14 2013
Return-Path: <owner-freebsd-virtualization@FreeBSD.ORG>
Delivered-To: freebsd-virtualization@freebsd.org
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
 by hub.freebsd.org (Postfix) with ESMTP id 070B5666
 for <freebsd-virtualization@freebsd.org>; Mon, 25 Mar 2013 18:54:14 +0000 (UTC)
 (envelope-from fivo.11235813@gmail.com)
Received: from mail-wi0-x22a.google.com (mail-wi0-x22a.google.com
 [IPv6:2a00:1450:400c:c05::22a])
 by mx1.freebsd.org (Postfix) with ESMTP id A0E45A24
 for <freebsd-virtualization@freebsd.org>; Mon, 25 Mar 2013 18:54:13 +0000 (UTC)
Received: by mail-wi0-f170.google.com with SMTP id hm11so11426311wib.1
 for <freebsd-virtualization@freebsd.org>; Mon, 25 Mar 2013 11:54:12 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=x-received:mime-version:from:date:message-id:subject:to
 :content-type; bh=mWJ9PThACJEk2foCs+mrczpSZXiJUo1sV1gfjvQi+GQ=;
 b=Zpn8dMTR1WcLhN+UW0Gn9GAq0XP/FVYdaUPcSacG+27XHW3g2c/dRNAaV/UD5YxE2k
 OfUeSfWD9GE380czUhj8Zu8LeKadqLoFn7TsDlw3oTDCKlrFqHvwsyULwSQpTd4rv6UP
 fo70FNFpucmwRXp6IONGe2SOvnQYvxv5PQUb+cE8ZAk31yLEltOIp7U6TJR/lr5eiQhx
 p5IpAwM4zZa3Zy2JpXyIvE24DvUunHR4uhHrqyvVFeAJnMmXoReXZ6t2bxgCNTP9Aln3
 YL4f6CTZjZCX0WTWJ08ZNadZw/uR4NcOzPB3BIVxAMX2+0VnZLgnyATsVbmExMJU1SWB
 cxNw==
X-Received: by 10.180.74.67 with SMTP id r3mr26404063wiv.14.1364237652911;
 Mon, 25 Mar 2013 11:54:12 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.194.169.161 with HTTP; Mon, 25 Mar 2013 11:53:51 -0700 (PDT)
From: =?UTF-8?Q?Iori=E3=80=80YONEJI?= <fivo.11235813@gmail.com>
Date: Tue, 26 Mar 2013 03:53:51 +0900
Message-ID: <CAJ-Y7VcZ=EbAesb+8Pup1YsSE_yaM_345KZjwibjaC=GMo0xfA@mail.gmail.com>
Subject: suspend/resume on BHyVe
To: freebsd-virtualization@freebsd.org
Content-Type: text/plain; charset=ISO-8859-1
X-BeenThere: freebsd-virtualization@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: "Discussion of various virtualization techniques FreeBSD supports."
 <freebsd-virtualization.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-virtualization>, 
 <mailto:freebsd-virtualization-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-virtualization>
List-Post: <mailto:freebsd-virtualization@freebsd.org>
List-Help: <mailto:freebsd-virtualization-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>, 
 <mailto:freebsd-virtualization-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Mar 2013 18:54:14 -0000

Hello.

I'm thinking of adding suspend/resume features on BHyVe.

For this, I think those below must be implemented.

  - virtual machine state command interface
  - saving registers per CPU
  - dumping physical memory
  - saving virt-io and other device emulation state


To save registers, the sysctl used in bhyvectl (vmmctl command
previously) is helpful,
however, it's interface is no good because getting register value
cause a sysctl call
so to context-switch per one register, and for getting all registers,
it's not efficient.
I think it's more preferable to make a struct to set or unset boolean
fields per register,
to tell which registers kernel should return, and kernel returns those
state with struct vmxctx.

struct vmxctx is such.
 struct vmxctx {
        register_t      tmpstk[32];             /* vmx_return() stack */
        register_t      tmpstktop;

        register_t      guest_rdi;              /* Guest state */
        register_t      guest_rsi;
                :
                :


And, considering memory dump, /dev/vmm/vmname is a file that is a map
of guest memory,
so memory dump doesn't seem hard, just stop vm, write back all guest
cache, and copy
memory file to a regular file.

Finally, I don't know much about device state, but I think there must
some state to be saved, like
network stack.

I'm not sure I wrote former, so I appreciate your ideas and suggestions.

Thanks, Iori.