From owner-freebsd-virtualization@FreeBSD.ORG Fri Feb 7 19:38:42 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5059B786 for ; Fri, 7 Feb 2014 19:38:42 +0000 (UTC) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id D2E1C11AE for ; Fri, 7 Feb 2014 19:38:41 +0000 (UTC) Received: from dommail.onthenet.com.au (dommail.OntheNet.com.au [203.13.70.57]) by alto.onthenet.com.au (Postfix) with ESMTPS id 3FA3812420; Sat, 8 Feb 2014 05:38:40 +1000 (EST) Received: from Peter-Grehans-MacBook-Pro-2.local ([64.245.0.210]) by dommail.onthenet.com.au (MOS 4.2.4-GA) with ESMTP id BRR37078 (AUTH peterg@ptree32.com.au); Sat, 8 Feb 2014 05:38:39 +1000 Message-ID: <52F5363D.8040102@freebsd.org> Date: Fri, 07 Feb 2014 11:38:37 -0800 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: =?ISO-8859-1?Q?Olivier_Cochard-Labb=E9?= Subject: Re: Report of my virtual network lab migrated from virtualbox to bhyve References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Feb 2014 19:38:42 -0000 Hi Olivier, > just a report of my migration to bhyve. Yeah !!! > First remark comparing the disk format: With an original nanobsd disk image > of 488MB. > - Virtualbox format disk size: 133M > - bhyve raw disk size: 488M If you create a sparse file for the bhyve raw disk (e.g. with truncate -s), du will show the actual blocks used rather than the total size. > Second remark regarding the difference between bhyve and virtualbox with > disk image management: > I'm using the "Linked clone" feature of virtualbox: This avoid to duplicate > full guest disk. Here is with the example of 10 VM: > - Virtualbox consumed disk space: 133M + 10x133K = about 135M in total > - bhyve consumed disk space: 488M x 10 = 4880M in total > => Hard drive space is not expensive today, but there is a huge difference > (I didn't compare perfs). bhyve doesn't support any VM sparse disk formats - it currently relies on the underlying filesystem for this (sparse files, ZFS compression/dedup etc). This can perform better than VM sparse formats, in both i/o speed and also space usage, at the expense of having to do a conversion to/from a flat file if you are importing/exporting. That being said, I'm sure bhyve will gather direct support for these at some point. > Third remark regarding memory usage on host (reported by top) with each VM > configured for 256M of RAM (the guest report 70M of used memory) in > SIZE/RES : > . Virtualbox consumed RAM: 490/308M x 10 + 125/20M (VBoxSVC) + 80/11M > (VBoxXPCOMIPCD) = total of 5100/4010M > - bhyve consumed RAM: 283/22M x 10 = total of 2830/220M > => Wow... bhyve memory usage is lot's lower than with Virtualbox ! To be fair to vbox, the memory usage reported for bhyve is only the pages that have been touched by the bhyve process in it's mmap() of guest memory. This may be less than that actually in use. We need to export the pages in use in the kernel vmspace that represents guest memory. > Forth remark regarding network card emulation: > Virtualbox permits to emulate em(4) NIC that supports altq(4) but bhyve > emulate only vtnet(4) that didn't have altq(4) support. > => I can't use bhyve for simulating network lab with altq(4) There is an em(4) emulation slowly being worked on. It should also be possible to add altq functionality to FreeBSD's virtio net driver. > Fifth remark regarding network card virtualization: > Virtualbox permits to generate "internal only" NIC between VM, but bhyve > supports only tap. Can you add epair(4) support into bhyve roadmap ? > => I have to create a huge number of bridge/tap interfaces on the host just > for internal VM Ethernet links. And a simple user can't create bridge/tap > interfaces. For internal-only networks, there will most likely be a user-space ethernet switch (ala VDE) that bhyve network interfaces can be pointed at. > Sixth remark regarding the use of nmdm device for serial redirection: > Often, I have to start a connection to the nmdm device for "un-pausing" the > starting of the bhyve guest. And once I connect to the nmdm dev, this error > is displayed (on head): > Assertion failed: (error == 0), function init_pci, file > /src/usr.sbin/bhyve/pci_emul.c, line 1047. Yeah, there are some issues with nmdm's simulated modem control :( Looking into this. > Seventh remark regarding user: I didn't have to use Virtualbox as root > (just to be in the virtualbox group), but it's mandatory with bhyve. Fixing this is on the roadmap. > And final remark: FreeBSD booting speed is very impressive with bhyve ! This may be a useful side-effect of the separate loader process. It runs in user-space with Unix process i/o as opposed to a BIOS loader which has to run in polled-mode. later, Peter.