From owner-freebsd-virtualization@FreeBSD.ORG Fri Feb 7 18:35:40 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9A8EB2F8 for ; Fri, 7 Feb 2014 18:35:40 +0000 (UTC) Received: from mail-vc0-x230.google.com (mail-vc0-x230.google.com [IPv6:2607:f8b0:400c:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 55F4F1BEF for ; Fri, 7 Feb 2014 18:35:40 +0000 (UTC) Received: by mail-vc0-f176.google.com with SMTP id la4so2915842vcb.21 for ; Fri, 07 Feb 2014 10:35:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to:content-type; bh=4NeFkv0mktvWYX+pwWyceOxbxUPJAz76dc7zAJ2yudM=; b=Qr3dupuxKP5QPPogaCr//GWi14Ni7KM6gTZ51wjrdshKi/yBwUiBdqLE61RYnLTzKr IAfQX+sSdm5N/ATSgEhRXczjckvnSa7b8DVpy8LTl8PTvZOTQkoNpVAZqY7sNVN5SeAT m0BIYBPXNVdUhmnOcihH+m0eJGSlyRIpeliTt+qLx5rDKSC+bd1qrmCaz0Mn08w2atCA Ho1MuJBYb807emsWk+HlI5ZvwXnzDDhcpppTgAnTNMgGBCzO68AafWtO0tah7KmTYUB5 CSAZ+crnbpWFurVB9GP+BIfNmxXi/h5YsA/q1t3fuKgTFzo1Zw/LPKx4qZTJCotFuKDl D/gQ== X-Received: by 10.52.106.107 with SMTP id gt11mr9792635vdb.7.1391798139432; Fri, 07 Feb 2014 10:35:39 -0800 (PST) MIME-Version: 1.0 Sender: cochard@gmail.com Received: by 10.58.188.35 with HTTP; Fri, 7 Feb 2014 10:35:19 -0800 (PST) From: =?ISO-8859-1?Q?Olivier_Cochard=2DLabb=E9?= Date: Fri, 7 Feb 2014 19:35:19 +0100 X-Google-Sender-Auth: ynhHLtaVEmmgc6vAWBxMNn5AWr4 Message-ID: Subject: Report of my virtual network lab migrated from virtualbox to bhyve To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Feb 2014 18:35:40 -0000 Hi, just a report of my migration to bhyve. I was using VirtualBox for generate full-meshed network lab of multiple VM (essentially nanobsd based) and have migrated my script to bhyve. My original script is resumed to this kind of usage: ./lab-script.sh -i FreeBSD-image-disk -n number-of-vm -l number-of-LAN For example I can easly generate this kind of topology: ./BSDRP-lab-bhyve.sh -i nanobsd-serial.img -n 4 -l 2 VM 1 have the following NIC: - vtnet0 connected to VM 2. - vtnet1 connected to VM 3. - vtnet2 connected to VM 4. - vtnet3 connected to LAN number 1 - vtnet4 connected to LAN number 2 VM 2 have the following NIC: - vtnet0 connected to VM 1. - vtnet1 connected to VM 3. - vtnet2 connected to VM 4. - vtnet3 connected to LAN number 1 - vtnet4 connected to LAN number 2 VM 3 have the following NIC: - vtnet0 connected to VM 1. - vtnet1 connected to VM 2. - vtnet2 connected to VM 4. - vtnet3 connected to LAN number 1 - vtnet4 connected to LAN number 2 VM 4 have the following NIC: - vtnet0 connected to VM 1. - vtnet1 connected to VM 2. - vtnet2 connected to VM 3. - vtnet3 connected to LAN number 1 - vtnet4 connected to LAN number 2 First remark comparing the disk format: With an original nanobsd disk image of 488MB. - Virtualbox format disk size: 133M - bhyve raw disk size: 488M Second remark regarding the difference between bhyve and virtualbox with disk image management: I'm using the "Linked clone" feature of virtualbox: This avoid to duplicate full guest disk. Here is with the example of 10 VM: - Virtualbox consumed disk space: 133M + 10x133K = about 135M in total - bhyve consumed disk space: 488M x 10 = 4880M in total => Hard drive space is not expensive today, but there is a huge difference (I didn't compare perfs). Third remark regarding memory usage on host (reported by top) with each VM configured for 256M of RAM (the guest report 70M of used memory) in SIZE/RES : . Virtualbox consumed RAM: 490/308M x 10 + 125/20M (VBoxSVC) + 80/11M (VBoxXPCOMIPCD) = total of 5100/4010M - bhyve consumed RAM: 283/22M x 10 = total of 2830/220M => Wow... bhyve memory usage is lot's lower than with Virtualbox ! Forth remark regarding network card emulation: Virtualbox permits to emulate em(4) NIC that supports altq(4) but bhyve emulate only vtnet(4) that didn't have altq(4) support. => I can't use bhyve for simulating network lab with altq(4) Fifth remark regarding network card virtualization: Virtualbox permits to generate "internal only" NIC between VM, but bhyve supports only tap. Can you add epair(4) support into bhyve roadmap ? => I have to create a huge number of bridge/tap interfaces on the host just for internal VM Ethernet links. And a simple user can't create bridge/tap interfaces. Sixth remark regarding the use of nmdm device for serial redirection: Often, I have to start a connection to the nmdm device for "un-pausing" the starting of the bhyve guest. And once I connect to the nmdm dev, this error is displayed (on head): Assertion failed: (error == 0), function init_pci, file /src/usr.sbin/bhyve/pci_emul.c, line 1047. Seventh remark regarding user: I didn't have to use Virtualbox as root (just to be in the virtualbox group), but it's mandatory with bhyve. And final remark: FreeBSD booting speed is very impressive with bhyve ! My bhyve and virtualbox scripts (and qemu) used are online here: http://bsdrp.net/documentation/examples/how_to_build_a_bsdrp_router_lab Regards, Olivier