From owner-freebsd-doc@FreeBSD.ORG Fri May 2 01:30:00 2014 Return-Path: Delivered-To: freebsd-doc@smarthost.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CB36F28D for ; Fri, 2 May 2014 01:30:00 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A436C15CC for ; Fri, 2 May 2014 01:30:00 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s421U0jM031056 for ; Fri, 2 May 2014 01:30:00 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s421U0sP031044; Fri, 2 May 2014 01:30:00 GMT (envelope-from gnats) Resent-Date: Fri, 2 May 2014 01:30:00 GMT Resent-Message-Id: <201405020130.s421U0sP031044@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-doc@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Allan Jude Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B4C8B108 for ; Fri, 2 May 2014 01:27:46 +0000 (UTC) Received: from cgiserv.freebsd.org (cgiserv.freebsd.org [IPv6:2001:1900:2254:206a::50:4]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A007C15C0 for ; Fri, 2 May 2014 01:27:46 +0000 (UTC) Received: from cgiserv.freebsd.org ([127.0.1.6]) by cgiserv.freebsd.org (8.14.8/8.14.8) with ESMTP id s421RkTH039408 for ; Fri, 2 May 2014 01:27:46 GMT (envelope-from nobody@cgiserv.freebsd.org) Received: (from nobody@localhost) by cgiserv.freebsd.org (8.14.8/8.14.8/Submit) id s421Rk5L039399; Fri, 2 May 2014 01:27:46 GMT (envelope-from nobody) Message-Id: <201405020127.s421Rk5L039399@cgiserv.freebsd.org> Date: Fri, 2 May 2014 01:27:46 GMT From: Allan Jude To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Subject: docs/189216: [patch] add a handbook section on hosting VMs with bhyve X-BeenThere: freebsd-doc@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Documentation project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2014 01:30:00 -0000 >Number: 189216 >Category: docs >Synopsis: [patch] add a handbook section on hosting VMs with bhyve >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-doc >State: open >Quarter: >Keywords: >Date-Required: >Class: change-request >Submitter-Id: current-users >Arrival-Date: Fri May 02 01:30:00 UTC 2014 >Closed-Date: >Last-Modified: >Originator: Allan Jude >Release: 10.0-STABLE >Organization: ScaleEngine Inc. >Environment: FreeBSD Trooper.HML3.ScaleEngine.net 10.0-STABLE FreeBSD 10.0-STABLE #0 Sat Mar 22 13:15:35 EDT 2014 root@Trooper.HML3.ScaleEngine.net:/usr/obj/media/10stable/sys/GENERIC amd64 >Description: This patch adds a bhyve section to the virtualization chapter, and separates the virtualbox chapter. Sponsored by: ScaleEngine Inc. >How-To-Repeat: >Fix: Patch attached with submission follows: Index: handbook/virtualization/chapter.xml =================================================================== --- handbook/virtualization/chapter.xml (revision 44736) +++ handbook/virtualization/chapter.xml (working copy) @@ -20,6 +20,16 @@ Contributed by + + + + + Allan + Jude + + bhyve section by + + @@ -1110,8 +1120,8 @@ - - &os; as a Host + + &os; as a Host with <application>VirtualBox</application> &virtualbox; is an actively developed, complete virtualization package, that is available @@ -1273,7 +1283,310 @@ &prompt.root; service devfs restart + + + &os; as a Host with + <application>bhyve</application> + + Starting with &os; 10.0-RELEASE the BSD licensed hypervisor + bhyve is part of the base system. + bhyve supports a number of guests + including &os;, OpenBSD, and many flavors of &linux;. + bhyve currently only supports a + serial console and does not emulate a graphical console. + bhyve is a legacy-free hypervisor, + meaning that instead of translating instructions, and manually + managing memory mappings, it relies on the virtualization + offload features of newer CPUs. + bhyve also avoids emulating + compatible hardware for the guest, and instead relies on the + paravirtualization drivers, called + VirtIO. + + Due to the design of bhyve, it is + only possible to use bhyve on + computers with newer processors that support &intel; + EPT (Extended Page Tables) or &amd; + RVI (Rapid Virtualization Indexing, also know + as NPT or Nested Page Tables). Most newer + processors, specifically the &intel; &core; i3/i5/i7 and + &intel; &xeon; E3/E5/E7 support this feature, for a + complete list of &intel; processors that support + EPT see the &intel; + ARK. RVI is found on the 3rd + generation and later of the &amd.opteron; (Barcelona) + processors. The easiest way to check for support of + EPT or RVI on a system is + to look for the POPCNT processor feature flag + on the Features2 line in + dmesg or + /var/run/dmesg.boot. + + + Preparing the Host + + The first step to creating a virtual machine in + bhyve is configuring the host + system. Load the bhyve kernel + module called vmm. Create a tap + interface for the network device in the virtual machine to + attach to. Optionally create a bridge interface and add the + tap interface as well as the physical + interface as members to allow the virtual machine to have + access to the network. + + &prompt.root; kldload vmm +&prompt.root; kldload nmdm +&prompt.root; ifconfig tap0 create +&prompt.root; sysctl net.link.tap.up_on_open=1 +net.link.tap.up_on_open: 0 -> 1 +&prompt.root; ifconfig bridge0 create +&prompt.root; ifconfig bridge0 addm igb0 addm tap0 +&prompt.root; ifconfig bridge0 up + + + + + Creating a FreeBSD Guest + + Create a file to use as the virtual disk for the guest + machine. + + &prompt.root; truncate -s 16G guest.img + + Download an installation image of &os; to install: + + &prompt.root; fetch ftp://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/10.0/FreeBSD-10.0-RELEASE-amd64-bootonly.iso +FreeBSD-10.0-RELEASE-amd64-bootonly.iso 100% of 209 MB 570 kBps 06m17s + + &os; comes with an example script for running a virtual + machine in bhyve. The script will + start the virtual machine and run it in a loop, so it will + automatically restart if it crashes. The script takes a + number of options to control the configuration of the machine. + controls the number of virtual CPUs. + limits the amount of memory available to + the guest. defines which + tap device to use. + indicates which disk image to use. tells + bhyve to boot from the CD image + instead of the disk, and defines which CD + image to use. Finally the last parameter is the name of the + virtual machine, used to track the running machines. Start + the virtual machine in installation mode: + + &prompt.root; sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-10.0-RELEASE-amd64-bootonly.iso guestname + + The system will boot and start the installer. After + installing a system in the virtual machine, when the system + asks about dropping in to a shell at the end of the + installation, choose Yes. A small + change needs to be made to make the system start with a serial + console. Edit /etc/ttys and replace the + existing console line with: + + console "/usr/libexec/getty std.9600" xterm on secure + + Reboot the virtual machine. Rebooting the virtual machine + causes bhyve to exit. However the + vmrun.sh script runs + bhyve in a loop and will automatically + restart it. When this happens, choose the reboot option from + the boot loader menu, and this will escape the loop. Now the + guest can be started from the virtual disk: + + &prompt.root; sh /usr/share/examples/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname + + + + Creating a &linux; Guest + + bhyve requires + sysutils/grub2-bhyve in order to boot + operating systems other than &os;. + + Create a file to use as the virtual disk for the guest + machine. + + &prompt.root; truncate -s 16G linux.img + + Starting a virtual machine with + bhyve is a two step process. First + a kernel must be loaded, then the guest can be started. + sysutils/grub2-bhyve is used to load the + &linux; kernel. Create a device.map that + grub will use to map the virtual + devices to the files on the host system: + + (hd0) ./linux.img +(cd0) ./somelinux.iso + + Use sysutils/grub2-bhyve to load the + &linux; kernel from the ISO image: + + &prompt.root; grub-bhyve -m device.map -r cd0 -M 1024M linuxguest + + This will start grub. If the installation CD contains a + grub.cfg then a menu will be displayed. + If not, the vmlinuz and + initrd files must be located and loaded + manually: + + grub> ls +(hd0) (cd0) (cd0,msdos1) (host) +grub> ls (cd0)/isolinux +boot.cat boot.msg grub.conf initrd.img isolinux.bin isolinux.cfg memtest +splash.jpg TRANS.TBL vesamenu.c32 vmlinuz +grub> linux (cd0)/isolinux/vmlinuz +grub> initrd (cd0)/isolinux/initrd.img +grub> boot + + Now that the &linux; kernel is loaded, the guest can be + started: + + &prompt.root; bhyve -AI -H -P \ +-s 0:0,hostbridge \ +-s 1:0,lpc \ +-s 2:0,virtio-net,tap1 \ +-s 3:0,virtio-blk,./linux.img \ +-s 4:0,ahci-cd,./somelinux.iso \ +-l com1,stdio \ +-c 4 -m 1024M linuxguest + + The system will boot and start the installer. After + installing a system in the virtual machine, reboot the virtual + machine. This will cause bhyve to + exit. The instance of the virtual machine needs to be + destroyed before it can be started again: + + &prompt.root; bhyvectl --destroy --vm=linuxguest + + Now the guest can be started directly from the virtual + disk. Load the kernel: + + &prompt.root; grub-bhyve -m device.map -r hd0,msdos1 -M 1024M linuxguest +grub> ls +(hd0) (hd0,msdos2) (hd0,msdos1) (cd0) (cd0,msdos1) (host) +(lvm/VolGroup-lv_swap) (lvm/VolGroup-lv_root) +grub> ls (hd0,msdos1)/ +lost+found/ grub/ efi/ System.map-2.6.32-431.el6.x86_64 config-2.6.32-431.el6.x +86_64 symvers-2.6.32-431.el6.x86_64.gz vmlinuz-2.6.32-431.el6.x86_64 +initramfs-2.6.32-431.el6.x86_64.img +grub> linux (hd0,msdos1)/vmlinuz-2.6.32-431.el6.x86_64 root=/dev/mapper/VolGroup-lv_root +grub> initrd (hd0,msdos1)/initramfs-2.6.32-431.el6.x86_64.img +grub> boot + + Boot the virtual machine: + + &prompt.root; bhyve -AI -H -P \ +-s 0:0,hostbridge \ +-s 1:0,lpc \ +-s 2:0,virtio-net,tap1 \ +-s 3:0,virtio-blk,./linux.img \ +-l com1,stdio \ +-c 4 -m 1024M linuxguest + + &linux; will now boot in the virtual machine and + eventually present you with the login prompt. Login and use + the virtual machine. When you are finished, reboot the + virtual machine to exit bhyve. + Destroy the virtual machine instance: + + &prompt.root; bhyvectl --destroy --vm=linuxguest + + + + Virtual Machines Consoles + + It is advantageous to wrap the + bhyve console in a session + management tool such as sysutils/tmux or + sysutils/screen in order to detach and + reattach to the console. It is also possible to have the + console of bhyve be a null modem + device that can be accessed with cu. Load + the nmdm kernel module, and replace + with + . The + /dev/nmdm devices are created + automatically as needed, each is a pair, + /dev/nmdm1A and + /dev/nmdm1B corresponding to the two ends + of the null modem cable. See &man.nmdm.4; for more + information. + + &prompt.root; bhyve -AI -H -P \ +-s 0:0,hostbridge \ +-s 1:0,lpc \ +-s 2:0,virtio-net,tap1 \ +-s 3:0,virtio-blk,./linux.img \ +-l com1,/dev/nmdm0A \ +-c 4 -m 1024M linuxguest +&prompt.root; cu -l /dev/nmdm0B -s 9600 +Connected + +Ubuntu 13.10 handbook ttyS0 + +handbook login: + + + + + Managing Virtual Machines + + A device node is created in /dev/vmm for each virtual + machine. This allows the administrator to easily see a list + of the running virtual machines: + + &prompt.root; ls -al /dev/vmm +total 1 +dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./ +dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../ +crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname +crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest +crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest + + Virtual machines can be destroyed using + bhyvectl: + + &prompt.root; bhyvectl --destroy --vm=guestname + + + + Persistent Configuration + + In order to make the system able to start + bhyve guests at boot time, the + following configurations must be made in the specified + files: + + + + <filename>/etc/sysctl.conf</filename> + + net.link.tap.up_on_open=1 + + + + <filename>/boot/loader.conf</filename> + + vmm_load="YES" +nmdm_load="YES" +if_bridge_load="YES" +if_tap_load="YES" + + + + <filename>/etc/rc.conf</filename> + + cloned_interfaces="bridge0 tap0" +ifconfig_bridge0="addm igb0 addm tap0" + + +