From owner-freebsd-doc@FreeBSD.ORG Fri May 2 01:40:30 2014 Return-Path: Delivered-To: freebsd-doc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04494654 for ; Fri, 2 May 2014 01:40:30 +0000 (UTC) Received: from mx1.scaleengine.net (beauharnois2.bhs1.scaleengine.net [142.4.218.15]) by mx1.freebsd.org (Postfix) with ESMTP id B7E991701 for ; Fri, 2 May 2014 01:40:29 +0000 (UTC) Received: from [10.1.1.1] (S01060001abad1dea.hm.shawcable.net [50.70.146.73]) (Authenticated sender: allan.jude@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 5741A70320 for ; Fri, 2 May 2014 01:40:22 +0000 (UTC) Message-ID: <5362F777.1010403@allanjude.com> Date: Thu, 01 May 2014 21:40:07 -0400 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: freebsd-doc@freebsd.org Subject: Re: docs/189216: [patch] add a handbook section on hosting VMs with bhyve References: <201405020127.s421Rk5L039399@cgiserv.freebsd.org> In-Reply-To: <201405020127.s421Rk5L039399@cgiserv.freebsd.org> X-Enigmail-Version: 1.6 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UaArI2Ofm7hl1NDljLIF5twKFsXskmDn6" X-BeenThere: freebsd-doc@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Documentation project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 May 2014 01:40:30 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --UaArI2Ofm7hl1NDljLIF5twKFsXskmDn6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 2014-05-01 21:27, Allan Jude wrote: >=20 >> Number: 189216 >> Category: docs >> Synopsis: [patch] add a handbook section on hosting VMs with bhy= ve >> Confidential: no >> Severity: non-critical >> Priority: low >> Responsible: freebsd-doc >> State: open >> Quarter: =20 >> Keywords: =20 >> Date-Required: >> Class: change-request >> Submitter-Id: current-users >> Arrival-Date: Fri May 02 01:30:00 UTC 2014 >> Closed-Date: >> Last-Modified: >> Originator: Allan Jude >> Release: 10.0-STABLE >> Organization: > ScaleEngine Inc. >> Environment: > FreeBSD Trooper.HML3.ScaleEngine.net 10.0-STABLE FreeBSD 10.0-STABLE #0= Sat Mar 22 13:15:35 EDT 2014 root@Trooper.HML3.ScaleEngine.net:/usr/= obj/media/10stable/sys/GENERIC amd64 >=20 >> Description: > This patch adds a bhyve section to the virtualization chapter, and sepa= rates the virtualbox chapter. >=20 >=20 > Sponsored by: ScaleEngine Inc. >> How-To-Repeat: >=20 >> Fix: >=20 >=20 > Patch attached with submission follows: >=20 > Index: handbook/virtualization/chapter.xml > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- handbook/virtualization/chapter.xml (revision 44736) > +++ handbook/virtualization/chapter.xml (working copy) > @@ -20,6 +20,16 @@ > Contributed by > > > + > + > + > + > + Allan > + Jude > + > + bhyve section by > + > + > > =20 > > @@ -1110,8 +1120,8 @@ > > > =20 > - > - &os; as a Host > + > + &os; as a Host with <application>VirtualBox</application></= title> > =20 > <para><application>&virtualbox;</application> is an actively > developed, complete virtualization package, that is available > @@ -1273,7 +1283,310 @@ > =20 > <screen>&prompt.root; <userinput>service devfs restart</userinpu= t></screen> > </sect2> > + </sect1> > =20 > + <sect1 xml:id=3D"virtualization-host-bhyve"> > + <title>&os; as a Host with > + <application>bhyve</application> > + > + Starting with &os; 10.0-RELEASE the BSD licensed hypervisor > + bhyve is part of the base system. > + bhyve supports a number of guests > + including &os;, OpenBSD, and many flavors of &linux;. > + bhyve currently only supports a > + serial console and does not emulate a graphical console. > + bhyve is a legacy-free hypervisor, > + meaning that instead of translating instructions, and manually > + managing memory mappings, it relies on the virtualization > + offload features of newer CPUs. > + bhyve also avoids emulating > + compatible hardware for the guest, and instead relies on the > + paravirtualization drivers, called > + VirtIO. > + > + Due to the design of bhyve, it is= > + only possible to use bhyve on > + computers with newer processors that support &intel; > + EPT (Extended Page Tables) or &amd; > + RVI (Rapid Virtualization Indexing, also know= > + as NPT or Nested Page Tables). Most newer > + processors, specifically the &intel; &core; i3/i5/i7 and > + &intel; &xeon; E3/E5/E7 support this feature, for a > + complete list of &intel; processors that support > + EPT see the + xlink:href=3D"http://ark.intel.com/search/advanced?s=3Dt&Ext= endedPageTables=3Dtrue">&intel; > + ARK. RVI is found on the 3rd > + generation and later of the &amd.opteron; (Barcelona) > + processors. The easiest way to check for support of > + EPT or RVI on a system is > + to look for the POPCNT processor feature flag= > + on the Features2 line in > + dmesg or > + /var/run/dmesg.boot. > + > + > + Preparing the Host > + > + The first step to creating a virtual machine in > + bhyve is configuring the host > + system. Load the bhyve kernel > + module called vmm. Create a tap > + interface for the network device in the virtual machine to > + attach to. Optionally create a bridge interface and add the > + tap interface as well as the physical > + interface as members to allow the virtual machine to have > + access to the network. > + > + &prompt.root; kldload vmm > +&prompt.root; kldload nmdm > +&prompt.root; ifconfig tap0 crea= te > +&prompt.root; sysctl net.link.tap.up_on_open=3D1 > +net.link.tap.up_on_open: 0 -> 1 > +&prompt.root; ifconfig bridge0 c= reate > +&prompt.root; ifconfig bridge0 a= ddm igb0 addm tap0<= /userinput> > +&prompt.root; ifconfig bridge0 u= p > + > + > + > + > + Creating a FreeBSD Guest > + > + Create a file to use as the virtual disk for the guest > + machine. > + > + &prompt.root; truncate -s 16G guest.img > + > + Download an installation image of &os; to install: > + > + &prompt.root; fetch ftp://ftp.fr= eebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/10.0/FreeBSD-10.0-RELEASE-amd64-bo= otonly.iso > +FreeBSD-10.0-RELEASE-amd64-bootonly.iso 100% of 209 MB 570 kBp= s 06m17s > + > + &os; comes with an example script for running a virtual > + machine in bhyve. The script will > + start the virtual machine and run it in a loop, so it will > + automatically restart if it crashes. The script takes a > + number of options to control the configuration of the machine. > + controls the number of virtual CPUs. > + limits the amount of memory available to > + the guest. defines which > + tap device to use. > + indicates which disk image to use. tells > + bhyve to boot from the CD image > + instead of the disk, and defines which CD > + image to use. Finally the last parameter is the name of the > + virtual machine, used to track the running machines. Start > + the virtual machine in installation mode: > + > + &prompt.root; sh /usr/share/example= s/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img -i -I FreeBSD-10.0-RELEASE-amd64-bootonly.iso g= uestname > + > + The system will boot and start the installer. After > + installing a system in the virtual machine, when the system > + asks about dropping in to a shell at the end of the > + installation, choose Yes. A small > + change needs to be made to make the system start with a serial > + console. Edit /etc/ttys and replace the > + existing console line with: > + > + console "/usr/libexec/getty std.9600" xterm = on secure > + > + Reboot the virtual machine. Rebooting the virtual machine= > + causes bhyve to exit. However the > + vmrun.sh script runs > + bhyve in a loop and will automatically > + restart it. When this happens, choose the reboot option from > + the boot loader menu, and this will escape the loop. Now the > + guest can be started from the virtual disk: > + > + &prompt.root; sh /usr/share/example= s/bhyve/vmrun.sh -c 4 -m 1024M -t tap0 -d guest.img guestname > + > + > + > + Creating a &linux; Guest > + > + bhyve requires > + sysutils/grub2-bhyve in order to boot > + operating systems other than &os;. > + > + Create a file to use as the virtual disk for the guest > + machine. > + > + &prompt.root; truncate -s 16G linux.img > + > + Starting a virtual machine with > + bhyve is a two step process. First > + a kernel must be loaded, then the guest can be started. > + sysutils/grub2-bhyve is used to load the > + &linux; kernel. Create a device.map that > + grub will use to map the virtual > + devices to the files on the host system: > + > + (hd0) ./linux.img > +(cd0) ./somelinux.iso > + > + Use sysutils/grub2-bhyve to load the > + &linux; kernel from the ISO image: > + > + &prompt.root; grub-bhyve -m device.= map -r cd0 -M 1024M li= nuxguest > + > + This will start grub. If the installation CD contains a > + grub.cfg then a menu will be displayed. > + If not, the vmlinuz and > + initrd files must be located and loaded > + manually: > + > + grub> ls > +(hd0) (cd0) (cd0,msdos1) (host) > +grub> ls (cd0)/isolinux > +boot.cat boot.msg grub.conf initrd.img isolinux.bin isolinux.cfg memte= st > +splash.jpg TRANS.TBL vesamenu.c32 vmlinuz > +grub> linux (cd0)/isolinux/vmlinuz > +grub> initrd (cd0)/isolinux/initrd.img > +grub> boot > + > + Now that the &linux; kernel is loaded, the guest can be > + started: > + > + &prompt.root; bhyve -AI -H -P \ > +-s 0:0,hostbridge \ > +-s 1:0,lpc \ > +-s 2:0,virtio-net,tap1 \ > +-s 3:0,virtio-blk,./linux.img \ > +-s 4:0,ahci-cd,./somelinux.iso \ > +-l com1,stdio \ > +-c 4 -m 1024M linuxguest > + > + The system will boot and start the installer. After > + installing a system in the virtual machine, reboot the virtual > + machine. This will cause bhyve to > + exit. The instance of the virtual machine needs to be > + destroyed before it can be started again: > + > + &prompt.root; bhyvectl --destroy --vm=3Dlinuxguest > + > + Now the guest can be started directly from the virtual > + disk. Load the kernel: > + > + &prompt.root; grub-bhyve -m device.= map -r hd0,msdos1 -M 1024M linuxguest > +grub> ls > +(hd0) (hd0,msdos2) (hd0,msdos1) (cd0) (cd0,msdos1) (host) > +(lvm/VolGroup-lv_swap) (lvm/VolGroup-lv_root) > +grub> ls (hd0,msdos1)/ > +lost+found/ grub/ efi/ System.map-2.6.32-431.el6.x86_64 config-2.6.32-= 431.el6.x > +86_64 symvers-2.6.32-431.el6.x86_64.gz vmlinuz-2.6.32-431.el6.x86_64 > +initramfs-2.6.32-431.el6.x86_64.img > +grub> linux (hd0,msdos1)/vmlinuz-2.6.32-431.el6.x86_64 root= =3D/dev/mapper/VolGroup-lv_root > +grub> initrd (hd0,msdos1)/initramfs-2.6.32-431.el6.x86_64.i= mg > +grub> boot > + > + Boot the virtual machine: > + > + &prompt.root; bhyve -AI -H -P \ > +-s 0:0,hostbridge \ > +-s 1:0,lpc \ > +-s 2:0,virtio-net,tap1 \ > +-s 3:0,virtio-blk,./linux.img \ > +-l com1,stdio \ > +-c 4 -m 1024M linuxguest > + > + &linux; will now boot in the virtual machine and > + eventually present you with the login prompt. Login and use > + the virtual machine. When you are finished, reboot the > + virtual machine to exit bhyve. > + Destroy the virtual machine instance: > + > + &prompt.root; bhyvectl --destroy --vm=3Dlinuxguest > + > + > + > + Virtual Machines Consoles > + > + It is advantageous to wrap the > + bhyve console in a session > + management tool such as sysutils/tmux or > + sysutils/screen in order to detach and > + reattach to the console. It is also possible to have the > + console of bhyve be a null modem > + device that can be accessed with cu. Load > + the nmdm kernel module, and replace > + with > + . The > + /dev/nmdm devices are created > + automatically as needed, each is a pair, > + /dev/nmdm1A and > + /dev/nmdm1B corresponding to the two ends > + of the null modem cable. See &man.nmdm.4; for more > + information. > + > + &prompt.root; bhyve -AI -H -P \ > +-s 0:0,hostbridge \ > +-s 1:0,lpc \ > +-s 2:0,virtio-net,tap1 \ > +-s 3:0,virtio-blk,./linux.img \ > +-l com1,/dev/nmdm0A \ > +-c 4 -m 1024M linuxguest > +&prompt.root; cu -l /dev/nmdm0B -s 9600 > +Connected > + > +Ubuntu 13.10 handbook ttyS0 > + > +handbook login: > + > + > + > + > + Managing Virtual Machines > + > + A device node is created in + role=3D"directory">/dev/vmm for each virtual > + machine. This allows the administrator to easily see a list > + of the running virtual machines: > + > + &prompt.root; ls -al /dev/vmm > +total 1 > +dr-xr-xr-x 2 root wheel 512 Mar 17 12:19 ./ > +dr-xr-xr-x 14 root wheel 512 Mar 17 06:38 ../ > +crw------- 1 root wheel 0x1a2 Mar 17 12:20 guestname > +crw------- 1 root wheel 0x19f Mar 17 12:19 linuxguest > +crw------- 1 root wheel 0x1a1 Mar 17 12:19 otherguest > + > + Virtual machines can be destroyed using > + bhyvectl: > + > + &prompt.root; bhyvectl --destroy --vm=3Dguestname > + > + > + > + Persistent Configuration > + > + In order to make the system able to start > + bhyve guests at boot time, the > + following configurations must be made in the specified > + files: > + > + > + > + <filename>/etc/sysctl.conf</filename> > + > + net.link.tap.up_on_open=3D1 > + > + > + > + <filename>/boot/loader.conf</filename> > + > + vmm_load=3D"YES" > +nmdm_load=3D"YES" > +if_bridge_load=3D"YES" > +if_tap_load=3D"YES" > + > + > + > + <filename>/etc/rc.conf</filename> > + > + cloned_interfaces=3D"bridge0 tap0" > +ifconfig_bridge0=3D"addm igb0 addm tap0" > + > + > + >