From owner-freebsd-virtualization@FreeBSD.ORG Mon Oct 27 16:29:20 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4E900C87 for ; Mon, 27 Oct 2014 16:29:20 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 275E3E80 for ; Mon, 27 Oct 2014 16:29:20 +0000 (UTC) Received: from ralph.baldwin.cx (pool-173-70-85-31.nwrknj.fios.verizon.net [173.70.85.31]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id D283DB941; Mon, 27 Oct 2014 12:29:18 -0400 (EDT) From: John Baldwin To: freebsd-virtualization@freebsd.org Subject: Re: NATed or Private Network Setups Date: Mon, 27 Oct 2014 12:21:52 -0400 Message-ID: <1666962.21oQs0XfTB@ralph.baldwin.cx> User-Agent: KMail/4.14.2 (FreeBSD/10.1-PRERELEASE; KDE/4.14.2; amd64; ; ) In-Reply-To: <544ADBEB.2030907@nomadlogic.org> References: <544ADBEB.2030907@nomadlogic.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 27 Oct 2014 12:29:18 -0400 (EDT) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Oct 2014 16:29:20 -0000 On Friday, October 24, 2014 04:08:27 PM Pete Wright wrote: > Hi All, > Has anyone deployed bhyve using NAT'd or private network setups? I've > been able to deploy bridged interfaces, but I was wondering if anyone > has done other network topologies. Is there anything preventing this > from happening code wise? I reckon it could be achieved by creating a > pseudo interface? I setup a bridge on my laptop and add all the tap interfaces for VMs as members to the bridge. I use a /24 for the internal "LAN" for these interfaces and assign the .1 to the bridge0 interface itself. I then run dnsmasq to provide DHCP/DNS to the VMs and use natd (ipfw_nat would also work) to allow the VMs NAT access to the outside world. There are more details in an article in the most recent issue of the FreeBSD Journal, but I'll push that into the regular FreeBSD docs at some point as well. With the dnsmasq setup, I put the vmname as the hostname so that it is sent in the dhclient request. dnsmasq then adds local overrides for VMs while they are active. (So you can 'ssh vm0' on the host, or from another vm.) The 'host' entry in /etc/hosts is also snarfed up by dnsmasq so that within a vm I can use 'host' as a hostname (e.g. for NFS mounting something off of my laptop). Some config file snippets: /etc/sysctl.conf: net.link.tap.up_on_open=1 /etc/rc.conf: # bhyve setup autobridge_interfaces="bridge0" autobridge_bridge0="tap*" cloned_interfaces="bridge0 tap0 tap1 tap2" ifconfig_bridge0="inet 192.168.16.1/24" gateway_enable="YES" natd_enable="YES" natd_interface="wlan0" dnsmasq_enable="YES" firewall_enable="YES" firewall_type="/etc/rc.firewall.pippin" /etc/hosts: 192.168.16.1 host /etc/resolvconf.conf: name_servers=127.0.0.1 dnsmasq_conf=/etc/dnsmasq-conf.conf dnsmasq_resolv=/etc/dnsmasq-resolv.conf /usr/local/etc/dnsmasq.conf: domain-needed bogus-priv resolv-file=/etc/dnsmasq-resolv.conf interface=bridge0 dhcp-range=192.168.16.10,192.168.16.200,12h conf-file=/etc/dnsmasq-conf.conf /etc/rc.firewall.pippin: # prevent inbound traffic for our guest /24 add deny all from any to 192.168.16.0/24 via em0 add deny all from any to 192.168.16.0/24 via wlan0 # divert packets between guest and outside world to natd add divert natd all from any to any via wlan0 # prevent outbound traffic for our guest /24 add deny all from 192.168.16.0/24 to any via em0 add deny all from 192.168.16.0/24 to any via wlan0 # pass everything else add allow all from any to any (I have not figured out a way to have the NAT prefer em0 if present and fail over to wlan0 if not, etc.) -- John Baldwin