From owner-freebsd-virtualization@FreeBSD.ORG Mon Oct 27 17:36:15 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 40B8CFF; Mon, 27 Oct 2014 17:36:15 +0000 (UTC) Received: from mail.nomadlogic.org (mail.nomadlogic.org [IPv6:2607:f2f8:a098::4]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1C1B68F3; Mon, 27 Oct 2014 17:36:15 +0000 (UTC) Received: from mail.nomadlogic.org (localhost [127.0.0.1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.nomadlogic.org (Postfix) with ESMTPS id CB726125EE1; Mon, 27 Oct 2014 10:36:08 -0700 (PDT) Received: from pop.rubicorp.com (unknown [72.34.113.100]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.nomadlogic.org (Postfix) with ESMTPSA id BA8A7125EBA; Mon, 27 Oct 2014 10:36:08 -0700 (PDT) Message-ID: <544E8288.9020001@nomadlogic.org> Date: Mon, 27 Oct 2014 10:36:08 -0700 From: Pete Wright User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: John Baldwin , freebsd-virtualization@freebsd.org Subject: Re: NATed or Private Network Setups References: <544ADBEB.2030907@nomadlogic.org> <1666962.21oQs0XfTB@ralph.baldwin.cx> In-Reply-To: <1666962.21oQs0XfTB@ralph.baldwin.cx> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Oct 2014 17:36:15 -0000 On 10/27/14 09:21, John Baldwin wrote: > On Friday, October 24, 2014 04:08:27 PM Pete Wright wrote: >> Hi All, >> Has anyone deployed bhyve using NAT'd or private network setups? I've >> been able to deploy bridged interfaces, but I was wondering if anyone >> has done other network topologies. Is there anything preventing this >> from happening code wise? I reckon it could be achieved by creating a >> pseudo interface? > > I setup a bridge on my laptop and add all the tap interfaces for VMs as > members to the bridge. I use a /24 for the internal "LAN" for these > interfaces and assign the .1 to the bridge0 interface itself. I then run > dnsmasq to provide DHCP/DNS to the VMs and use natd (ipfw_nat would also work) > to allow the VMs NAT access to the outside world. There are more details in > an article in the most recent issue of the FreeBSD Journal, but I'll push that > into the regular FreeBSD docs at some point as well. > > With the dnsmasq setup, I put the vmname as the hostname so that it is sent in > the dhclient request. dnsmasq then adds local overrides for VMs while they > are active. (So you can 'ssh vm0' on the host, or from another vm.) The > 'host' entry in /etc/hosts is also snarfed up by dnsmasq so that within a vm I > can use 'host' as a hostname (e.g. for NFS mounting something off of my > laptop). > > Some config file snippets: > > /etc/sysctl.conf: > > net.link.tap.up_on_open=1 > > /etc/rc.conf: > > # bhyve setup > autobridge_interfaces="bridge0" > autobridge_bridge0="tap*" > cloned_interfaces="bridge0 tap0 tap1 tap2" > ifconfig_bridge0="inet 192.168.16.1/24" > gateway_enable="YES" > natd_enable="YES" > natd_interface="wlan0" > dnsmasq_enable="YES" > firewall_enable="YES" > firewall_type="/etc/rc.firewall.pippin" > > /etc/hosts: > > 192.168.16.1 host > > /etc/resolvconf.conf: > > name_servers=127.0.0.1 > dnsmasq_conf=/etc/dnsmasq-conf.conf > dnsmasq_resolv=/etc/dnsmasq-resolv.conf > > /usr/local/etc/dnsmasq.conf: > > domain-needed > bogus-priv > resolv-file=/etc/dnsmasq-resolv.conf > interface=bridge0 > dhcp-range=192.168.16.10,192.168.16.200,12h > conf-file=/etc/dnsmasq-conf.conf > > /etc/rc.firewall.pippin: > > # prevent inbound traffic for our guest /24 > add deny all from any to 192.168.16.0/24 via em0 > add deny all from any to 192.168.16.0/24 via wlan0 > > # divert packets between guest and outside world to natd > add divert natd all from any to any via wlan0 > > # prevent outbound traffic for our guest /24 > add deny all from 192.168.16.0/24 to any via em0 > add deny all from 192.168.16.0/24 to any via wlan0 > > # pass everything else > add allow all from any to any > > (I have not figured out a way to have the NAT prefer em0 if present and fail > over to wlan0 if not, etc.) > Thanks for this detailed explanation John! Using dnsmasq sounds great, especially for my environment since we already leverage it for openstack on our linux systems extensively. Cheers, -pete -- Pete Wright pete@nomadlogic.org twitter => @nomadlogicLA