From owner-freebsd-virtualization@freebsd.org Sun Aug 19 16:13:08 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3C219106E6A0 for ; Sun, 19 Aug 2018 16:13:08 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from vps-mail.nomadlogic.org (mail.nomadlogic.org [IPv6:2607:f2f8:a098::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AAD6C7D827 for ; Sun, 19 Aug 2018 16:13:07 +0000 (UTC) (envelope-from pete@nomadlogic.org) Received: from [192.168.1.106] (cpe-23-243-162-239.socal.res.rr.com [23.243.162.239]) by vps-mail.nomadlogic.org (OpenSMTPD) with ESMTPSA id b070e3c8 TLS version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO for ; Sun, 19 Aug 2018 09:13:04 -0700 (PDT) To: freebsd-virtualization@freebsd.org From: Pete Wright Subject: bhyve NAT issue Message-ID: Date: Sun, 19 Aug 2018 09:13:00 -0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Aug 2018 16:13:08 -0000 Hello, I have a baremetal server hosted by Vultr that I would like to host several bhyve instances on.  I have been given one public ipv4 address, so my goal is to run the bhyve instances on a private subnet (172.16.0.0/24) then use pf to NAT this subnet and do port forwarding to.  I am having an issues though getting the VM instances network to work. Here is how the hybervisor is configured: uname: FreeBSD cojo 11.2-STABLE FreeBSD 11.2-STABLE #1 r337947: Fri Aug 17 03:22:33 PDT 2018 pete@cojo:/usr/obj/usr/home/pete/src/freebsd-stable/sys/GENERIC amd64 rc.conf: cloned_interfaces="bridge0 tap0" ifconfig_bridge0="inet 172.16.0.1 netmask 255.255.255.0 addm tap0 up" gateway_enable="YES" ipv6_gateway_enable="YES" pf_enable="YES" pf.conf: ext_if=ix0 scrub in all nat on $ext_if inet from 172.16.0.0/24 to any -> ($ext_if) pass in all pass out all and my script to invoke the VM: /usr/sbin/bhyve -AHP -s 31:0,lpc \   -s 2:0,virtio-net,tap0 \   -s 3:0,virtio-blk,/vms/freebsd0 \   -s 29,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait \   -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \   -c 2 -m 1024M \   test0 I am able to bring up the FreeBSD VM and can attach to it via tightvnc, so that's great.  I have configured the VM to have an IPv4 address of 172.16.0.20/24 with a gateway of 172.16.0.1.  This is where I get stuck - when I attempt to ping the gateway from the VM I get "ping: sendto host down" errors.  Then I run "arp -an" to see what's up from the VM and see the following: ? (172.16.0.1) at (incomplete) on vtnet0 expired [ethernet] when I run tcpdump against the bridge0 interface on the hypervisor while ping is run I see the following output: 15:56:48.995284 ARP, Request who-has 172.16.0.1 tell 172.16.0.20, length 46 15:56:48.995292 ARP, Reply 172.16.0.1 is-at 02:46:2f:56:ab:00 (oui Unknown), length 28 And I can verify that that is the correct MAC addr of the bridge0 interface: $ ifconfig bridge0|grep ether     ether 02:46:2f:56:ab:00 yet, tcpdump on the vtnet0 interface from the VM does not show the responses to the ARP requests: ARP: Request who-has 172.16.0.1 tell 172.16.0.20, length 28 ARP: Request who-has 172.16.0.1 tell 172.16.0.20, length 28 ARP: Request who-has 172.16.0.1 tell 172.16.0.20, length 28 I suspect I'm missing something trivial here, so any input would be appreciated :) Oh, one last bit - here are my sysctl settings: security.bsd.see_other_uids=0 security.bsd.see_other_gids=0 security.bsd.unprivileged_read_msgbuf=0 security.bsd.unprivileged_proc_debug=0 kern.randompid=1 security.bsd.stack_guard_page=1 vfs.zfs.min_auto_ashift=12 net.link.tap.up_on_open=1 net.inet.ip.forwarding=1 net.inet6.ip6.forwarding=1 Cheers, -pete -- Pete Wright pete@nomadlogic.org @nomadlogicLA