From owner-freebsd-virtualization@freebsd.org Thu Mar 3 14:49:20 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0FD08A931E5 for ; Thu, 3 Mar 2016 14:49:20 +0000 (UTC) (envelope-from dweimer@dweimer.net) Received: from webmail.dweimer.net (24-240-198-187.static.stls.mo.charter.com [24.240.198.187]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E55C017C for ; Thu, 3 Mar 2016 14:49:18 +0000 (UTC) (envelope-from dweimer@dweimer.net) Received: from webmail.dweimer.local (localhost [10.9.5.2]) by webmail.dweimer.net (8.15.2/8.15.2) with ESMTPS id u23EnCea005056 (version=TLSv1.2 cipher=DHE-RSA-CHACHA20-POLY1305 bits=256 verify=NO); Thu, 3 Mar 2016 08:49:12 -0600 (CST) (envelope-from dweimer@dweimer.net) Received: (from www@localhost) by webmail.dweimer.local (8.15.2/8.15.2/Submit) id u23EnBFu005055; Thu, 3 Mar 2016 08:49:11 -0600 (CST) (envelope-from dweimer@dweimer.net) X-Authentication-Warning: webmail.dweimer.local: www set sender to dweimer@dweimer.net using -f To: Nikos Vassiliadis Subject: Re: bhyve and CARP? X-PHP-Script: www.dweimer.net/webmail/index.php for 71.86.41.122, 10.9.5.3 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 03 Mar 2016 08:49:11 -0600 From: dweimer Cc: freebsd-virtualization@freebsd.org Organization: dweimer.net Reply-To: dweimer@dweimer.net Mail-Reply-To: dweimer@dweimer.net In-Reply-To: <56D6022A.8030808@gmx.com> References: <56D6022A.8030808@gmx.com> Message-ID: X-Sender: dweimer@dweimer.net User-Agent: Roundcube Webmail/1.1.4 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Mar 2016 14:49:20 -0000 On 2016-03-01 2:57 pm, Nikos Vassiliadis wrote: > Hi, > > On 03/01/16 18:43, dweimer wrote: >> I am considering setting up a bhyve virtual machine to run pfSense. >> Not >> too thrilled with the CPU heat on the PC Engines APU1D4 when under >> heavy >> load, but don't want to rely entirely on a VM. As I like still having >> internet if I would have to take my server offline for disk >> replacement >> or other issues, having web access to search for errors is a big plus. >> So in order to avoid spending money on a new piece of hardware I >> thought >> why not do a VM with CARP fail over to the physical. I am not finding >> much searching on FreeBSD byhve and CARP, I know its somewhat of an >> issue withing VMware on ESX making sure you enable the right options >> on >> the virtual switches and interfaces. >> >> Enable promiscuous mode on the vSwitch >> Enable "MAC Address changes" >> Enable "Forged transmits" >> >> Before I got started on the setup I was curious if anyone has done >> something similar, or know if this isn't possible on bhyve at the >> current version? I am running my system currently on 10.3-BETA3. >> > > I am running two postgres VMs with DRBD and not CARP but UCARP which > should be 100% compatible with CARP. Each VM has a tap interface and > each tap is bridged to a bridge interface. There is no need for special > configuration. Everything works as expected. > Well so far I have it mostly working, one issue though, that I can't quite find the source of the problem. I have multiple port forwards setup and use NAT reflection to make those accessible from the same host name internally and externally. I am redirecting ports 80, 443, 7443, and 8443 among others on of the virtual carp IP addresses. 80 and 443 are redirected to my proxy jail running Squid as a reverse proxy, jail is on same host as bhyve. 7443 redirects to Ubiquiti UniFi Video server for HTTPS running on another bhyve Linux virtual machine. 8443 redirects to Ubiquiti UniFi Wireless controller for HTTPS on another jail on the same host as the bhyve virtual machines. Everything that is running with NAT reflection works except for the port 443 traffic from the bhyve host machine, any jails running on it, and the other bhyve virtual machine. However it works fine from other network clients. Of course the NAT reflection is so that the same certificate can be used on all the HTTPS connections and show as valid. As near as I can tell the initial request makes it through the pfSense, to the Proxy. The Proxy's response makes it back to the pfSense. The pfSense system sends it to the client, but the client doesn't acknowledge that it received it. I have used tcpdump on the system to verify that it does receive the packets. I initially suspected something with the HTTPS was rejecting the virtual IPs used with CARP but that doesn't explain why it works on the other HTTPS ports. And failing over to the old physical APU1D4 it all works. As well as it working from other clients. I plan to add a second HTTPS port to the squid reverse proxy configuration to see if its isolated to the port 443 or if its isolated to the HTTPS on squid. I will also try redirecting straight to the Apache jail that the Proxy forwards to, Squid is only used as a reverse proxy on this setup so that I can test Squid updates here before installing them on the Reverse proxy I maintain at work. -- Thanks, Dean E. Weimer http://www.dweimer.net/