From owner-freebsd-net@freebsd.org  Sun Dec 11 06:01:10 2016
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5344BC715DA
 for <freebsd-net@mailman.ysv.freebsd.org>;
 Sun, 11 Dec 2016 06:01:10 +0000 (UTC)
 (envelope-from dkleinh@phy.ucsf.edu)
Received: from mail.cin.ucsf.edu (ns.cin.ucsf.edu [169.230.188.10])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 37F2A773
 for <freebsd-net@freebsd.org>; Sun, 11 Dec 2016 06:01:09 +0000 (UTC)
 (envelope-from dkleinh@phy.ucsf.edu)
Received: by mail.cin.ucsf.edu (Postfix, from userid 33)
 id C887D140D26; Sat, 10 Dec 2016 21:54:05 -0800 (PST)
Received: from adsl-71-131-0-97.dsl.sntc01.pacbell.net
 (adsl-71-131-0-97.dsl.sntc01.pacbell.net [71.131.0.97]) by keck.ucsf.edu
 (Horde Framework) with HTTP; Sat, 10 Dec 2016 21:54:05 -0800
Message-ID: <20161210215405.886061vp9d04ld6l@keck.ucsf.edu>
Date: Sat, 10 Dec 2016 21:54:05 -0800
From: dkleinh@phy.ucsf.edu
To: freebsd-net@freebsd.org
Subject: tcp between tap interfaces
MIME-Version: 1.0
Content-Type: text/plain;
 charset=ISO-8859-1;
 DelSp="Yes";
 format="flowed"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit
User-Agent: Internet Messaging Program (IMP) H3 (4.3.11)
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 11 Dec 2016 06:01:10 -0000

I'm trying to setup a private testing environment using the bhyve
hypervisor and some virtual machines connected with tap interfaces
to a bridge.  My network configuration for this environment looks like
this:

I have a bridge interface with 5 tap interfaces, but no real interface as
this is to be virtual.  The bridge interface has interface: 192.168.1.1
This is the gateway for the VMs.  Each tap interface on the (virtual)  
bridge to each VM is on the 192.168.1.0/24 network.  I nat the private  
network out through a real interface on the host.

I use the pf packet filter and nat is working great, each VM can  
connect out to the world.  The host can connect into each VM through  
the bridge and icmp and udp seem to work great between the VMs on the  
private network, but tcp does not seem to work.  That is, I cannot ssh  
between the VMs, but ping works and I've setup a DNS server on one of  
the VMs and that works for resolving the different private VM host  
names and external names.  The host can ssh into each VM OK.

I'm totally at a loss where to go with this.

I'm running FreeBSD 10.1 on the host.