From owner-freebsd-net@FreeBSD.ORG Thu Apr 23 18:26:54 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EE3DF499 for ; Thu, 23 Apr 2015 18:26:54 +0000 (UTC) Received: from mx2.shrew.net (mx2.shrew.net [38.97.5.132]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B27811AD for ; Thu, 23 Apr 2015 18:26:54 +0000 (UTC) Received: from mail.shrew.net (mail.shrew.prv [10.24.10.20]) by mx2.shrew.net (8.14.7/8.14.7) with ESMTP id t3NIPJ2K026758 for ; Thu, 23 Apr 2015 13:25:19 -0500 (CDT) (envelope-from mgrooms@shrew.net) Received: from [10.16.32.30] (72-48-144-84.static.grandenetworks.net [72.48.144.84]) by mail.shrew.net (Postfix) with ESMTPSA id 949AB18AF96 for ; Thu, 23 Apr 2015 13:25:08 -0500 (CDT) Message-ID: <5539395D.9040304@shrew.net> Date: Thu, 23 Apr 2015 13:26:37 -0500 From: Matthew Grooms User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: bhyve with vlans - host and vm can't pass traffic References: <5537C5F8.1090000@sds.com> <55382C0A.1040505@shrew.net> <55384C24.5040607@sds.com> In-Reply-To: <55384C24.5040607@sds.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mx2.shrew.net [10.24.10.11]); Thu, 23 Apr 2015 13:25:19 -0500 (CDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Apr 2015 18:26:55 -0000 On 4/22/2015 8:34 PM, Scott O'Connell wrote: > I tried your suggestions. > > I was successful in changing the vmhost01 bridge to include vlan100 > and tap0, and in the vm (dev) binding the address directly to vtnet0. > > On the VMHOST: > tap0: flags=8943 > metric 0 mtu 1500 > options=80000 > ether 00:bd:4c:d1:02:00 > media: Ethernet autoselect > status: active > Opened by PID 888 > bridge0: flags=8843 metric 0 > mtu 1500 > ether 02:d3:e4:02:03:00 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap0 flags=143 > ifmaxaddr 0 port 6 priority 128 path cost 2000000 > member: vlan100 flags=143 > ifmaxaddr 0 port 5 priority 128 path cost 2000000 > > In the VM: > vtnet0: flags=8943 > metric 0 mtu 1500 > options=80028 > ether 00:a0:98:2b:34:37 > inet 10.0.1.6 netmask 0xffffff00 broadcast 10.0.1.255 > nd6 options=29 > media: Ethernet 10Gbase-T > status: active > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > > The same results with regard to connectivity. Both the VMHOST and > the VM can communicate everywhere, except with each other. > > I'm not sure how much detail to post, or what protocol I should be > testing from the tcpdump, but here are a couple of relevant portions. > Captured on the VMHOST with "tcpdump -i tap0 -n -vv", and on the VM > with "tcpdump -i vtnet0 -n -vv" > > A ping from the VM (10.0.1.6) to VMHOST (10.0.1.17): > > Captured on tap0: > 18:18:40.656407 IP (tos 0x0, ttl 64, id 2398, offset 0, flags [none], > proto ICMP (1), length 84) > 10.0.1.6 > 10.0.1.17: ICMP echo request, id 46082, seq 689, length 64 > 18:18:40.656429 IP (tos 0x0, ttl 64, id 3824, offset 0, flags [none], > proto ICMP (1), length 84, bad cksum 0 (->55a3)!) > 10.0.1.17 > 10.0.1.6: ICMP echo reply, id 46082, seq 689, length 64 > > Captured on vtnet0: > 18:18:40.906203 IP (tos 0x0, ttl 64, id 2398, offset 0, flags [none], > proto ICMP (1), length 84) > 10.0.1.6 > 10.0.1.17: ICMP echo request, id 46082, seq 689, length 64 > 18:18:40.906366 IP (tos 0x0, ttl 64, id 3824, offset 0, flags [none], > proto ICMP (1), length 84, bad cksum 0 (->55a3)!) > 10.0.1.17 > 10.0.1.6: ICMP echo reply, id 46082, seq 689, length 64 > > 100% packet loss on the ping. > > Here is the same traffic from both systems between the VM (10.0.1.6) > and the switch (10.0.1.1) through the VMHOST: > > Captured on tap0: > 18:23:42.712065 IP (tos 0x0, ttl 64, id 2858, offset 0, flags [none], > proto ICMP (1), length 84) > 10.0.1.6 > 10.0.1.1: ICMP echo request, id 58626, seq 2, length 64 > 18:23:42.712595 IP (tos 0x0, ttl 255, id 2858, offset 0, flags [none], > proto ICMP (1), length 84) > 10.0.1.1 > 10.0.1.6: ICMP echo reply, id 58626, seq 2, length 64 > > Captured on vtnet0: > 18:23:43.141890 IP (tos 0x0, ttl 64, id 2858, offset 0, flags [none], > proto ICMP (1), length 84) > 10.0.1.6 > 10.0.1.1: ICMP echo request, id 58626, seq 2, length 64 > 18:23:43.142553 IP (tos 0x0, ttl 255, id 2858, offset 0, flags [none], > proto ICMP (1), length 84) > 10.0.1.1 > 10.0.1.6: ICMP echo reply, id 58626, seq 2, length 64 > > 100% packet success on the ping. > > I'm never quite sure when checksum's with TCP Dump or Wireshark are > expected, and when they aren't, but it appears that is where the > problem lies here. > > With that said, if I'm understanding this correctly, and checksums are > the problem, I'm not sure what to try next. > > Thanks again! > Hi Scott, I certainly appears that ICMP echo reply packets are being returned but the host isn't processing them for some reason. Do you have any firewalls running on either system? You might try including a -e in the tcpdump command line arguments. IIRC, that will also show you VLAN and MAC address info from the packet headers. Maybe one of the network kernel developers could provide some additional insight as to what may be happening in this scenario. -Matthew