From owner-freebsd-net@freebsd.org Tue Sep 18 07:19:11 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 615691094A52 for ; Tue, 18 Sep 2018 07:19:11 +0000 (UTC) (envelope-from pekka.jarvinen@gmail.com) Received: from mail-oi0-x236.google.com (mail-oi0-x236.google.com [IPv6:2607:f8b0:4003:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E8CFF819D4 for ; Tue, 18 Sep 2018 07:19:10 +0000 (UTC) (envelope-from pekka.jarvinen@gmail.com) Received: by mail-oi0-x236.google.com with SMTP id x197-v6so836673oix.5 for ; Tue, 18 Sep 2018 00:19:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to :content-transfer-encoding; bh=hz4sfldPARYdZ+rUJR30qrOswbo0YkZAiUC/qKXeah0=; b=LrRBIiGtyS0qwXyzax1SjYXpp57wiphwbh4LE5EtwZ7TfatjOyjrydhbz0BgQWvocm 2/yIJw7kiJxlao6vJyElYPckQaTOrx+PF2vPlFo8j7p03rPSNr8+Pbw8lO2yS2zZew+Z FofLDXg+OR5o2nN3yVEareVdXRAWgKrnbmB4Y6EqFx4HA9uTJh20OoN5DCE2VxufRrvM ikcAHugcwRSXRy52/wyiI7f0XcaRoes0/dkYPOugA72z96JyNnDknss4nblzLAPMGA2M pI3MBUkCWfpg2IpNUcEK4mIul80qNjvDMSgYnZj5pShd2mQ6onU8h5WL/sH//jrgcmYQ yJrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to :content-transfer-encoding; bh=hz4sfldPARYdZ+rUJR30qrOswbo0YkZAiUC/qKXeah0=; b=nzqFtqZJwySKvRKC49+JDq4EthMgqwb8frLX0Y5xqvCaBpoAKvs7N8dFM54BfPZ7hz ziIXRpkB4ZWGexSUvjjKb2uAj/F5i+8vdS30ac05J4lu9Zko/UFBp+QO9kPTDbN9S+Ax 9ZtOFahgcxMuFzs5n8VED4TxcR5Sjb9LGEhMqbfqp4mOumH34XCbkg2/PMAvTps152wx E48UTDvHjS1oSU2N+hH/G87x9Cr8/OdNThSbGpcHYGsiNHgGIQEuNT7mS+yi30D2a80p lCadp5hQrFD8N3bhEHFlhoklZ5VJqLmQXk9ljPozqH0ATcBPz8VRoHtSmWNi4eCe7za/ uXyg== X-Gm-Message-State: APzg51AENIRejtEgNIIAZoevuYyMGZ50/DqvHVqxB51DJBeMB8pCHl+g //z/3Cp8Cla7pyNPtP+Ke9qjiVWeyUQsVPBO0++CyV1B X-Google-Smtp-Source: ANB0VdaWIP00KxJ2efrSxO8gRpIp5v61XJPYqFbZCBZUYSP1ViQidjqpLDn7nrs1HZNidtQWqO66kZQeNFWrrjXnfJE= X-Received: by 2002:aca:650d:: with SMTP id m13-v6mr746851oim.178.1537255149968; Tue, 18 Sep 2018 00:19:09 -0700 (PDT) MIME-Version: 1.0 From: =?UTF-8?Q?Pekka_J=C3=A4rvinen?= Date: Tue, 18 Sep 2018 10:18:43 +0300 Message-ID: Subject: Bridge, VLANs and breaking packets To: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Sep 2018 07:19:11 -0000 I'm trying out FreeBSD with bhyve hypervisor and trying to replicate VMware ESXi vSwitch config in FreeBSD so that pfSense VM gets network traffic properly. VLANs: * 111 =3D LAN, 192.168.101.0/24 * 333 =3D Internet (WAN), DHCP pfSense virtual machine setup (ESXi & FreeBSD): NICS: * vnic0 * vnic0.111 192.168.101.1/24 * vnic0.333 DHCP from ISP * NAT: VLAN 333 <-> VLAN 111 & DHCP server * HW offloads off Old ESXi setup: * 192.168.101.6/24 on VLAN 111 GW 192.168.101.1 * Only vSwitch, no dvSwitches vSwitch0: ----------------------- ----------------------- | allvlans | | Physical adapters | | VLAN ID: 4095 (all) |----| * vmnic0, 1000 Mbps | | * pfSense24 | | ----------------------- ----------------------- | | ----------------------- | | Management Network |--/ | VLAN ID: 111 | | VMkernel ports (1): | | * vmk0: 192.168.101.6 | ----------------------- Security policy: * Allow promiscuous mode: **no** * Allow forged transmits: **no** * Allow MAC changes: **no** Port group allvlans: * VLAN ID: 4095 (allow all tagged VLAN traffic) * Allow promiscuous mode: **Inherit from vSwitch** (no) * Allow forged transmits: **Inherit from vSwitch** (no) * Allow MAC changes: **Inherit from vSwitch** (no) FreeBSD ESXi replacement setup attempt: sysrc -f /boot/loader.conf vmm_load=3D"YES" sysrc -f /boot/loader.conf nmdm_load=3D"YES" sysrc -f /boot/loader.conf if_bridge_load=3D"YES" sysrc -f /boot/loader.conf if_tap_load=3D"YES" sysrc if_vlan_load=3D"YES" sysrc cloned_interfaces=3D"bridge0 tap0" sysrc ifconfig_bridge0=3D"addm em0 addm tap0" echo "net.link.tap.up_on_open=3D1" > /etc/sysctl.d/vm_network.conf sysrc defaultrouter=3D"192.168.101.1" sysrc ifconfig_em0=3D"up" sysrc gateway_enable=3D"YES" ESXi's *Management Network* equivalent(?) for SSH access: sysrc vlans_em0=3D"111" sysrc ifconfig_em0_111=3D"inet 192.168.101.6/24" Interfaces: em0: flags=3D8943 metric 0 mtu 1500 options=3D852099 ether 00:25:90:14:95:8c nd6 options=3D29 media: Ethernet autoselect (1000baseT ) status: active bridge0: flags=3D8843 metric 0 mtu 1500 ether 02:eb:00:40:63:00 nd6 options=3D9 groups: bridge id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap0 flags=3D143 ifmaxaddr 0 port 5 priority 128 path cost 2000000 member: em0 flags=3D143 ifmaxaddr 0 port 1 priority 128 path cost 2000000 tap0: flags=3D8943 metric 0 mtu 1500 options=3D80000 ether 00:bd:f0:02:f7:00 nd6 options=3D29 media: Ethernet autoselect status: active groups: tap Opened by PID 45408 em0.111: flags=3D8843 metric 0 mtu 1500 options=3D103 ether 00:25:90:14:95:8c inet 192.168.101.6 netmask 0xffffff00 broadcast 192.168.101.255 nd6 options=3D29 media: Ethernet autoselect (1000baseT ) status: active vlan: 111 vlanpcp: 0 parent interface: em0 groups: vlan `pciconf -lv`: em0@pci0:1:0:0: class=3D0x020000 card=3D0x10d315d9 chip=3D0x10d38086 rev=3D0x00 hdr=3D0x00 vendor =3D 'Intel Corporation' device =3D '82574L Gigabit Network Connection' class =3D network subclass =3D ethernet Running pfSense VM with: sh /usr/share/examples/bhyve/vmrun.sh -m 2048M -d /dev/zvol/tank/pfsense0 pfsensevm It uses `tap0`. Currently I can access pfSense's web admin but `vmnet0.333` doesn't get IP from my ISP. I'd like to set VLAN bridging as securely as possible (`private` for `bridge0`?). If it's possible to send all traffic to the physical switch first, that's what I would prefer. openvSwitch is also ok, but I'm not familiar with it. DHCP request packet is correctly seen in tap0, bridge0, and em0 with `tcpdump -lnexv -i "vlan 333"`and not in em0.111, which is correct when requesting new IP in pfSense. There's no DHCP response from ISP. em0 and tap0 are in promisc mode which is disabled in vSwitch version. More debugging: Connected Raspberry Pi to my VDSL modem and the DHCP packet is seen. I also tried spoofing the pfSense's MAC address with RPi and it works. So possible culprits currently are packet checksum and packet truncating. At least Linux bridge implementation is very well known to break packets with bridging and VLANs so FreeBSD might be doing it as well? Adding VLAN 333 to em0 DHCP works. So what is bridge0 and/or tap0 doing? What I am missing? You can see this same message properly formatted @ https://unix.stackexchange.com/questions/469514/freebsd-ovs-equivalent-of-e= sxi-vswitch-vlan-config --=20 Pekka J=C3=A4rvinen