Date: Fri, 08 Feb 2019 17:08:14 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 235607] Incorrect checksums with NAT on vtnet with offloading Message-ID: <bug-235607-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235607 Bug ID: 235607 Summary: Incorrect checksums with NAT on vtnet with offloading Product: Base System Version: 12.0-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: sjorge+signup@blackdot.be ### description The issue only pops up when there is no valid tcp checksum present on the source traffic, it 'works' when the csum is valid. I did verify with the illumos people about if a bad checksum will be pass along as is or not with offloading enabled: [17:03:15] <sjorge> rzezeski qq, if a guest with csum enabled sends a checksummed packet does it get discarded and recalculate when it hits the physical nic before it goes on the wire? Or is it kept as is? [17:03:26] <sjorge> From my captures it seems to be kept as is? [17:05:26] <rzezeski> sjorge: It depends on the guest OS. In illumos, either the IP stack calcs the checksum on Tx or places a flag on the dblk to have = it calculated by hardware. If you do both then the hardware will calculate a second sum over the current one and it will be incorrect. [17:05:42] <rzezeski> 1) <reply to diff question> 2) I have no idea how FBSD/Linux work in that regard, but they would have the same issue to deal with. I beleive the later case is what is happening here, pf nat adds checksum ba= sed on empty initial checksum and then it gets pass as-is out the nic. That wou= ld also explain why a packet with a valid checksum that comes in will pass back out correctly after pf nat. <-> =3D phsyical link, aka cat5/cat6 plugged into a NIC or Switch <~> =3D loopback link, aka traffic between bhyve guests, between host and b= hyve guest, ... stuff that never hit the MAC layer. <.> =3D wireless link This flow that is currently broken (only when pf nat is involved): fbsd_guest1 <~> fbsd_guest_fw <-> switch(1) <-> modem win10_guest <~> fbsd_guest_fw <-> switch(1) <-> modem This flow is OK: macbook <-> switch <->(2) fbsd_guest_fw <-> switch <-> modem macbook <.> AP <-> switch <-> fbsd_guest_fw <-> switch <-> modem 1) using port-mirror on the switch I was able to confirm packets with a bad checksum end up on the wire 2) the bhyve guest has a vnic on the physical nic (they are considered brid= ge for the host OS) #### workaround but comes at a performance hit root@nattest:~ # ifconfig vtnet0 -rxcsum -txcsum -rxcsum6 -txcsum6 -tso4 -l= ro root@nattest:~ # ifconfig vtnet1 -rxcsum -txcsum -rxcsum6 -txcsum6 -tso4 -l= ro Re-enabeling these on will make the behavior return, this can be done live! ### uname -a FreeBSD nattest 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64 ### ifconfig output vtnet0: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric= 0 mtu 1500 =20=20=20=20=20=20=20 options=3D6c05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,TSO4,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:22:06:05:01:0a inet 192.168.0.212 netmask 0xffffff00 broadcast 192.168.0.255 media: Ethernet 10Gbase-T <full-duplex> status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> vtnet1: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric= 0 mtu 1500 =20=20=20=20=20=20=20 options=3D6c05bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSU= M,TSO4,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:22:06:0a:01:01 inet 10.23.10.87 netmask 0xffffff00 broadcast 10.23.10.255 media: Ethernet 10Gbase-T <full-duplex> status: active nd6 options=3D29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=3D680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=3D141<UP,RUNNING,PROMISC> metric 0 mtu 33160 groups: pflog ### /etc/pf.conf if_wan=3D"vtnet0" if_lan=3D"vtnet1" net_lan=3D$if_lan:network scrub on lo0 all random-id scrub on vtnet0 all random-id scrub on vtnet1 all random-id nat on $if_wan inet from $net_lan to any -> ($if_wan) port 1024:65535 antispoof log for vtnet0 antispoof log for vtnet1 block in all pass on lo0 all pass inet proto icmp all icmp-type { echoreq, unreach } keep state pass out all keep state pass in on $if_lan proto tcp from $net_lan to any port { 22, 80, 443 } ### /etc/rc.conf hostname=3D"nattest" gateway_enable=3D"YES" ifconfig_vtnet0=3D"DHCP" ifconfig_vtnet1=3D"10.23.10.87 netmask 255.255.255.0" zfs_enable=3D"YES" clear_tmp_enable=3D"YES" sshd_enable=3D"YES" dumpdev=3D"AUTO" pf_enable=3D"YES" pflog_enable=3D"YES" --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-235607-227>