From owner-freebsd-bugs@freebsd.org Fri Feb 8 17:08:18 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E1B014B640C for ; Fri, 8 Feb 2019 17:08:18 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BB7CA8AE87 for ; Fri, 8 Feb 2019 17:08:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 7066614B640B; Fri, 8 Feb 2019 17:08:17 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 46D7714B6409 for ; Fri, 8 Feb 2019 17:08:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C350D8AE86 for ; Fri, 8 Feb 2019 17:08:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id F3C3F19631 for ; Fri, 8 Feb 2019 17:08:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id x18H8F0D083628 for ; Fri, 8 Feb 2019 17:08:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id x18H8FPF083627 for bugs@FreeBSD.org; Fri, 8 Feb 2019 17:08:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 235607] Incorrect checksums with NAT on vtnet with offloading Date: Fri, 08 Feb 2019 17:08:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sjorge+signup@blackdot.be X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Feb 2019 17:08:18 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235607 Bug ID: 235607 Summary: Incorrect checksums with NAT on vtnet with offloading Product: Base System Version: 12.0-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: sjorge+signup@blackdot.be ### description The issue only pops up when there is no valid tcp checksum present on the source traffic, it 'works' when the csum is valid. I did verify with the illumos people about if a bad checksum will be pass along as is or not with offloading enabled: [17:03:15] rzezeski qq, if a guest with csum enabled sends a checksummed packet does it get discarded and recalculate when it hits the physical nic before it goes on the wire? Or is it kept as is? [17:03:26] From my captures it seems to be kept as is? [17:05:26] sjorge: It depends on the guest OS. In illumos, either the IP stack calcs the checksum on Tx or places a flag on the dblk to have = it calculated by hardware. If you do both then the hardware will calculate a second sum over the current one and it will be incorrect. [17:05:42] 1) 2) I have no idea how FBSD/Linux work in that regard, but they would have the same issue to deal with. I beleive the later case is what is happening here, pf nat adds checksum ba= sed on empty initial checksum and then it gets pass as-is out the nic. That wou= ld also explain why a packet with a valid checksum that comes in will pass back out correctly after pf nat. <-> =3D phsyical link, aka cat5/cat6 plugged into a NIC or Switch <~> =3D loopback link, aka traffic between bhyve guests, between host and b= hyve guest, ... stuff that never hit the MAC layer. <.> =3D wireless link This flow that is currently broken (only when pf nat is involved): fbsd_guest1 <~> fbsd_guest_fw <-> switch(1) <-> modem win10_guest <~> fbsd_guest_fw <-> switch(1) <-> modem This flow is OK: macbook <-> switch <->(2) fbsd_guest_fw <-> switch <-> modem macbook <.> AP <-> switch <-> fbsd_guest_fw <-> switch <-> modem 1) using port-mirror on the switch I was able to confirm packets with a bad checksum end up on the wire 2) the bhyve guest has a vnic on the physical nic (they are considered brid= ge for the host OS) #### workaround but comes at a performance hit root@nattest:~ # ifconfig vtnet0 -rxcsum -txcsum -rxcsum6 -txcsum6 -tso4 -l= ro root@nattest:~ # ifconfig vtnet1 -rxcsum -txcsum -rxcsum6 -txcsum6 -tso4 -l= ro Re-enabeling these on will make the behavior return, this can be done live! ### uname -a FreeBSD nattest 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64 ### ifconfig output vtnet0: flags=3D8943 metric= 0 mtu 1500 =20=20=20=20=20=20=20 options=3D6c05bb ether 00:22:06:05:01:0a inet 192.168.0.212 netmask 0xffffff00 broadcast 192.168.0.255 media: Ethernet 10Gbase-T status: active nd6 options=3D29 vtnet1: flags=3D8943 metric= 0 mtu 1500 =20=20=20=20=20=20=20 options=3D6c05bb ether 00:22:06:0a:01:01 inet 10.23.10.87 netmask 0xffffff00 broadcast 10.23.10.255 media: Ethernet 10Gbase-T status: active nd6 options=3D29 lo0: flags=3D8049 metric 0 mtu 16384 options=3D680003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=3D21 pflog0: flags=3D141 metric 0 mtu 33160 groups: pflog ### /etc/pf.conf if_wan=3D"vtnet0" if_lan=3D"vtnet1" net_lan=3D$if_lan:network scrub on lo0 all random-id scrub on vtnet0 all random-id scrub on vtnet1 all random-id nat on $if_wan inet from $net_lan to any -> ($if_wan) port 1024:65535 antispoof log for vtnet0 antispoof log for vtnet1 block in all pass on lo0 all pass inet proto icmp all icmp-type { echoreq, unreach } keep state pass out all keep state pass in on $if_lan proto tcp from $net_lan to any port { 22, 80, 443 } ### /etc/rc.conf hostname=3D"nattest" gateway_enable=3D"YES" ifconfig_vtnet0=3D"DHCP" ifconfig_vtnet1=3D"10.23.10.87 netmask 255.255.255.0" zfs_enable=3D"YES" clear_tmp_enable=3D"YES" sshd_enable=3D"YES" dumpdev=3D"AUTO" pf_enable=3D"YES" pflog_enable=3D"YES" --=20 You are receiving this mail because: You are the assignee for the bug.=