From owner-freebsd-net@freebsd.org Tue Mar 16 14:35:32 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8A5EE5AFDC9 for ; Tue, 16 Mar 2021 14:35:32 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0G6X2Pbvz4d18 for ; Tue, 16 Mar 2021 14:35:32 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from [IPv6:2a02:8109:1140:c3d:98e:1307:de88:8980] (unknown [IPv6:2a02:8109:1140:c3d:98e:1307:de88:8980]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTPSA id E295476938665; Tue, 16 Mar 2021 15:35:25 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: Severe IPv6 TCP transfer issues on 13.0-RC1 and RC2 From: tuexen@freebsd.org In-Reply-To: Date: Tue, 16 Mar 2021 15:35:25 +0100 Cc: Blake Hartshorn , freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <947D1E51-CCB0-4A1E-B671-708F70420DA1@freebsd.org> References: <5753280.1HxCrU2fYu@thinkbook> <10847992.4GmZMkJedg@thinkbook> <2808496E-C2B5-4B6F-8FDB-310B7F69E540@freebsd.org> To: Marek Zarychta X-Mailer: Apple Mail (2.3654.60.0.2.21) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, URIBL_BLOCKED autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Rspamd-Queue-Id: 4F0G6X2Pbvz4d18 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 14:35:32 -0000 > On 16. Mar 2021, at 15:18, Marek Zarychta = wrote: >=20 > W dniu 16.03.2021 o 12:50, tuexen@freebsd.org pisze: >>> On 16. Mar 2021, at 11:55, Blake Hartshorn = wrote: >>>=20 >>> Hi Michael, >>>=20 >>> I've attached tcpdumps for port 80 on both sides of a bad transfer, = using 2 VMs in the same datacenter, FreeBSD 13 serving and 12 as a = client. A friend of mine suggested I also run some tests with iperf3, so = pasting those results below. You'll see it going fast in one direction = and crawling in the other on TCP. There's also some disparity on UDP.=20 >> The problem is that the server provides TCP segments larger than the = MTU >> to the NIC. These are dropped and needs to be retransmitted. That is = why >> it takes so long. So I guess TSO is enabled on the NIC and not = working correctly. >>=20 >> What is the output of ifconfig? Can you disable TSO? Does that work = around >> the problem? >>=20 >> Best regards >> Michael >>>=20 >>> TCP: >>> [ ID][Role] Interval Transfer Bitrate Retr = Cwnd >>> [ 5][TX-C] 0.00-1.00 sec 27.7 MBytes 233 Mbits/sec 342 = 26.7 KBytes =20 >>> [ 7][RX-C] 0.00-1.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>> [ 5][TX-C] 1.00-2.00 sec 15.8 MBytes 132 Mbits/sec 249 = 52.0 KBytes =20 >>> [ 7][RX-C] 1.00-2.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>> [ 5][TX-C] 2.00-3.00 sec 13.7 MBytes 115 Mbits/sec 307 = 15.4 KBytes =20 >>> [ 7][RX-C] 2.00-3.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>> [ 5][TX-C] 3.00-4.00 sec 14.5 MBytes 121 Mbits/sec 260 = 22.4 KBytes =20 >>> [ 7][RX-C] 3.00-4.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>> [ 5][TX-C] 4.00-5.00 sec 14.3 MBytes 120 Mbits/sec 240 = 37.9 KBytes =20 >>> [ 7][RX-C] 4.00-5.00 sec 5.58 KBytes 45.7 Kbits/sec = =20 >>> [ 5][TX-C] 5.00-6.00 sec 17.7 MBytes 149 Mbits/sec 363 = 15.4 KBytes =20 >>> [ 7][RX-C] 5.00-6.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>> [ 5][TX-C] 6.00-7.00 sec 14.8 MBytes 124 Mbits/sec 287 = 8.38 KBytes =20 >>> [ 7][RX-C] 6.00-7.00 sec 5.58 KBytes 45.7 Kbits/sec = =20 >>> [ 5][TX-C] 7.00-8.00 sec 14.7 MBytes 123 Mbits/sec 293 = 28.1 KBytes =20 >>> [ 7][RX-C] 7.00-8.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>> [ 5][TX-C] 8.00-9.00 sec 11.9 MBytes 100 Mbits/sec 325 = 18.3 KBytes =20 >>> [ 7][RX-C] 8.00-9.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>> [ 5][TX-C] 9.00-10.00 sec 14.3 MBytes 120 Mbits/sec 315 = 39.3 KBytes =20 >>> [ 7][RX-C] 9.00-10.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>> - - - - - - - - - - - - - - - - - - - - - - - - - >>> [ ID][Role] Interval Transfer Bitrate Retr >>> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec 2981 = sender >>> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec = receiver >>> [ 7][RX-C] 0.00-10.00 sec 77.0 KBytes 63.1 Kbits/sec 65 = sender >>> [ 7][RX-C] 0.00-10.00 sec 44.6 KBytes 36.6 Kbits/sec = receiver >>> = --------------------------------------------------------------------------= --------------------------------- >>>=20 >>>=20 >>> UDP: >>> [ ID][Role] Interval Transfer Bitrate Jitter = Lost/Total Datagrams >>> [ 5][TX-C] 0.00-1.00 sec 81.6 MBytes 685 Mbits/sec = 67798 =20 >>> [ 7][RX-C] 0.00-1.00 sec 8.80 MBytes 73.8 Mbits/sec 0.255 ms = 54070/60475 (89%) =20 >>> [ 5][TX-C] 1.00-2.00 sec 72.7 MBytes 610 Mbits/sec = 64802 =20 >>> [ 7][RX-C] 1.00-2.00 sec 8.52 MBytes 71.5 Mbits/sec 0.154 ms = 68912/75116 (92%) =20 >>> [ 5][TX-C] 2.00-3.00 sec 73.7 MBytes 618 Mbits/sec = 64158 =20 >>> [ 7][RX-C] 2.00-3.00 sec 8.52 MBytes 71.5 Mbits/sec 0.276 ms = 67738/73945 (92%) =20 >>> [ 5][TX-C] 3.00-4.00 sec 76.6 MBytes 643 Mbits/sec = 63521 =20 >>> [ 7][RX-C] 3.00-4.00 sec 8.55 MBytes 71.8 Mbits/sec 0.160 ms = 68647/74874 (92%) =20 >>> [ 5][TX-C] 4.00-5.00 sec 76.1 MBytes 638 Mbits/sec = 64614 =20 >>> [ 7][RX-C] 4.00-5.00 sec 8.55 MBytes 71.7 Mbits/sec 0.461 ms = 67542/73767 (92%) =20 >>> [ 5][TX-C] 5.00-6.00 sec 75.9 MBytes 637 Mbits/sec = 64834 =20 >>> [ 7][RX-C] 5.00-6.00 sec 8.57 MBytes 71.9 Mbits/sec 0.297 ms = 71565/77806 (92%) =20 >>> [ 5][TX-C] 6.00-7.00 sec 73.0 MBytes 613 Mbits/sec = 63639 =20 >>> [ 7][RX-C] 6.00-7.00 sec 8.40 MBytes 70.5 Mbits/sec 0.199 ms = 69545/75663 (92%) =20 >>> [ 5][TX-C] 7.00-8.00 sec 74.6 MBytes 626 Mbits/sec = 65030 =20 >>> [ 7][RX-C] 7.00-8.00 sec 8.78 MBytes 73.6 Mbits/sec 0.254 ms = 67173/73566 (91%) =20 >>> [ 5][TX-C] 8.00-9.00 sec 75.0 MBytes 629 Mbits/sec = 64848 =20 >>> [ 7][RX-C] 8.00-9.00 sec 8.77 MBytes 73.5 Mbits/sec 0.298 ms = 70932/77315 (92%) =20 >>> [ 5][TX-C] 9.00-10.00 sec 74.5 MBytes 625 Mbits/sec = 64487 =20 >>> [ 7][RX-C] 9.00-10.00 sec 8.71 MBytes 73.1 Mbits/sec 0.185 ms = 68268/74612 (91%) =20 >>> - - - - - - - - - - - - - - - - - - - - - - - - - >>> [ ID][Role] Interval Transfer Bitrate Jitter = Lost/Total Datagrams >>> [ 5][TX-C] 0.00-10.00 sec 754 MBytes 632 Mbits/sec 0.000 ms = 0/647731 (0%) sender >>> [ 5][TX-C] 0.00-10.12 sec 105 MBytes 87.2 Mbits/sec 0.245 ms = 571090/647649 (88%) receiver >>> [ 7][RX-C] 0.00-10.00 sec 1009 MBytes 846 Mbits/sec 0.000 ms = 0/761013 (0%) sender >>> [ 7][RX-C] 0.00-10.12 sec 86.2 MBytes 71.4 Mbits/sec 0.185 ms = 674392/737139 (91%) receiver >>>=20 >>>=20 >=20 > Taking a look at this iperf output I recalled that 80% of my setups > suffered from similar issue after transitioning from 1{1,2}-STABLE to > 13-STABLE about a mounth ago. I have asked on IRC but nobody confirmed > similar problems so I have reduced MTU to 8900 from the original 9000 = on > some vlan(4) interfaces to solve the issue. I am using mostly vlan(4)s > over LACP lagg(4)s created on NICs. So far (for FreeBSD 11 and 12) > setting MTU 9000 on physical NIC was sufficient to make it work, now I > have set MTU 9000 on NICs and reduced MTU 8900 on vlan(4)s. TCP announces an MSS of 1440, which corresponds to an MTU of 1500 byte. So I guess your problem is different. Is your physical MTU 9000 bytes? Best regards Michael >=20 >>>=20 >>>=20 >>> On Tuesday, March 16, 2021 4:16:15 AM EDT tuexen@freebsd.org wrote: >>>>> On 15. Mar 2021, at 12:56, Blake Hartshorn = wrote: >>>>>=20 >>>>> The short version, when I use FreeBSD 13, delivering data can take = 5 minutes for 1MB over SSH or HTTP when using IPv6. This problem does = not happen with IPv4. I installed FreeBSD 12 and Linux on that same = device, neither had the problem. >>>>>=20 >>>>> Did some troubleshooting with Linode, have ultimately ruled the = network itself out at this point. When the server is on FreeBSD 13, it = can download quickly over IPv6, but not deliver. Started investigating = after noticing my SSH session was lagging when cat'ing large files or = running builds. This problem even occurs between VMs in the same = datacenter. I generated a 1MB file of base64 garbage served by nginx for = testing. IPv6 is being configured by SLAAC and on both 12 and 13 = installs was setup by the installer. Linode uses Linux/KVM hosts for = their virtual machines so it's running on that virtual adapter. >>>>>=20 >>>>> I asked on the forums, another user recommended going to the = mailing lists instead. Does anyone know if config settings need to be = different on 13? Did I maybe just find a real issue? I can provide any = requested details. Thanks! >>>> Could you prove a .pcap tracefile, one from the sender, one from = the recevier, of >>>> a TCP/IPv6 connection, which doesn't work as expected. For example, = use your 1MB >>>> base64 garbage transfer. >>>>=20 >>>> Best regardes >>>> Michael >>>>>=20 >>>>>=20 >>>>> _______________________________________________ >>>>> freebsd-net@freebsd.org mailing list >>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >>>>=20 >>>>=20 >>> >>=20 >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to = "freebsd-net-unsubscribe@freebsd.org" >>=20 >=20 >=20 > --=20 > Marek Zarychta >=20