From owner-freebsd-net@freebsd.org Tue Mar 16 14:53:02 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5872F5B05D3 for ; Tue, 16 Mar 2021 14:53:02 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (plan-b.pwste.edu.pl [IPv6:2001:678:618::40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "plan-b.pwste.edu.pl", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0GVj3bYMz4f54; Tue, 16 Mar 2021 14:53:01 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from fomalhaut.potoki.eu ([IPv6:2001:678:618:c2:2:0:0:1]) (authenticated bits=0) by plan-b.pwste.edu.pl (8.16.1/8.16.1) with ESMTPSA id 12GEq2Gq016273 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Tue, 16 Mar 2021 15:52:13 +0100 (CET) (envelope-from zarychtam@plan-b.pwste.edu.pl) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=plan-b.pwste.edu.pl; s=plan-b-mailer; t=1615906348; bh=f+PInN9np8nQIE5h+CZl9yEix0t0My2pkBWgrsd+xs8=; h=To:Cc:References:From:Subject:Date:In-Reply-To; b=PeaeZvaRU0rS8L3whe6jKaAs3UIA3TGzqG90DoIhdWlCf/6RMvkFh7Lw5IETY7gU4 2Rqzo1RRAfSBlhfYanq6WDtNfrmGvZh80u2zkfhRF2xHsBNs7aFat1KunAO8rbmsWY v6r9pFPm8O3HjiN0tRc+/VWA9xWXTCASUyimUMgLJh/6unSCp0xXZVYTURcVrhFvfE 6JnztsUdurK/S4U4yZEUiBoBL+y4PxXrzluq/VHJn98wFB9F6/K+s1ZCnDFh9VLiwQ a5O9MohkUrscvzeKCvZDFNUyYzHb8du3IlumzmlruIioM/BTSGcqy7Eug9GpENC2F6 o6KmfpdFEFfkg== X-Authentication-Warning: plan-b.pwste.edu.pl: Host [IPv6:2001:678:618:c2:2:0:0:1] claimed to be fomalhaut.potoki.eu To: tuexen@freebsd.org Cc: Blake Hartshorn , freebsd-net@freebsd.org References: <5753280.1HxCrU2fYu@thinkbook> <10847992.4GmZMkJedg@thinkbook> <2808496E-C2B5-4B6F-8FDB-310B7F69E540@freebsd.org> <947D1E51-CCB0-4A1E-B671-708F70420DA1@freebsd.org> From: Marek Zarychta Subject: Re: Severe IPv6 TCP transfer issues on 13.0-RC1 and RC2 Message-ID: Date: Tue, 16 Mar 2021 15:51:52 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 In-Reply-To: <947D1E51-CCB0-4A1E-B671-708F70420DA1@freebsd.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="qMLh9QWrbst0hZe7ZMrmKN8QG9pvAfcx5" X-Rspamd-Queue-Id: 4F0GVj3bYMz4f54 X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=plan-b.pwste.edu.pl header.s=plan-b-mailer header.b=PeaeZvaR; dmarc=pass (policy=none) header.from=plan-b.pwste.edu.pl; spf=none (mx1.freebsd.org: domain of zarychtam@plan-b.pwste.edu.pl has no SPF policy when checking 2001:678:618::40) smtp.mailfrom=zarychtam@plan-b.pwste.edu.pl X-Spamd-Result: default: False [-7.90 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; HAS_ATTACHMENT(0.00)[]; HAS_XAW(0.00)[]; DKIM_TRACE(0.00)[plan-b.pwste.edu.pl:+]; DMARC_POLICY_ALLOW(-0.50)[plan-b.pwste.edu.pl,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:678:618::40:from]; ASN(0.00)[asn:206006, ipnet:2001:678:618::/48, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[plan-b.pwste.edu.pl:s=plan-b-mailer]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DWL_DNSWL_MED(-2.00)[pwste.edu.pl:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; SPAMHAUS_ZRD(0.00)[2001:678:618::40:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[no SPF record]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 14:53:02 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --qMLh9QWrbst0hZe7ZMrmKN8QG9pvAfcx5 Content-Type: multipart/mixed; boundary="J3sia0MV05ZtZmHbnbTrGyi8A4wcOA5yg"; protected-headers="v1" From: Marek Zarychta To: tuexen@freebsd.org Cc: Blake Hartshorn , freebsd-net@freebsd.org Message-ID: Subject: Re: Severe IPv6 TCP transfer issues on 13.0-RC1 and RC2 References: <5753280.1HxCrU2fYu@thinkbook> <10847992.4GmZMkJedg@thinkbook> <2808496E-C2B5-4B6F-8FDB-310B7F69E540@freebsd.org> <947D1E51-CCB0-4A1E-B671-708F70420DA1@freebsd.org> In-Reply-To: <947D1E51-CCB0-4A1E-B671-708F70420DA1@freebsd.org> --J3sia0MV05ZtZmHbnbTrGyi8A4wcOA5yg Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable W dniu 16.03.2021 o=C2=A015:35, tuexen@freebsd.org pisze: >> On 16. Mar 2021, at 15:18, Marek Zarychta wrote: >> >> W dniu 16.03.2021 o 12:50, tuexen@freebsd.org pisze: >>>> On 16. Mar 2021, at 11:55, Blake Hartshorn wrote: >>>> >>>> Hi Michael, >>>> >>>> I've attached tcpdumps for port 80 on both sides of a bad transfer, = using 2 VMs in the same datacenter, FreeBSD 13 serving and 12 as a client= =2E A friend of mine suggested I also run some tests with iperf3, so past= ing those results below. You'll see it going fast in one direction and cr= awling in the other on TCP. There's also some disparity on UDP.=20 >>> The problem is that the server provides TCP segments larger than the = MTU >>> to the NIC. These are dropped and needs to be retransmitted. That is = why >>> it takes so long. So I guess TSO is enabled on the NIC and not workin= g correctly. >>> >>> What is the output of ifconfig? Can you disable TSO? Does that work a= round >>> the problem? >>> >>> Best regards >>> Michael >>>> >>>> TCP: >>>> [ ID][Role] Interval Transfer Bitrate Retr Cw= nd >>>> [ 5][TX-C] 0.00-1.00 sec 27.7 MBytes 233 Mbits/sec 342 26= =2E7 KBytes =20 >>>> [ 7][RX-C] 0.00-1.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>>> [ 5][TX-C] 1.00-2.00 sec 15.8 MBytes 132 Mbits/sec 249 52= =2E0 KBytes =20 >>>> [ 7][RX-C] 1.00-2.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>>> [ 5][TX-C] 2.00-3.00 sec 13.7 MBytes 115 Mbits/sec 307 15= =2E4 KBytes =20 >>>> [ 7][RX-C] 2.00-3.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>>> [ 5][TX-C] 3.00-4.00 sec 14.5 MBytes 121 Mbits/sec 260 22= =2E4 KBytes =20 >>>> [ 7][RX-C] 3.00-4.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>>> [ 5][TX-C] 4.00-5.00 sec 14.3 MBytes 120 Mbits/sec 240 37= =2E9 KBytes =20 >>>> [ 7][RX-C] 4.00-5.00 sec 5.58 KBytes 45.7 Kbits/sec = =20 >>>> [ 5][TX-C] 5.00-6.00 sec 17.7 MBytes 149 Mbits/sec 363 15= =2E4 KBytes =20 >>>> [ 7][RX-C] 5.00-6.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>>> [ 5][TX-C] 6.00-7.00 sec 14.8 MBytes 124 Mbits/sec 287 8.= 38 KBytes =20 >>>> [ 7][RX-C] 6.00-7.00 sec 5.58 KBytes 45.7 Kbits/sec = =20 >>>> [ 5][TX-C] 7.00-8.00 sec 14.7 MBytes 123 Mbits/sec 293 28= =2E1 KBytes =20 >>>> [ 7][RX-C] 7.00-8.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>>> [ 5][TX-C] 8.00-9.00 sec 11.9 MBytes 100 Mbits/sec 325 18= =2E3 KBytes =20 >>>> [ 7][RX-C] 8.00-9.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>>> [ 5][TX-C] 9.00-10.00 sec 14.3 MBytes 120 Mbits/sec 315 39= =2E3 KBytes =20 >>>> [ 7][RX-C] 9.00-10.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >>>> - - - - - - - - - - - - - - - - - - - - - - - - - >>>> [ ID][Role] Interval Transfer Bitrate Retr >>>> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec 2981 = sender >>>> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec = receiver >>>> [ 7][RX-C] 0.00-10.00 sec 77.0 KBytes 63.1 Kbits/sec 65 = sender >>>> [ 7][RX-C] 0.00-10.00 sec 44.6 KBytes 36.6 Kbits/sec = receiver >>>> --------------------------------------------------------------------= --------------------------------------- >>>> >>>> >>>> UDP: >>>> [ ID][Role] Interval Transfer Bitrate Jitter = Lost/Total Datagrams >>>> [ 5][TX-C] 0.00-1.00 sec 81.6 MBytes 685 Mbits/sec = 67798 =20 >>>> [ 7][RX-C] 0.00-1.00 sec 8.80 MBytes 73.8 Mbits/sec 0.255 ms= 54070/60475 (89%) =20 >>>> [ 5][TX-C] 1.00-2.00 sec 72.7 MBytes 610 Mbits/sec = 64802 =20 >>>> [ 7][RX-C] 1.00-2.00 sec 8.52 MBytes 71.5 Mbits/sec 0.154 ms= 68912/75116 (92%) =20 >>>> [ 5][TX-C] 2.00-3.00 sec 73.7 MBytes 618 Mbits/sec = 64158 =20 >>>> [ 7][RX-C] 2.00-3.00 sec 8.52 MBytes 71.5 Mbits/sec 0.276 ms= 67738/73945 (92%) =20 >>>> [ 5][TX-C] 3.00-4.00 sec 76.6 MBytes 643 Mbits/sec = 63521 =20 >>>> [ 7][RX-C] 3.00-4.00 sec 8.55 MBytes 71.8 Mbits/sec 0.160 ms= 68647/74874 (92%) =20 >>>> [ 5][TX-C] 4.00-5.00 sec 76.1 MBytes 638 Mbits/sec = 64614 =20 >>>> [ 7][RX-C] 4.00-5.00 sec 8.55 MBytes 71.7 Mbits/sec 0.461 ms= 67542/73767 (92%) =20 >>>> [ 5][TX-C] 5.00-6.00 sec 75.9 MBytes 637 Mbits/sec = 64834 =20 >>>> [ 7][RX-C] 5.00-6.00 sec 8.57 MBytes 71.9 Mbits/sec 0.297 ms= 71565/77806 (92%) =20 >>>> [ 5][TX-C] 6.00-7.00 sec 73.0 MBytes 613 Mbits/sec = 63639 =20 >>>> [ 7][RX-C] 6.00-7.00 sec 8.40 MBytes 70.5 Mbits/sec 0.199 ms= 69545/75663 (92%) =20 >>>> [ 5][TX-C] 7.00-8.00 sec 74.6 MBytes 626 Mbits/sec = 65030 =20 >>>> [ 7][RX-C] 7.00-8.00 sec 8.78 MBytes 73.6 Mbits/sec 0.254 ms= 67173/73566 (91%) =20 >>>> [ 5][TX-C] 8.00-9.00 sec 75.0 MBytes 629 Mbits/sec = 64848 =20 >>>> [ 7][RX-C] 8.00-9.00 sec 8.77 MBytes 73.5 Mbits/sec 0.298 ms= 70932/77315 (92%) =20 >>>> [ 5][TX-C] 9.00-10.00 sec 74.5 MBytes 625 Mbits/sec = 64487 =20 >>>> [ 7][RX-C] 9.00-10.00 sec 8.71 MBytes 73.1 Mbits/sec 0.185 ms= 68268/74612 (91%) =20 >>>> - - - - - - - - - - - - - - - - - - - - - - - - - >>>> [ ID][Role] Interval Transfer Bitrate Jitter = Lost/Total Datagrams >>>> [ 5][TX-C] 0.00-10.00 sec 754 MBytes 632 Mbits/sec 0.000 ms= 0/647731 (0%) sender >>>> [ 5][TX-C] 0.00-10.12 sec 105 MBytes 87.2 Mbits/sec 0.245 ms= 571090/647649 (88%) receiver >>>> [ 7][RX-C] 0.00-10.00 sec 1009 MBytes 846 Mbits/sec 0.000 ms= 0/761013 (0%) sender >>>> [ 7][RX-C] 0.00-10.12 sec 86.2 MBytes 71.4 Mbits/sec 0.185 ms= 674392/737139 (91%) receiver >>>> >>>> >> >> Taking a look at this iperf output I recalled that 80% of my setups >> suffered from similar issue after transitioning from 1{1,2}-STABLE to >> 13-STABLE about a mounth ago. I have asked on IRC but nobody confirmed= >> similar problems so I have reduced MTU to 8900 from the original 9000 = on >> some vlan(4) interfaces to solve the issue. I am using mostly vlan(4)s= >> over LACP lagg(4)s created on NICs. So far (for FreeBSD 11 and 12) >> setting MTU 9000 on physical NIC was sufficient to make it work, now I= >> have set MTU 9000 on NICs and reduced MTU 8900 on vlan(4)s. > TCP announces an MSS of 1440, which corresponds to an MTU of 1500 byte.= > So I guess your problem is different. > Is your physical MTU 9000 bytes? >=20 > Best regards > Michael Yes, MTU for some VLANs in this network is 9000 bytes and I performed only software upgrades, hardware was untouched. I haven't had similar issues while running 11.4-STABLE or 12.2-STABLE. The aforementioned 8900 is not a critical value, but the reduction from an original 9000 was mandatory. With kind regards, Marek >>>> >>>> >>>> On Tuesday, March 16, 2021 4:16:15 AM EDT tuexen@freebsd.org wrote: >>>>>> On 15. Mar 2021, at 12:56, Blake Hartshorn wrote: >>>>>> >>>>>> The short version, when I use FreeBSD 13, delivering data can take= 5 minutes for 1MB over SSH or HTTP when using IPv6. This problem does no= t happen with IPv4. I installed FreeBSD 12 and Linux on that same device,= neither had the problem. >>>>>> >>>>>> Did some troubleshooting with Linode, have ultimately ruled the ne= twork itself out at this point. When the server is on FreeBSD 13, it can = download quickly over IPv6, but not deliver. Started investigating after = noticing my SSH session was lagging when cat'ing large files or running b= uilds. This problem even occurs between VMs in the same datacenter. I gen= erated a 1MB file of base64 garbage served by nginx for testing. IPv6 is = being configured by SLAAC and on both 12 and 13 installs was setup by the= installer. Linode uses Linux/KVM hosts for their virtual machines so it'= s running on that virtual adapter. >>>>>> >>>>>> I asked on the forums, another user recommended going to the maili= ng lists instead. Does anyone know if config settings need to be differen= t on 13? Did I maybe just find a real issue? I can provide any requested = details. Thanks! >>>>> Could you prove a .pcap tracefile, one from the sender, one from th= e recevier, of >>>>> a TCP/IPv6 connection, which doesn't work as expected. For example,= use your 1MB >>>>> base64 garbage transfer. >>>>> >>>>> Best regardes >>>>> Michael >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> freebsd-net@freebsd.org mailing list >>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.= org" >>>>> >>>>> >>>> >>> >>> _______________________________________________ >>> freebsd-net@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " >>> >> >> >> --=20 >> Marek Zarychta >> >=20 >=20 --=20 Marek Zarychta --J3sia0MV05ZtZmHbnbTrGyi8A4wcOA5yg-- --qMLh9QWrbst0hZe7ZMrmKN8QG9pvAfcx5 Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEMOqvKm6wKvS1/ZeCdZ/s//1SjSwFAmBQxggFAwAAAAAACgkQdZ/s//1SjSw2 CAf+LfNwxhHnBBzKAxhvaR4FZwYZYvfFlVNjX25sEIl1wILs+0KqXeKWjpdAnKGKsO5znLhasocV s2BdCwmf5LI8yAtN7VzAZuZCkkXU3yBbKVldYYaAFvMfXQaL/DQZ/nYABn3hMtLNZsNQA+lcQmsz 4ah/uBG4l6cCffnTLmXKCTK2SmTCcdJBFggppuP/ez1gVezofZFRP1J6B/38kSI07OsLAgoDv3hn gc947WYsuyDTMO1blzwkPZnryJfuwVjrZpzy+iwi35UBO9c3ozXTLBCYlyykKBsSDjOsll8SWWKV 7/6MpZAHDdbhmp6OzoWOJHOYElsQ/gUUsq01EnsYYQ== =BoOn -----END PGP SIGNATURE----- --qMLh9QWrbst0hZe7ZMrmKN8QG9pvAfcx5--