From owner-freebsd-net@freebsd.org Tue Mar 16 14:18:48 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4561C5AF430 for ; Tue, 16 Mar 2021 14:18:48 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from plan-b.pwste.edu.pl (plan-b.pwste.edu.pl [IPv6:2001:678:618::40]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "plan-b.pwste.edu.pl", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0FlB5Bjwz4cD6; Tue, 16 Mar 2021 14:18:46 +0000 (UTC) (envelope-from zarychtam@plan-b.pwste.edu.pl) Received: from fomalhaut.potoki.eu ([IPv6:2001:678:618:c2:2:0:0:1]) (authenticated bits=0) by plan-b.pwste.edu.pl (8.16.1/8.16.1) with ESMTPSA id 12GEIYt1009239 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Tue, 16 Mar 2021 15:18:35 +0100 (CET) (envelope-from zarychtam@plan-b.pwste.edu.pl) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=plan-b.pwste.edu.pl; s=plan-b-mailer; t=1615904315; bh=LEB2a8YZukv1z0L/mUL87rLCaen1w/kJqpqgqx4ljog=; h=To:Cc:References:From:Subject:Date:In-Reply-To; b=HActbdLgG3sg++Gpd9sIeSFSx9hJaxZ05PN9i7LuepMh0vbGZCQYgmkQpsbaqrjDO Fo4q1Ocy3Nfsc3cgD2gsrULg61zEz3ewwsSC3z9xvXbZKp6IWL4QKf80yzEhLQ9REM okjfhBHzONxdPQDD7DcG5BFA5A60GkR8KbxjAzdxgT3BykknUgINcBHRKgZOzEhzEw 7uHgjvWhV4abCxOPTi5bDs+CzRtPUfX/OwXtwpnRKQofdAxJK1TXlrcz/wdF5db059 pl2Ne9HEpKETcZBxW0XO2lLrMu+ayJK7D1EgXNlwWPpMrjS/Fol0ldn1JV4yEXLjue a/Jp0A+CO3qXA== X-Authentication-Warning: plan-b.pwste.edu.pl: Host [IPv6:2001:678:618:c2:2:0:0:1] claimed to be fomalhaut.potoki.eu To: tuexen@freebsd.org, Blake Hartshorn Cc: freebsd-net@freebsd.org References: <5753280.1HxCrU2fYu@thinkbook> <10847992.4GmZMkJedg@thinkbook> <2808496E-C2B5-4B6F-8FDB-310B7F69E540@freebsd.org> From: Marek Zarychta Subject: Re: Severe IPv6 TCP transfer issues on 13.0-RC1 and RC2 Message-ID: Date: Tue, 16 Mar 2021 15:18:33 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 In-Reply-To: <2808496E-C2B5-4B6F-8FDB-310B7F69E540@freebsd.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="aGGnVIZwMR5UCA0W97ECtZo5jud9vtNg4" X-Rspamd-Queue-Id: 4F0FlB5Bjwz4cD6 X-Spamd-Bar: ------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=plan-b.pwste.edu.pl header.s=plan-b-mailer header.b=HActbdLg; dmarc=pass (policy=none) header.from=plan-b.pwste.edu.pl; spf=none (mx1.freebsd.org: domain of zarychtam@plan-b.pwste.edu.pl has no SPF policy when checking 2001:678:618::40) smtp.mailfrom=zarychtam@plan-b.pwste.edu.pl X-Spamd-Result: default: False [-7.90 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; HAS_ATTACHMENT(0.00)[]; HAS_XAW(0.00)[]; DKIM_TRACE(0.00)[plan-b.pwste.edu.pl:+]; DMARC_POLICY_ALLOW(-0.50)[plan-b.pwste.edu.pl,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2001:678:618::40:from]; ASN(0.00)[asn:206006, ipnet:2001:678:618::/48, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[plan-b.pwste.edu.pl:s=plan-b-mailer]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DWL_DNSWL_MED(-2.00)[pwste.edu.pl:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; SPAMHAUS_ZRD(0.00)[2001:678:618::40:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_NA(0.00)[no SPF record]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 14:18:48 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --aGGnVIZwMR5UCA0W97ECtZo5jud9vtNg4 Content-Type: multipart/mixed; boundary="J4fBDqdWX9uGzCu8zZ3iITrbO2Uz1hLcV"; protected-headers="v1" From: Marek Zarychta To: tuexen@freebsd.org, Blake Hartshorn Cc: freebsd-net@freebsd.org Message-ID: Subject: Re: Severe IPv6 TCP transfer issues on 13.0-RC1 and RC2 References: <5753280.1HxCrU2fYu@thinkbook> <10847992.4GmZMkJedg@thinkbook> <2808496E-C2B5-4B6F-8FDB-310B7F69E540@freebsd.org> In-Reply-To: <2808496E-C2B5-4B6F-8FDB-310B7F69E540@freebsd.org> --J4fBDqdWX9uGzCu8zZ3iITrbO2Uz1hLcV Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable W dniu 16.03.2021 o=C2=A012:50, tuexen@freebsd.org pisze: >> On 16. Mar 2021, at 11:55, Blake Hartshorn wrote: >> >> Hi Michael, >> >> I've attached tcpdumps for port 80 on both sides of a bad transfer, us= ing 2 VMs in the same datacenter, FreeBSD 13 serving and 12 as a client. = A friend of mine suggested I also run some tests with iperf3, so pasting = those results below. You'll see it going fast in one direction and crawli= ng in the other on TCP. There's also some disparity on UDP.=20 > The problem is that the server provides TCP segments larger than the MT= U > to the NIC. These are dropped and needs to be retransmitted. That is wh= y > it takes so long. So I guess TSO is enabled on the NIC and not working = correctly. >=20 > What is the output of ifconfig? Can you disable TSO? Does that work aro= und > the problem? >=20 > Best regards > Michael >> >> TCP: >> [ ID][Role] Interval Transfer Bitrate Retr Cwnd= >> [ 5][TX-C] 0.00-1.00 sec 27.7 MBytes 233 Mbits/sec 342 26.7= KBytes =20 >> [ 7][RX-C] 0.00-1.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >> [ 5][TX-C] 1.00-2.00 sec 15.8 MBytes 132 Mbits/sec 249 52.0= KBytes =20 >> [ 7][RX-C] 1.00-2.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >> [ 5][TX-C] 2.00-3.00 sec 13.7 MBytes 115 Mbits/sec 307 15.4= KBytes =20 >> [ 7][RX-C] 2.00-3.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >> [ 5][TX-C] 3.00-4.00 sec 14.5 MBytes 121 Mbits/sec 260 22.4= KBytes =20 >> [ 7][RX-C] 3.00-4.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >> [ 5][TX-C] 4.00-5.00 sec 14.3 MBytes 120 Mbits/sec 240 37.9= KBytes =20 >> [ 7][RX-C] 4.00-5.00 sec 5.58 KBytes 45.7 Kbits/sec = =20 >> [ 5][TX-C] 5.00-6.00 sec 17.7 MBytes 149 Mbits/sec 363 15.4= KBytes =20 >> [ 7][RX-C] 5.00-6.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >> [ 5][TX-C] 6.00-7.00 sec 14.8 MBytes 124 Mbits/sec 287 8.38= KBytes =20 >> [ 7][RX-C] 6.00-7.00 sec 5.58 KBytes 45.7 Kbits/sec = =20 >> [ 5][TX-C] 7.00-8.00 sec 14.7 MBytes 123 Mbits/sec 293 28.1= KBytes =20 >> [ 7][RX-C] 7.00-8.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >> [ 5][TX-C] 8.00-9.00 sec 11.9 MBytes 100 Mbits/sec 325 18.3= KBytes =20 >> [ 7][RX-C] 8.00-9.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >> [ 5][TX-C] 9.00-10.00 sec 14.3 MBytes 120 Mbits/sec 315 39.3= KBytes =20 >> [ 7][RX-C] 9.00-10.00 sec 4.18 KBytes 34.3 Kbits/sec = =20 >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID][Role] Interval Transfer Bitrate Retr >> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec 2981 = sender >> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec = receiver >> [ 7][RX-C] 0.00-10.00 sec 77.0 KBytes 63.1 Kbits/sec 65 = sender >> [ 7][RX-C] 0.00-10.00 sec 44.6 KBytes 36.6 Kbits/sec = receiver >> ----------------------------------------------------------------------= ------------------------------------- >> >> >> UDP: >> [ ID][Role] Interval Transfer Bitrate Jitter = Lost/Total Datagrams >> [ 5][TX-C] 0.00-1.00 sec 81.6 MBytes 685 Mbits/sec = 67798 =20 >> [ 7][RX-C] 0.00-1.00 sec 8.80 MBytes 73.8 Mbits/sec 0.255 ms = 54070/60475 (89%) =20 >> [ 5][TX-C] 1.00-2.00 sec 72.7 MBytes 610 Mbits/sec = 64802 =20 >> [ 7][RX-C] 1.00-2.00 sec 8.52 MBytes 71.5 Mbits/sec 0.154 ms = 68912/75116 (92%) =20 >> [ 5][TX-C] 2.00-3.00 sec 73.7 MBytes 618 Mbits/sec = 64158 =20 >> [ 7][RX-C] 2.00-3.00 sec 8.52 MBytes 71.5 Mbits/sec 0.276 ms = 67738/73945 (92%) =20 >> [ 5][TX-C] 3.00-4.00 sec 76.6 MBytes 643 Mbits/sec = 63521 =20 >> [ 7][RX-C] 3.00-4.00 sec 8.55 MBytes 71.8 Mbits/sec 0.160 ms = 68647/74874 (92%) =20 >> [ 5][TX-C] 4.00-5.00 sec 76.1 MBytes 638 Mbits/sec = 64614 =20 >> [ 7][RX-C] 4.00-5.00 sec 8.55 MBytes 71.7 Mbits/sec 0.461 ms = 67542/73767 (92%) =20 >> [ 5][TX-C] 5.00-6.00 sec 75.9 MBytes 637 Mbits/sec = 64834 =20 >> [ 7][RX-C] 5.00-6.00 sec 8.57 MBytes 71.9 Mbits/sec 0.297 ms = 71565/77806 (92%) =20 >> [ 5][TX-C] 6.00-7.00 sec 73.0 MBytes 613 Mbits/sec = 63639 =20 >> [ 7][RX-C] 6.00-7.00 sec 8.40 MBytes 70.5 Mbits/sec 0.199 ms = 69545/75663 (92%) =20 >> [ 5][TX-C] 7.00-8.00 sec 74.6 MBytes 626 Mbits/sec = 65030 =20 >> [ 7][RX-C] 7.00-8.00 sec 8.78 MBytes 73.6 Mbits/sec 0.254 ms = 67173/73566 (91%) =20 >> [ 5][TX-C] 8.00-9.00 sec 75.0 MBytes 629 Mbits/sec = 64848 =20 >> [ 7][RX-C] 8.00-9.00 sec 8.77 MBytes 73.5 Mbits/sec 0.298 ms = 70932/77315 (92%) =20 >> [ 5][TX-C] 9.00-10.00 sec 74.5 MBytes 625 Mbits/sec = 64487 =20 >> [ 7][RX-C] 9.00-10.00 sec 8.71 MBytes 73.1 Mbits/sec 0.185 ms = 68268/74612 (91%) =20 >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID][Role] Interval Transfer Bitrate Jitter = Lost/Total Datagrams >> [ 5][TX-C] 0.00-10.00 sec 754 MBytes 632 Mbits/sec 0.000 ms = 0/647731 (0%) sender >> [ 5][TX-C] 0.00-10.12 sec 105 MBytes 87.2 Mbits/sec 0.245 ms = 571090/647649 (88%) receiver >> [ 7][RX-C] 0.00-10.00 sec 1009 MBytes 846 Mbits/sec 0.000 ms = 0/761013 (0%) sender >> [ 7][RX-C] 0.00-10.12 sec 86.2 MBytes 71.4 Mbits/sec 0.185 ms = 674392/737139 (91%) receiver >> >> Taking a look at this iperf output I recalled that 80% of my setups suffered from similar issue after transitioning from 1{1,2}-STABLE to 13-STABLE about a mounth ago. I have asked on IRC but nobody confirmed similar problems so I have reduced MTU to 8900 from the original 9000 on some vlan(4) interfaces to solve the issue. I am using mostly vlan(4)s over LACP lagg(4)s created on NICs. So far (for FreeBSD 11 and 12) setting MTU 9000 on physical NIC was sufficient to make it work, now I have set MTU 9000 on NICs and reduced MTU 8900 on vlan(4)s. >> >> >> On Tuesday, March 16, 2021 4:16:15 AM EDT tuexen@freebsd.org wrote: >>>> On 15. Mar 2021, at 12:56, Blake Hartshorn wrote: >>>> >>>> The short version, when I use FreeBSD 13, delivering data can take 5= minutes for 1MB over SSH or HTTP when using IPv6. This problem does not = happen with IPv4. I installed FreeBSD 12 and Linux on that same device, n= either had the problem. >>>> >>>> Did some troubleshooting with Linode, have ultimately ruled the netw= ork itself out at this point. When the server is on FreeBSD 13, it can do= wnload quickly over IPv6, but not deliver. Started investigating after no= ticing my SSH session was lagging when cat'ing large files or running bui= lds. This problem even occurs between VMs in the same datacenter. I gener= ated a 1MB file of base64 garbage served by nginx for testing. IPv6 is be= ing configured by SLAAC and on both 12 and 13 installs was setup by the i= nstaller. Linode uses Linux/KVM hosts for their virtual machines so it's = running on that virtual adapter. >>>> >>>> I asked on the forums, another user recommended going to the mailing= lists instead. Does anyone know if config settings need to be different = on 13? Did I maybe just find a real issue? I can provide any requested de= tails. Thanks! >>> Could you prove a .pcap tracefile, one from the sender, one from the = recevier, of >>> a TCP/IPv6 connection, which doesn't work as expected. For example, u= se your 1MB >>> base64 garbage transfer. >>> >>> Best regardes >>> Michael >>>> >>>> >>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.or= g" >>> >>> >> >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >=20 --=20 Marek Zarychta --J4fBDqdWX9uGzCu8zZ3iITrbO2Uz1hLcV-- --aGGnVIZwMR5UCA0W97ECtZo5jud9vtNg4 Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEMOqvKm6wKvS1/ZeCdZ/s//1SjSwFAmBQvjkFAwAAAAAACgkQdZ/s//1SjSxn 7wf+MhimjJ6ZtsEjkshNAz4ZRAqL5a3tMQHrRIuda9elIgVmXkbSloUqqlDt3jBNkj31pGKtBB9E GuLemV19r82w0X9jxiypS0WYk1fdXZDDrrXHcDI5VGfVUabGbwJ1z884gzpmXw6GZiI543YfvByO HJ5tjpKbuWHWreNlymiX4DjG/noFbB5GrvXODJY2chqB/RfWh93yfrjBsqQyUJHv/57g8paNjH4j WesZnAQPp8PF+FolAxMB/wRAJ6h2p2CGfuGhP0galVGvXJevsNw9GmcGwYosiizzlPmq6VZiHX8J LZDo/vAOTpUrs88JZpjAF4rjZLj1Q2kFXo64HqOGVA== =gutX -----END PGP SIGNATURE----- --aGGnVIZwMR5UCA0W97ECtZo5jud9vtNg4--