Date: Tue, 16 Mar 2021 15:51:52 +0100 From: Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> To: tuexen@freebsd.org Cc: Blake Hartshorn <contact@blakehartshorn.com>, freebsd-net@freebsd.org Subject: Re: Severe IPv6 TCP transfer issues on 13.0-RC1 and RC2 Message-ID: <bdfdf126-db3d-cf45-dfe7-4e251ffeb8f9@plan-b.pwste.edu.pl> In-Reply-To: <947D1E51-CCB0-4A1E-B671-708F70420DA1@freebsd.org> References: <5753280.1HxCrU2fYu@thinkbook> <E1346D9D-F1E0-4ABC-89D6-33BC77705771@freebsd.org> <10847992.4GmZMkJedg@thinkbook> <2808496E-C2B5-4B6F-8FDB-310B7F69E540@freebsd.org> <d71bd76f-d01a-0af7-9b02-c3a473de5ef9@plan-b.pwste.edu.pl> <947D1E51-CCB0-4A1E-B671-708F70420DA1@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--qMLh9QWrbst0hZe7ZMrmKN8QG9pvAfcx5
Content-Type: multipart/mixed; boundary="J3sia0MV05ZtZmHbnbTrGyi8A4wcOA5yg";
protected-headers="v1"
From: Marek Zarychta <zarychtam@plan-b.pwste.edu.pl>
To: tuexen@freebsd.org
Cc: Blake Hartshorn <contact@blakehartshorn.com>, freebsd-net@freebsd.org
Message-ID: <bdfdf126-db3d-cf45-dfe7-4e251ffeb8f9@plan-b.pwste.edu.pl>
Subject: Re: Severe IPv6 TCP transfer issues on 13.0-RC1 and RC2
References: <5753280.1HxCrU2fYu@thinkbook>
<E1346D9D-F1E0-4ABC-89D6-33BC77705771@freebsd.org>
<10847992.4GmZMkJedg@thinkbook>
<2808496E-C2B5-4B6F-8FDB-310B7F69E540@freebsd.org>
<d71bd76f-d01a-0af7-9b02-c3a473de5ef9@plan-b.pwste.edu.pl>
<947D1E51-CCB0-4A1E-B671-708F70420DA1@freebsd.org>
In-Reply-To: <947D1E51-CCB0-4A1E-B671-708F70420DA1@freebsd.org>
--J3sia0MV05ZtZmHbnbTrGyi8A4wcOA5yg
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
W dniu 16.03.2021 o=C2=A015:35, tuexen@freebsd.org pisze:
>> On 16. Mar 2021, at 15:18, Marek Zarychta <zarychtam@plan-b.pwste.edu.=
pl> wrote:
>>
>> W dniu 16.03.2021 o 12:50, tuexen@freebsd.org pisze:
>>>> On 16. Mar 2021, at 11:55, Blake Hartshorn <contact@blakehartshorn.c=
om> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> I've attached tcpdumps for port 80 on both sides of a bad transfer, =
using 2 VMs in the same datacenter, FreeBSD 13 serving and 12 as a client=
=2E A friend of mine suggested I also run some tests with iperf3, so past=
ing those results below. You'll see it going fast in one direction and cr=
awling in the other on TCP. There's also some disparity on UDP.=20
>>> The problem is that the server provides TCP segments larger than the =
MTU
>>> to the NIC. These are dropped and needs to be retransmitted. That is =
why
>>> it takes so long. So I guess TSO is enabled on the NIC and not workin=
g correctly.
>>>
>>> What is the output of ifconfig? Can you disable TSO? Does that work a=
round
>>> the problem?
>>>
>>> Best regards
>>> Michael
>>>>
>>>> TCP:
>>>> [ ID][Role] Interval Transfer Bitrate Retr Cw=
nd
>>>> [ 5][TX-C] 0.00-1.00 sec 27.7 MBytes 233 Mbits/sec 342 26=
=2E7 KBytes =20
>>>> [ 7][RX-C] 0.00-1.00 sec 4.18 KBytes 34.3 Kbits/sec =
=20
>>>> [ 5][TX-C] 1.00-2.00 sec 15.8 MBytes 132 Mbits/sec 249 52=
=2E0 KBytes =20
>>>> [ 7][RX-C] 1.00-2.00 sec 4.18 KBytes 34.3 Kbits/sec =
=20
>>>> [ 5][TX-C] 2.00-3.00 sec 13.7 MBytes 115 Mbits/sec 307 15=
=2E4 KBytes =20
>>>> [ 7][RX-C] 2.00-3.00 sec 4.18 KBytes 34.3 Kbits/sec =
=20
>>>> [ 5][TX-C] 3.00-4.00 sec 14.5 MBytes 121 Mbits/sec 260 22=
=2E4 KBytes =20
>>>> [ 7][RX-C] 3.00-4.00 sec 4.18 KBytes 34.3 Kbits/sec =
=20
>>>> [ 5][TX-C] 4.00-5.00 sec 14.3 MBytes 120 Mbits/sec 240 37=
=2E9 KBytes =20
>>>> [ 7][RX-C] 4.00-5.00 sec 5.58 KBytes 45.7 Kbits/sec =
=20
>>>> [ 5][TX-C] 5.00-6.00 sec 17.7 MBytes 149 Mbits/sec 363 15=
=2E4 KBytes =20
>>>> [ 7][RX-C] 5.00-6.00 sec 4.18 KBytes 34.3 Kbits/sec =
=20
>>>> [ 5][TX-C] 6.00-7.00 sec 14.8 MBytes 124 Mbits/sec 287 8.=
38 KBytes =20
>>>> [ 7][RX-C] 6.00-7.00 sec 5.58 KBytes 45.7 Kbits/sec =
=20
>>>> [ 5][TX-C] 7.00-8.00 sec 14.7 MBytes 123 Mbits/sec 293 28=
=2E1 KBytes =20
>>>> [ 7][RX-C] 7.00-8.00 sec 4.18 KBytes 34.3 Kbits/sec =
=20
>>>> [ 5][TX-C] 8.00-9.00 sec 11.9 MBytes 100 Mbits/sec 325 18=
=2E3 KBytes =20
>>>> [ 7][RX-C] 8.00-9.00 sec 4.18 KBytes 34.3 Kbits/sec =
=20
>>>> [ 5][TX-C] 9.00-10.00 sec 14.3 MBytes 120 Mbits/sec 315 39=
=2E3 KBytes =20
>>>> [ 7][RX-C] 9.00-10.00 sec 4.18 KBytes 34.3 Kbits/sec =
=20
>>>> - - - - - - - - - - - - - - - - - - - - - - - - -
>>>> [ ID][Role] Interval Transfer Bitrate Retr
>>>> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec 2981 =
sender
>>>> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec =
receiver
>>>> [ 7][RX-C] 0.00-10.00 sec 77.0 KBytes 63.1 Kbits/sec 65 =
sender
>>>> [ 7][RX-C] 0.00-10.00 sec 44.6 KBytes 36.6 Kbits/sec =
receiver
>>>> --------------------------------------------------------------------=
---------------------------------------
>>>>
>>>>
>>>> UDP:
>>>> [ ID][Role] Interval Transfer Bitrate Jitter =
Lost/Total Datagrams
>>>> [ 5][TX-C] 0.00-1.00 sec 81.6 MBytes 685 Mbits/sec =
67798 =20
>>>> [ 7][RX-C] 0.00-1.00 sec 8.80 MBytes 73.8 Mbits/sec 0.255 ms=
54070/60475 (89%) =20
>>>> [ 5][TX-C] 1.00-2.00 sec 72.7 MBytes 610 Mbits/sec =
64802 =20
>>>> [ 7][RX-C] 1.00-2.00 sec 8.52 MBytes 71.5 Mbits/sec 0.154 ms=
68912/75116 (92%) =20
>>>> [ 5][TX-C] 2.00-3.00 sec 73.7 MBytes 618 Mbits/sec =
64158 =20
>>>> [ 7][RX-C] 2.00-3.00 sec 8.52 MBytes 71.5 Mbits/sec 0.276 ms=
67738/73945 (92%) =20
>>>> [ 5][TX-C] 3.00-4.00 sec 76.6 MBytes 643 Mbits/sec =
63521 =20
>>>> [ 7][RX-C] 3.00-4.00 sec 8.55 MBytes 71.8 Mbits/sec 0.160 ms=
68647/74874 (92%) =20
>>>> [ 5][TX-C] 4.00-5.00 sec 76.1 MBytes 638 Mbits/sec =
64614 =20
>>>> [ 7][RX-C] 4.00-5.00 sec 8.55 MBytes 71.7 Mbits/sec 0.461 ms=
67542/73767 (92%) =20
>>>> [ 5][TX-C] 5.00-6.00 sec 75.9 MBytes 637 Mbits/sec =
64834 =20
>>>> [ 7][RX-C] 5.00-6.00 sec 8.57 MBytes 71.9 Mbits/sec 0.297 ms=
71565/77806 (92%) =20
>>>> [ 5][TX-C] 6.00-7.00 sec 73.0 MBytes 613 Mbits/sec =
63639 =20
>>>> [ 7][RX-C] 6.00-7.00 sec 8.40 MBytes 70.5 Mbits/sec 0.199 ms=
69545/75663 (92%) =20
>>>> [ 5][TX-C] 7.00-8.00 sec 74.6 MBytes 626 Mbits/sec =
65030 =20
>>>> [ 7][RX-C] 7.00-8.00 sec 8.78 MBytes 73.6 Mbits/sec 0.254 ms=
67173/73566 (91%) =20
>>>> [ 5][TX-C] 8.00-9.00 sec 75.0 MBytes 629 Mbits/sec =
64848 =20
>>>> [ 7][RX-C] 8.00-9.00 sec 8.77 MBytes 73.5 Mbits/sec 0.298 ms=
70932/77315 (92%) =20
>>>> [ 5][TX-C] 9.00-10.00 sec 74.5 MBytes 625 Mbits/sec =
64487 =20
>>>> [ 7][RX-C] 9.00-10.00 sec 8.71 MBytes 73.1 Mbits/sec 0.185 ms=
68268/74612 (91%) =20
>>>> - - - - - - - - - - - - - - - - - - - - - - - - -
>>>> [ ID][Role] Interval Transfer Bitrate Jitter =
Lost/Total Datagrams
>>>> [ 5][TX-C] 0.00-10.00 sec 754 MBytes 632 Mbits/sec 0.000 ms=
0/647731 (0%) sender
>>>> [ 5][TX-C] 0.00-10.12 sec 105 MBytes 87.2 Mbits/sec 0.245 ms=
571090/647649 (88%) receiver
>>>> [ 7][RX-C] 0.00-10.00 sec 1009 MBytes 846 Mbits/sec 0.000 ms=
0/761013 (0%) sender
>>>> [ 7][RX-C] 0.00-10.12 sec 86.2 MBytes 71.4 Mbits/sec 0.185 ms=
674392/737139 (91%) receiver
>>>>
>>>>
>>
>> Taking a look at this iperf output I recalled that 80% of my setups
>> suffered from similar issue after transitioning from 1{1,2}-STABLE to
>> 13-STABLE about a mounth ago. I have asked on IRC but nobody confirmed=
>> similar problems so I have reduced MTU to 8900 from the original 9000 =
on
>> some vlan(4) interfaces to solve the issue. I am using mostly vlan(4)s=
>> over LACP lagg(4)s created on NICs. So far (for FreeBSD 11 and 12)
>> setting MTU 9000 on physical NIC was sufficient to make it work, now I=
>> have set MTU 9000 on NICs and reduced MTU 8900 on vlan(4)s.
> TCP announces an MSS of 1440, which corresponds to an MTU of 1500 byte.=
> So I guess your problem is different.
> Is your physical MTU 9000 bytes?
>=20
> Best regards
> Michael
Yes, MTU for some VLANs in this network is 9000 bytes and I performed
only software upgrades, hardware was untouched. I haven't had similar
issues while running 11.4-STABLE or 12.2-STABLE. The aforementioned 8900
is not a critical value, but the reduction from an original 9000 was
mandatory.
With kind regards,
Marek
>>>>
>>>>
>>>> On Tuesday, March 16, 2021 4:16:15 AM EDT tuexen@freebsd.org wrote:
>>>>>> On 15. Mar 2021, at 12:56, Blake Hartshorn <contact@blakehartshorn=
=2Ecom> wrote:
>>>>>>
>>>>>> The short version, when I use FreeBSD 13, delivering data can take=
5 minutes for 1MB over SSH or HTTP when using IPv6. This problem does no=
t happen with IPv4. I installed FreeBSD 12 and Linux on that same device,=
neither had the problem.
>>>>>>
>>>>>> Did some troubleshooting with Linode, have ultimately ruled the ne=
twork itself out at this point. When the server is on FreeBSD 13, it can =
download quickly over IPv6, but not deliver. Started investigating after =
noticing my SSH session was lagging when cat'ing large files or running b=
uilds. This problem even occurs between VMs in the same datacenter. I gen=
erated a 1MB file of base64 garbage served by nginx for testing. IPv6 is =
being configured by SLAAC and on both 12 and 13 installs was setup by the=
installer. Linode uses Linux/KVM hosts for their virtual machines so it'=
s running on that virtual adapter.
>>>>>>
>>>>>> I asked on the forums, another user recommended going to the maili=
ng lists instead. Does anyone know if config settings need to be differen=
t on 13? Did I maybe just find a real issue? I can provide any requested =
details. Thanks!
>>>>> Could you prove a .pcap tracefile, one from the sender, one from th=
e recevier, of
>>>>> a TCP/IPv6 connection, which doesn't work as expected. For example,=
use your 1MB
>>>>> base64 garbage transfer.
>>>>>
>>>>> Best regardes
>>>>> Michael
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> freebsd-net@freebsd.org mailing list
>>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.=
org"
>>>>>
>>>>>
>>>> <freebsd12-client.pcap.gz><freebsd13-server.pcap.gz>
>>>
>>> _______________________________________________
>>> freebsd-net@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org=
"
>>>
>>
>>
>> --=20
>> Marek Zarychta
>>
>=20
>=20
--=20
Marek Zarychta
--J3sia0MV05ZtZmHbnbTrGyi8A4wcOA5yg--
--qMLh9QWrbst0hZe7ZMrmKN8QG9pvAfcx5
Content-Type: application/pgp-signature; name="OpenPGP_signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="OpenPGP_signature"
-----BEGIN PGP SIGNATURE-----
wsB5BAABCAAjFiEEMOqvKm6wKvS1/ZeCdZ/s//1SjSwFAmBQxggFAwAAAAAACgkQdZ/s//1SjSw2
CAf+LfNwxhHnBBzKAxhvaR4FZwYZYvfFlVNjX25sEIl1wILs+0KqXeKWjpdAnKGKsO5znLhasocV
s2BdCwmf5LI8yAtN7VzAZuZCkkXU3yBbKVldYYaAFvMfXQaL/DQZ/nYABn3hMtLNZsNQA+lcQmsz
4ah/uBG4l6cCffnTLmXKCTK2SmTCcdJBFggppuP/ez1gVezofZFRP1J6B/38kSI07OsLAgoDv3hn
gc947WYsuyDTMO1blzwkPZnryJfuwVjrZpzy+iwi35UBO9c3ozXTLBCYlyykKBsSDjOsll8SWWKV
7/6MpZAHDdbhmp6OzoWOJHOYElsQ/gUUsq01EnsYYQ==
=BoOn
-----END PGP SIGNATURE-----
--qMLh9QWrbst0hZe7ZMrmKN8QG9pvAfcx5--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bdfdf126-db3d-cf45-dfe7-4e251ffeb8f9>