From owner-freebsd-net@freebsd.org Tue Mar 16 15:07:57 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1BFF45B0F80 for ; Tue, 16 Mar 2021 15:07:57 +0000 (UTC) (envelope-from mtrm79@gmail.com) Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F0Gqx00lLz4frH; Tue, 16 Mar 2021 15:07:56 +0000 (UTC) (envelope-from mtrm79@gmail.com) Received: by mail-ot1-x329.google.com with SMTP id t23-20020a0568301e37b02901b65ab30024so7012080otr.4; Tue, 16 Mar 2021 08:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=jEQU127UXyNDW7/efPrEQJc9qq+vnNfXz+gMNmrJIIw=; b=F3XFABQpZDbfGtVpYnnJQvl3ZdLPFCLRLWyfkP1ePEP7zdgTpdw0kJ7gUfvzD/SmP1 CUj+Hv1ifDp1zvWAQ2Wp3vRKddQhFGCZQy/+CbBFwxohntOzmUfJicmByiTmsMm/VA8o tFIK0dLDQ6sq+Z9ULE7cL2qksYznnZ94+FeTEgLdvwDskhqo1M1hVied4tW+k5YDxLWQ 8piaBiCX7Ono9eBiCrMBgBh8blGNsQLLUkOegYcHaXx69LiqCwjUsTq+xt6I8zpGHNZ3 4my/qftA7YVPFpH3eGN+LsprJZGbJBJ4cK2vt0UK9D11i+V4CPvv2oEsGfgvKDzWsnxI biCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=jEQU127UXyNDW7/efPrEQJc9qq+vnNfXz+gMNmrJIIw=; b=rZcCWGurkPFTKn1hocJDMgWrEk0gVvbCDtbKGEPCdWjg47hF21LU4SROXzMuMiMl0d g2rRn08HvzkRA14EHcV7/0+UijFiWebR9t+BfearG1rfLn7YX31SwcKNJn2boEGn3D2L Geunzh3CE7u7l0K2Ov+uDGXfImOPVRpnGtGM4J5/vdWDarnYCSRShyJn2/WdcBT6H5Rp rK0JTjhnxet/2DLqbauy2viU9oJcZ68W+KH9Hc6FTH9K0BbM2GJ+A27Hno8DSTw6i3ge AHuHXPhYhvP73hjZob8zp1xo2I++2GbIAFTkeJwfLZzaJYz6kVm1/A49FidrnIU5bZ9A jHeg== X-Gm-Message-State: AOAM532KLsfQwOr9/0jP6GRigXjBccU3uGUGCPRwRE4Q8TQeswRDdfIB CgnRioiHzj8UMdN0kQQGxo4mEN8ji5EY1NPAXn0BlR39OMNulA== X-Google-Smtp-Source: ABdhPJyXqwLvyPV+G2r720hv/fMVx0DE1/SwE59YcFBkX/UrWCo/vF3dY+HUKcPjuASzs+bnCczM611a8bLuTdRi3Hk= X-Received: by 2002:a9d:4048:: with SMTP id o8mr4029158oti.152.1615907275218; Tue, 16 Mar 2021 08:07:55 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a4a:d5c8:0:0:0:0:0 with HTTP; Tue, 16 Mar 2021 08:07:54 -0700 (PDT) In-Reply-To: References: <5753280.1HxCrU2fYu@thinkbook> <10847992.4GmZMkJedg@thinkbook> <2808496E-C2B5-4B6F-8FDB-310B7F69E540@freebsd.org> <947D1E51-CCB0-4A1E-B671-708F70420DA1@freebsd.org> From: mtrm79 Date: Tue, 16 Mar 2021 08:07:54 -0700 Message-ID: Subject: Re: Severe IPv6 TCP transfer issues on 13.0-RC1 and RC2 To: Marek Zarychta Cc: "tuexen@freebsd.org" , Blake Hartshorn , "freebsd-net@freebsd.org" X-Rspamd-Queue-Id: 4F0Gqx00lLz4frH X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 15:07:57 -0000 On Tuesday, March 16, 2021, Marek Zarychta wrote: > W dniu 16.03.2021 o 15:35, tuexen@freebsd.org pisze: > >> On 16. Mar 2021, at 15:18, Marek Zarychta < > zarychtam@plan-b.pwste.edu.pl> wrote: > >> > >> W dniu 16.03.2021 o 12:50, tuexen@freebsd.org pisze: > >>>> On 16. Mar 2021, at 11:55, Blake Hartshorn < > contact@blakehartshorn.com> wrote: > >>>> > >>>> Hi Michael, > >>>> > >>>> I've attached tcpdumps for port 80 on both sides of a bad transfer, > using 2 VMs in the same datacenter, FreeBSD 13 serving and 12 as a client= . > A friend of mine suggested I also run some tests with iperf3, so pasting > those results below. You'll see it going fast in one direction and crawli= ng > in the other on TCP. There's also some disparity on UDP. > >>> The problem is that the server provides TCP segments larger than the > MTU > >>> to the NIC. These are dropped and needs to be retransmitted. That is > why > >>> it takes so long. So I guess TSO is enabled on the NIC and not workin= g > correctly. > >>> > >>> What is the output of ifconfig? Can you disable TSO? Does that work > around > >>> the problem? > >>> > >>> Best regards > >>> Michael > >>>> > >>>> TCP: > >>>> [ ID][Role] Interval Transfer Bitrate Retr Cw= nd > >>>> [ 5][TX-C] 0.00-1.00 sec 27.7 MBytes 233 Mbits/sec 342 > 26.7 KBytes > >>>> [ 7][RX-C] 0.00-1.00 sec 4.18 KBytes 34.3 Kbits/sec > > >>>> [ 5][TX-C] 1.00-2.00 sec 15.8 MBytes 132 Mbits/sec 249 > 52.0 KBytes > >>>> [ 7][RX-C] 1.00-2.00 sec 4.18 KBytes 34.3 Kbits/sec > > >>>> [ 5][TX-C] 2.00-3.00 sec 13.7 MBytes 115 Mbits/sec 307 > 15.4 KBytes > >>>> [ 7][RX-C] 2.00-3.00 sec 4.18 KBytes 34.3 Kbits/sec > > >>>> [ 5][TX-C] 3.00-4.00 sec 14.5 MBytes 121 Mbits/sec 260 > 22.4 KBytes > >>>> [ 7][RX-C] 3.00-4.00 sec 4.18 KBytes 34.3 Kbits/sec > > >>>> [ 5][TX-C] 4.00-5.00 sec 14.3 MBytes 120 Mbits/sec 240 > 37.9 KBytes > >>>> [ 7][RX-C] 4.00-5.00 sec 5.58 KBytes 45.7 Kbits/sec > > >>>> [ 5][TX-C] 5.00-6.00 sec 17.7 MBytes 149 Mbits/sec 363 > 15.4 KBytes > >>>> [ 7][RX-C] 5.00-6.00 sec 4.18 KBytes 34.3 Kbits/sec > > >>>> [ 5][TX-C] 6.00-7.00 sec 14.8 MBytes 124 Mbits/sec 287 > 8.38 KBytes > >>>> [ 7][RX-C] 6.00-7.00 sec 5.58 KBytes 45.7 Kbits/sec > > >>>> [ 5][TX-C] 7.00-8.00 sec 14.7 MBytes 123 Mbits/sec 293 > 28.1 KBytes > >>>> [ 7][RX-C] 7.00-8.00 sec 4.18 KBytes 34.3 Kbits/sec > > >>>> [ 5][TX-C] 8.00-9.00 sec 11.9 MBytes 100 Mbits/sec 325 > 18.3 KBytes > >>>> [ 7][RX-C] 8.00-9.00 sec 4.18 KBytes 34.3 Kbits/sec > > >>>> [ 5][TX-C] 9.00-10.00 sec 14.3 MBytes 120 Mbits/sec 315 > 39.3 KBytes > >>>> [ 7][RX-C] 9.00-10.00 sec 4.18 KBytes 34.3 Kbits/sec > > >>>> - - - - - - - - - - - - - - - - - - - - - - - - - > >>>> [ ID][Role] Interval Transfer Bitrate Retr > >>>> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec 2981 > sender > >>>> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec > receiver > >>>> [ 7][RX-C] 0.00-10.00 sec 77.0 KBytes 63.1 Kbits/sec 65 > sender > >>>> [ 7][RX-C] 0.00-10.00 sec 44.6 KBytes 36.6 Kbits/sec > receiver > >>>> ------------------------------------------------------------ > ----------------------------------------------- > >>>> > >>>> > >>>> UDP: > >>>> [ ID][Role] Interval Transfer Bitrate Jitter > Lost/Total Datagrams > >>>> [ 5][TX-C] 0.00-1.00 sec 81.6 MBytes 685 Mbits/sec > 67798 > >>>> [ 7][RX-C] 0.00-1.00 sec 8.80 MBytes 73.8 Mbits/sec 0.255 ms > 54070/60475 (89%) > >>>> [ 5][TX-C] 1.00-2.00 sec 72.7 MBytes 610 Mbits/sec > 64802 > >>>> [ 7][RX-C] 1.00-2.00 sec 8.52 MBytes 71.5 Mbits/sec 0.154 ms > 68912/75116 (92%) > >>>> [ 5][TX-C] 2.00-3.00 sec 73.7 MBytes 618 Mbits/sec > 64158 > >>>> [ 7][RX-C] 2.00-3.00 sec 8.52 MBytes 71.5 Mbits/sec 0.276 ms > 67738/73945 (92%) > >>>> [ 5][TX-C] 3.00-4.00 sec 76.6 MBytes 643 Mbits/sec > 63521 > >>>> [ 7][RX-C] 3.00-4.00 sec 8.55 MBytes 71.8 Mbits/sec 0.160 ms > 68647/74874 (92%) > >>>> [ 5][TX-C] 4.00-5.00 sec 76.1 MBytes 638 Mbits/sec > 64614 > >>>> [ 7][RX-C] 4.00-5.00 sec 8.55 MBytes 71.7 Mbits/sec 0.461 ms > 67542/73767 (92%) > >>>> [ 5][TX-C] 5.00-6.00 sec 75.9 MBytes 637 Mbits/sec > 64834 > >>>> [ 7][RX-C] 5.00-6.00 sec 8.57 MBytes 71.9 Mbits/sec 0.297 ms > 71565/77806 (92%) > >>>> [ 5][TX-C] 6.00-7.00 sec 73.0 MBytes 613 Mbits/sec > 63639 > >>>> [ 7][RX-C] 6.00-7.00 sec 8.40 MBytes 70.5 Mbits/sec 0.199 ms > 69545/75663 (92%) > >>>> [ 5][TX-C] 7.00-8.00 sec 74.6 MBytes 626 Mbits/sec > 65030 > >>>> [ 7][RX-C] 7.00-8.00 sec 8.78 MBytes 73.6 Mbits/sec 0.254 ms > 67173/73566 (91%) > >>>> [ 5][TX-C] 8.00-9.00 sec 75.0 MBytes 629 Mbits/sec > 64848 > >>>> [ 7][RX-C] 8.00-9.00 sec 8.77 MBytes 73.5 Mbits/sec 0.298 ms > 70932/77315 (92%) > >>>> [ 5][TX-C] 9.00-10.00 sec 74.5 MBytes 625 Mbits/sec > 64487 > >>>> [ 7][RX-C] 9.00-10.00 sec 8.71 MBytes 73.1 Mbits/sec 0.185 ms > 68268/74612 (91%) > >>>> - - - - - - - - - - - - - - - - - - - - - - - - - > >>>> [ ID][Role] Interval Transfer Bitrate Jitter > Lost/Total Datagrams > >>>> [ 5][TX-C] 0.00-10.00 sec 754 MBytes 632 Mbits/sec 0.000 ms > 0/647731 (0%) sender > >>>> [ 5][TX-C] 0.00-10.12 sec 105 MBytes 87.2 Mbits/sec 0.245 ms > 571090/647649 (88%) receiver > >>>> [ 7][RX-C] 0.00-10.00 sec 1009 MBytes 846 Mbits/sec 0.000 ms > 0/761013 (0%) sender > >>>> [ 7][RX-C] 0.00-10.12 sec 86.2 MBytes 71.4 Mbits/sec 0.185 ms > 674392/737139 (91%) receiver > >>>> > >>>> > >> > >> Taking a look at this iperf output I recalled that 80% of my setups > >> suffered from similar issue after transitioning from 1{1,2}-STABLE to > >> 13-STABLE about a mounth ago. I have asked on IRC but nobody confirmed > >> similar problems so I have reduced MTU to 8900 from the original 9000 = on > >> some vlan(4) interfaces to solve the issue. I am using mostly vlan(4)s > >> over LACP lagg(4)s created on NICs. So far (for FreeBSD 11 and 12) > >> setting MTU 9000 on physical NIC was sufficient to make it work, now I > >> have set MTU 9000 on NICs and reduced MTU 8900 on vlan(4)s. > > TCP announces an MSS of 1440, which corresponds to an MTU of 1500 byte. > > So I guess your problem is different. > > Is your physical MTU 9000 bytes? > > > > Best regards > > Michael > > Yes, MTU for some VLANs in this network is 9000 bytes and I performed > only software upgrades, hardware was untouched. I haven't had similar > issues while running 11.4-STABLE or 12.2-STABLE. The aforementioned 8900 > is not a critical value, but the reduction from an original 9000 was > mandatory. > > With kind regards, > Marek > > I also experienced that same issue with 13.0-RC2 after upgrading some VMs. The same issue also exists in Beta4. It only affects IPv6. For me it is extremely slow in both directions if both VMs are on the same host and are running 13.0-RC2. It is easy to reproduce. Simply setup two VMs using virt-manager or proxmox running FreeBSD 13.0-RC2. Configure the network to use virtio and setup IPv6. Run iperf using IPv6 and we are down to modem speeds. :-( If you test using ipv4 no issues at all. To only way to get better speeds is to disable tso6. ifconfig vtnet0 -tso6 The host-host speeds after that are still not as good as IPv4 but at least it=E2=80=99s in multiple Gbps vs just some Kbps. FreeBSD 12.2 has none of t= hose issues. Could it have something to do with the if_vtnet changes on 2021-01-19 committed by Bryan Venteicher https://reviews.freebsd.org/D27904 https://cgit.freebsd.org/src/log/sys/dev/virtio/network?h=3Dreleng/13.0 Haven=E2=80=99t had time to compile a kernel before those changes or right = after to see if the issue goes away. I only tested the ipv6 slowness on virtio. Don=E2=80=99t know if the issue = exists on bare metal hardware as well. Thanks, Michael Tratz > >>>> > >>>> > >>>> On Tuesday, March 16, 2021 4:16:15 AM EDT tuexen@freebsd.org wrote: > >>>>>> On 15. Mar 2021, at 12:56, Blake Hartshorn < > contact@blakehartshorn.com> wrote: > >>>>>> > >>>>>> The short version, when I use FreeBSD 13, delivering data can take > 5 minutes for 1MB over SSH or HTTP when using IPv6. This problem does not > happen with IPv4. I installed FreeBSD 12 and Linux on that same device, > neither had the problem. > >>>>>> > >>>>>> Did some troubleshooting with Linode, have ultimately ruled the > network itself out at this point. When the server is on FreeBSD 13, it ca= n > download quickly over IPv6, but not deliver. Started investigating after > noticing my SSH session was lagging when cat'ing large files or running > builds. This problem even occurs between VMs in the same datacenter. I > generated a 1MB file of base64 garbage served by nginx for testing. IPv6 = is > being configured by SLAAC and on both 12 and 13 installs was setup by the > installer. Linode uses Linux/KVM hosts for their virtual machines so it's > running on that virtual adapter. > >>>>>> > >>>>>> I asked on the forums, another user recommended going to the > mailing lists instead. Does anyone know if config settings need to be > different on 13? Did I maybe just find a real issue? I can provide any > requested details. Thanks! > >>>>> Could you prove a .pcap tracefile, one from the sender, one from th= e > recevier, of > >>>>> a TCP/IPv6 connection, which doesn't work as expected. For example, > use your 1MB > >>>>> base64 garbage transfer. > >>>>> > >>>>> Best regardes > >>>>> Michael > >>>>>> > >>>>>> > >>>>>> _______________________________________________ > >>>>>> freebsd-net@freebsd.org mailing list > >>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net > >>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@ > freebsd.org" > >>>>> > >>>>> > >>>> > >>> > >>> _______________________________________________ > >>> freebsd-net@freebsd.org mailing list > >>> https://lists.freebsd.org/mailman/listinfo/freebsd-net > >>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org= " > >>> > >> > >> > >> -- > >> Marek Zarychta > >> > > > > > > > -- > Marek Zarychta > >