From owner-freebsd-net@freebsd.org Tue Mar 16 10:56:02 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9D4B05A897B for ; Tue, 16 Mar 2021 10:56:02 +0000 (UTC) (envelope-from contact@blakehartshorn.com) Received: from mail-qk1-x734.google.com (mail-qk1-x734.google.com [IPv6:2607:f8b0:4864:20::734]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F09FF5ZpCz3tcr for ; Tue, 16 Mar 2021 10:56:01 +0000 (UTC) (envelope-from contact@blakehartshorn.com) Received: by mail-qk1-x734.google.com with SMTP id f124so34679973qkj.5 for ; Tue, 16 Mar 2021 03:56:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=blakehartshorn-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wJsnDVZ+/YJQjV5yOMFjdc9VgNkToFHSWA+Y/+219fk=; b=WMxtPKc53UVfnn9pP5HRsSxaTpKdH1wWZrizdDmmVgqn1lc5uBbA7JJqvd1RTa9qUn i6T0NdNMEkAvYdGTTkFhJ8kQqSegkQNVT9tPImczl5TALd4ywv0UQion4G4wvs9F9Wjd /B/UKlZa6URB5I6gWxO+3JXm/6V1fOdIoZnJpEt1Be4WHN0dySo2ZoQc7lc6B+ya3Qdh B/tSwP1JD8R9v5NVw8tr4WcFzG6CQzaY3E3awATk6vRm4oMFpJj/zcV5tqtLA4JpSct6 TXLnIqnD9+Gzi8KZlYflb/VwrnosrHU9tLsRrYpPJ6kApH2PJStna2N+YchoCbcrWg7H MnDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wJsnDVZ+/YJQjV5yOMFjdc9VgNkToFHSWA+Y/+219fk=; b=G+X0Oxfr/46QonlS6q1C04cfc3u2TWxfKL0p9rLBcsw6ZdFBfGS2x7R/bthlA8avZC YxfXs6nGFGNCPsRKd7SlAhMITUZRnbK7YTnaLF2B/CXYqp6aXhh/aqJOpVgtq08ESKgn 4psZcHleQC9FFWfoCkHiKpQaCt9c9Y+QAT2cYMUKKZxdFqYrEcV7P8PtjJ6Q37xk0l7F MdfncnTvyicn6LKRAS2ZsoRcEbeA2mLj5Y2YGnrVvdbF5nQ3dlP8kxZ81tmhbFWC26f5 cPU0u619fXU76itpMBTwiAHwHlx7cLeT4zXm8E3WnO7VIRpcfFyTEWw8aHeipEFMFMED o7GQ== X-Gm-Message-State: AOAM531RhFfPMa3Kuw9aXmPQKRNIRwLKV5IDWKmHHUGyVI8u7OJNtUN8 e/UedZ9KdEsukkZbv8B3EKRhcjTAKDVW8xQn X-Google-Smtp-Source: ABdhPJxXfiIBMwKVzqDq9hzUduUoekIObiv9f6LAePqTzM1yBukUvvHWBYZqkqqIBA+ORbu7Uf1CBw== X-Received: by 2002:a37:a44:: with SMTP id 65mr28685131qkk.479.1615892159773; Tue, 16 Mar 2021 03:55:59 -0700 (PDT) Received: from thinkbook.localnet ([2601:18d:97f:d245:5328:24df:14af:ed29]) by smtp.gmail.com with ESMTPSA id e2sm12959282qto.50.2021.03.16.03.55.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 03:55:55 -0700 (PDT) From: Blake Hartshorn To: tuexen@freebsd.org Cc: freebsd-net@freebsd.org Subject: Re: Severe IPv6 TCP transfer issues on 13.0-RC1 and RC2 Date: Tue, 16 Mar 2021 06:55:54 -0400 Message-ID: <10847992.4GmZMkJedg@thinkbook> In-Reply-To: References: <5753280.1HxCrU2fYu@thinkbook> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4F09FF5ZpCz3tcr X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=blakehartshorn-com.20150623.gappssmtp.com header.s=20150623 header.b=WMxtPKc5; dmarc=none; spf=none (mx1.freebsd.org: domain of contact@blakehartshorn.com has no SPF policy when checking 2607:f8b0:4864:20::734) smtp.mailfrom=contact@blakehartshorn.com X-Spamd-Result: default: False [-2.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[blakehartshorn-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~,3:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::734:from]; CTE_CASE(0.50)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[blakehartshorn-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[blakehartshorn.com]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::734:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::734:from]; R_SPF_NA(0.00)[no SPF record]; MID_RHS_NOT_FQDN(0.50)[]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-Mailman-Approved-At: Tue, 16 Mar 2021 13:02:42 +0000 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7Bit X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Mar 2021 10:56:02 -0000 Hi Michael, I've attached tcpdumps for port 80 on both sides of a bad transfer, using 2 VMs in the same datacenter, FreeBSD 13 serving and 12 as a client. A friend of mine suggested I also run some tests with iperf3, so pasting those results below. You'll see it going fast in one direction and crawling in the other on TCP. There's also some disparity on UDP. TCP: [ ID][Role] Interval Transfer Bitrate Retr Cwnd [ 5][TX-C] 0.00-1.00 sec 27.7 MBytes 233 Mbits/sec 342 26.7 KBytes [ 7][RX-C] 0.00-1.00 sec 4.18 KBytes 34.3 Kbits/sec [ 5][TX-C] 1.00-2.00 sec 15.8 MBytes 132 Mbits/sec 249 52.0 KBytes [ 7][RX-C] 1.00-2.00 sec 4.18 KBytes 34.3 Kbits/sec [ 5][TX-C] 2.00-3.00 sec 13.7 MBytes 115 Mbits/sec 307 15.4 KBytes [ 7][RX-C] 2.00-3.00 sec 4.18 KBytes 34.3 Kbits/sec [ 5][TX-C] 3.00-4.00 sec 14.5 MBytes 121 Mbits/sec 260 22.4 KBytes [ 7][RX-C] 3.00-4.00 sec 4.18 KBytes 34.3 Kbits/sec [ 5][TX-C] 4.00-5.00 sec 14.3 MBytes 120 Mbits/sec 240 37.9 KBytes [ 7][RX-C] 4.00-5.00 sec 5.58 KBytes 45.7 Kbits/sec [ 5][TX-C] 5.00-6.00 sec 17.7 MBytes 149 Mbits/sec 363 15.4 KBytes [ 7][RX-C] 5.00-6.00 sec 4.18 KBytes 34.3 Kbits/sec [ 5][TX-C] 6.00-7.00 sec 14.8 MBytes 124 Mbits/sec 287 8.38 KBytes [ 7][RX-C] 6.00-7.00 sec 5.58 KBytes 45.7 Kbits/sec [ 5][TX-C] 7.00-8.00 sec 14.7 MBytes 123 Mbits/sec 293 28.1 KBytes [ 7][RX-C] 7.00-8.00 sec 4.18 KBytes 34.3 Kbits/sec [ 5][TX-C] 8.00-9.00 sec 11.9 MBytes 100 Mbits/sec 325 18.3 KBytes [ 7][RX-C] 8.00-9.00 sec 4.18 KBytes 34.3 Kbits/sec [ 5][TX-C] 9.00-10.00 sec 14.3 MBytes 120 Mbits/sec 315 39.3 KBytes [ 7][RX-C] 9.00-10.00 sec 4.18 KBytes 34.3 Kbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID][Role] Interval Transfer Bitrate Retr [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec 2981 sender [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec receiver [ 7][RX-C] 0.00-10.00 sec 77.0 KBytes 63.1 Kbits/sec 65 sender [ 7][RX-C] 0.00-10.00 sec 44.6 KBytes 36.6 Kbits/sec receiver ----------------------------------------------------------------------------------------------------------- UDP: [ ID][Role] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5][TX-C] 0.00-1.00 sec 81.6 MBytes 685 Mbits/sec 67798 [ 7][RX-C] 0.00-1.00 sec 8.80 MBytes 73.8 Mbits/sec 0.255 ms 54070/60475 (89%) [ 5][TX-C] 1.00-2.00 sec 72.7 MBytes 610 Mbits/sec 64802 [ 7][RX-C] 1.00-2.00 sec 8.52 MBytes 71.5 Mbits/sec 0.154 ms 68912/75116 (92%) [ 5][TX-C] 2.00-3.00 sec 73.7 MBytes 618 Mbits/sec 64158 [ 7][RX-C] 2.00-3.00 sec 8.52 MBytes 71.5 Mbits/sec 0.276 ms 67738/73945 (92%) [ 5][TX-C] 3.00-4.00 sec 76.6 MBytes 643 Mbits/sec 63521 [ 7][RX-C] 3.00-4.00 sec 8.55 MBytes 71.8 Mbits/sec 0.160 ms 68647/74874 (92%) [ 5][TX-C] 4.00-5.00 sec 76.1 MBytes 638 Mbits/sec 64614 [ 7][RX-C] 4.00-5.00 sec 8.55 MBytes 71.7 Mbits/sec 0.461 ms 67542/73767 (92%) [ 5][TX-C] 5.00-6.00 sec 75.9 MBytes 637 Mbits/sec 64834 [ 7][RX-C] 5.00-6.00 sec 8.57 MBytes 71.9 Mbits/sec 0.297 ms 71565/77806 (92%) [ 5][TX-C] 6.00-7.00 sec 73.0 MBytes 613 Mbits/sec 63639 [ 7][RX-C] 6.00-7.00 sec 8.40 MBytes 70.5 Mbits/sec 0.199 ms 69545/75663 (92%) [ 5][TX-C] 7.00-8.00 sec 74.6 MBytes 626 Mbits/sec 65030 [ 7][RX-C] 7.00-8.00 sec 8.78 MBytes 73.6 Mbits/sec 0.254 ms 67173/73566 (91%) [ 5][TX-C] 8.00-9.00 sec 75.0 MBytes 629 Mbits/sec 64848 [ 7][RX-C] 8.00-9.00 sec 8.77 MBytes 73.5 Mbits/sec 0.298 ms 70932/77315 (92%) [ 5][TX-C] 9.00-10.00 sec 74.5 MBytes 625 Mbits/sec 64487 [ 7][RX-C] 9.00-10.00 sec 8.71 MBytes 73.1 Mbits/sec 0.185 ms 68268/74612 (91%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID][Role] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5][TX-C] 0.00-10.00 sec 754 MBytes 632 Mbits/sec 0.000 ms 0/647731 (0%) sender [ 5][TX-C] 0.00-10.12 sec 105 MBytes 87.2 Mbits/sec 0.245 ms 571090/647649 (88%) receiver [ 7][RX-C] 0.00-10.00 sec 1009 MBytes 846 Mbits/sec 0.000 ms 0/761013 (0%) sender [ 7][RX-C] 0.00-10.12 sec 86.2 MBytes 71.4 Mbits/sec 0.185 ms 674392/737139 (91%) receiver On Tuesday, March 16, 2021 4:16:15 AM EDT tuexen@freebsd.org wrote: > > On 15. Mar 2021, at 12:56, Blake Hartshorn wrote: > > > > The short version, when I use FreeBSD 13, delivering data can take 5 minutes for 1MB over SSH or HTTP when using IPv6. This problem does not happen with IPv4. I installed FreeBSD 12 and Linux on that same device, neither had the problem. > > > > Did some troubleshooting with Linode, have ultimately ruled the network itself out at this point. When the server is on FreeBSD 13, it can download quickly over IPv6, but not deliver. Started investigating after noticing my SSH session was lagging when cat'ing large files or running builds. This problem even occurs between VMs in the same datacenter. I generated a 1MB file of base64 garbage served by nginx for testing. IPv6 is being configured by SLAAC and on both 12 and 13 installs was setup by the installer. Linode uses Linux/KVM hosts for their virtual machines so it's running on that virtual adapter. > > > > I asked on the forums, another user recommended going to the mailing lists instead. Does anyone know if config settings need to be different on 13? Did I maybe just find a real issue? I can provide any requested details. Thanks! > Could you prove a .pcap tracefile, one from the sender, one from the recevier, of > a TCP/IPv6 connection, which doesn't work as expected. For example, use your 1MB > base64 garbage transfer. > > Best regardes > Michael > > > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >