From nobody Fri Oct 21 14:19:19 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Mv66d2mSfz4fVHF for ; Fri, 21 Oct 2022 14:19:37 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Mv66b6PGdz3nmw for ; Fri, 21 Oct 2022 14:19:35 +0000 (UTC) (envelope-from zlei.huang@gmail.com) Received: by mail-pj1-x1036.google.com with SMTP id i3-20020a17090a3d8300b00212cf2e2af9so1844398pjc.1 for ; Fri, 21 Oct 2022 07:19:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:date:message-id:subject:mime-version:from:from:to:cc:subject :date:message-id:reply-to; bh=zB9E5YPDvArBjW8dq3D4jwDTM1vk9Mx4CAALqEU6uk4=; b=cykW5M8CFABaY+NWkNTi1M2kVSW5nyv/O4GZo+qxwOgP6mae5BF7jLSD69V0ZdiEAZ zEPbaVZ9dCZT5UoSbhx+llvJIIUn6q2+fmnzs+n07cqvmTu4M5BE2wB6KUCuk8iu4lyZ 9f9mXnbXFzZD78a/qNnlwdaLGe80nrKdIfAMEFmYqxdyFX7+W5e1ijrfjGzDggjpNSBJ U82ePPvEGq8lH9beVZ0Zik09U7OPFdG6yy22uBBeWJ3GPdt9qqbT/79+vBYYeZyzv0g3 TGET9lOp2qR3WlnVeVBftknCKDpHf3U+1P6pkWJL+QVJamDf1+5r5H5Pv+lWN8qz2FlX urgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:date:message-id:subject:mime-version:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zB9E5YPDvArBjW8dq3D4jwDTM1vk9Mx4CAALqEU6uk4=; b=XnLTIEW4360ZdPflh9K7wd78oHZFghp7H3A+msHnm0nqyeANG7dAwIDL/gVe3xaPCd 8sb99b003FUoUkG5w/mwZ8dh8opm29xlmPhWfY+BtJrbuOPrcjnLynKgsPYpp2bgWJlA XhQPl/sYs+fHTnQaROe5u0AX01eFhsk9/bQAs3TeSD5G56z9G+brPVaCtLJp5uSlRf2a ecyaGgWuSxuGOAm6tn7wRVUWDOMN8EB6O6/s9l8ybll9UCF5EQjEKClj9CNENrpODryB FCxw7754JGdh5lozKP2FyRO70qAAjoQXYsomuWD2Nxkh2CT4bPfoUIQZSPj3HNjvGkwW EP3w== X-Gm-Message-State: ACrzQf36XLSQddEqVTMEvjXFEv6KZprDxsJ+J1K8XpVeFH3hX+F0Et1l lijf7TbG8KLCtwuQjxOecjU9QnduRzy3BiK6 X-Google-Smtp-Source: AMsMyM7Gkp/b2GRnl57JhDSgveOWpd2DbSGyU8hQX9dpp+rO4divsahXJJlrb9KvUkGUIiwKCaS6xw== X-Received: by 2002:a17:902:820b:b0:185:b9a:8ac1 with SMTP id x11-20020a170902820b00b001850b9a8ac1mr19634852pln.111.1666361974273; Fri, 21 Oct 2022 07:19:34 -0700 (PDT) Received: from [172.17.252.129] (ns1.oxydns.net. [45.32.91.63]) by smtp.gmail.com with ESMTPSA id kx7-20020a17090b228700b00211923b8edesm1632133pjb.40.2022.10.21.07.19.30 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Oct 2022 07:19:33 -0700 (PDT) From: Zhenlei Huang Content-Type: multipart/alternative; boundary="Apple-Mail=_F1C4CA8E-F708-46BD-B8F1-F2DE4E290809" List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.7\)) Subject: Too aggressive TCP ACKs Message-Id: <75D35F36-7759-4168-ADBA-C2414F5B53BC@gmail.com> Date: Fri, 21 Oct 2022 22:19:19 +0800 To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.3608.120.23.2.7) X-Rspamd-Queue-Id: 4Mv66b6PGdz3nmw X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=cykW5M8C; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zlei.huang@gmail.com designates 2607:f8b0:4864:20::1036 as permitted sender) smtp.mailfrom=zlei.huang@gmail.com X-Spamd-Result: default: False [-3.46 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.956]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1036:from]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TAGGED_FROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org] X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_F1C4CA8E-F708-46BD-B8F1-F2DE4E290809 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, While I was repeating = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258755, I observed a strange behavior. The TCP ACKs from FreeBSD host are too aggressive. My setup is simple: A B [ MacOS ] <=3D=3D=3D=3D> [ FreeBSD VM ] 192.168.120.1 192.168.12.134 (disable tso and lro) While A <--- B, i.e. A as server and B as client, the packets rate looks = good. One session on B: root@:~ # iperf3 -c 192.168.120.1 -b 10m Connecting to host 192.168.120.1, port 5201 [ 5] local 192.168.120.134 port 54459 connected to 192.168.120.1 port = 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 [ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 [ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes = =20 [ 5] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 [ 5] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes = =20 [ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 [ 5] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes = =20 [ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 [ 5] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes = =20 [ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec 0 = sender [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec = receiver iperf Done. Another session on B: root@:~ # netstat -w 1 -I vmx0 input vmx0 output packets errs idrops bytes packets errs bytes colls 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 342 0 0 22600 526 0 775724 0 150 0 0 9900 851 0 1281454 0 109 0 0 7194 901 0 1357850 0 126 0 0 8316 828 0 1246632 0 122 0 0 8052 910 0 1370780 0 109 0 0 7194 819 0 1233702 0 120 0 0 7920 910 0 1370780 0 110 0 0 7260 819 0 1233702 0 123 0 0 8118 910 0 1370780 0 109 0 0 7194 819 0 1233702 0 73 0 0 5088 465 0 686342 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D While A ---> B, i.e. A as client and B as server, the ACKs sent from B = looks strange. Session on A: % iperf3 -c 192.168.120.134 -b 10m Connecting to host 192.168.120.134, port 5201 [ 5] local 192.168.120.1 port 52370 connected to 192.168.120.134 port = 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec =20 [ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec =20 [ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec =20 [ 5] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec =20 [ 5] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec =20 [ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec =20 [ 5] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec =20 [ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec =20 [ 5] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec =20 [ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec = sender [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec = receiver iperf Done. Session on B: root@:~ # netstat -w 1 -I vmx0 input vmx0 output packets errs idrops bytes packets errs bytes colls 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 649 0 0 960562 330 0 21800 0 819 0 0 1233702 415 0 27390 0 910 0 0 1370780 459 0 30294 0 819 0 0 1233702 415 0 27390 0 910 0 0 1370780 459 0 30294 0 910 0 0 1370780 460 0 30360 0 819 0 0 1233702 414 0 27324 0 910 0 0 1370780 460 0 30360 0 819 0 0 1233702 414 0 27324 0 910 0 0 1370780 460 0 30360 0 285 0 0 412287 147 0 9981 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 The ACK packets replied from B (the FreeBSD VM) are too aggressive. They = are about one half of TCP packets received from A. I've tested with different bitrates, from 10m to 300m, all behave the = same. Tested with baremetal FreeBSD 13.1 Box as B (with intel em driver), the=20= bitrates is 1g, also behaves the same. Also tried different FreeBSD versions, 11.4, 12.3, stable/13 and = current/14 all=20 behave the same. My question is, is that the expected behavior of current default TCP = stack? Best regards, Zhenlei --Apple-Mail=_F1C4CA8E-F708-46BD-B8F1-F2DE4E290809 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
Hi,

While I was repeating https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258755,= I observed a
strange behavior. The TCP ACKs from = FreeBSD host are too aggressive.

My setup is simple:
         A       =                     =       B
   [ MacOS ] =  <=3D=3D=3D=3D> [ FreeBSD VM ]
192.168.120.1           =  192.168.12.134 (disable tso and lro)
While A = <--- B, i.e. A as server and B as client, the packets rate looks = good.

One = session on B:

root@:~ # iperf3 -c 192.168.120.1 -b = 10m
Connecting to host 192.168.120.1, port = 5201
[  5] local 192.168.120.134 port 54459 = connected to 192.168.120.1 port 5201
[ ID] Interval =           Transfer     Bitrate   =       Retr  Cwnd
[  5] =   0.00-1.00   sec  1.25 MBytes  10.5 Mbits/sec =    0    257 KBytes     =   
[  5]   1.00-2.00   sec =  1.25 MBytes  10.5 Mbits/sec    0    257 = KBytes       
[  5]   = 2.00-3.00   sec  1.12 MBytes  9.44 Mbits/sec   =  0    257 KBytes       
[  5]   3.00-4.00   sec  1.25 MBytes =  10.5 Mbits/sec    0    257 KBytes   =     
[  5]   4.00-5.00 =   sec  1.12 MBytes  9.44 Mbits/sec    0   =  257 KBytes       
[ =  5]   5.00-6.00   sec  1.25 MBytes  10.5 = Mbits/sec    0    257 KBytes     =   
[  5]   6.00-7.00   sec =  1.12 MBytes  9.44 Mbits/sec    0    257 = KBytes       
[  5]   = 7.00-8.00   sec  1.25 MBytes  10.5 Mbits/sec   =  0    257 KBytes       
[  5]   8.00-9.00   sec  1.12 MBytes =  9.44 Mbits/sec    0    257 KBytes   =     
[  5]   9.00-10.00 =  sec  1.25 MBytes  10.5 Mbits/sec    0   =  257 KBytes       
- - - - = - - - - - - - - - - - - - - - - - - - - -
[ ID] = Interval           Transfer     = Bitrate         Retr
[  5] =   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec =    0             = sender
[  5]   0.00-10.00  sec =  12.0 MBytes  10.1 Mbits/sec         =          receiver

iperf Done.

Another session on = B:

root@:~ # netstat -w 1 -I vmx0
  =           input         =   vmx0           output
   packets  errs idrops     =  bytes    packets  errs      bytes = colls
         0   =   0     0          0   =        0     0         =  0     0
        =  0     0     0         =  0          0     0   =        0     0
  =      342     0     0     =  22600        526     0     = 775724     0
      =  150     0     0       9900 =        851     0    1281454 =     0
       109 =     0     0       7194     =    901     0    1357850     = 0
       126     0 =     0       8316        828 =     0    1246632     0
       122     0     = 0       8052        910     = 0    1370780     0
  =      109     0     0     =   7194        819     0   =  1233702     0
      =  120     0     0       7920 =        910     0    1370780 =     0
       110 =     0     0       7260     =    819     0    1233702     = 0
       123     0 =     0       8118        910 =     0    1370780     0
       109     0     = 0       7194        819     = 0    1233702     0
  =       73     0     0     =   5088        465     0     = 686342     0
        =  0     0     0         =  0          0     0   =        0     0
  =        0     0     0   =        0          0   =   0          0     = 0



=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D


While A ---> B, i.e. A as client and B as server, = the ACKs sent from B looks strange.

Session on A:

% iperf3 -c = 192.168.120.134 -b 10m
Connecting to host = 192.168.120.134, port 5201
[  5] local = 192.168.120.1 port 52370 connected to 192.168.120.134 port = 5201
[ ID] Interval         =   Transfer     Bitrate
[  5] =   0.00-1.00   sec  1.25 MBytes  10.5 Mbits/sec =                  
[  5]   1.00-2.00   sec  1.25 MBytes =  10.5 Mbits/sec               =    
[  5]   2.00-3.00   = sec  1.12 MBytes  9.44 Mbits/sec         =          
[  5] =   3.00-4.00   sec  1.25 MBytes  10.5 Mbits/sec =                  
[  5]   4.00-5.00   sec  1.12 MBytes =  9.44 Mbits/sec               =    
[  5]   5.00-6.00   = sec  1.25 MBytes  10.5 Mbits/sec         =          
[  5] =   6.00-7.00   sec  1.12 MBytes  9.44 Mbits/sec =                  
[  5]   7.00-8.00   sec  1.25 MBytes =  10.5 Mbits/sec               =    
[  5]   8.00-9.00   = sec  1.12 MBytes  9.44 Mbits/sec         =          
[  5] =   9.00-10.00  sec  1.25 MBytes  10.5 Mbits/sec =                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer =     Bitrate
[  5]   0.00-10.00 =  sec  12.0 MBytes  10.1 Mbits/sec       =            sender
[ =  5]   0.00-10.00  sec  12.0 MBytes  10.1 = Mbits/sec                 =  receiver

iperf Done.

Session on B:

root@:~ # netstat -w 1 -I vmx0
  =           input         =   vmx0           output
   packets  errs idrops     =  bytes    packets  errs      bytes = colls
         0   =   0     0          0   =        0     0         =  0     0
        =  0     0     0         =  0          0     0   =        0     0
  =      649     0     0     = 960562        330     0     =  21800     0
      =  819     0     0    1233702   =      415     0      27390   =   0
       910   =   0     0    1370780       =  459     0      30294     = 0
       819     0 =     0    1233702        415 =     0      27390     0
       910     0     = 0    1370780        459     0 =      30294     0
  =      910     0     0   =  1370780        460     0   =    30360     0
    =    819     0     0    1233702 =        414     0      27324 =     0
       910 =     0     0    1370780     =    460     0      30360     = 0
       819     0 =     0    1233702        414 =     0      27324     0
       910     0     = 0    1370780        460     0 =      30360     0
  =      285     0     0     = 412287        147     0     =   9981     0
      =    0     0     0       =    0          0     0 =          0     0
         0     0   =   0          0         =  0     0          0   =   0
         0   =   0     0          0   =        0     0         =  0     0


The = ACK packets replied from B (the FreeBSD VM) are too aggressive. They = are
about one half of TCP packets received from = A.

I've tested = with different bitrates, from 10m to 300m, all behave the = same.
Tested with baremetal FreeBSD 13.1 Box as B = (with intel em driver), the 
bitrates is 1g, also  behaves the = same.

Also tried different FreeBSD versions, 11.4, 12.3, stable/13 = and current/14 all 
behave the = same.


My = question is, is that the expected behavior of current default TCP stack?



Best regards,
Zhenlei

= --Apple-Mail=_F1C4CA8E-F708-46BD-B8F1-F2DE4E290809--