Date: Fri, 21 Oct 2022 22:19:19 +0800 From: Zhenlei Huang <zlei.huang@gmail.com> To: freebsd-net@freebsd.org Subject: Too aggressive TCP ACKs Message-ID: <75D35F36-7759-4168-ADBA-C2414F5B53BC@gmail.com>
next in thread | raw e-mail | index | archive | help
--Apple-Mail=_F1C4CA8E-F708-46BD-B8F1-F2DE4E290809 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, While I was repeating = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258755, I observed a strange behavior. The TCP ACKs from FreeBSD host are too aggressive. My setup is simple: A B [ MacOS ] <=3D=3D=3D=3D> [ FreeBSD VM ] 192.168.120.1 192.168.12.134 (disable tso and lro) While A <--- B, i.e. A as server and B as client, the packets rate looks = good. One session on B: root@:~ # iperf3 -c 192.168.120.1 -b 10m Connecting to host 192.168.120.1, port 5201 [ 5] local 192.168.120.134 port 54459 connected to 192.168.120.1 port = 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 [ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 [ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes = =20 [ 5] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 [ 5] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes = =20 [ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 [ 5] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes = =20 [ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 [ 5] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes = =20 [ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes = =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec 0 = sender [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec = receiver iperf Done. Another session on B: root@:~ # netstat -w 1 -I vmx0 input vmx0 output packets errs idrops bytes packets errs bytes colls 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 342 0 0 22600 526 0 775724 0 150 0 0 9900 851 0 1281454 0 109 0 0 7194 901 0 1357850 0 126 0 0 8316 828 0 1246632 0 122 0 0 8052 910 0 1370780 0 109 0 0 7194 819 0 1233702 0 120 0 0 7920 910 0 1370780 0 110 0 0 7260 819 0 1233702 0 123 0 0 8118 910 0 1370780 0 109 0 0 7194 819 0 1233702 0 73 0 0 5088 465 0 686342 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D While A ---> B, i.e. A as client and B as server, the ACKs sent from B = looks strange. Session on A: % iperf3 -c 192.168.120.134 -b 10m Connecting to host 192.168.120.134, port 5201 [ 5] local 192.168.120.1 port 52370 connected to 192.168.120.134 port = 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec =20 [ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec =20 [ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec =20 [ 5] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec =20 [ 5] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec =20 [ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec =20 [ 5] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec =20 [ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec =20 [ 5] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec =20 [ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec = sender [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec = receiver iperf Done. Session on B: root@:~ # netstat -w 1 -I vmx0 input vmx0 output packets errs idrops bytes packets errs bytes colls 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 649 0 0 960562 330 0 21800 0 819 0 0 1233702 415 0 27390 0 910 0 0 1370780 459 0 30294 0 819 0 0 1233702 415 0 27390 0 910 0 0 1370780 459 0 30294 0 910 0 0 1370780 460 0 30360 0 819 0 0 1233702 414 0 27324 0 910 0 0 1370780 460 0 30360 0 819 0 0 1233702 414 0 27324 0 910 0 0 1370780 460 0 30360 0 285 0 0 412287 147 0 9981 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 The ACK packets replied from B (the FreeBSD VM) are too aggressive. They = are about one half of TCP packets received from A. I've tested with different bitrates, from 10m to 300m, all behave the = same. Tested with baremetal FreeBSD 13.1 Box as B (with intel em driver), the=20= bitrates is 1g, also behaves the same. Also tried different FreeBSD versions, 11.4, 12.3, stable/13 and = current/14 all=20 behave the same. My question is, is that the expected behavior of current default TCP = stack? Best regards, Zhenlei --Apple-Mail=_F1C4CA8E-F708-46BD-B8F1-F2DE4E290809 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><div = class=3D"">Hi,</div><div class=3D""><br class=3D""></div><div = class=3D"">While I was repeating <a = href=3D"https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258755" = class=3D"">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258755</a>,= I observed a</div><div class=3D"">strange behavior. The TCP ACKs from = FreeBSD host are too aggressive.</div><div class=3D""><br = class=3D""></div><div class=3D"">My setup is simple:</div><div = class=3D""> A = = B</div><div class=3D""> [ MacOS ] = <=3D=3D=3D=3D> [ FreeBSD VM ]</div><div = class=3D"">192.168.120.1 = 192.168.12.134 (disable tso and lro)</div><div class=3D"">While A = <--- B, i.e. A as server and B as client, the packets rate looks = good.</div><div class=3D""><br class=3D""></div><div class=3D"">One = session on B:</div><div class=3D""><br class=3D""></div><div = class=3D""><div class=3D"">root@:~ # iperf3 -c 192.168.120.1 -b = 10m</div><div class=3D"">Connecting to host 192.168.120.1, port = 5201</div><div class=3D"">[ 5] local 192.168.120.134 port 54459 = connected to 192.168.120.1 port 5201</div><div class=3D"">[ ID] Interval = Transfer Bitrate = Retr Cwnd</div><div class=3D"">[ 5] = 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec = 0 257 KBytes = </div><div class=3D"">[ 5] 1.00-2.00 sec = 1.25 MBytes 10.5 Mbits/sec 0 257 = KBytes </div><div class=3D"">[ 5] = 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec = 0 257 KBytes </div><div = class=3D"">[ 5] 3.00-4.00 sec 1.25 MBytes = 10.5 Mbits/sec 0 257 KBytes = </div><div class=3D"">[ 5] 4.00-5.00 = sec 1.12 MBytes 9.44 Mbits/sec 0 = 257 KBytes </div><div class=3D"">[ = 5] 5.00-6.00 sec 1.25 MBytes 10.5 = Mbits/sec 0 257 KBytes = </div><div class=3D"">[ 5] 6.00-7.00 sec = 1.12 MBytes 9.44 Mbits/sec 0 257 = KBytes </div><div class=3D"">[ 5] = 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec = 0 257 KBytes </div><div = class=3D"">[ 5] 8.00-9.00 sec 1.12 MBytes = 9.44 Mbits/sec 0 257 KBytes = </div><div class=3D"">[ 5] 9.00-10.00 = sec 1.25 MBytes 10.5 Mbits/sec 0 = 257 KBytes </div><div class=3D"">- - - - = - - - - - - - - - - - - - - - - - - - - -</div><div class=3D"">[ ID] = Interval Transfer = Bitrate Retr</div><div class=3D"">[ 5] = 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec = 0 = sender</div><div class=3D"">[ 5] 0.00-10.00 sec = 12.0 MBytes 10.1 Mbits/sec = receiver</div><div class=3D""><br = class=3D""></div><div class=3D"">iperf Done.</div></div><div = class=3D""><br class=3D""></div><div class=3D"">Another session on = B:</div><div class=3D""><br class=3D""></div><div class=3D""><div = class=3D"">root@:~ # netstat -w 1 -I vmx0</div><div class=3D""> = input = vmx0 output</div><div = class=3D""> packets errs idrops = bytes packets errs bytes = colls</div><div class=3D""> 0 = 0 0 0 = 0 0 = 0 0</div><div class=3D""> = 0 0 0 = 0 0 0 = 0 0</div><div class=3D""> = 342 0 0 = 22600 526 0 = 775724 0</div><div class=3D""> = 150 0 0 9900 = 851 0 1281454 = 0</div><div class=3D""> 109 = 0 0 7194 = 901 0 1357850 = 0</div><div class=3D""> 126 0 = 0 8316 828 = 0 1246632 0</div><div = class=3D""> 122 0 = 0 8052 910 = 0 1370780 0</div><div class=3D""> = 109 0 0 = 7194 819 0 = 1233702 0</div><div class=3D""> = 120 0 0 7920 = 910 0 1370780 = 0</div><div class=3D""> 110 = 0 0 7260 = 819 0 1233702 = 0</div><div class=3D""> 123 0 = 0 8118 910 = 0 1370780 0</div><div = class=3D""> 109 0 = 0 7194 819 = 0 1233702 0</div><div class=3D""> = 73 0 0 = 5088 465 0 = 686342 0</div><div class=3D""> = 0 0 0 = 0 0 0 = 0 0</div><div class=3D""> = 0 0 0 = 0 0 = 0 0 = 0</div></div><div class=3D""><br class=3D""></div><div class=3D""><br = class=3D""></div><div class=3D""><br class=3D""></div><div = class=3D"">=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D</div><div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""></div><div = class=3D"">While <span style=3D"caret-color: rgb(0, 0, 0); color: = rgb(0, 0, 0);" class=3D"">A ---> B, i.e. A as client and B as server, = the ACKs sent from B looks strange.</span></div><div class=3D""><br = class=3D""></div><div class=3D"">Session on A:</div><div class=3D""><br = class=3D""></div><div class=3D""><div class=3D"">% iperf3 -c = 192.168.120.134 -b 10m</div><div class=3D"">Connecting to host = 192.168.120.134, port 5201</div><div class=3D"">[ 5] local = 192.168.120.1 port 52370 connected to 192.168.120.134 port = 5201</div><div class=3D"">[ ID] Interval = Transfer Bitrate</div><div class=3D"">[ 5] = 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec = </div><div = class=3D"">[ 5] 1.00-2.00 sec 1.25 MBytes = 10.5 Mbits/sec = </div><div class=3D"">[ 5] 2.00-3.00 = sec 1.12 MBytes 9.44 Mbits/sec = </div><div class=3D"">[ 5] = 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec = </div><div = class=3D"">[ 5] 4.00-5.00 sec 1.12 MBytes = 9.44 Mbits/sec = </div><div class=3D"">[ 5] 5.00-6.00 = sec 1.25 MBytes 10.5 Mbits/sec = </div><div class=3D"">[ 5] = 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec = </div><div = class=3D"">[ 5] 7.00-8.00 sec 1.25 MBytes = 10.5 Mbits/sec = </div><div class=3D"">[ 5] 8.00-9.00 = sec 1.12 MBytes 9.44 Mbits/sec = </div><div class=3D"">[ 5] = 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec = </div><div = class=3D"">- - - - - - - - - - - - - - - - - - - - - - - - -</div><div = class=3D"">[ ID] Interval Transfer = Bitrate</div><div class=3D"">[ 5] 0.00-10.00 = sec 12.0 MBytes 10.1 Mbits/sec = sender</div><div class=3D"">[ = 5] 0.00-10.00 sec 12.0 MBytes 10.1 = Mbits/sec = receiver</div><div class=3D""><br class=3D""></div><div = class=3D"">iperf Done.</div><div class=3D""><br class=3D""></div><div = class=3D"">Session on B:</div><div class=3D""><br class=3D""></div><div = class=3D"">root@:~ # netstat -w 1 -I vmx0</div><div class=3D""> = input = vmx0 output</div><div = class=3D""> packets errs idrops = bytes packets errs bytes = colls</div><div class=3D""> 0 = 0 0 0 = 0 0 = 0 0</div><div class=3D""> = 0 0 0 = 0 0 0 = 0 0</div><div class=3D""> = 649 0 0 = 960562 330 0 = 21800 0</div><div class=3D""> = 819 0 0 1233702 = 415 0 27390 = 0</div><div class=3D""> 910 = 0 0 1370780 = 459 0 30294 = 0</div><div class=3D""> 819 0 = 0 1233702 415 = 0 27390 0</div><div = class=3D""> 910 0 = 0 1370780 459 0 = 30294 0</div><div class=3D""> = 910 0 0 = 1370780 460 0 = 30360 0</div><div class=3D""> = 819 0 0 1233702 = 414 0 27324 = 0</div><div class=3D""> 910 = 0 0 1370780 = 460 0 30360 = 0</div><div class=3D""> 819 0 = 0 1233702 414 = 0 27324 0</div><div = class=3D""> 910 0 = 0 1370780 460 0 = 30360 0</div><div class=3D""> = 285 0 0 = 412287 147 0 = 9981 0</div><div class=3D""> = 0 0 0 = 0 0 0 = 0 0</div><div = class=3D""> 0 0 = 0 0 = 0 0 0 = 0</div><div class=3D""> 0 = 0 0 0 = 0 0 = 0 0</div></div><div class=3D""><br = class=3D""></div><div class=3D""><br class=3D""></div><div class=3D"">The = ACK packets replied from B (the FreeBSD VM) are too aggressive. They = are</div><div class=3D"">about one half of TCP packets received from = A.</div><div class=3D""><br class=3D""></div><div class=3D"">I've tested = with different bitrates, from 10m to 300m, all behave the = same.</div><div class=3D"">Tested with baremetal FreeBSD 13.1 Box as B = (with intel em driver), the </div><div class=3D""><span = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" = class=3D"">bitrates is 1g</span>, also <span style=3D"caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);" class=3D"">behaves the = same.</span></div><div class=3D""><br class=3D""></div><div = class=3D"">Also tried different FreeBSD versions, 11.4, 12.3, stable/13 = and current/14 all </div><div class=3D""><span style=3D"caret-color: = rgb(0, 0, 0); color: rgb(0, 0, 0);" class=3D"">behave the = same.</span></div><div class=3D""><span style=3D"caret-color: rgb(0, 0, = 0); color: rgb(0, 0, 0);" class=3D""><br class=3D""></span></div><div = class=3D""><br class=3D""></div><div class=3D""><font color=3D"#000000" = class=3D""><span style=3D"caret-color: rgb(0, 0, 0);" class=3D"">My = question is, is that the expected </span></font><span = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" = class=3D"">behavior of current default TCP stack?</span></div><div = class=3D""><span style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);" class=3D""><br class=3D""></span></div><div class=3D""><br = class=3D""></div><br class=3D""><div class=3D""> <div>Best regards,</div><div>Zhenlei</div> </div> <br class=3D""></body></html>= --Apple-Mail=_F1C4CA8E-F708-46BD-B8F1-F2DE4E290809--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?75D35F36-7759-4168-ADBA-C2414F5B53BC>