Date: Fri, 21 Oct 2022 22:19:19 +0800 From: Zhenlei Huang <zlei.huang@gmail.com> To: freebsd-net@freebsd.org Subject: Too aggressive TCP ACKs Message-ID: <75D35F36-7759-4168-ADBA-C2414F5B53BC@gmail.com>
index | next in thread | raw e-mail
[-- Attachment #1 --] Hi, While I was repeating https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258755, I observed a strange behavior. The TCP ACKs from FreeBSD host are too aggressive. My setup is simple: A B [ MacOS ] <====> [ FreeBSD VM ] 192.168.120.1 192.168.12.134 (disable tso and lro) While A <--- B, i.e. A as server and B as client, the packets rate looks good. One session on B: root@:~ # iperf3 -c 192.168.120.1 -b 10m Connecting to host 192.168.120.1, port 5201 [ 5] local 192.168.120.134 port 54459 connected to 192.168.120.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes [ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes [ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes [ 5] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes [ 5] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes [ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes [ 5] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes [ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes [ 5] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes [ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec receiver iperf Done. Another session on B: root@:~ # netstat -w 1 -I vmx0 input vmx0 output packets errs idrops bytes packets errs bytes colls 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 342 0 0 22600 526 0 775724 0 150 0 0 9900 851 0 1281454 0 109 0 0 7194 901 0 1357850 0 126 0 0 8316 828 0 1246632 0 122 0 0 8052 910 0 1370780 0 109 0 0 7194 819 0 1233702 0 120 0 0 7920 910 0 1370780 0 110 0 0 7260 819 0 1233702 0 123 0 0 8118 910 0 1370780 0 109 0 0 7194 819 0 1233702 0 73 0 0 5088 465 0 686342 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ================================================================ While A ---> B, i.e. A as client and B as server, the ACKs sent from B looks strange. Session on A: % iperf3 -c 192.168.120.134 -b 10m Connecting to host 192.168.120.134, port 5201 [ 5] local 192.168.120.1 port 52370 connected to 192.168.120.134 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec [ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec [ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec [ 5] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec [ 5] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec [ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec [ 5] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec [ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec [ 5] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec [ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec sender [ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec receiver iperf Done. Session on B: root@:~ # netstat -w 1 -I vmx0 input vmx0 output packets errs idrops bytes packets errs bytes colls 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 649 0 0 960562 330 0 21800 0 819 0 0 1233702 415 0 27390 0 910 0 0 1370780 459 0 30294 0 819 0 0 1233702 415 0 27390 0 910 0 0 1370780 459 0 30294 0 910 0 0 1370780 460 0 30360 0 819 0 0 1233702 414 0 27324 0 910 0 0 1370780 460 0 30360 0 819 0 0 1233702 414 0 27324 0 910 0 0 1370780 460 0 30360 0 285 0 0 412287 147 0 9981 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 The ACK packets replied from B (the FreeBSD VM) are too aggressive. They are about one half of TCP packets received from A. I've tested with different bitrates, from 10m to 300m, all behave the same. Tested with baremetal FreeBSD 13.1 Box as B (with intel em driver), the bitrates is 1g, also behaves the same. Also tried different FreeBSD versions, 11.4, 12.3, stable/13 and current/14 all behave the same. My question is, is that the expected behavior of current default TCP stack? Best regards, Zhenlei [-- Attachment #2 --] <html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Hi,</div><div class=""><br class=""></div><div class="">While I was repeating <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258755" class="">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258755</a>, I observed a</div><div class="">strange behavior. The TCP ACKs from FreeBSD host are too aggressive.</div><div class=""><br class=""></div><div class="">My setup is simple:</div><div class=""> A B</div><div class=""> [ MacOS ] <====> [ FreeBSD VM ]</div><div class="">192.168.120.1 192.168.12.134 (disable tso and lro)</div><div class="">While A <--- B, i.e. A as server and B as client, the packets rate looks good.</div><div class=""><br class=""></div><div class="">One session on B:</div><div class=""><br class=""></div><div class=""><div class="">root@:~ # iperf3 -c 192.168.120.1 -b 10m</div><div class="">Connecting to host 192.168.120.1, port 5201</div><div class="">[ 5] local 192.168.120.134 port 54459 connected to 192.168.120.1 port 5201</div><div class="">[ ID] Interval Transfer Bitrate Retr Cwnd</div><div class="">[ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes </div><div class="">[ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes </div><div class="">[ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes </div><div class="">[ 5] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes </div><div class="">[ 5] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes </div><div class="">[ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes </div><div class="">[ 5] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes </div><div class="">[ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes </div><div class="">[ 5] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec 0 257 KBytes </div><div class="">[ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec 0 257 KBytes </div><div class="">- - - - - - - - - - - - - - - - - - - - - - - - -</div><div class="">[ ID] Interval Transfer Bitrate Retr</div><div class="">[ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec 0 sender</div><div class="">[ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec receiver</div><div class=""><br class=""></div><div class="">iperf Done.</div></div><div class=""><br class=""></div><div class="">Another session on B:</div><div class=""><br class=""></div><div class=""><div class="">root@:~ # netstat -w 1 -I vmx0</div><div class=""> input vmx0 output</div><div class=""> packets errs idrops bytes packets errs bytes colls</div><div class=""> 0 0 0 0 0 0 0 0</div><div class=""> 0 0 0 0 0 0 0 0</div><div class=""> 342 0 0 22600 526 0 775724 0</div><div class=""> 150 0 0 9900 851 0 1281454 0</div><div class=""> 109 0 0 7194 901 0 1357850 0</div><div class=""> 126 0 0 8316 828 0 1246632 0</div><div class=""> 122 0 0 8052 910 0 1370780 0</div><div class=""> 109 0 0 7194 819 0 1233702 0</div><div class=""> 120 0 0 7920 910 0 1370780 0</div><div class=""> 110 0 0 7260 819 0 1233702 0</div><div class=""> 123 0 0 8118 910 0 1370780 0</div><div class=""> 109 0 0 7194 819 0 1233702 0</div><div class=""> 73 0 0 5088 465 0 686342 0</div><div class=""> 0 0 0 0 0 0 0 0</div><div class=""> 0 0 0 0 0 0 0 0</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">================================================================</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">While <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">A ---> B, i.e. A as client and B as server, the ACKs sent from B looks strange.</span></div><div class=""><br class=""></div><div class="">Session on A:</div><div class=""><br class=""></div><div class=""><div class="">% iperf3 -c 192.168.120.134 -b 10m</div><div class="">Connecting to host 192.168.120.134, port 5201</div><div class="">[ 5] local 192.168.120.1 port 52370 connected to 192.168.120.134 port 5201</div><div class="">[ ID] Interval Transfer Bitrate</div><div class="">[ 5] 0.00-1.00 sec 1.25 MBytes 10.5 Mbits/sec </div><div class="">[ 5] 1.00-2.00 sec 1.25 MBytes 10.5 Mbits/sec </div><div class="">[ 5] 2.00-3.00 sec 1.12 MBytes 9.44 Mbits/sec </div><div class="">[ 5] 3.00-4.00 sec 1.25 MBytes 10.5 Mbits/sec </div><div class="">[ 5] 4.00-5.00 sec 1.12 MBytes 9.44 Mbits/sec </div><div class="">[ 5] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec </div><div class="">[ 5] 6.00-7.00 sec 1.12 MBytes 9.44 Mbits/sec </div><div class="">[ 5] 7.00-8.00 sec 1.25 MBytes 10.5 Mbits/sec </div><div class="">[ 5] 8.00-9.00 sec 1.12 MBytes 9.44 Mbits/sec </div><div class="">[ 5] 9.00-10.00 sec 1.25 MBytes 10.5 Mbits/sec </div><div class="">- - - - - - - - - - - - - - - - - - - - - - - - -</div><div class="">[ ID] Interval Transfer Bitrate</div><div class="">[ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec sender</div><div class="">[ 5] 0.00-10.00 sec 12.0 MBytes 10.1 Mbits/sec receiver</div><div class=""><br class=""></div><div class="">iperf Done.</div><div class=""><br class=""></div><div class="">Session on B:</div><div class=""><br class=""></div><div class="">root@:~ # netstat -w 1 -I vmx0</div><div class=""> input vmx0 output</div><div class=""> packets errs idrops bytes packets errs bytes colls</div><div class=""> 0 0 0 0 0 0 0 0</div><div class=""> 0 0 0 0 0 0 0 0</div><div class=""> 649 0 0 960562 330 0 21800 0</div><div class=""> 819 0 0 1233702 415 0 27390 0</div><div class=""> 910 0 0 1370780 459 0 30294 0</div><div class=""> 819 0 0 1233702 415 0 27390 0</div><div class=""> 910 0 0 1370780 459 0 30294 0</div><div class=""> 910 0 0 1370780 460 0 30360 0</div><div class=""> 819 0 0 1233702 414 0 27324 0</div><div class=""> 910 0 0 1370780 460 0 30360 0</div><div class=""> 819 0 0 1233702 414 0 27324 0</div><div class=""> 910 0 0 1370780 460 0 30360 0</div><div class=""> 285 0 0 412287 147 0 9981 0</div><div class=""> 0 0 0 0 0 0 0 0</div><div class=""> 0 0 0 0 0 0 0 0</div><div class=""> 0 0 0 0 0 0 0 0</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">The ACK packets replied from B (the FreeBSD VM) are too aggressive. They are</div><div class="">about one half of TCP packets received from A.</div><div class=""><br class=""></div><div class="">I've tested with different bitrates, from 10m to 300m, all behave the same.</div><div class="">Tested with baremetal FreeBSD 13.1 Box as B (with intel em driver), the </div><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">bitrates is 1g</span>, also <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">behaves the same.</span></div><div class=""><br class=""></div><div class="">Also tried different FreeBSD versions, 11.4, 12.3, stable/13 and current/14 all </div><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">behave the same.</span></div><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></span></div><div class=""><br class=""></div><div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class="">My question is, is that the expected </span></font><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">behavior of current default TCP stack?</span></div><div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class=""></span></div><div class=""><br class=""></div><br class=""><div class=""> <div>Best regards,</div><div>Zhenlei</div> </div> <br class=""></body></html>help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?75D35F36-7759-4168-ADBA-C2414F5B53BC>