Date: Thu, 16 Jun 2022 20:38:55 -0400 From: Mike Jakubik <mike.jakubik@swiftsmsgateway.com> To: "freebsd-net" <freebsd-net@FreeBSD.org> Cc: "Alexander V. Chernikov" <melifaro@ipfw.ru> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) Message-ID: <1816f19416a.b852ce5b2189187.4131912798685804323@swiftsmsgateway.com> In-Reply-To: <C54B2F28-D02F-4589-82D7-1C84705C2C23@ipfw.ru> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> <1816e469bdf.126cdb81b2139485.369352368493375815@swiftsmsgateway.com> <C54B2F28-D02F-4589-82D7-1C84705C2C23@ipfw.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hi, I believe you hit the nail on the head! I am now getting consistent high speeds, even higher than on Linux! Is this a problem with the scheduler? Should someone in that area of expertise be made aware of this? More importantly i guess, would this affect real world performance, these servers will be running RabbitMQ (it uses quite a bit of bandwidth) and PostgreSQL w/ replication. [root@db-01 ~]# cpuset -l 1 iperf3 -s ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- [root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [ 5] local 192.168.10.31 port 22445 connected to 192.168.10.30 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 2.17 GBytes 18.7 Gbits/sec 263 1.13 MBytes [ 5] 1.00-2.00 sec 2.11 GBytes 18.1 Gbits/sec 0 1.14 MBytes [ 5] 2.00-3.00 sec 2.46 GBytes 21.2 Gbits/sec 696 893 KBytes [ 5] 3.00-4.00 sec 2.49 GBytes 21.4 Gbits/sec 857 447 KBytes [ 5] 4.00-5.00 sec 2.36 GBytes 20.3 Gbits/sec 281 566 KBytes [ 5] 5.00-6.00 sec 2.36 GBytes 20.3 Gbits/sec 348 593 KBytes [ 5] 6.00-7.00 sec 2.28 GBytes 19.6 Gbits/sec 0 694 KBytes [ 5] 7.00-8.00 sec 2.20 GBytes 18.9 Gbits/sec 165 611 KBytes [ 5] 8.00-9.00 sec 2.26 GBytes 19.5 Gbits/sec 183 541 KBytes [ 5] 9.00-10.00 sec 2.26 GBytes 19.4 Gbits/sec 0 625 KBytes [ 5] 10.00-11.00 sec 2.27 GBytes 19.5 Gbits/sec 165 608 KBytes [ 5] 11.00-12.00 sec 2.22 GBytes 19.0 Gbits/sec 0 679 KBytes [ 5] 12.00-13.00 sec 2.22 GBytes 19.0 Gbits/sec 0 726 KBytes [ 5] 13.00-14.00 sec 2.25 GBytes 19.3 Gbits/sec 174 610 KBytes [ 5] 14.00-15.00 sec 2.19 GBytes 18.8 Gbits/sec 0 644 KBytes [ 5] 15.00-16.00 sec 2.20 GBytes 18.9 Gbits/sec 0 704 KBytes [ 5] 16.00-17.00 sec 2.18 GBytes 18.7 Gbits/sec 0 784 KBytes [ 5] 17.00-18.00 sec 2.18 GBytes 18.7 Gbits/sec 0 820 KBytes [ 5] 18.00-19.00 sec 2.19 GBytes 18.8 Gbits/sec 0 874 KBytes [ 5] 19.00-20.00 sec 2.19 GBytes 18.8 Gbits/sec 0 902 KBytes [ 5] 20.00-21.00 sec 2.23 GBytes 19.1 Gbits/sec 202 661 KBytes [ 5] 21.00-22.00 sec 2.30 GBytes 19.8 Gbits/sec 171 894 KBytes [ 5] 22.00-23.00 sec 2.33 GBytes 20.0 Gbits/sec 341 409 KBytes [ 5] 23.00-24.00 sec 2.29 GBytes 19.7 Gbits/sec 167 594 KBytes [ 5] 24.00-25.00 sec 2.28 GBytes 19.6 Gbits/sec 0 714 KBytes [ 5] 25.00-26.00 sec 2.29 GBytes 19.7 Gbits/sec 166 604 KBytes [ 5] 26.00-27.00 sec 2.26 GBytes 19.4 Gbits/sec 0 705 KBytes [ 5] 27.00-28.00 sec 1.86 GBytes 16.0 Gbits/sec 0 763 KBytes [ 5] 28.00-29.00 sec 2.25 GBytes 19.3 Gbits/sec 0 816 KBytes [ 5] 29.00-30.00 sec 2.24 GBytes 19.2 Gbits/sec 0 869 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 67.4 GBytes 19.3 Gbits/sec 4179 sender [ 5] 0.00-30.00 sec 67.4 GBytes 19.3 Gbits/sec receiver iperf Done. [root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [ 5] local 192.168.10.31 port 51998 connected to 192.168.10.30 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.99 GBytes 17.1 Gbits/sec 276 456 KBytes [ 5] 1.00-2.00 sec 2.00 GBytes 17.2 Gbits/sec 0 593 KBytes [ 5] 2.00-3.00 sec 2.27 GBytes 19.5 Gbits/sec 166 556 KBytes [ 5] 3.00-4.00 sec 2.31 GBytes 19.8 Gbits/sec 186 558 KBytes [ 5] 4.00-5.00 sec 2.31 GBytes 19.8 Gbits/sec 168 558 KBytes [ 5] 5.00-6.00 sec 2.29 GBytes 19.6 Gbits/sec 182 545 KBytes [ 5] 6.00-7.00 sec 2.29 GBytes 19.7 Gbits/sec 171 507 KBytes [ 5] 7.00-8.00 sec 2.29 GBytes 19.7 Gbits/sec 180 517 KBytes [ 5] 8.00-9.00 sec 2.30 GBytes 19.7 Gbits/sec 174 568 KBytes [ 5] 9.00-10.00 sec 2.28 GBytes 19.6 Gbits/sec 178 434 KBytes [ 5] 10.00-11.00 sec 2.26 GBytes 19.4 Gbits/sec 0 583 KBytes [ 5] 11.00-12.00 sec 2.31 GBytes 19.8 Gbits/sec 180 443 KBytes [ 5] 12.00-13.00 sec 2.32 GBytes 19.9 Gbits/sec 166 564 KBytes [ 5] 13.00-14.00 sec 2.35 GBytes 20.2 Gbits/sec 350 424 KBytes [ 5] 14.00-15.02 sec 1.49 GBytes 12.5 Gbits/sec 175 1.41 KBytes [ 5] 15.02-16.00 sec 1.73 GBytes 15.2 Gbits/sec 0 1.12 MBytes [ 5] 16.00-17.00 sec 2.22 GBytes 19.1 Gbits/sec 171 784 KBytes [ 5] 17.00-18.00 sec 2.34 GBytes 20.1 Gbits/sec 348 436 KBytes [ 5] 18.00-19.00 sec 2.34 GBytes 20.1 Gbits/sec 350 547 KBytes [ 5] 19.00-20.00 sec 2.31 GBytes 19.9 Gbits/sec 173 595 KBytes [ 5] 20.00-21.00 sec 2.35 GBytes 20.2 Gbits/sec 360 439 KBytes [ 5] 21.00-22.00 sec 2.41 GBytes 20.7 Gbits/sec 525 580 KBytes [ 5] 22.00-23.00 sec 2.43 GBytes 20.9 Gbits/sec 510 397 KBytes [ 5] 23.00-24.00 sec 2.38 GBytes 20.4 Gbits/sec 532 533 KBytes [ 5] 24.00-25.00 sec 2.37 GBytes 20.4 Gbits/sec 344 547 KBytes [ 5] 25.00-26.00 sec 2.36 GBytes 20.2 Gbits/sec 354 389 KBytes [ 5] 26.00-27.00 sec 2.30 GBytes 19.8 Gbits/sec 165 592 KBytes [ 5] 27.00-28.00 sec 2.30 GBytes 19.8 Gbits/sec 173 584 KBytes [ 5] 28.00-29.00 sec 2.27 GBytes 19.5 Gbits/sec 0 701 KBytes [ 5] 29.00-30.00 sec 2.29 GBytes 19.7 Gbits/sec 0 790 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 67.4 GBytes 19.3 Gbits/sec 6557 sender [ 5] 0.00-30.00 sec 67.4 GBytes 19.3 Gbits/sec receiver iperf Done. Thank You! ---- On Thu, 16 Jun 2022 17:00:25 -0400 Alexander V. Chernikov <melifaro@ipfw.ru> wrote ---- > On 16 Jun 2022, at 21:48, Mike Jakubik <mailto:mike.jakubik@swiftsmsgateway.com> wrote: > > After multiple tests and tweaks i believe the issue is not with the HW or Numa related (Infinity fabric should do around 32GB) but rather with FreeBSD TCP/IP stack. It's like it cant figure itself out properly for the speed that the HW can do, i keep getting widely varying results when testing. Below is an example of two tests, with about a 15 second break in between the two. Does pinning iperf to a specific CPU core (or range) address the variety part? e.g. cpuset -l 1 perf … The output you shared above shows CPU#83 as the core iperf is running on. Just wondering whether the scheduler migrates iperf too often, thrashing the caches. [-- Attachment #2 --] <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt;"><div>Hi,<br></div><div><br></div><div>I believe you hit the nail on the head! I am now getting consistent high speeds, even higher than on Linux! Is this a problem with the scheduler? Should someone in that area of expertise be made aware of this? More importantly i guess, would this affect real world performance, these servers will be running RabbitMQ (it uses quite a bit of bandwidth) and PostgreSQL w/ replication.<br></div><div><br></div><div>[root@db-01 ~]# cpuset -l 1 iperf3 -s <br></div><div>-----------------------------------------------------------<br></div><div>Server listening on 5201 (test #1)<br></div><div>-----------------------------------------------------------<br></div><div><br></div><div>[root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01 <br></div><div>Connecting to host db-01, port 5201<br></div><div>[ 5] local 192.168.10.31 port 22445 connected to 192.168.10.30 port 5201<br></div><div>[ ID] Interval Transfer Bitrate Retr Cwnd<br></div><div>[ 5] 0.00-1.00 sec 2.17 GBytes 18.7 Gbits/sec 263 1.13 MBytes<br></div><div>[ 5] 1.00-2.00 sec 2.11 GBytes 18.1 Gbits/sec 0 1.14 MBytes<br></div><div>[ 5] 2.00-3.00 sec 2.46 GBytes 21.2 Gbits/sec 696 893 KBytes<br></div><div>[ 5] 3.00-4.00 sec 2.49 GBytes 21.4 Gbits/sec 857 447 KBytes<br></div><div>[ 5] 4.00-5.00 sec 2.36 GBytes 20.3 Gbits/sec 281 566 KBytes<br></div><div>[ 5] 5.00-6.00 sec 2.36 GBytes 20.3 Gbits/sec 348 593 KBytes<br></div><div>[ 5] 6.00-7.00 sec 2.28 GBytes 19.6 Gbits/sec 0 694 KBytes<br></div><div>[ 5] 7.00-8.00 sec 2.20 GBytes 18.9 Gbits/sec 165 611 KBytes<br></div><div>[ 5] 8.00-9.00 sec 2.26 GBytes 19.5 Gbits/sec 183 541 KBytes<br></div><div>[ 5] 9.00-10.00 sec 2.26 GBytes 19.4 Gbits/sec 0 625 KBytes<br></div><div>[ 5] 10.00-11.00 sec 2.27 GBytes 19.5 Gbits/sec 165 608 KBytes<br></div><div>[ 5] 11.00-12.00 sec 2.22 GBytes 19.0 Gbits/sec 0 679 KBytes<br></div><div>[ 5] 12.00-13.00 sec 2.22 GBytes 19.0 Gbits/sec 0 726 KBytes<br></div><div>[ 5] 13.00-14.00 sec 2.25 GBytes 19.3 Gbits/sec 174 610 KBytes<br></div><div>[ 5] 14.00-15.00 sec 2.19 GBytes 18.8 Gbits/sec 0 644 KBytes<br></div><div>[ 5] 15.00-16.00 sec 2.20 GBytes 18.9 Gbits/sec 0 704 KBytes<br></div><div>[ 5] 16.00-17.00 sec 2.18 GBytes 18.7 Gbits/sec 0 784 KBytes<br></div><div>[ 5] 17.00-18.00 sec 2.18 GBytes 18.7 Gbits/sec 0 820 KBytes<br></div><div>[ 5] 18.00-19.00 sec 2.19 GBytes 18.8 Gbits/sec 0 874 KBytes<br></div><div>[ 5] 19.00-20.00 sec 2.19 GBytes 18.8 Gbits/sec 0 902 KBytes<br></div><div>[ 5] 20.00-21.00 sec 2.23 GBytes 19.1 Gbits/sec 202 661 KBytes<br></div><div>[ 5] 21.00-22.00 sec 2.30 GBytes 19.8 Gbits/sec 171 894 KBytes<br></div><div>[ 5] 22.00-23.00 sec 2.33 GBytes 20.0 Gbits/sec 341 409 KBytes<br></div><div>[ 5] 23.00-24.00 sec 2.29 GBytes 19.7 Gbits/sec 167 594 KBytes<br></div><div>[ 5] 24.00-25.00 sec 2.28 GBytes 19.6 Gbits/sec 0 714 KBytes<br></div><div>[ 5] 25.00-26.00 sec 2.29 GBytes 19.7 Gbits/sec 166 604 KBytes<br></div><div>[ 5] 26.00-27.00 sec 2.26 GBytes 19.4 Gbits/sec 0 705 KBytes<br></div><div>[ 5] 27.00-28.00 sec 1.86 GBytes 16.0 Gbits/sec 0 763 KBytes<br></div><div>[ 5] 28.00-29.00 sec 2.25 GBytes 19.3 Gbits/sec 0 816 KBytes<br></div><div>[ 5] 29.00-30.00 sec 2.24 GBytes 19.2 Gbits/sec 0 869 KBytes<br></div><div>- - - - - - - - - - - - - - - - - - - - - - - - -<br></div><div>[ ID] Interval Transfer Bitrate Retr<br></div><div>[ 5] 0.00-30.00 sec 67.4 GBytes 19.3 Gbits/sec 4179 sender<br></div><div>[ 5] 0.00-30.00 sec 67.4 GBytes 19.3 Gbits/sec receiver<br></div><div><br></div><div>iperf Done.<br></div><div>[root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01<br></div><div>Connecting to host db-01, port 5201<br></div><div>[ 5] local 192.168.10.31 port 51998 connected to 192.168.10.30 port 5201<br></div><div>[ ID] Interval Transfer Bitrate Retr Cwnd<br></div><div>[ 5] 0.00-1.00 sec 1.99 GBytes 17.1 Gbits/sec 276 456 KBytes<br></div><div>[ 5] 1.00-2.00 sec 2.00 GBytes 17.2 Gbits/sec 0 593 KBytes<br></div><div>[ 5] 2.00-3.00 sec 2.27 GBytes 19.5 Gbits/sec 166 556 KBytes<br></div><div>[ 5] 3.00-4.00 sec 2.31 GBytes 19.8 Gbits/sec 186 558 KBytes<br></div><div>[ 5] 4.00-5.00 sec 2.31 GBytes 19.8 Gbits/sec 168 558 KBytes<br></div><div>[ 5] 5.00-6.00 sec 2.29 GBytes 19.6 Gbits/sec 182 545 KBytes<br></div><div>[ 5] 6.00-7.00 sec 2.29 GBytes 19.7 Gbits/sec 171 507 KBytes<br></div><div>[ 5] 7.00-8.00 sec 2.29 GBytes 19.7 Gbits/sec 180 517 KBytes<br></div><div>[ 5] 8.00-9.00 sec 2.30 GBytes 19.7 Gbits/sec 174 568 KBytes<br></div><div>[ 5] 9.00-10.00 sec 2.28 GBytes 19.6 Gbits/sec 178 434 KBytes<br></div><div>[ 5] 10.00-11.00 sec 2.26 GBytes 19.4 Gbits/sec 0 583 KBytes<br></div><div>[ 5] 11.00-12.00 sec 2.31 GBytes 19.8 Gbits/sec 180 443 KBytes<br></div><div>[ 5] 12.00-13.00 sec 2.32 GBytes 19.9 Gbits/sec 166 564 KBytes<br></div><div>[ 5] 13.00-14.00 sec 2.35 GBytes 20.2 Gbits/sec 350 424 KBytes<br></div><div>[ 5] 14.00-15.02 sec 1.49 GBytes 12.5 Gbits/sec 175 1.41 KBytes<br></div><div>[ 5] 15.02-16.00 sec 1.73 GBytes 15.2 Gbits/sec 0 1.12 MBytes<br></div><div>[ 5] 16.00-17.00 sec 2.22 GBytes 19.1 Gbits/sec 171 784 KBytes<br></div><div>[ 5] 17.00-18.00 sec 2.34 GBytes 20.1 Gbits/sec 348 436 KBytes<br></div><div>[ 5] 18.00-19.00 sec 2.34 GBytes 20.1 Gbits/sec 350 547 KBytes<br></div><div>[ 5] 19.00-20.00 sec 2.31 GBytes 19.9 Gbits/sec 173 595 KBytes<br></div><div>[ 5] 20.00-21.00 sec 2.35 GBytes 20.2 Gbits/sec 360 439 KBytes<br></div><div>[ 5] 21.00-22.00 sec 2.41 GBytes 20.7 Gbits/sec 525 580 KBytes<br></div><div>[ 5] 22.00-23.00 sec 2.43 GBytes 20.9 Gbits/sec 510 397 KBytes<br></div><div>[ 5] 23.00-24.00 sec 2.38 GBytes 20.4 Gbits/sec 532 533 KBytes<br></div><div>[ 5] 24.00-25.00 sec 2.37 GBytes 20.4 Gbits/sec 344 547 KBytes<br></div><div>[ 5] 25.00-26.00 sec 2.36 GBytes 20.2 Gbits/sec 354 389 KBytes<br></div><div>[ 5] 26.00-27.00 sec 2.30 GBytes 19.8 Gbits/sec 165 592 KBytes<br></div><div>[ 5] 27.00-28.00 sec 2.30 GBytes 19.8 Gbits/sec 173 584 KBytes<br></div><div>[ 5] 28.00-29.00 sec 2.27 GBytes 19.5 Gbits/sec 0 701 KBytes<br></div><div>[ 5] 29.00-30.00 sec 2.29 GBytes 19.7 Gbits/sec 0 790 KBytes<br></div><div>- - - - - - - - - - - - - - - - - - - - - - - - -<br></div><div>[ ID] Interval Transfer Bitrate Retr<br></div><div>[ 5] 0.00-30.00 sec 67.4 GBytes 19.3 Gbits/sec 6557 sender<br></div><div>[ 5] 0.00-30.00 sec 67.4 GBytes 19.3 Gbits/sec receiver<br></div><div><br></div><div>iperf Done.<br></div><div><br></div><div><br></div><div>Thank You!</div><div><br></div><div class="zmail_extra_hr" style="border-top: 1px solid rgb(204, 204, 204); height: 0px; margin-top: 10px; margin-bottom: 10px; line-height: 0px;"><br></div><div class="zmail_extra" data-zbluepencil-ignore="true"><div><br></div><div id="Zm-_Id_-Sgn1">---- On Thu, 16 Jun 2022 17:00:25 -0400 <b>Alexander V. Chernikov <melifaro@ipfw.ru></b> wrote ----<br></div><div><br></div><blockquote style="margin: 0px;"><div><br><br>> On 16 Jun 2022, at 21:48, Mike Jakubik <<a href="mailto:mike.jakubik@swiftsmsgateway.com" target="_blank">mike.jakubik@swiftsmsgateway.com</a>> wrote: <br>> <br>> After multiple tests and tweaks i believe the issue is not with the HW or Numa related (Infinity fabric should do around 32GB) but rather with FreeBSD TCP/IP stack. It's like it cant figure itself out properly for the speed that the HW can do, i keep getting widely varying results when testing. Below is an example of two tests, with about a 15 second break in between the two. <br>Does pinning iperf to a specific CPU core (or range) address the variety part? e.g. cpuset -l 1 perf … <br>The output you shared above shows CPU#83 as the core iperf is running on. Just wondering whether the scheduler migrates iperf too often, thrashing the caches. <br> <br></div></blockquote></div><div><br></div></div><br></body></html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1816f19416a.b852ce5b2189187.4131912798685804323>