Date: Thu, 16 Jun 2022 20:38:55 -0400 From: Mike Jakubik <mike.jakubik@swiftsmsgateway.com> To: "freebsd-net" <freebsd-net@FreeBSD.org> Cc: "Alexander V. Chernikov" <melifaro@ipfw.ru> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) Message-ID: <1816f19416a.b852ce5b2189187.4131912798685804323@swiftsmsgateway.com> In-Reply-To: <C54B2F28-D02F-4589-82D7-1C84705C2C23@ipfw.ru> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> <1816e469bdf.126cdb81b2139485.369352368493375815@swiftsmsgateway.com> <C54B2F28-D02F-4589-82D7-1C84705C2C23@ipfw.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_6963461_1051991644.1655426335083 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi, I believe you hit the nail on the head! I am now getting consistent high sp= eeds, even higher than on Linux! Is this a problem with the scheduler? Shou= ld someone in that area of expertise be made aware of this? More importantl= y i guess, would this affect real world performance, these servers will be = running RabbitMQ (it uses quite a bit of bandwidth) and PostgreSQL w/ repli= cation. [root@db-01 ~]# cpuset -l 1 iperf3 -s=20 ----------------------------------------------------------- Server listening on 5201 (test #1) ----------------------------------------------------------- [root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 22445 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 2.17 GBytes=C2=A0 18= .7 Gbits/sec=C2=A0 263=C2=A0=C2=A0 1.13 MBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 2.11 GBytes=C2=A0 18= .1 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.14 MBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 2.46 GBytes=C2=A0 21= .2 Gbits/sec=C2=A0 696=C2=A0=C2=A0=C2=A0 893 KBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 2.49 GBytes=C2=A0 21= .4 Gbits/sec=C2=A0 857=C2=A0=C2=A0=C2=A0 447 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 2.36 GBytes=C2=A0 20= .3 Gbits/sec=C2=A0 281=C2=A0=C2=A0=C2=A0 566 KBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 2.36 GBytes=C2=A0 20= .3 Gbits/sec=C2=A0 348=C2=A0=C2=A0=C2=A0 593 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 2.28 GBytes=C2=A0 19= .6 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 694 KBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 2.20 GBytes=C2=A0 18= .9 Gbits/sec=C2=A0 165=C2=A0=C2=A0=C2=A0 611 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 2.26 GBytes=C2=A0 19= .5 Gbits/sec=C2=A0 183=C2=A0=C2=A0=C2=A0 541 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 2.26 GBytes=C2=A0 19.4 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 625 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 2.27 GBytes=C2=A0 19.5 Gbits/s= ec=C2=A0 165=C2=A0=C2=A0=C2=A0 608 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 2.22 GBytes=C2=A0 19.0 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 679 KBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 2.22 GBytes=C2=A0 19.0 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 726 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 2.25 GBytes=C2=A0 19.3 Gbits/s= ec=C2=A0 174=C2=A0=C2=A0=C2=A0 610 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 2.19 GBytes=C2=A0 18.8 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 644 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 2.20 GBytes=C2=A0 18.9 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 704 KBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 2.18 GBytes=C2=A0 18.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 784 KBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 2.18 GBytes=C2=A0 18.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 820 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 2.19 GBytes=C2=A0 18.8 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 874 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 2.19 GBytes=C2=A0 18.8 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 902 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 2.23 GBytes=C2=A0 19.1 Gbits/s= ec=C2=A0 202=C2=A0=C2=A0=C2=A0 661 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 2.30 GBytes=C2=A0 19.8 Gbits/s= ec=C2=A0 171=C2=A0=C2=A0=C2=A0 894 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 2.33 GBytes=C2=A0 20.0 Gbits/s= ec=C2=A0 341=C2=A0=C2=A0=C2=A0 409 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0 167=C2=A0=C2=A0=C2=A0 594 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 2.28 GBytes=C2=A0 19.6 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 714 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0 166=C2=A0=C2=A0=C2=A0 604 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 2.26 GBytes=C2=A0 19.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 705 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.86 GBytes=C2=A0 16.0 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 763 KBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 2.25 GBytes=C2=A0 19.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 816 KBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 2.24 GBytes=C2=A0 19.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 869 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 67.4 GBytes=C2=A0 19.3 Gb= its/sec=C2=A0 4179=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 67.4 GBytes=C2=A0 19.3 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver iperf Done. [root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 51998 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.99 GBytes=C2=A0 17= .1 Gbits/sec=C2=A0 276=C2=A0=C2=A0=C2=A0 456 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 2.00 GBytes=C2=A0 17= .2 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 593 KBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 2.27 GBytes=C2=A0 19= .5 Gbits/sec=C2=A0 166=C2=A0=C2=A0=C2=A0 556 KBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 2.31 GBytes=C2=A0 19= .8 Gbits/sec=C2=A0 186=C2=A0=C2=A0=C2=A0 558 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 2.31 GBytes=C2=A0 19= .8 Gbits/sec=C2=A0 168=C2=A0=C2=A0=C2=A0 558 KBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19= .6 Gbits/sec=C2=A0 182=C2=A0=C2=A0=C2=A0 545 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19= .7 Gbits/sec=C2=A0 171=C2=A0=C2=A0=C2=A0 507 KBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19= .7 Gbits/sec=C2=A0 180=C2=A0=C2=A0=C2=A0 517 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 2.30 GBytes=C2=A0 19= .7 Gbits/sec=C2=A0 174=C2=A0=C2=A0=C2=A0 568 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 2.28 GBytes=C2=A0 19.6 Gb= its/sec=C2=A0 178=C2=A0=C2=A0=C2=A0 434 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 2.26 GBytes=C2=A0 19.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 583 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 2.31 GBytes=C2=A0 19.8 Gbits/s= ec=C2=A0 180=C2=A0=C2=A0=C2=A0 443 KBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 2.32 GBytes=C2=A0 19.9 Gbits/s= ec=C2=A0 166=C2=A0=C2=A0=C2=A0 564 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 2.35 GBytes=C2=A0 20.2 Gbits/s= ec=C2=A0 350=C2=A0=C2=A0=C2=A0 424 KBytes [=C2=A0 5]=C2=A0 14.00-15.02=C2=A0 sec=C2=A0 1.49 GBytes=C2=A0 12.5 Gbits/s= ec=C2=A0 175=C2=A0=C2=A0 1.41 KBytes [=C2=A0 5]=C2=A0 15.02-16.00=C2=A0 sec=C2=A0 1.73 GBytes=C2=A0 15.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.12 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 2.22 GBytes=C2=A0 19.1 Gbits/s= ec=C2=A0 171=C2=A0=C2=A0=C2=A0 784 KBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 2.34 GBytes=C2=A0 20.1 Gbits/s= ec=C2=A0 348=C2=A0=C2=A0=C2=A0 436 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 2.34 GBytes=C2=A0 20.1 Gbits/s= ec=C2=A0 350=C2=A0=C2=A0=C2=A0 547 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 2.31 GBytes=C2=A0 19.9 Gbits/s= ec=C2=A0 173=C2=A0=C2=A0=C2=A0 595 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 2.35 GBytes=C2=A0 20.2 Gbits/s= ec=C2=A0 360=C2=A0=C2=A0=C2=A0 439 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 2.41 GBytes=C2=A0 20.7 Gbits/s= ec=C2=A0 525=C2=A0=C2=A0=C2=A0 580 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 2.43 GBytes=C2=A0 20.9 Gbits/s= ec=C2=A0 510=C2=A0=C2=A0=C2=A0 397 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 2.38 GBytes=C2=A0 20.4 Gbits/s= ec=C2=A0 532=C2=A0=C2=A0=C2=A0 533 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 2.37 GBytes=C2=A0 20.4 Gbits/s= ec=C2=A0 344=C2=A0=C2=A0=C2=A0 547 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 2.36 GBytes=C2=A0 20.2 Gbits/s= ec=C2=A0 354=C2=A0=C2=A0=C2=A0 389 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 2.30 GBytes=C2=A0 19.8 Gbits/s= ec=C2=A0 165=C2=A0=C2=A0=C2=A0 592 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 2.30 GBytes=C2=A0 19.8 Gbits/s= ec=C2=A0 173=C2=A0=C2=A0=C2=A0 584 KBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 2.27 GBytes=C2=A0 19.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 701 KBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 790 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 67.4 GBytes=C2=A0 19.3 Gb= its/sec=C2=A0 6557=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 67.4 GBytes=C2=A0 19.3 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver iperf Done. Thank You! ---- On Thu, 16 Jun 2022 17:00:25 -0400 Alexander V. Chernikov <melifaro@ip= fw.ru> wrote ---- > On 16 Jun 2022, at 21:48, Mike Jakubik <mailto:mike.jakubik@swiftsmsgatew= ay.com> wrote:=20 >=20 > After multiple tests and tweaks i believe the issue is not with the HW or= Numa related (Infinity fabric should do around 32GB) but rather with FreeB= SD TCP/IP stack. It's like it cant figure itself out properly for the speed= that the HW can do, i keep getting widely varying results when testing. Be= low is an example of two tests, with about a 15 second break in between the= two.=20 Does pinning iperf to a specific CPU core (or range) address the variety pa= rt? e.g. cpuset -l 1 perf =E2=80=A6=20 The output you shared above shows CPU#83 as the core iperf is running on. J= ust wondering whether the scheduler migrates iperf too often, thrashing the= caches. ------=_Part_6963461_1051991644.1655426335083 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head>= <meta content=3D"text/html;charset=3DUTF-8" http-equiv=3D"Content-Type"></h= ead><body ><div style=3D"font-family: Verdana, Arial, Helvetica, sans-serif= ; font-size: 10pt;"><div>Hi,<br></div><div><br></div><div>I believe you hit= the nail on the head! I am now getting consistent high speeds, even higher= than on Linux! Is this a problem with the scheduler? Should someone in tha= t area of expertise be made aware of this? More importantly i guess, would = this affect real world performance, these servers will be running RabbitMQ = (it uses quite a bit of bandwidth) and PostgreSQL w/ replication.<br></div>= <div><br></div><div>[root@db-01 ~]# cpuset -l 1 iperf3 -s <br></div><div>--= ---------------------------------------------------------<br></div><div>Ser= ver listening on 5201 (test #1)<br></div><div>-----------------------------= ------------------------------<br></div><div><br></div><div>[root@db-02 ~]#= cpuset -l 1 iperf3 -i 1 -t 30 -c db-01 <br></div><div>Connecting to host d= b-01, port 5201<br></div><div>[ 5] local 192.168.10.31 port 22445 con= nected to 192.168.10.30 port 5201<br></div><div>[ ID] Interval &= nbsp; Transfer &= nbsp; Bitrate Retr Cw= nd<br></div><div>[ 5] 0.00-1.00 sec 2.1= 7 GBytes 18.7 Gbits/sec 263 1.13 MBytes<br></div><d= iv>[ 5] 1.00-2.00 sec 2.11 GBytes = 18.1 Gbits/sec 0 1.14 MBytes<br></div><div>[= 5] 2.00-3.00 sec 2.46 GBytes 21.= 2 Gbits/sec 696 893 KBytes<br></div><div>[ 5]= 3.00-4.00 sec 2.49 GBytes 21.4 Gbits/s= ec 857 447 KBytes<br></div><div>[ 5] &nb= sp; 4.00-5.00 sec 2.36 GBytes 20.3 Gbits/sec = 281 566 KBytes<br></div><div>[ 5] 5.00-= 6.00 sec 2.36 GBytes 20.3 Gbits/sec 348 = 593 KBytes<br></div><div>[ 5] 6.00-7.00 = ; sec 2.28 GBytes 19.6 Gbits/sec 0 = ; 694 KBytes<br></div><div>[ 5] 7.00-8.00&nbs= p; sec 2.20 GBytes 18.9 Gbits/sec 165 &n= bsp; 611 KBytes<br></div><div>[ 5] 8.00-9.00 = sec 2.26 GBytes 19.5 Gbits/sec 183 541 = KBytes<br></div><div>[ 5] 9.00-10.00 sec 2.26= GBytes 19.4 Gbits/sec 0 625 KByt= es<br></div><div>[ 5] 10.00-11.00 sec 2.27 GBytes&n= bsp; 19.5 Gbits/sec 165 608 KBytes<br></div><div>[&= nbsp; 5] 11.00-12.00 sec 2.22 GBytes 19.0 Gbits/sec= 0 679 KBytes<br></div><div>[ 5]&= nbsp; 12.00-13.00 sec 2.22 GBytes 19.0 Gbits/sec &nb= sp; 0 726 KBytes<br></div><div>[ 5] 13.= 00-14.00 sec 2.25 GBytes 19.3 Gbits/sec 174 &n= bsp; 610 KBytes<br></div><div>[ 5] 14.00-15.00 sec&= nbsp; 2.19 GBytes 18.8 Gbits/sec 0  = ; 644 KBytes<br></div><div>[ 5] 15.00-16.00 sec 2.2= 0 GBytes 18.9 Gbits/sec 0 704 KBy= tes<br></div><div>[ 5] 16.00-17.00 sec 2.18 GBytes&= nbsp; 18.7 Gbits/sec 0 784 KBytes<br></= div><div>[ 5] 17.00-18.00 sec 2.18 GBytes 18.= 7 Gbits/sec 0 820 KBytes<br></div><div>= [ 5] 18.00-19.00 sec 2.19 GBytes 18.8 Gbits/s= ec 0 874 KBytes<br></div><div>[ 5= ] 19.00-20.00 sec 2.19 GBytes 18.8 Gbits/sec &= nbsp; 0 902 KBytes<br></div><div>[ 5] 2= 0.00-21.00 sec 2.23 GBytes 19.1 Gbits/sec 202 = 661 KBytes<br></div><div>[ 5] 21.00-22.00 se= c 2.30 GBytes 19.8 Gbits/sec 171 894 KB= ytes<br></div><div>[ 5] 22.00-23.00 sec 2.33 GBytes= 20.0 Gbits/sec 341 409 KBytes<br></div><div>= [ 5] 23.00-24.00 sec 2.29 GBytes 19.7 Gbits/s= ec 167 594 KBytes<br></div><div>[ 5] 24= .00-25.00 sec 2.28 GBytes 19.6 Gbits/sec  = ; 0 714 KBytes<br></div><div>[ 5] 25.00-26.00= sec 2.29 GBytes 19.7 Gbits/sec 166 &nbs= p; 604 KBytes<br></div><div>[ 5] 26.00-27.00 sec 2.= 26 GBytes 19.4 Gbits/sec 0 705 KB= ytes<br></div><div>[ 5] 27.00-28.00 sec 1.86 GBytes= 16.0 Gbits/sec 0 763 KBytes<br><= /div><div>[ 5] 28.00-29.00 sec 2.25 GBytes 19= .3 Gbits/sec 0 816 KBytes<br></div><div= >[ 5] 29.00-30.00 sec 2.24 GBytes 19.2 Gbits/= sec 0 869 KBytes<br></div><div>- - - - = - - - - - - - - - - - - - - - - - - - - -<br></div><div>[ ID] Interval = ; Transfer  = ; Bitrate Retr<= br></div><div>[ 5] 0.00-30.00 sec 67.4 GBytes= 19.3 Gbits/sec 4179 &= nbsp; sender<br></div><div>[ 5] 0= .00-30.00 sec 67.4 GBytes 19.3 Gbits/sec  = ; &n= bsp; receiver<br></div><div><br></div><div>iperf Done.<br></div><div>= [root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01<br></div><div>Connec= ting to host db-01, port 5201<br></div><div>[ 5] local 192.168.10.31 = port 51998 connected to 192.168.10.30 port 5201<br></div><div>[ ID] Interva= l Transfer = ; Bitrate = Retr Cwnd<br></div><div>[ 5] 0.00-1.00 = sec 1.99 GBytes 17.1 Gbits/sec 276 456= KBytes<br></div><div>[ 5] 1.00-2.00 sec = ; 2.00 GBytes 17.2 Gbits/sec 0 59= 3 KBytes<br></div><div>[ 5] 2.00-3.00 sec&nbs= p; 2.27 GBytes 19.5 Gbits/sec 166 556 KBytes<= br></div><div>[ 5] 3.00-4.00 sec 2.31 G= Bytes 19.8 Gbits/sec 186 558 KBytes<br></div>= <div>[ 5] 4.00-5.00 sec 2.31 GBytes&nbs= p; 19.8 Gbits/sec 168 558 KBytes<br></div><div>[&nb= sp; 5] 5.00-6.00 sec 2.29 GBytes 19.6 G= bits/sec 182 545 KBytes<br></div><div>[ 5]&nb= sp; 6.00-7.00 sec 2.29 GBytes 19.7 Gbits/sec&= nbsp; 171 507 KBytes<br></div><div>[ 5] = 7.00-8.00 sec 2.29 GBytes 19.7 Gbits/sec 180= 517 KBytes<br></div><div>[ 5] 8.00-9.0= 0 sec 2.30 GBytes 19.7 Gbits/sec 174 &nb= sp; 568 KBytes<br></div><div>[ 5] 9.00-10.00 = sec 2.28 GBytes 19.6 Gbits/sec 178 434 = KBytes<br></div><div>[ 5] 10.00-11.00 sec 2.26 GByt= es 19.4 Gbits/sec 0 583 KBytes<br= ></div><div>[ 5] 11.00-12.00 sec 2.31 GBytes = 19.8 Gbits/sec 180 443 KBytes<br></div><div>[ = 5] 12.00-13.00 sec 2.32 GBytes 19.9 Gbits/sec = ; 166 564 KBytes<br></div><div>[ 5] 13.00-14.= 00 sec 2.35 GBytes 20.2 Gbits/sec 350 &n= bsp; 424 KBytes<br></div><div>[ 5] 14.00-15.02 sec = 1.49 GBytes 12.5 Gbits/sec 175 1.41 KBytes<br></div= ><div>[ 5] 15.02-16.00 sec 1.73 GBytes 15.2 G= bits/sec 0 1.12 MBytes<br></div><div>[ = 5] 16.00-17.00 sec 2.22 GBytes 19.1 Gbits/sec = 171 784 KBytes<br></div><div>[ 5] 17.00-18.0= 0 sec 2.34 GBytes 20.1 Gbits/sec 348 &nb= sp; 436 KBytes<br></div><div>[ 5] 18.00-19.00 sec 2= .34 GBytes 20.1 Gbits/sec 350 547 KBytes<br><= /div><div>[ 5] 19.00-20.00 sec 2.31 GBytes 19= .9 Gbits/sec 173 595 KBytes<br></div><div>[ 5= ] 20.00-21.00 sec 2.35 GBytes 20.2 Gbits/sec = 360 439 KBytes<br></div><div>[ 5] 21.00-22.00= sec 2.41 GBytes 20.7 Gbits/sec 525 &nbs= p; 580 KBytes<br></div><div>[ 5] 22.00-23.00 sec 2.= 43 GBytes 20.9 Gbits/sec 510 397 KBytes<br></= div><div>[ 5] 23.00-24.00 sec 2.38 GBytes 20.= 4 Gbits/sec 532 533 KBytes<br></div><div>[ 5]= 24.00-25.00 sec 2.37 GBytes 20.4 Gbits/sec 3= 44 547 KBytes<br></div><div>[ 5] 25.00-26.00&= nbsp; sec 2.36 GBytes 20.2 Gbits/sec 354  = ; 389 KBytes<br></div><div>[ 5] 26.00-27.00 sec 2.3= 0 GBytes 19.8 Gbits/sec 165 592 KBytes<br></d= iv><div>[ 5] 27.00-28.00 sec 2.30 GBytes 19.8= Gbits/sec 173 584 KBytes<br></div><div>[ 5]&= nbsp; 28.00-29.00 sec 2.27 GBytes 19.5 Gbits/sec &nb= sp; 0 701 KBytes<br></div><div>[ 5] 29.= 00-30.00 sec 2.29 GBytes 19.7 Gbits/sec = 0 790 KBytes<br></div><div>- - - - - - - - - - - - - - -= - - - - - - - - - -<br></div><div>[ ID] Interval &n= bsp; Transfer Bitrate= Retr<br></div><div>[ = 5] 0.00-30.00 sec 67.4 GBytes 19.3 Gbits/sec= 6557 &nbs= p; sender<br></div><div>[ 5] 0.00-30.00 sec&n= bsp; 67.4 GBytes 19.3 Gbits/sec &n= bsp; receiver<b= r></div><div><br></div><div>iperf Done.<br></div><div><br></div><div><br></= div><div>Thank You!</div><div><br></div><div class=3D"zmail_extra_hr" style= =3D"border-top: 1px solid rgb(204, 204, 204); height: 0px; margin-top: 10px= ; margin-bottom: 10px; line-height: 0px;"><br></div><div class=3D"zmail_ext= ra" data-zbluepencil-ignore=3D"true"><div><br></div><div id=3D"Zm-_Id_-Sgn1= ">---- On Thu, 16 Jun 2022 17:00:25 -0400 <b>Alexander V. Chernikov <mel= ifaro@ipfw.ru></b> wrote ----<br></div><div><br></div><blockquote style= =3D"margin: 0px;"><div><br><br>> On 16 Jun 2022, at 21:48, Mike Jakubik = <<a href=3D"mailto:mike.jakubik@swiftsmsgateway.com" target=3D"_blank">m= ike.jakubik@swiftsmsgateway.com</a>> wrote: <br>> <br>> After mult= iple tests and tweaks i believe the issue is not with the HW or Numa relate= d (Infinity fabric should do around 32GB) but rather with FreeBSD TCP/IP st= ack. It's like it cant figure itself out properly for the speed that the HW= can do, i keep getting widely varying results when testing. Below is an ex= ample of two tests, with about a 15 second break in between the two. <br>Do= es pinning iperf to a specific CPU core (or range) address the variety part= ? e.g. cpuset -l 1 perf =E2=80=A6 <br>The output you shared above shows CPU= #83 as the core iperf is running on. Just wondering whether the scheduler m= igrates iperf too often, thrashing the caches. <br> <br></div></blockquote>= </div><div><br></div></div><br></body></html> ------=_Part_6963461_1051991644.1655426335083--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1816f19416a.b852ce5b2189187.4131912798685804323>