Date: Tue, 14 Jun 2022 10:21:51 -0400 From: Mike Jakubik <mike.jakubik@swiftsmsgateway.com> To: "Hans Petter Selasky" <hps@selasky.org> Cc: "freebsd-net" <freebsd-net@FreeBSD.org> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) Message-ID: <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> In-Reply-To: <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_4634877_126180685.1655216511632 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Disabling rx/tx pause seems to produce higher peaks. [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.89 GBytes=C2=A0 16= .2 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.10 MBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.86 GBytes=C2=A0 15= .9 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.10 MBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 2.05 GBytes=C2=A0 17= .6 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 859 MBytes=C2= =A0 7.20 Gbits/sec=C2=A0=C2=A0 21=C2=A0=C2=A0=C2=A0 938 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 652 MBytes=C2= =A0 5.47 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 659 MBytes=C2= =A0 5.53 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.03 MBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 666 MBytes=C2= =A0 5.59 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.05 MBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 657 MBytes=C2= =A0 5.51 Gbits/sec=C2=A0=C2=A0 98=C2=A0=C2=A0=C2=A0 989 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 665 MBytes=C2= =A0 5.58 Gbits/sec=C2=A0 139=C2=A0=C2=A0=C2=A0 712 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 647 MBytes=C2=A0 5.= 43 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.02 MBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 650 MBytes=C2=A0 5.45 Gb= its/sec=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0=C2=A0 606 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.53 GBytes=C2=A0 13.1 Gbits/s= ec=C2=A0 358=C2=A0=C2=A0 1.07 MBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 2.10 GBytes=C2=A0 18.1 Gbits/s= ec=C2=A0 162=C2=A0=C2=A0=C2=A0 837 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 2.09 GBytes=C2=A0 18.0 Gbits/s= ec=C2=A0 332=C2=A0=C2=A0=C2=A0 838 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 2.43 GBytes=C2=A0 20.9 Gbits/s= ec=C2=A0 639=C2=A0=C2=A0=C2=A0 747 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 2.38 GBytes=C2=A0 20.4 Gbits/s= ec=C2=A0 612=C2=A0=C2=A0 1.02 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 2.25 GBytes=C2=A0 19.3 Gbits/s= ec=C2=A0 535=C2=A0=C2=A0 1.24 MBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 2.52 GBytes=C2=A0 21.6 Gbits/s= ec=C2=A0 818=C2=A0=C2=A0=C2=A0 423 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0 218=C2=A0=C2=A0=C2=A0 444 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0 114=C2=A0=C2=A0=C2=A0 859 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14.1 Gbits/s= ec=C2=A0 100=C2=A0=C2=A0=C2=A0 541 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 1.01 GBytes=C2=A0 8.67 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 639 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0=C2=A0 625 MBytes=C2=A0 5.24 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 648 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 630 MBytes=C2=A0 5.28 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 648 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 1.56 GBytes=C2=A0 13.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 702 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 1.78 GBytes=C2=A0 15.3 Gbits/s= ec=C2=A0 118=C2=A0=C2=A0=C2=A0 406 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.37 GBytes=C2=A0 11.8 Gbits/s= ec=C2=A0 105=C2=A0=C2=A0=C2=A0 890 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.82 GBytes=C2=A0 15.6 Gbits/s= ec=C2=A0 104=C2=A0=C2=A0=C2=A0 963 KBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.68 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.20 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.38 MBytes - - - - - - - - - - - - - - - - - - - - - - - - -=20 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 44.8 GBytes=C2=A0 12.8 Gb= its/sec=C2=A0 4477=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.01=C2=A0 sec=C2=A0 44.8 GBytes=C2=A0 12.8 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver After a few runs: [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 52152 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.91 GBytes=C2=A0 16= .4 Gbits/sec=C2=A0=C2=A0 67=C2=A0=C2=A0=C2=A0 606 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.78 GBytes=C2=A0 15= .3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.07 MBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 1.60 GBytes=C2=A0 13= .7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.54 MBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 1.61 GBytes=C2=A0 13= .8 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14= .3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14= .3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14= .1 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 1.70 GBytes=C2=A0 14= .6 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 1.72 GBytes=C2=A0 14= .8 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 1.85 GBytes=C2=A0 15.9 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 1.81 GBytes=C2=A0 15.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 1.18 GBytes=C2=A0 10.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 794 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 1.73 GBytes=C2=A0 14.8 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 1.73 GBytes=C2=A0 14.9 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 1.77 GBytes=C2=A0 15.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.80 GBytes=C2=A0 15.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 2.03 GBytes=C2=A0 17.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 1.88 GBytes=C2=A0 16.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 1.80 GBytes=C2=A0 15.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 24.00-25.01=C2=A0 sec=C2=A0 1.59 GBytes=C2=A0 13.6 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 25.01-26.00=C2=A0 sec=C2=A0 1.33 GBytes=C2=A0 11.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s= ec=C2=A0=C2=A0 97=C2=A0=C2=A0 1.01 MBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0=C2=A0 719 MBytes=C2=A0 6.03 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0=C2=A0 727 MBytes=C2=A0 6.10 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 49.3 GBytes=C2=A0 14.1 Gb= its/sec=C2=A0 164=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 49.3 GBytes=C2=A0 14.1 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver The CPU usage is rather low, the NIC is in the OCP port, so im sure thats d= esigned accordingly and the NIC is bound to numa0. CPU:=C2=A0 0.0% user,=C2=A0 0.0% nice,=C2=A0 0.5% system,=C2=A0 0.7% interr= upt, 98.8% idle =C2=A0 PID USERNAME=C2=A0=C2=A0=C2=A0 THR PRI NICE=C2=A0=C2=A0 SIZE=C2=A0= =C2=A0=C2=A0 RES STATE=C2=A0=C2=A0=C2=A0 C=C2=A0=C2=A0 TIME=C2=A0=C2=A0=C2= =A0 WCPU COMMAND=20 2195 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0 52= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 17M=C2=A0 6884K select=C2=A0 83=C2= =A0=C2=A0 0:14=C2=A0 27.99% iperf3 # vmstat -i -w1|grep mlx5 irq671: mlx5_core0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49969=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 47008 (this drops to about 14k with HW LRO enabled) The dump is rather large, i dont think i can attach it in the mailing list,= but if you wish to see it i can upload it somewhere. Thank You. ---- On Mon, 13 Jun 2022 16:42:59 -0400 Hans Petter Selasky <hps@selasky.or= g> wrote ---- Some ideas:=20 =20 Try to disable "rxpause,txpause" when setting the media.=20 =20 Keep HW LRO off for now, it doesn't work for large number of connections.= =20 =20 What is the CPU usage during test? Is iperf3 running on a CPU-core which=20 has direct access to the NIC's numa domain?=20 =20 Is the NIC installed in the "correct" PCI high-performance slot?=20 =20 There are some sysctl knobs which may tell where the problem is, if it's=20 PCI backpressure or something else.=20 =20 sysctl -a | grep diag_pci_enable=20 sysctl -a | grep diag_general_enable=20 =20 Set these two to 1, then run some traffic and dump all mce sysctls:=20 =20 sysctl -a | grep mce > dump.txt=20 =20 --HPS=20 =20 Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use o= f the addressee(s) and may contain information that is privileged or confid= ential. If you are not the intended recipient, or responsible for deliverin= g the information to the intended recipient, you are hereby notified that a= ny dissemination, distribution, printing or copying of this e-mail and any = attachments is strictly prohibited. If this e-mail and any attachments were= received in error, please notify the sender by reply e-mail and delete the= original message. ------=_Part_4634877_126180685.1655216511632 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head>= <meta content=3D"text/html;charset=3DUTF-8" http-equiv=3D"Content-Type"></h= ead><body ><div style=3D"font-family: Verdana, Arial, Helvetica, sans-serif= ; font-size: 10pt;"><div>Disabling rx/tx pause seems to produce higher peak= s.<br></div><div><br></div><div>[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 = <br></div><div>Connecting to host db-01, port 5201<br></div><div>[ 5]= local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5201<br></d= iv><div>[ ID] Interval  = ; Transfer Bitrate &nb= sp; Retr Cwnd<br></div><div>[ 5] = 0.00-1.00 sec 1.89 GBytes 16.2 Gbits/sec  = ; 0 1.10 MBytes<br></div><div>[ 5] 1.00= -2.00 sec 1.86 GBytes 15.9 Gbits/sec &nb= sp; 0 1.10 MBytes<br></div><div>[ 5] 2.00-3.0= 0 sec 2.05 GBytes 17.6 Gbits/sec = 0 1.11 MBytes<br></div><div>[ 5] 3.00-4.00&nb= sp; sec 859 MBytes 7.20 Gbits/sec 21&nb= sp; 938 KBytes<br></div><div>[ 5] 4.00-5.00&n= bsp; sec 652 MBytes 5.47 Gbits/sec  = ; 0 1.01 MBytes<br></div><div>[ 5] 5.00-6.00&= nbsp; sec 659 MBytes 5.53 Gbits/sec &nbs= p; 0 1.03 MBytes<br></div><div>[ 5] 6.00-7.00= sec 666 MBytes 5.59 Gbits/sec &nb= sp; 0 1.05 MBytes<br></div><div>[ 5] 7.00-8.0= 0 sec 657 MBytes 5.51 Gbits/sec 9= 8 989 KBytes<br></div><div>[ 5] 8.00-9.= 00 sec 665 MBytes 5.58 Gbits/sec 139&nb= sp; 712 KBytes<br></div><div>[ 5] 9.00-10.00&= nbsp; sec 647 MBytes 5.43 Gbits/sec 0&n= bsp; 1.02 MBytes<br></div><div>[ 5] 10.00-11.00 sec= 650 MBytes 5.45 Gbits/sec 4  = ; 606 KBytes<br></div><div>[ 5] 11.00-12.00 sec&nbs= p; 1.53 GBytes 13.1 Gbits/sec 358 1.07 MBytes<br></= div><div>[ 5] 12.00-13.00 sec 2.10 GBytes 18.= 1 Gbits/sec 162 837 KBytes<br></div><div>[ 5]= 13.00-14.00 sec 2.09 GBytes 18.0 Gbits/sec 3= 32 838 KBytes<br></div><div>[ 5] 14.00-15.00&= nbsp; sec 2.43 GBytes 20.9 Gbits/sec 639  = ; 747 KBytes<br></div><div>[ 5] 15.00-16.00 sec 2.3= 8 GBytes 20.4 Gbits/sec 612 1.02 MBytes<br></div><d= iv>[ 5] 16.00-17.00 sec 2.25 GBytes 19.3 Gbit= s/sec 535 1.24 MBytes<br></div><div>[ 5] 17.0= 0-18.00 sec 2.52 GBytes 21.6 Gbits/sec 818 &nb= sp; 423 KBytes<br></div><div>[ 5] 18.00-19.00 sec&n= bsp; 2.29 GBytes 19.7 Gbits/sec 218 444 KByte= s<br></div><div>[ 5] 19.00-20.00 sec 2.29 GBytes&nb= sp; 19.7 Gbits/sec 114 859 KBytes<br></div><div>[&n= bsp; 5] 20.00-21.00 sec 1.65 GBytes 14.1 Gbits/sec&= nbsp; 100 541 KBytes<br></div><div>[ 5] 21.00= -22.00 sec 1.01 GBytes 8.67 Gbits/sec 0= 639 KBytes<br></div><div>[ 5] 22.00-23.00&nb= sp; sec 625 MBytes 5.24 Gbits/sec 0&nbs= p; 648 KBytes<br></div><div>[ 5] 23.00-24.00 = sec 630 MBytes 5.28 Gbits/sec 0 &n= bsp; 648 KBytes<br></div><div>[ 5] 24.00-25.00 sec&= nbsp; 1.56 GBytes 13.4 Gbits/sec 0  = ; 702 KBytes<br></div><div>[ 5] 25.00-26.00 sec 1.7= 8 GBytes 15.3 Gbits/sec 118 406 KBytes<br></d= iv><div>[ 5] 26.00-27.00 sec 1.37 GBytes 11.8= Gbits/sec 105 890 KBytes<br></div><div>[ 5]&= nbsp; 27.00-28.00 sec 1.82 GBytes 15.6 Gbits/sec 10= 4 963 KBytes<br></div><div>[ 5] 28.00-29.00&n= bsp; sec 1.68 GBytes 14.4 Gbits/sec 0 &n= bsp; 1.20 MBytes<br></div><div>[ 5] 29.00-30.00 sec = 1.67 GBytes 14.4 Gbits/sec 0 1.38 MByt= es<br></div><div>- - - - - - - - - - - - - - - - - - - - - - - - - <br></di= v><div>[ ID] Interval = Transfer Bitrate &nbs= p; Retr<br></div><div>[ 5] 0.00-30.00&n= bsp; sec 44.8 GBytes 12.8 Gbits/sec 4477  = ; sender<br></div><di= v>[ 5] 0.00-30.01 sec 44.8 GBytes 12.8 = Gbits/sec = receiver<br></div><div><br></div><div>= After a few runs:<br></div><div><br></div><div>[root@db-02 ~]# iperf3 -i 1 = -t 30 -c db-01 <br></div><div>Connecting to host db-01, port 5201<br></div>= <div>[ 5] local 192.168.10.31 port 52152 connected to 192.168.10.30 p= ort 5201<br></div><div>[ ID] Interval &n= bsp; Transfer Bitrate = Retr Cwnd<br></div><div>[ = 5] 0.00-1.00 sec 1.91 GBytes 16.4 Gbits= /sec 67 606 KBytes<br></div><div>[ 5]&n= bsp; 1.00-2.00 sec 1.78 GBytes 15.3 Gbits/sec= 0 1.07 MBytes<br></div><div>[ 5] = 2.00-3.00 sec 1.60 GBytes 13.7 Gbits/sec&nbs= p; 0 1.54 MBytes<br></div><div>[ 5] &nbs= p; 3.00-4.00 sec 1.61 GBytes 13.8 Gbits/sec &n= bsp; 0 1.61 MBytes<br></div><div>[ 5] 4= .00-5.00 sec 1.66 GBytes 14.3 Gbits/sec = 0 1.61 MBytes<br></div><div>[ 5] 5.00-= 6.00 sec 1.67 GBytes 14.3 Gbits/sec &nbs= p; 0 1.61 MBytes<br></div><div>[ 5] 6.00-7.00= sec 1.65 GBytes 14.1 Gbits/sec 0= 1.61 MBytes<br></div><div>[ 5] 7.00-8.00&nbs= p; sec 1.70 GBytes 14.6 Gbits/sec 0&nbs= p; 1.61 MBytes<br></div><div>[ 5] 8.00-9.00 &n= bsp; sec 1.72 GBytes 14.8 Gbits/sec 0 &n= bsp; 1.61 MBytes<br></div><div>[ 5] 9.00-10.00 sec&= nbsp; 1.85 GBytes 15.9 Gbits/sec 0 1.61= MBytes<br></div><div>[ 5] 10.00-11.00 sec 1.81 GBy= tes 15.5 Gbits/sec 0 1.61 MBytes<br></d= iv><div>[ 5] 11.00-12.00 sec 1.67 GBytes 14.3= Gbits/sec 0 1.61 MBytes<br></div><div>[ = ; 5] 12.00-13.00 sec 1.66 GBytes 14.3 Gbits/sec&nbs= p; 0 1.61 MBytes<br></div><div>[ 5] 13.= 00-14.00 sec 1.83 GBytes 15.7 Gbits/sec = 0 1.61 MBytes<br></div><div>[ 5] 14.00-15.00 = sec 1.18 GBytes 10.1 Gbits/sec 0 = 794 KBytes<br></div><div>[ 5] 15.00-16.00 sec = ; 1.67 GBytes 14.4 Gbits/sec 0 1.60 MBy= tes<br></div><div>[ 5] 16.00-17.00 sec 1.73 GBytes&= nbsp; 14.8 Gbits/sec 0 1.60 MBytes<br></div><= div>[ 5] 17.00-18.00 sec 1.73 GBytes 14.9 Gbi= ts/sec 0 1.60 MBytes<br></div><div>[ 5]= 18.00-19.00 sec 1.83 GBytes 15.7 Gbits/sec &n= bsp; 0 1.61 MBytes<br></div><div>[ 5] 19.00-2= 0.00 sec 1.77 GBytes 15.2 Gbits/sec 0&n= bsp; 1.61 MBytes<br></div><div>[ 5] 20.00-21.00 sec= 1.80 GBytes 15.5 Gbits/sec 0 1.6= 1 MBytes<br></div><div>[ 5] 21.00-22.00 sec 2.03 GB= ytes 17.4 Gbits/sec 0 1.61 MBytes<br></= div><div>[ 5] 22.00-23.00 sec 1.88 GBytes 16.= 1 Gbits/sec 0 1.61 MBytes<br></div><div>[&nbs= p; 5] 23.00-24.00 sec 1.80 GBytes 15.5 Gbits/sec&nb= sp; 0 1.61 MBytes<br></div><div>[ 5] 24= .00-25.01 sec 1.59 GBytes 13.6 Gbits/sec  = ; 0 1.61 MBytes<br></div><div>[ 5] 25.01-26.00 = ; sec 1.33 GBytes 11.4 Gbits/sec 0  = ; 1.61 MBytes<br></div><div>[ 5] 26.00-27.00 sec 1.= 71 GBytes 14.7 Gbits/sec 0 1.61 MBytes<= br></div><div>[ 5] 27.00-28.00 sec 1.71 GBytes = ; 14.7 Gbits/sec 97 1.01 MBytes<br></div><div>[&nbs= p; 5] 28.00-29.00 sec 719 MBytes 6.03 Gbits/s= ec 0 1.01 MBytes<br></div><div>[ 5]&nbs= p; 29.00-30.00 sec 727 MBytes 6.10 Gbits/sec &= nbsp; 0 1.01 MBytes<br></div><div>- - - - - - - - - - - -= - - - - - - - - - - - - -<br></div><div>[ ID] Interval &n= bsp; Transfer B= itrate Retr<br></div><div>[= 5] 0.00-30.00 sec 49.3 GBytes 14.1 Gbi= ts/sec 164  = ; sender<br></div><div>[ 5] 0.00-30.00 = sec 49.3 GBytes 14.1 Gbits/sec &nb= sp; recei= ver<br></div><div><br></div><div><br></div><div><br></div><div>The CPU usag= e is rather low, the NIC is in the OCP port, so im sure thats designed acco= rdingly and the NIC is bound to numa0.<br></div><div><br></div><div>CPU:&nb= sp; 0.0% user, 0.0% nice, 0.5% system, 0.7% interrupt, 98= .8% idle<br></div><div><br></div><div> PID USERNAME = THR PRI NICE SIZE RES STATE  = ; C TIME WCPU COMMAND <br></div><div>2195 roo= t 1 52 &nb= sp; 0 17M 6884K select 83 0= :14 27.99% iperf3<br></div><div><br></div><div># vmstat -i -w1|grep m= lx5<br></div><div>irq671: mlx5_core0 &nb= sp; 49969 = 47008<br></div><div><br></div><div>(this drops to about = 14k with HW LRO enabled)<br></div><div><br></div><div>The dump is rather la= rge, i dont think i can attach it in the mailing list, but if you wish to s= ee it i can upload it somewhere.<br></div><div><br></div><div>Thank You.</d= iv><div><br></div><div><br></div><div class=3D"zmail_extra_hr" style=3D"bor= der-top: 1px solid rgb(204, 204, 204); height: 0px; margin-top: 10px; margi= n-bottom: 10px; line-height: 0px;"><br></div><div class=3D"zmail_extra" dat= a-zbluepencil-ignore=3D"true"><div><br></div><div id=3D"Zm-_Id_-Sgn1">---- = On Mon, 13 Jun 2022 16:42:59 -0400 <b>Hans Petter Selasky <hps@selasky.o= rg></b> wrote ----<br></div><div><br></div><blockquote style=3D"margin: = 0px;"><div><br>Some ideas: <br> <br>Try to disable "rxpause,txpause" when s= etting the media. <br> <br>Keep HW LRO off for now, it doesn't work for lar= ge number of connections. <br> <br>What is the CPU usage during test? Is ip= erf3 running on a CPU-core which <br>has direct access to the NIC's numa do= main? <br> <br>Is the NIC installed in the "correct" PCI high-performance s= lot? <br> <br>There are some sysctl knobs which may tell where the problem = is, if it's <br>PCI backpressure or something else. <br> <br>sysctl -a | gr= ep diag_pci_enable <br>sysctl -a | grep diag_general_enable <br> <br>Set th= ese two to 1, then run some traffic and dump all mce sysctls: <br> <br>sysc= tl -a | grep mce > dump.txt <br> <br>--HPS <br> <br></div></blockquote><= /div><div><br></div><div class=3D"zmail_signature_below"><div id=3D"Zm-_Id_= -Sgn" data-zbluepencil-ignore=3D"true" data-sigid=3D"7886077000000096004"><= div>Mike Jakubik<br></div><div><a target=3D"_blank" href=3D"https://www.swi= ftsmsgateway.com/">https://www.swiftsmsgateway.com/</a><br></div><div><br><= /div><div><span class=3D"size" style=3D"font-size:10px">Disclaimer: This e-= mail and any attachments are intended only for the use of the addressee(s) = and may contain information that is privileged or confidential. If you are = not the intended recipient, or responsible for delivering the information t= o the intended recipient, you are hereby notified that any dissemination, d= istribution, printing or copying of this e-mail and any attachments is stri= ctly prohibited. If this e-mail and any attachments were received in error,= please notify the sender by reply e-mail and delete the original message.<= /span><br></div></div></div><div><br></div></div><br></body></html> ------=_Part_4634877_126180685.1655216511632--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?18162979a8f.e81f383a1466900.9104319828015733292>