Skip site navigation (1)Skip section navigation (2) 
Date:      Tue, 14 Jun 2022 10:36:06 -0400
From:      Mike Jakubik <mike.jakubik@swiftsmsgateway.com>
To:        "mikejakubik" <mike.jakubik@swiftsmsgateway.com>
Cc:        "Hans Petter Selasky" <hps@selasky.org>, "freebsd-net" <freebsd-net@FreeBSD.org>
Subject:   Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)
Message-ID:  <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com>
In-Reply-To: <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com>
References:  <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com>
next in thread | previous in thread | raw e-mail | index | archive | help 
------=_Part_4650864_1970601159.1655217366007
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Actually, i believe its the disabling to HW LRO that makes the difference (=
i disabled it and rx/tx pause previously). With rx/tx pause on and LRO off =
i get similar results. The throughput is still very sporadic though.



Connecting to host db-01, port 5201=20

[=C2=A0 5] local 192.168.10.31 port 59055 connected to 192.168.10.30 port 5=
201

[ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd

[=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15=
.7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes

[=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15=
.7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes

[=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 2.06 GBytes=C2=A0 17=
.7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.12 MBytes

[=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18=
.5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.12 MBytes

[=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18=
.4 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.12 MBytes

[=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18=
.5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.13 MBytes

[=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18=
.5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.13 MBytes

[=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18=
.5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.13 MBytes

[=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18=
.5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.13 MBytes

[=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18.5 Gb=
its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.13 MBytes

[=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 1.72 GBytes=C2=A0 14.7 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.14 MBytes

[=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.18 GBytes=C2=A0 10.1 Gbits/s=
ec=C2=A0 131=C2=A0=C2=A0=C2=A0 932 KBytes

[=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0=C2=A0 739 MBytes=C2=A0 6.20 Gb=
its/sec=C2=A0 352=C2=A0=C2=A0=C2=A0 913 KBytes

[=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0=C2=A0 724 MBytes=C2=A0 6.07 Gb=
its/sec=C2=A0 142=C2=A0=C2=A0=C2=A0 636 KBytes

[=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0=C2=A0 759 MBytes=C2=A0 6.37 Gb=
its/sec=C2=A0 246=C2=A0=C2=A0=C2=A0 748 KBytes

[=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0=C2=A0 748 MBytes=C2=A0 6.27 Gb=
its/sec=C2=A0=C2=A0 22=C2=A0=C2=A0=C2=A0 962 KBytes

[=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0=C2=A0 753 MBytes=C2=A0 6.32 Gb=
its/sec=C2=A0 170=C2=A0=C2=A0=C2=A0 711 KBytes

[=C2=A0 5]=C2=A0 17.00-18.01=C2=A0 sec=C2=A0=C2=A0 690 MBytes=C2=A0 5.76 Gb=
its/sec=C2=A0 389=C2=A0=C2=A0=C2=A0 981 KBytes

[=C2=A0 5]=C2=A0 18.01-19.00=C2=A0 sec=C2=A0=C2=A0 771 MBytes=C2=A0 6.50 Gb=
its/sec=C2=A0 145=C2=A0=C2=A0=C2=A0 896 KBytes

[=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0=C2=A0 741 MBytes=C2=A0 6.22 Gb=
its/sec=C2=A0=C2=A0 13=C2=A0=C2=A0=C2=A0 952 KBytes

[=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0=C2=A0 743 MBytes=C2=A0 6.23 Gb=
its/sec=C2=A0=C2=A0 86=C2=A0=C2=A0=C2=A0 985 KBytes

[=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0=C2=A0 751 MBytes=C2=A0 6.30 Gb=
its/sec=C2=A0 245=C2=A0=C2=A0=C2=A0 944 KBytes

[=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0=C2=A0 724 MBytes=C2=A0 6.07 Gb=
its/sec=C2=A0 178=C2=A0=C2=A0=C2=A0 774 KBytes

[=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 728 MBytes=C2=A0 6.11 Gb=
its/sec=C2=A0 348=C2=A0=C2=A0=C2=A0 546 KBytes

[=C2=A0 5]=C2=A0 24.00-25.01=C2=A0 sec=C2=A0=C2=A0 622 MBytes=C2=A0 5.18 Gb=
its/sec=C2=A0 215=C2=A0=C2=A0=C2=A0 842 KBytes

[=C2=A0 5]=C2=A0 25.01-26.00=C2=A0 sec=C2=A0=C2=A0 920 MBytes=C2=A0 7.78 Gb=
its/sec=C2=A0 359=C2=A0=C2=A0 1.03 MBytes

[=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.02 GBytes=C2=A0 8.80 Gbits/s=
ec=C2=A0 385=C2=A0=C2=A0 1011 KBytes

[=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0=C2=A0 999 MBytes=C2=A0 8.38 Gb=
its/sec=C2=A0 161=C2=A0=C2=A0=C2=A0 757 KBytes

[=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.68 GBytes=C2=A0 14.4 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.03 MBytes

[=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14.2 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.51 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 Retr

[=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 39.2 GBytes=C2=A0 11.2 Gb=
its/sec=C2=A0 3587=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 sender

[=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 39.2 GBytes=C2=A0 11.2 Gb=
its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver





[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20

Connecting to host db-01, port 5201

[=C2=A0 5] local 192.168.10.31 port 60931 connected to 192.168.10.30 port 5=
201

[ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd

[=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 931 MBytes=C2=
=A0 7.81 Gbits/sec=C2=A0 146=C2=A0=C2=A0=C2=A0 689 KBytes

[=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 674 MBytes=C2=
=A0 5.66 Gbits/sec=C2=A0 267=C2=A0=C2=A0=C2=A0 575 KBytes

[=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 641 MBytes=C2=
=A0 5.38 Gbits/sec=C2=A0 270=C2=A0=C2=A0=C2=A0 485 KBytes

[=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 668 MBytes=C2=
=A0 5.60 Gbits/sec=C2=A0 221=C2=A0=C2=A0=C2=A0 801 KBytes

[=C2=A0 5]=C2=A0=C2=A0 4.00-5.01=C2=A0=C2=A0 sec=C2=A0=C2=A0 532 MBytes=C2=
=A0 4.40 Gbits/sec=C2=A0 208=C2=A0=C2=A0=C2=A0 981 KBytes

[=C2=A0 5]=C2=A0=C2=A0 5.01-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 838 MBytes=C2=
=A0 7.13 Gbits/sec=C2=A0 291=C2=A0=C2=A0=C2=A0 949 KBytes

[=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 703 MBytes=C2=
=A0 5.90 Gbits/sec=C2=A0 178=C2=A0=C2=A0=C2=A0 628 KBytes

[=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 696 MBytes=C2=
=A0 5.84 Gbits/sec=C2=A0=C2=A0 85=C2=A0=C2=A0=C2=A0 928 KBytes

[=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 1.06 GBytes=C2=A0 9.=
14 Gbits/sec=C2=A0=C2=A0 66=C2=A0=C2=A0=C2=A0 565 KBytes

[=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 975 MBytes=C2=A0 8.=
18 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 801 KBytes

[=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 679 MBytes=C2=A0 5.70 Gb=
its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 859 KBytes

[=C2=A0 5]=C2=A0 11.00-12.01=C2=A0 sec=C2=A0=C2=A0 628 MBytes=C2=A0 5.23 Gb=
its/sec=C2=A0 123=C2=A0=C2=A0 1.12 MBytes

[=C2=A0 5]=C2=A0 12.01-13.00=C2=A0 sec=C2=A0=C2=A0 731 MBytes=C2=A0 6.16 Gb=
its/sec=C2=A0 311=C2=A0=C2=A0=C2=A0 864 KBytes

[=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0=C2=A0 696 MBytes=C2=A0 5.86 Gb=
its/sec=C2=A0=C2=A0 39=C2=A0=C2=A0=C2=A0 798 KBytes

[=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0=C2=A0 914 MBytes=C2=A0 7.66 Gb=
its/sec=C2=A0=C2=A0 65=C2=A0=C2=A0 1024 KBytes

[=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14.2 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.47 MBytes

[=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0=C2=A0 955 MBytes=C2=A0 8.01 Gb=
its/sec=C2=A0 222=C2=A0=C2=A0=C2=A0 260 KBytes

[=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0=C2=A0 769 MBytes=C2=A0 6.45 Gb=
its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 520 KBytes

[=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0=C2=A0 781 MBytes=C2=A0 6.55 Gb=
its/sec=C2=A0 119=C2=A0=C2=A0=C2=A0 432 KBytes

[=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0=C2=A0 779 MBytes=C2=A0 6.54 Gb=
its/sec=C2=A0 148=C2=A0=C2=A0=C2=A0 816 KBytes

[=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0=C2=A0 771 MBytes=C2=A0 6.47 Gb=
its/sec=C2=A0 127=C2=A0=C2=A0=C2=A0 965 KBytes

[=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0=C2=A0 745 MBytes=C2=A0 6.25 Gb=
its/sec=C2=A0 337=C2=A0=C2=A0=C2=A0 927 KBytes

[=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0=C2=A0 637 MBytes=C2=A0 5.35 Gb=
its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 938 KBytes

[=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 626 MBytes=C2=A0 5.25 Gb=
its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 938 KBytes

[=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 1.96 GBytes=C2=A0 16.8 Gbits/s=
ec=C2=A0 185=C2=A0=C2=A0=C2=A0 862 KBytes

[=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s=
ec=C2=A0=C2=A0 22=C2=A0=C2=A0=C2=A0 945 KBytes

[=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 2.14 GBytes=C2=A0 18.3 Gbits/s=
ec=C2=A0 365=C2=A0=C2=A0=C2=A0 593 KBytes

[=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.56 GBytes=C2=A0 13.4 Gbits/s=
ec=C2=A0=C2=A0 33=C2=A0=C2=A0 1.23 MBytes

[=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.4 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14.2 Gbits/s=
ec=C2=A0=C2=A0 35=C2=A0=C2=A0 1.14 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 Retr

[=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 29.4 GBytes=C2=A0 8.42 Gb=
its/sec=C2=A0 3863=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 sender

[=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 29.4 GBytes=C2=A0 8.42 Gb=
its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver











---- On Tue, 14 Jun 2022 10:21:51 -0400 Mike Jakubik <mike.jakubik@swiftsms=
gateway.com> wrote ----



Disabling rx/tx pause seems to produce higher peaks.



[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20

Connecting to host db-01, port 5201

[=C2=A0 5] local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5=
201

[ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd

[=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.89 GBytes=C2=A0 16=
.2 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.10 MBytes

[=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.86 GBytes=C2=A0 15=
.9 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.10 MBytes

[=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 2.05 GBytes=C2=A0 17=
.6 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes

[=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 859 MBytes=C2=
=A0 7.20 Gbits/sec=C2=A0=C2=A0 21=C2=A0=C2=A0=C2=A0 938 KBytes

[=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 652 MBytes=C2=
=A0 5.47 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes

[=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 659 MBytes=C2=
=A0 5.53 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.03 MBytes

[=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 666 MBytes=C2=
=A0 5.59 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.05 MBytes

[=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 657 MBytes=C2=
=A0 5.51 Gbits/sec=C2=A0=C2=A0 98=C2=A0=C2=A0=C2=A0 989 KBytes

[=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 665 MBytes=C2=
=A0 5.58 Gbits/sec=C2=A0 139=C2=A0=C2=A0=C2=A0 712 KBytes

[=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 647 MBytes=C2=A0 5.=
43 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.02 MBytes

[=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 650 MBytes=C2=A0 5.45 Gb=
its/sec=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0=C2=A0 606 KBytes

[=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.53 GBytes=C2=A0 13.1 Gbits/s=
ec=C2=A0 358=C2=A0=C2=A0 1.07 MBytes

[=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 2.10 GBytes=C2=A0 18.1 Gbits/s=
ec=C2=A0 162=C2=A0=C2=A0=C2=A0 837 KBytes

[=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 2.09 GBytes=C2=A0 18.0 Gbits/s=
ec=C2=A0 332=C2=A0=C2=A0=C2=A0 838 KBytes

[=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 2.43 GBytes=C2=A0 20.9 Gbits/s=
ec=C2=A0 639=C2=A0=C2=A0=C2=A0 747 KBytes

[=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 2.38 GBytes=C2=A0 20.4 Gbits/s=
ec=C2=A0 612=C2=A0=C2=A0 1.02 MBytes

[=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 2.25 GBytes=C2=A0 19.3 Gbits/s=
ec=C2=A0 535=C2=A0=C2=A0 1.24 MBytes

[=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 2.52 GBytes=C2=A0 21.6 Gbits/s=
ec=C2=A0 818=C2=A0=C2=A0=C2=A0 423 KBytes

[=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s=
ec=C2=A0 218=C2=A0=C2=A0=C2=A0 444 KBytes

[=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s=
ec=C2=A0 114=C2=A0=C2=A0=C2=A0 859 KBytes

[=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14.1 Gbits/s=
ec=C2=A0 100=C2=A0=C2=A0=C2=A0 541 KBytes

[=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 1.01 GBytes=C2=A0 8.67 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 639 KBytes

[=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0=C2=A0 625 MBytes=C2=A0 5.24 Gb=
its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 648 KBytes

[=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 630 MBytes=C2=A0 5.28 Gb=
its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 648 KBytes

[=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 1.56 GBytes=C2=A0 13.4 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 702 KBytes

[=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 1.78 GBytes=C2=A0 15.3 Gbits/s=
ec=C2=A0 118=C2=A0=C2=A0=C2=A0 406 KBytes

[=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.37 GBytes=C2=A0 11.8 Gbits/s=
ec=C2=A0 105=C2=A0=C2=A0=C2=A0 890 KBytes

[=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.82 GBytes=C2=A0 15.6 Gbits/s=
ec=C2=A0 104=C2=A0=C2=A0=C2=A0 963 KBytes

[=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.68 GBytes=C2=A0 14.4 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.20 MBytes

[=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.4 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.38 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -=20

[ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 Retr

[=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 44.8 GBytes=C2=A0 12.8 Gb=
its/sec=C2=A0 4477=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 sender

[=C2=A0 5]=C2=A0=C2=A0 0.00-30.01=C2=A0 sec=C2=A0 44.8 GBytes=C2=A0 12.8 Gb=
its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver



After a few runs:



[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20

Connecting to host db-01, port 5201

[=C2=A0 5] local 192.168.10.31 port 52152 connected to 192.168.10.30 port 5=
201

[ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd

[=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.91 GBytes=C2=A0 16=
.4 Gbits/sec=C2=A0=C2=A0 67=C2=A0=C2=A0=C2=A0 606 KBytes

[=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.78 GBytes=C2=A0 15=
.3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.07 MBytes

[=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 1.60 GBytes=C2=A0 13=
.7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.54 MBytes

[=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 1.61 GBytes=C2=A0 13=
.8 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14=
.3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14=
.3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14=
.1 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 1.70 GBytes=C2=A0 14=
.6 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 1.72 GBytes=C2=A0 14=
.8 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 1.85 GBytes=C2=A0 15.9 Gb=
its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 1.81 GBytes=C2=A0 15.5 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.3 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14.3 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15.7 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 1.18 GBytes=C2=A0 10.1 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 794 KBytes

[=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.4 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes

[=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 1.73 GBytes=C2=A0 14.8 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes

[=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 1.73 GBytes=C2=A0 14.9 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes

[=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15.7 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 1.77 GBytes=C2=A0 15.2 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.80 GBytes=C2=A0 15.5 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 2.03 GBytes=C2=A0 17.4 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 1.88 GBytes=C2=A0 16.1 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 1.80 GBytes=C2=A0 15.5 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 24.00-25.01=C2=A0 sec=C2=A0 1.59 GBytes=C2=A0 13.6 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 25.01-26.00=C2=A0 sec=C2=A0 1.33 GBytes=C2=A0 11.4 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s=
ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes

[=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s=
ec=C2=A0=C2=A0 97=C2=A0=C2=A0 1.01 MBytes

[=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0=C2=A0 719 MBytes=C2=A0 6.03 Gb=
its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes

[=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0=C2=A0 727 MBytes=C2=A0 6.10 Gb=
its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 Retr

[=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 49.3 GBytes=C2=A0 14.1 Gb=
its/sec=C2=A0 164=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 sender

[=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 49.3 GBytes=C2=A0 14.1 Gb=
its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver







The CPU usage is rather low, the NIC is in the OCP port, so im sure thats d=
esigned accordingly and the NIC is bound to numa0.



CPU:=C2=A0 0.0% user,=C2=A0 0.0% nice,=C2=A0 0.5% system,=C2=A0 0.7% interr=
upt, 98.8% idle



=C2=A0 PID USERNAME=C2=A0=C2=A0=C2=A0 THR PRI NICE=C2=A0=C2=A0 SIZE=C2=A0=
=C2=A0=C2=A0 RES STATE=C2=A0=C2=A0=C2=A0 C=C2=A0=C2=A0 TIME=C2=A0=C2=A0=C2=
=A0 WCPU COMMAND=20

2195 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0 52=
=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 17M=C2=A0 6884K select=C2=A0 83=C2=
=A0=C2=A0 0:14=C2=A0 27.99% iperf3



# vmstat -i -w1|grep mlx5

irq671: mlx5_core0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49969=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
 47008



(this drops to about 14k with HW LRO enabled)



The dump is rather large, i dont think i can attach it in the mailing list,=
 but if you wish to see it i can upload it somewhere.



Thank You.









---- On Mon, 13 Jun 2022 16:42:59 -0400 Hans Petter Selasky <mailto:hps@sel=
asky.org> wrote ----




Some ideas:=20
=20
Try to disable "rxpause,txpause" when setting the media.=20
=20
Keep HW LRO off for now, it doesn't work for large number of connections.=
=20
=20
What is the CPU usage during test? Is iperf3 running on a CPU-core which=20
has direct access to the NIC's numa domain?=20
=20
Is the NIC installed in the "correct" PCI high-performance slot?=20
=20
There are some sysctl knobs which may tell where the problem is, if it's=20
PCI backpressure or something else.=20
=20
sysctl -a | grep diag_pci_enable=20
sysctl -a | grep diag_general_enable=20
=20
Set these two to 1, then run some traffic and dump all mce sysctls:=20
=20
sysctl -a | grep mce > dump.txt=20
=20
--HPS=20
=20





Mike Jakubik

https://www.swiftsmsgateway.com/



Disclaimer: This e-mail and any attachments are intended only for the use o=
f the addressee(s) and may contain information that is privileged or confid=
ential. If you are not the intended recipient, or responsible for deliverin=
g the information to the intended recipient, you are hereby notified that a=
ny dissemination, distribution, printing or copying of this e-mail and any =
attachments is strictly prohibited. If this e-mail and any attachments were=
 received in error, please notify the sender by reply e-mail and delete the=
 original message.












Mike Jakubik

https://www.swiftsmsgateway.com/



Disclaimer: This e-mail and any attachments are intended only for the use o=
f the addressee(s) and may contain information that is privileged or confid=
ential. If you are not the intended recipient, or responsible for deliverin=
g the information to the intended recipient, you are hereby notified that a=
ny dissemination, distribution, printing or copying of this e-mail and any =
attachments is strictly prohibited. If this e-mail and any attachments were=
 received in error, please notify the sender by reply e-mail and delete the=
 original message.
------=_Part_4650864_1970601159.1655217366007
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head>=
<meta content=3D"text/html;charset=3DUTF-8" http-equiv=3D"Content-Type"></h=
ead><body ><div style=3D"font-family: Verdana, Arial, Helvetica, sans-serif=
; font-size: 10pt;"><div>Actually, i believe its the disabling to HW LRO th=
at makes the difference (i disabled it and rx/tx pause previously). With rx=
/tx pause on and LRO off i get similar results. The throughput is still ver=
y sporadic though.<br></div><div><br></div><div>Connecting to host db-01, p=
ort 5201 <br></div><div>[&nbsp; 5] local 192.168.10.31 port 59055 connected=
 to 192.168.10.30 port 5201<br></div><div>[ ID] Interval&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Transfer&nbsp;&nbsp;&nbsp;&nbsp; =
Bitrate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retr&nbsp; Cwnd<br>=
</div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-1.00&nbsp;&nbsp; sec&nbsp; 1.83 GByt=
es&nbsp; 15.7 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.11 MBytes<br></di=
v><div>[&nbsp; 5]&nbsp;&nbsp; 1.00-2.00&nbsp;&nbsp; sec&nbsp; 1.83 GBytes&n=
bsp; 15.7 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.11 MBytes<br></div><d=
iv>[&nbsp; 5]&nbsp;&nbsp; 2.00-3.00&nbsp;&nbsp; sec&nbsp; 2.06 GBytes&nbsp;=
 17.7 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.12 MBytes<br></div><div>[=
&nbsp; 5]&nbsp;&nbsp; 3.00-4.00&nbsp;&nbsp; sec&nbsp; 2.15 GBytes&nbsp; 18.=
5 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.12 MBytes<br></div><div>[&nbs=
p; 5]&nbsp;&nbsp; 4.00-5.00&nbsp;&nbsp; sec&nbsp; 2.15 GBytes&nbsp; 18.4 Gb=
its/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.12 MBytes<br></div><div>[&nbsp; 5=
]&nbsp;&nbsp; 5.00-6.00&nbsp;&nbsp; sec&nbsp; 2.15 GBytes&nbsp; 18.5 Gbits/=
sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.13 MBytes<br></div><div>[&nbsp; 5]&nb=
sp;&nbsp; 6.00-7.00&nbsp;&nbsp; sec&nbsp; 2.15 GBytes&nbsp; 18.5 Gbits/sec&=
nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.13 MBytes<br></div><div>[&nbsp; 5]&nbsp;&=
nbsp; 7.00-8.00&nbsp;&nbsp; sec&nbsp; 2.15 GBytes&nbsp; 18.5 Gbits/sec&nbsp=
;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.13 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp=
; 8.00-9.00&nbsp;&nbsp; sec&nbsp; 2.15 GBytes&nbsp; 18.5 Gbits/sec&nbsp;&nb=
sp;&nbsp; 0&nbsp;&nbsp; 1.13 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 9.=
00-10.00&nbsp; sec&nbsp; 2.15 GBytes&nbsp; 18.5 Gbits/sec&nbsp;&nbsp;&nbsp;=
 0&nbsp;&nbsp; 1.13 MBytes<br></div><div>[&nbsp; 5]&nbsp; 10.00-11.00&nbsp;=
 sec&nbsp; 1.72 GBytes&nbsp; 14.7 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;=
 1.14 MBytes<br></div><div>[&nbsp; 5]&nbsp; 11.00-12.00&nbsp; sec&nbsp; 1.1=
8 GBytes&nbsp; 10.1 Gbits/sec&nbsp; 131&nbsp;&nbsp;&nbsp; 932 KBytes<br></d=
iv><div>[&nbsp; 5]&nbsp; 12.00-13.00&nbsp; sec&nbsp;&nbsp; 739 MBytes&nbsp;=
 6.20 Gbits/sec&nbsp; 352&nbsp;&nbsp;&nbsp; 913 KBytes<br></div><div>[&nbsp=
; 5]&nbsp; 13.00-14.00&nbsp; sec&nbsp;&nbsp; 724 MBytes&nbsp; 6.07 Gbits/se=
c&nbsp; 142&nbsp;&nbsp;&nbsp; 636 KBytes<br></div><div>[&nbsp; 5]&nbsp; 14.=
00-15.00&nbsp; sec&nbsp;&nbsp; 759 MBytes&nbsp; 6.37 Gbits/sec&nbsp; 246&nb=
sp;&nbsp;&nbsp; 748 KBytes<br></div><div>[&nbsp; 5]&nbsp; 15.00-16.00&nbsp;=
 sec&nbsp;&nbsp; 748 MBytes&nbsp; 6.27 Gbits/sec&nbsp;&nbsp; 22&nbsp;&nbsp;=
&nbsp; 962 KBytes<br></div><div>[&nbsp; 5]&nbsp; 16.00-17.00&nbsp; sec&nbsp=
;&nbsp; 753 MBytes&nbsp; 6.32 Gbits/sec&nbsp; 170&nbsp;&nbsp;&nbsp; 711 KBy=
tes<br></div><div>[&nbsp; 5]&nbsp; 17.00-18.01&nbsp; sec&nbsp;&nbsp; 690 MB=
ytes&nbsp; 5.76 Gbits/sec&nbsp; 389&nbsp;&nbsp;&nbsp; 981 KBytes<br></div><=
div>[&nbsp; 5]&nbsp; 18.01-19.00&nbsp; sec&nbsp;&nbsp; 771 MBytes&nbsp; 6.5=
0 Gbits/sec&nbsp; 145&nbsp;&nbsp;&nbsp; 896 KBytes<br></div><div>[&nbsp; 5]=
&nbsp; 19.00-20.00&nbsp; sec&nbsp;&nbsp; 741 MBytes&nbsp; 6.22 Gbits/sec&nb=
sp;&nbsp; 13&nbsp;&nbsp;&nbsp; 952 KBytes<br></div><div>[&nbsp; 5]&nbsp; 20=
.00-21.00&nbsp; sec&nbsp;&nbsp; 743 MBytes&nbsp; 6.23 Gbits/sec&nbsp;&nbsp;=
 86&nbsp;&nbsp;&nbsp; 985 KBytes<br></div><div>[&nbsp; 5]&nbsp; 21.00-22.00=
&nbsp; sec&nbsp;&nbsp; 751 MBytes&nbsp; 6.30 Gbits/sec&nbsp; 245&nbsp;&nbsp=
;&nbsp; 944 KBytes<br></div><div>[&nbsp; 5]&nbsp; 22.00-23.00&nbsp; sec&nbs=
p;&nbsp; 724 MBytes&nbsp; 6.07 Gbits/sec&nbsp; 178&nbsp;&nbsp;&nbsp; 774 KB=
ytes<br></div><div>[&nbsp; 5]&nbsp; 23.00-24.00&nbsp; sec&nbsp;&nbsp; 728 M=
Bytes&nbsp; 6.11 Gbits/sec&nbsp; 348&nbsp;&nbsp;&nbsp; 546 KBytes<br></div>=
<div>[&nbsp; 5]&nbsp; 24.00-25.01&nbsp; sec&nbsp;&nbsp; 622 MBytes&nbsp; 5.=
18 Gbits/sec&nbsp; 215&nbsp;&nbsp;&nbsp; 842 KBytes<br></div><div>[&nbsp; 5=
]&nbsp; 25.01-26.00&nbsp; sec&nbsp;&nbsp; 920 MBytes&nbsp; 7.78 Gbits/sec&n=
bsp; 359&nbsp;&nbsp; 1.03 MBytes<br></div><div>[&nbsp; 5]&nbsp; 26.00-27.00=
&nbsp; sec&nbsp; 1.02 GBytes&nbsp; 8.80 Gbits/sec&nbsp; 385&nbsp;&nbsp; 101=
1 KBytes<br></div><div>[&nbsp; 5]&nbsp; 27.00-28.00&nbsp; sec&nbsp;&nbsp; 9=
99 MBytes&nbsp; 8.38 Gbits/sec&nbsp; 161&nbsp;&nbsp;&nbsp; 757 KBytes<br></=
div><div>[&nbsp; 5]&nbsp; 28.00-29.00&nbsp; sec&nbsp; 1.68 GBytes&nbsp; 14.=
4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.03 MBytes<br></div><div>[&nbs=
p; 5]&nbsp; 29.00-30.00&nbsp; sec&nbsp; 1.65 GBytes&nbsp; 14.2 Gbits/sec&nb=
sp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.51 MBytes<br></div><div>- - - - - - - - - -=
 - - - - - - - - - - - - - - -<br></div><div>[ ID] Interval&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Transfer&nbsp;&nbsp;&nbsp;&nbs=
p; Bitrate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retr<br></div><d=
iv>[&nbsp; 5]&nbsp;&nbsp; 0.00-30.00&nbsp; sec&nbsp; 39.2 GBytes&nbsp; 11.2=
 Gbits/sec&nbsp; 3587&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp; sender<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-30.00&n=
bsp; sec&nbsp; 39.2 GBytes&nbsp; 11.2 Gbits/sec&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
receiver<br></div><div><br></div><div><br></div><div>[root@db-02 ~]# iperf3=
 -i 1 -t 30 -c db-01 <br></div><div>Connecting to host db-01, port 5201<br>=
</div><div>[&nbsp; 5] local 192.168.10.31 port 60931 connected to 192.168.1=
0.30 port 5201<br></div><div>[ ID] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp;&nbsp;&nbsp;&nbsp; Transfer&nbsp;&nbsp;&nbsp;&nbsp; Bitrate&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retr&nbsp; Cwnd<br></div><div>[&=
nbsp; 5]&nbsp;&nbsp; 0.00-1.00&nbsp;&nbsp; sec&nbsp;&nbsp; 931 MBytes&nbsp;=
 7.81 Gbits/sec&nbsp; 146&nbsp;&nbsp;&nbsp; 689 KBytes<br></div><div>[&nbsp=
; 5]&nbsp;&nbsp; 1.00-2.00&nbsp;&nbsp; sec&nbsp;&nbsp; 674 MBytes&nbsp; 5.6=
6 Gbits/sec&nbsp; 267&nbsp;&nbsp;&nbsp; 575 KBytes<br></div><div>[&nbsp; 5]=
&nbsp;&nbsp; 2.00-3.00&nbsp;&nbsp; sec&nbsp;&nbsp; 641 MBytes&nbsp; 5.38 Gb=
its/sec&nbsp; 270&nbsp;&nbsp;&nbsp; 485 KBytes<br></div><div>[&nbsp; 5]&nbs=
p;&nbsp; 3.00-4.00&nbsp;&nbsp; sec&nbsp;&nbsp; 668 MBytes&nbsp; 5.60 Gbits/=
sec&nbsp; 221&nbsp;&nbsp;&nbsp; 801 KBytes<br></div><div>[&nbsp; 5]&nbsp;&n=
bsp; 4.00-5.01&nbsp;&nbsp; sec&nbsp;&nbsp; 532 MBytes&nbsp; 4.40 Gbits/sec&=
nbsp; 208&nbsp;&nbsp;&nbsp; 981 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp;=
 5.01-6.00&nbsp;&nbsp; sec&nbsp;&nbsp; 838 MBytes&nbsp; 7.13 Gbits/sec&nbsp=
; 291&nbsp;&nbsp;&nbsp; 949 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 6.0=
0-7.00&nbsp;&nbsp; sec&nbsp;&nbsp; 703 MBytes&nbsp; 5.90 Gbits/sec&nbsp; 17=
8&nbsp;&nbsp;&nbsp; 628 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 7.00-8.=
00&nbsp;&nbsp; sec&nbsp;&nbsp; 696 MBytes&nbsp; 5.84 Gbits/sec&nbsp;&nbsp; =
85&nbsp;&nbsp;&nbsp; 928 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 8.00-9=
.00&nbsp;&nbsp; sec&nbsp; 1.06 GBytes&nbsp; 9.14 Gbits/sec&nbsp;&nbsp; 66&n=
bsp;&nbsp;&nbsp; 565 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 9.00-10.00=
&nbsp; sec&nbsp;&nbsp; 975 MBytes&nbsp; 8.18 Gbits/sec&nbsp;&nbsp;&nbsp; 0&=
nbsp;&nbsp;&nbsp; 801 KBytes<br></div><div>[&nbsp; 5]&nbsp; 10.00-11.00&nbs=
p; sec&nbsp;&nbsp; 679 MBytes&nbsp; 5.70 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp=
;&nbsp;&nbsp; 859 KBytes<br></div><div>[&nbsp; 5]&nbsp; 11.00-12.01&nbsp; s=
ec&nbsp;&nbsp; 628 MBytes&nbsp; 5.23 Gbits/sec&nbsp; 123&nbsp;&nbsp; 1.12 M=
Bytes<br></div><div>[&nbsp; 5]&nbsp; 12.01-13.00&nbsp; sec&nbsp;&nbsp; 731 =
MBytes&nbsp; 6.16 Gbits/sec&nbsp; 311&nbsp;&nbsp;&nbsp; 864 KBytes<br></div=
><div>[&nbsp; 5]&nbsp; 13.00-14.00&nbsp; sec&nbsp;&nbsp; 696 MBytes&nbsp; 5=
.86 Gbits/sec&nbsp;&nbsp; 39&nbsp;&nbsp;&nbsp; 798 KBytes<br></div><div>[&n=
bsp; 5]&nbsp; 14.00-15.00&nbsp; sec&nbsp;&nbsp; 914 MBytes&nbsp; 7.66 Gbits=
/sec&nbsp;&nbsp; 65&nbsp;&nbsp; 1024 KBytes<br></div><div>[&nbsp; 5]&nbsp; =
15.00-16.00&nbsp; sec&nbsp; 1.66 GBytes&nbsp; 14.2 Gbits/sec&nbsp;&nbsp;&nb=
sp; 0&nbsp;&nbsp; 1.47 MBytes<br></div><div>[&nbsp; 5]&nbsp; 16.00-17.00&nb=
sp; sec&nbsp;&nbsp; 955 MBytes&nbsp; 8.01 Gbits/sec&nbsp; 222&nbsp;&nbsp;&n=
bsp; 260 KBytes<br></div><div>[&nbsp; 5]&nbsp; 17.00-18.00&nbsp; sec&nbsp;&=
nbsp; 769 MBytes&nbsp; 6.45 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;=
 520 KBytes<br></div><div>[&nbsp; 5]&nbsp; 18.00-19.00&nbsp; sec&nbsp;&nbsp=
; 781 MBytes&nbsp; 6.55 Gbits/sec&nbsp; 119&nbsp;&nbsp;&nbsp; 432 KBytes<br=
></div><div>[&nbsp; 5]&nbsp; 19.00-20.00&nbsp; sec&nbsp;&nbsp; 779 MBytes&n=
bsp; 6.54 Gbits/sec&nbsp; 148&nbsp;&nbsp;&nbsp; 816 KBytes<br></div><div>[&=
nbsp; 5]&nbsp; 20.00-21.00&nbsp; sec&nbsp;&nbsp; 771 MBytes&nbsp; 6.47 Gbit=
s/sec&nbsp; 127&nbsp;&nbsp;&nbsp; 965 KBytes<br></div><div>[&nbsp; 5]&nbsp;=
 21.00-22.00&nbsp; sec&nbsp;&nbsp; 745 MBytes&nbsp; 6.25 Gbits/sec&nbsp; 33=
7&nbsp;&nbsp;&nbsp; 927 KBytes<br></div><div>[&nbsp; 5]&nbsp; 22.00-23.00&n=
bsp; sec&nbsp;&nbsp; 637 MBytes&nbsp; 5.35 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nb=
sp;&nbsp;&nbsp; 938 KBytes<br></div><div>[&nbsp; 5]&nbsp; 23.00-24.00&nbsp;=
 sec&nbsp;&nbsp; 626 MBytes&nbsp; 5.25 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&=
nbsp;&nbsp; 938 KBytes<br></div><div>[&nbsp; 5]&nbsp; 24.00-25.00&nbsp; sec=
&nbsp; 1.96 GBytes&nbsp; 16.8 Gbits/sec&nbsp; 185&nbsp;&nbsp;&nbsp; 862 KBy=
tes<br></div><div>[&nbsp; 5]&nbsp; 25.00-26.00&nbsp; sec&nbsp; 1.71 GBytes&=
nbsp; 14.7 Gbits/sec&nbsp;&nbsp; 22&nbsp;&nbsp;&nbsp; 945 KBytes<br></div><=
div>[&nbsp; 5]&nbsp; 26.00-27.00&nbsp; sec&nbsp; 2.14 GBytes&nbsp; 18.3 Gbi=
ts/sec&nbsp; 365&nbsp;&nbsp;&nbsp; 593 KBytes<br></div><div>[&nbsp; 5]&nbsp=
; 27.00-28.00&nbsp; sec&nbsp; 1.56 GBytes&nbsp; 13.4 Gbits/sec&nbsp;&nbsp; =
33&nbsp;&nbsp; 1.23 MBytes<br></div><div>[&nbsp; 5]&nbsp; 28.00-29.00&nbsp;=
 sec&nbsp; 1.67 GBytes&nbsp; 14.4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;=
 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 29.00-30.00&nbsp; sec&nbsp; 1.6=
5 GBytes&nbsp; 14.2 Gbits/sec&nbsp;&nbsp; 35&nbsp;&nbsp; 1.14 MBytes<br></d=
iv><div>- - - - - - - - - - - - - - - - - - - - - - - - -<br></div><div>[ I=
D] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Tra=
nsfer&nbsp;&nbsp;&nbsp;&nbsp; Bitrate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n=
bsp;&nbsp; Retr<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-30.00&nbsp; sec&n=
bsp; 29.4 GBytes&nbsp; 8.42 Gbits/sec&nbsp; 3863&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sender<br></div><div>[&nbsp; =
5]&nbsp;&nbsp; 0.00-30.00&nbsp; sec&nbsp; 29.4 GBytes&nbsp; 8.42 Gbits/sec&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp; receiver<br></div><div><br></div><div><br></div>=
<div><br></div><div class=3D"zmail_extra_hr" style=3D"border-top: 1px solid=
 rgb(204, 204, 204); height: 0px; margin-top: 10px; margin-bottom: 10px; li=
ne-height: 0px;"><br></div><div class=3D"zmail_extra" data-zbluepencil-igno=
re=3D"true"><div><br></div><div id=3D"Zm-_Id_-Sgn1">---- On Tue, 14 Jun 202=
2 10:21:51 -0400 <b>Mike Jakubik &lt;mike.jakubik@swiftsmsgateway.com&gt;</=
b> wrote ----<br></div><div><br></div><blockquote style=3D"margin: 0px;"><d=
iv><div style=3D"font-family: Verdana, Arial, Helvetica, sans-serif;font-si=
ze: 10.0pt;"><div>Disabling rx/tx pause seems to produce higher peaks.<br><=
/div><div><br></div><div>[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 <br></d=
iv><div>Connecting to host db-01, port 5201<br></div><div>[&nbsp; 5] local =
192.168.10.31 port 10146 connected to 192.168.10.30 port 5201<br></div><div=
>[ ID] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
 Transfer&nbsp;&nbsp;&nbsp;&nbsp; Bitrate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp; Retr&nbsp; Cwnd<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-1.=
00&nbsp;&nbsp; sec&nbsp; 1.89 GBytes&nbsp; 16.2 Gbits/sec&nbsp;&nbsp;&nbsp;=
 0&nbsp;&nbsp; 1.10 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 1.00-2.00&n=
bsp;&nbsp; sec&nbsp; 1.86 GBytes&nbsp; 15.9 Gbits/sec&nbsp;&nbsp;&nbsp; 0&n=
bsp;&nbsp; 1.10 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 2.00-3.00&nbsp;=
&nbsp; sec&nbsp; 2.05 GBytes&nbsp; 17.6 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;=
&nbsp; 1.11 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 3.00-4.00&nbsp;&nbs=
p; sec&nbsp;&nbsp; 859 MBytes&nbsp; 7.20 Gbits/sec&nbsp;&nbsp; 21&nbsp;&nbs=
p;&nbsp; 938 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 4.00-5.00&nbsp;&nb=
sp; sec&nbsp;&nbsp; 652 MBytes&nbsp; 5.47 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbs=
p;&nbsp; 1.01 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 5.00-6.00&nbsp;&n=
bsp; sec&nbsp;&nbsp; 659 MBytes&nbsp; 5.53 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nb=
sp;&nbsp; 1.03 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 6.00-7.00&nbsp;&=
nbsp; sec&nbsp;&nbsp; 666 MBytes&nbsp; 5.59 Gbits/sec&nbsp;&nbsp;&nbsp; 0&n=
bsp;&nbsp; 1.05 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 7.00-8.00&nbsp;=
&nbsp; sec&nbsp;&nbsp; 657 MBytes&nbsp; 5.51 Gbits/sec&nbsp;&nbsp; 98&nbsp;=
&nbsp;&nbsp; 989 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 8.00-9.00&nbsp=
;&nbsp; sec&nbsp;&nbsp; 665 MBytes&nbsp; 5.58 Gbits/sec&nbsp; 139&nbsp;&nbs=
p;&nbsp; 712 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 9.00-10.00&nbsp; s=
ec&nbsp;&nbsp; 647 MBytes&nbsp; 5.43 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nb=
sp; 1.02 MBytes<br></div><div>[&nbsp; 5]&nbsp; 10.00-11.00&nbsp; sec&nbsp;&=
nbsp; 650 MBytes&nbsp; 5.45 Gbits/sec&nbsp;&nbsp;&nbsp; 4&nbsp;&nbsp;&nbsp;=
 606 KBytes<br></div><div>[&nbsp; 5]&nbsp; 11.00-12.00&nbsp; sec&nbsp; 1.53=
 GBytes&nbsp; 13.1 Gbits/sec&nbsp; 358&nbsp;&nbsp; 1.07 MBytes<br></div><di=
v>[&nbsp; 5]&nbsp; 12.00-13.00&nbsp; sec&nbsp; 2.10 GBytes&nbsp; 18.1 Gbits=
/sec&nbsp; 162&nbsp;&nbsp;&nbsp; 837 KBytes<br></div><div>[&nbsp; 5]&nbsp; =
13.00-14.00&nbsp; sec&nbsp; 2.09 GBytes&nbsp; 18.0 Gbits/sec&nbsp; 332&nbsp=
;&nbsp;&nbsp; 838 KBytes<br></div><div>[&nbsp; 5]&nbsp; 14.00-15.00&nbsp; s=
ec&nbsp; 2.43 GBytes&nbsp; 20.9 Gbits/sec&nbsp; 639&nbsp;&nbsp;&nbsp; 747 K=
Bytes<br></div><div>[&nbsp; 5]&nbsp; 15.00-16.00&nbsp; sec&nbsp; 2.38 GByte=
s&nbsp; 20.4 Gbits/sec&nbsp; 612&nbsp;&nbsp; 1.02 MBytes<br></div><div>[&nb=
sp; 5]&nbsp; 16.00-17.00&nbsp; sec&nbsp; 2.25 GBytes&nbsp; 19.3 Gbits/sec&n=
bsp; 535&nbsp;&nbsp; 1.24 MBytes<br></div><div>[&nbsp; 5]&nbsp; 17.00-18.00=
&nbsp; sec&nbsp; 2.52 GBytes&nbsp; 21.6 Gbits/sec&nbsp; 818&nbsp;&nbsp;&nbs=
p; 423 KBytes<br></div><div>[&nbsp; 5]&nbsp; 18.00-19.00&nbsp; sec&nbsp; 2.=
29 GBytes&nbsp; 19.7 Gbits/sec&nbsp; 218&nbsp;&nbsp;&nbsp; 444 KBytes<br></=
div><div>[&nbsp; 5]&nbsp; 19.00-20.00&nbsp; sec&nbsp; 2.29 GBytes&nbsp; 19.=
7 Gbits/sec&nbsp; 114&nbsp;&nbsp;&nbsp; 859 KBytes<br></div><div>[&nbsp; 5]=
&nbsp; 20.00-21.00&nbsp; sec&nbsp; 1.65 GBytes&nbsp; 14.1 Gbits/sec&nbsp; 1=
00&nbsp;&nbsp;&nbsp; 541 KBytes<br></div><div>[&nbsp; 5]&nbsp; 21.00-22.00&=
nbsp; sec&nbsp; 1.01 GBytes&nbsp; 8.67 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&=
nbsp;&nbsp; 639 KBytes<br></div><div>[&nbsp; 5]&nbsp; 22.00-23.00&nbsp; sec=
&nbsp;&nbsp; 625 MBytes&nbsp; 5.24 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp=
;&nbsp; 648 KBytes<br></div><div>[&nbsp; 5]&nbsp; 23.00-24.00&nbsp; sec&nbs=
p;&nbsp; 630 MBytes&nbsp; 5.28 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nb=
sp; 648 KBytes<br></div><div>[&nbsp; 5]&nbsp; 24.00-25.00&nbsp; sec&nbsp; 1=
.56 GBytes&nbsp; 13.4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; 702 K=
Bytes<br></div><div>[&nbsp; 5]&nbsp; 25.00-26.00&nbsp; sec&nbsp; 1.78 GByte=
s&nbsp; 15.3 Gbits/sec&nbsp; 118&nbsp;&nbsp;&nbsp; 406 KBytes<br></div><div=
>[&nbsp; 5]&nbsp; 26.00-27.00&nbsp; sec&nbsp; 1.37 GBytes&nbsp; 11.8 Gbits/=
sec&nbsp; 105&nbsp;&nbsp;&nbsp; 890 KBytes<br></div><div>[&nbsp; 5]&nbsp; 2=
7.00-28.00&nbsp; sec&nbsp; 1.82 GBytes&nbsp; 15.6 Gbits/sec&nbsp; 104&nbsp;=
&nbsp;&nbsp; 963 KBytes<br></div><div>[&nbsp; 5]&nbsp; 28.00-29.00&nbsp; se=
c&nbsp; 1.68 GBytes&nbsp; 14.4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.=
20 MBytes<br></div><div>[&nbsp; 5]&nbsp; 29.00-30.00&nbsp; sec&nbsp; 1.67 G=
Bytes&nbsp; 14.4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.38 MBytes<br><=
/div><div>- - - - - - - - - - - - - - - - - - - - - - - - - <br></div><div>=
[ ID] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
Transfer&nbsp;&nbsp;&nbsp;&nbsp; Bitrate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp=
;&nbsp;&nbsp; Retr<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-30.00&nbsp; se=
c&nbsp; 44.8 GBytes&nbsp; 12.8 Gbits/sec&nbsp; 4477&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sender<br></div><div>[&nbs=
p; 5]&nbsp;&nbsp; 0.00-30.01&nbsp; sec&nbsp; 44.8 GBytes&nbsp; 12.8 Gbits/s=
ec&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; receiver<br></div><div><br></div><div>After a=
 few runs:<br></div><div><br></div><div>[root@db-02 ~]# iperf3 -i 1 -t 30 -=
c db-01 <br></div><div>Connecting to host db-01, port 5201<br></div><div>[&=
nbsp; 5] local 192.168.10.31 port 52152 connected to 192.168.10.30 port 520=
1<br></div><div>[ ID] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp; Transfer&nbsp;&nbsp;&nbsp;&nbsp; Bitrate&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retr&nbsp; Cwnd<br></div><div>[&nbsp; 5]&nbsp=
;&nbsp; 0.00-1.00&nbsp;&nbsp; sec&nbsp; 1.91 GBytes&nbsp; 16.4 Gbits/sec&nb=
sp;&nbsp; 67&nbsp;&nbsp;&nbsp; 606 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nb=
sp; 1.00-2.00&nbsp;&nbsp; sec&nbsp; 1.78 GBytes&nbsp; 15.3 Gbits/sec&nbsp;&=
nbsp;&nbsp; 0&nbsp;&nbsp; 1.07 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; =
2.00-3.00&nbsp;&nbsp; sec&nbsp; 1.60 GBytes&nbsp; 13.7 Gbits/sec&nbsp;&nbsp=
;&nbsp; 0&nbsp;&nbsp; 1.54 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 3.00=
-4.00&nbsp;&nbsp; sec&nbsp; 1.61 GBytes&nbsp; 13.8 Gbits/sec&nbsp;&nbsp;&nb=
sp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 4.00-5.0=
0&nbsp;&nbsp; sec&nbsp; 1.66 GBytes&nbsp; 14.3 Gbits/sec&nbsp;&nbsp;&nbsp; =
0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 5.00-6.00&nb=
sp;&nbsp; sec&nbsp; 1.67 GBytes&nbsp; 14.3 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nb=
sp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 6.00-7.00&nbsp;&=
nbsp; sec&nbsp; 1.65 GBytes&nbsp; 14.1 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&=
nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 7.00-8.00&nbsp;&nbsp=
; sec&nbsp; 1.70 GBytes&nbsp; 14.6 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp=
; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 8.00-9.00&nbsp;&nbsp; se=
c&nbsp; 1.72 GBytes&nbsp; 14.8 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.=
61 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 9.00-10.00&nbsp; sec&nbsp; 1=
.85 GBytes&nbsp; 15.9 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes=
<br></div><div>[&nbsp; 5]&nbsp; 10.00-11.00&nbsp; sec&nbsp; 1.81 GBytes&nbs=
p; 15.5 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div=
>[&nbsp; 5]&nbsp; 11.00-12.00&nbsp; sec&nbsp; 1.67 GBytes&nbsp; 14.3 Gbits/=
sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nb=
sp; 12.00-13.00&nbsp; sec&nbsp; 1.66 GBytes&nbsp; 14.3 Gbits/sec&nbsp;&nbsp=
;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 13.00-14.0=
0&nbsp; sec&nbsp; 1.83 GBytes&nbsp; 15.7 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp=
;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 14.00-15.00&nbsp; sec&nb=
sp; 1.18 GBytes&nbsp; 10.1 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp; =
794 KBytes<br></div><div>[&nbsp; 5]&nbsp; 15.00-16.00&nbsp; sec&nbsp; 1.67 =
GBytes&nbsp; 14.4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.60 MBytes<br>=
</div><div>[&nbsp; 5]&nbsp; 16.00-17.00&nbsp; sec&nbsp; 1.73 GBytes&nbsp; 1=
4.8 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.60 MBytes<br></div><div>[&n=
bsp; 5]&nbsp; 17.00-18.00&nbsp; sec&nbsp; 1.73 GBytes&nbsp; 14.9 Gbits/sec&=
nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.60 MBytes<br></div><div>[&nbsp; 5]&nbsp; =
18.00-19.00&nbsp; sec&nbsp; 1.83 GBytes&nbsp; 15.7 Gbits/sec&nbsp;&nbsp;&nb=
sp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 19.00-20.00&nb=
sp; sec&nbsp; 1.77 GBytes&nbsp; 15.2 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nb=
sp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 20.00-21.00&nbsp; sec&nbsp; =
1.80 GBytes&nbsp; 15.5 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MByte=
s<br></div><div>[&nbsp; 5]&nbsp; 21.00-22.00&nbsp; sec&nbsp; 2.03 GBytes&nb=
sp; 17.4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><di=
v>[&nbsp; 5]&nbsp; 22.00-23.00&nbsp; sec&nbsp; 1.88 GBytes&nbsp; 16.1 Gbits=
/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&n=
bsp; 23.00-24.00&nbsp; sec&nbsp; 1.80 GBytes&nbsp; 15.5 Gbits/sec&nbsp;&nbs=
p;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 24.00-25.=
01&nbsp; sec&nbsp; 1.59 GBytes&nbsp; 13.6 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbs=
p;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 25.01-26.00&nbsp; sec&n=
bsp; 1.33 GBytes&nbsp; 11.4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 =
MBytes<br></div><div>[&nbsp; 5]&nbsp; 26.00-27.00&nbsp; sec&nbsp; 1.71 GByt=
es&nbsp; 14.7 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></di=
v><div>[&nbsp; 5]&nbsp; 27.00-28.00&nbsp; sec&nbsp; 1.71 GBytes&nbsp; 14.7 =
Gbits/sec&nbsp;&nbsp; 97&nbsp;&nbsp; 1.01 MBytes<br></div><div>[&nbsp; 5]&n=
bsp; 28.00-29.00&nbsp; sec&nbsp;&nbsp; 719 MBytes&nbsp; 6.03 Gbits/sec&nbsp=
;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.01 MBytes<br></div><div>[&nbsp; 5]&nbsp; 29.0=
0-30.00&nbsp; sec&nbsp;&nbsp; 727 MBytes&nbsp; 6.10 Gbits/sec&nbsp;&nbsp;&n=
bsp; 0&nbsp;&nbsp; 1.01 MBytes<br></div><div>- - - - - - - - - - - - - - - =
- - - - - - - - - -<br></div><div>[ ID] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nb=
sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Transfer&nbsp;&nbsp;&nbsp;&nbsp; Bitrate&=
nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retr<br></div><div>[&nbsp; =
5]&nbsp;&nbsp; 0.00-30.00&nbsp; sec&nbsp; 49.3 GBytes&nbsp; 14.1 Gbits/sec&=
nbsp; 164&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;=
&nbsp; sender<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-30.00&nbsp; sec&nbs=
p; 49.3 GBytes&nbsp; 14.1 Gbits/sec&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; receiver<br>=
</div><div><br></div><div><br></div><div><br></div><div>The CPU usage is ra=
ther low, the NIC is in the OCP port, so im sure thats designed accordingly=
 and the NIC is bound to numa0.<br></div><div><br></div><div>CPU:&nbsp; 0.0=
% user,&nbsp; 0.0% nice,&nbsp; 0.5% system,&nbsp; 0.7% interrupt, 98.8% idl=
e<br></div><div><br></div><div>&nbsp; PID USERNAME&nbsp;&nbsp;&nbsp; THR PR=
I NICE&nbsp;&nbsp; SIZE&nbsp;&nbsp;&nbsp; RES STATE&nbsp;&nbsp;&nbsp; C&nbs=
p;&nbsp; TIME&nbsp;&nbsp;&nbsp; WCPU COMMAND <br></div><div>2195 root&nbsp;=
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1&nbsp; 52&nbsp;&nbsp;&nbs=
p; 0&nbsp;&nbsp;&nbsp; 17M&nbsp; 6884K select&nbsp; 83&nbsp;&nbsp; 0:14&nbs=
p; 27.99% iperf3<br></div><div><br></div><div># vmstat -i -w1|grep mlx5<br>=
</div><div>irq671: mlx5_core0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs=
p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 49969&nbsp;&nbsp;&nbsp;&=
nbsp;&nbsp; 47008<br></div><div><br></div><div>(this drops to about 14k wit=
h HW LRO enabled)<br></div><div><br></div><div>The dump is rather large, i =
dont think i can attach it in the mailing list, but if you wish to see it i=
 can upload it somewhere.<br></div><div><br></div><div>Thank You.<br></div>=
<div><br></div><div><br></div><div class=3D"x_-614958649zmail_extra_hr" sty=
le=3D"border-top: 1.0px solid rgb(204,204,204);min-height: 0.0px;margin-top=
: 10.0px;margin-bottom: 10.0px;line-height: 0.0px;"><br></div><div class=3D=
"x_-614958649zmail_extra"><div><br></div><div id=3D"x_-614958649Zm-_Id_-Sgn=
1">---- On Mon, 13 Jun 2022 16:42:59 -0400 <b>Hans Petter Selasky &lt;<a hr=
ef=3D"mailto:hps@selasky.org" target=3D"_blank">hps@selasky.org</a>&gt;</b>=
 wrote ----<br></div><div><br></div><blockquote style=3D"margin: 0.0px;"><d=
iv><br>Some ideas: <br> <br>Try to disable "rxpause,txpause" when setting t=
he media. <br> <br>Keep HW LRO off for now, it doesn't work for large numbe=
r of connections. <br> <br>What is the CPU usage during test? Is iperf3 run=
ning on a CPU-core which <br>has direct access to the NIC's numa domain? <b=
r> <br>Is the NIC installed in the "correct" PCI high-performance slot? <br=
> <br>There are some sysctl knobs which may tell where the problem is, if i=
t's <br>PCI backpressure or something else. <br> <br>sysctl -a | grep diag_=
pci_enable <br>sysctl -a | grep diag_general_enable <br> <br>Set these two =
to 1, then run some traffic and dump all mce sysctls: <br> <br>sysctl -a | =
grep mce &gt; dump.txt <br> <br>--HPS <br> <br></div></blockquote></div><di=
v><br></div><div class=3D"x_-614958649zmail_signature_below"><div id=3D"x_-=
614958649Zm-_Id_-Sgn"><div>Mike Jakubik<br></div><div><a target=3D"_blank" =
href=3D"https://www.swiftsmsgateway.com/">https://www.swiftsmsgateway.com/<=
/a><br></div><div><br></div><div><span class=3D"size" style=3D"font-size:10=
px">Disclaimer: This e-mail and any attachments are intended only for the u=
se of the addressee(s) and may contain information that is privileged or co=
nfidential. If you are not the intended recipient, or responsible for deliv=
ering the information to the intended recipient, you are hereby notified th=
at any dissemination, distribution, printing or copying of this e-mail and =
any attachments is strictly prohibited. If this e-mail and any attachments =
were received in error, please notify the sender by reply e-mail and delete=
 the original message.</span><br></div></div></div><div><br></div></div><br=
></div></blockquote></div><div><br></div><div class=3D"zmail_signature_belo=
w"><div id=3D"Zm-_Id_-Sgn" data-zbluepencil-ignore=3D"true" data-sigid=3D"7=
886077000000096004"><div>Mike Jakubik<br></div><div><a target=3D"_blank" hr=
ef=3D"https://www.swiftsmsgateway.com/">https://www.swiftsmsgateway.com/</a=
><br></div><div><br></div><div><span class=3D"size" style=3D"font-size:10px=
">Disclaimer: This e-mail and any attachments are intended only for the use=
 of the addressee(s) and may contain information that is privileged or conf=
idential. If you are not the intended recipient, or responsible for deliver=
ing the information to the intended recipient, you are hereby notified that=
 any dissemination, distribution, printing or copying of this e-mail and an=
y attachments is strictly prohibited. If this e-mail and any attachments we=
re received in error, please notify the sender by reply e-mail and delete t=
he original message.</span><br></div></div></div><div><br></div></div><br><=
/body></html>
------=_Part_4650864_1970601159.1655217366007--

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?18162a4a3f6.10a1a03d11472072.3783895140221599214>