Date: Tue, 14 Jun 2022 10:36:06 -0400 From: Mike Jakubik <mike.jakubik@swiftsmsgateway.com> To: "mikejakubik" <mike.jakubik@swiftsmsgateway.com> Cc: "Hans Petter Selasky" <hps@selasky.org>, "freebsd-net" <freebsd-net@FreeBSD.org> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) Message-ID: <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> In-Reply-To: <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Actually, i believe its the disabling to HW LRO that makes the difference (i disabled it and rx/tx pause previously). With rx/tx pause on and LRO off i get similar results. The throughput is still very sporadic though. Connecting to host db-01, port 5201 [ 5] local 192.168.10.31 port 59055 connected to 192.168.10.30 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.83 GBytes 15.7 Gbits/sec 0 1.11 MBytes [ 5] 1.00-2.00 sec 1.83 GBytes 15.7 Gbits/sec 0 1.11 MBytes [ 5] 2.00-3.00 sec 2.06 GBytes 17.7 Gbits/sec 0 1.12 MBytes [ 5] 3.00-4.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.12 MBytes [ 5] 4.00-5.00 sec 2.15 GBytes 18.4 Gbits/sec 0 1.12 MBytes [ 5] 5.00-6.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.13 MBytes [ 5] 6.00-7.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.13 MBytes [ 5] 7.00-8.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.13 MBytes [ 5] 8.00-9.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.13 MBytes [ 5] 9.00-10.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.13 MBytes [ 5] 10.00-11.00 sec 1.72 GBytes 14.7 Gbits/sec 0 1.14 MBytes [ 5] 11.00-12.00 sec 1.18 GBytes 10.1 Gbits/sec 131 932 KBytes [ 5] 12.00-13.00 sec 739 MBytes 6.20 Gbits/sec 352 913 KBytes [ 5] 13.00-14.00 sec 724 MBytes 6.07 Gbits/sec 142 636 KBytes [ 5] 14.00-15.00 sec 759 MBytes 6.37 Gbits/sec 246 748 KBytes [ 5] 15.00-16.00 sec 748 MBytes 6.27 Gbits/sec 22 962 KBytes [ 5] 16.00-17.00 sec 753 MBytes 6.32 Gbits/sec 170 711 KBytes [ 5] 17.00-18.01 sec 690 MBytes 5.76 Gbits/sec 389 981 KBytes [ 5] 18.01-19.00 sec 771 MBytes 6.50 Gbits/sec 145 896 KBytes [ 5] 19.00-20.00 sec 741 MBytes 6.22 Gbits/sec 13 952 KBytes [ 5] 20.00-21.00 sec 743 MBytes 6.23 Gbits/sec 86 985 KBytes [ 5] 21.00-22.00 sec 751 MBytes 6.30 Gbits/sec 245 944 KBytes [ 5] 22.00-23.00 sec 724 MBytes 6.07 Gbits/sec 178 774 KBytes [ 5] 23.00-24.00 sec 728 MBytes 6.11 Gbits/sec 348 546 KBytes [ 5] 24.00-25.01 sec 622 MBytes 5.18 Gbits/sec 215 842 KBytes [ 5] 25.01-26.00 sec 920 MBytes 7.78 Gbits/sec 359 1.03 MBytes [ 5] 26.00-27.00 sec 1.02 GBytes 8.80 Gbits/sec 385 1011 KBytes [ 5] 27.00-28.00 sec 999 MBytes 8.38 Gbits/sec 161 757 KBytes [ 5] 28.00-29.00 sec 1.68 GBytes 14.4 Gbits/sec 0 1.03 MBytes [ 5] 29.00-30.00 sec 1.65 GBytes 14.2 Gbits/sec 0 1.51 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 39.2 GBytes 11.2 Gbits/sec 3587 sender [ 5] 0.00-30.00 sec 39.2 GBytes 11.2 Gbits/sec receiver [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [ 5] local 192.168.10.31 port 60931 connected to 192.168.10.30 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 931 MBytes 7.81 Gbits/sec 146 689 KBytes [ 5] 1.00-2.00 sec 674 MBytes 5.66 Gbits/sec 267 575 KBytes [ 5] 2.00-3.00 sec 641 MBytes 5.38 Gbits/sec 270 485 KBytes [ 5] 3.00-4.00 sec 668 MBytes 5.60 Gbits/sec 221 801 KBytes [ 5] 4.00-5.01 sec 532 MBytes 4.40 Gbits/sec 208 981 KBytes [ 5] 5.01-6.00 sec 838 MBytes 7.13 Gbits/sec 291 949 KBytes [ 5] 6.00-7.00 sec 703 MBytes 5.90 Gbits/sec 178 628 KBytes [ 5] 7.00-8.00 sec 696 MBytes 5.84 Gbits/sec 85 928 KBytes [ 5] 8.00-9.00 sec 1.06 GBytes 9.14 Gbits/sec 66 565 KBytes [ 5] 9.00-10.00 sec 975 MBytes 8.18 Gbits/sec 0 801 KBytes [ 5] 10.00-11.00 sec 679 MBytes 5.70 Gbits/sec 0 859 KBytes [ 5] 11.00-12.01 sec 628 MBytes 5.23 Gbits/sec 123 1.12 MBytes [ 5] 12.01-13.00 sec 731 MBytes 6.16 Gbits/sec 311 864 KBytes [ 5] 13.00-14.00 sec 696 MBytes 5.86 Gbits/sec 39 798 KBytes [ 5] 14.00-15.00 sec 914 MBytes 7.66 Gbits/sec 65 1024 KBytes [ 5] 15.00-16.00 sec 1.66 GBytes 14.2 Gbits/sec 0 1.47 MBytes [ 5] 16.00-17.00 sec 955 MBytes 8.01 Gbits/sec 222 260 KBytes [ 5] 17.00-18.00 sec 769 MBytes 6.45 Gbits/sec 0 520 KBytes [ 5] 18.00-19.00 sec 781 MBytes 6.55 Gbits/sec 119 432 KBytes [ 5] 19.00-20.00 sec 779 MBytes 6.54 Gbits/sec 148 816 KBytes [ 5] 20.00-21.00 sec 771 MBytes 6.47 Gbits/sec 127 965 KBytes [ 5] 21.00-22.00 sec 745 MBytes 6.25 Gbits/sec 337 927 KBytes [ 5] 22.00-23.00 sec 637 MBytes 5.35 Gbits/sec 0 938 KBytes [ 5] 23.00-24.00 sec 626 MBytes 5.25 Gbits/sec 0 938 KBytes [ 5] 24.00-25.00 sec 1.96 GBytes 16.8 Gbits/sec 185 862 KBytes [ 5] 25.00-26.00 sec 1.71 GBytes 14.7 Gbits/sec 22 945 KBytes [ 5] 26.00-27.00 sec 2.14 GBytes 18.3 Gbits/sec 365 593 KBytes [ 5] 27.00-28.00 sec 1.56 GBytes 13.4 Gbits/sec 33 1.23 MBytes [ 5] 28.00-29.00 sec 1.67 GBytes 14.4 Gbits/sec 0 1.61 MBytes [ 5] 29.00-30.00 sec 1.65 GBytes 14.2 Gbits/sec 35 1.14 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 29.4 GBytes 8.42 Gbits/sec 3863 sender [ 5] 0.00-30.00 sec 29.4 GBytes 8.42 Gbits/sec receiver ---- On Tue, 14 Jun 2022 10:21:51 -0400 Mike Jakubik <mike.jakubik@swiftsmsgateway.com> wrote ---- Disabling rx/tx pause seems to produce higher peaks. [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [ 5] local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.89 GBytes 16.2 Gbits/sec 0 1.10 MBytes [ 5] 1.00-2.00 sec 1.86 GBytes 15.9 Gbits/sec 0 1.10 MBytes [ 5] 2.00-3.00 sec 2.05 GBytes 17.6 Gbits/sec 0 1.11 MBytes [ 5] 3.00-4.00 sec 859 MBytes 7.20 Gbits/sec 21 938 KBytes [ 5] 4.00-5.00 sec 652 MBytes 5.47 Gbits/sec 0 1.01 MBytes [ 5] 5.00-6.00 sec 659 MBytes 5.53 Gbits/sec 0 1.03 MBytes [ 5] 6.00-7.00 sec 666 MBytes 5.59 Gbits/sec 0 1.05 MBytes [ 5] 7.00-8.00 sec 657 MBytes 5.51 Gbits/sec 98 989 KBytes [ 5] 8.00-9.00 sec 665 MBytes 5.58 Gbits/sec 139 712 KBytes [ 5] 9.00-10.00 sec 647 MBytes 5.43 Gbits/sec 0 1.02 MBytes [ 5] 10.00-11.00 sec 650 MBytes 5.45 Gbits/sec 4 606 KBytes [ 5] 11.00-12.00 sec 1.53 GBytes 13.1 Gbits/sec 358 1.07 MBytes [ 5] 12.00-13.00 sec 2.10 GBytes 18.1 Gbits/sec 162 837 KBytes [ 5] 13.00-14.00 sec 2.09 GBytes 18.0 Gbits/sec 332 838 KBytes [ 5] 14.00-15.00 sec 2.43 GBytes 20.9 Gbits/sec 639 747 KBytes [ 5] 15.00-16.00 sec 2.38 GBytes 20.4 Gbits/sec 612 1.02 MBytes [ 5] 16.00-17.00 sec 2.25 GBytes 19.3 Gbits/sec 535 1.24 MBytes [ 5] 17.00-18.00 sec 2.52 GBytes 21.6 Gbits/sec 818 423 KBytes [ 5] 18.00-19.00 sec 2.29 GBytes 19.7 Gbits/sec 218 444 KBytes [ 5] 19.00-20.00 sec 2.29 GBytes 19.7 Gbits/sec 114 859 KBytes [ 5] 20.00-21.00 sec 1.65 GBytes 14.1 Gbits/sec 100 541 KBytes [ 5] 21.00-22.00 sec 1.01 GBytes 8.67 Gbits/sec 0 639 KBytes [ 5] 22.00-23.00 sec 625 MBytes 5.24 Gbits/sec 0 648 KBytes [ 5] 23.00-24.00 sec 630 MBytes 5.28 Gbits/sec 0 648 KBytes [ 5] 24.00-25.00 sec 1.56 GBytes 13.4 Gbits/sec 0 702 KBytes [ 5] 25.00-26.00 sec 1.78 GBytes 15.3 Gbits/sec 118 406 KBytes [ 5] 26.00-27.00 sec 1.37 GBytes 11.8 Gbits/sec 105 890 KBytes [ 5] 27.00-28.00 sec 1.82 GBytes 15.6 Gbits/sec 104 963 KBytes [ 5] 28.00-29.00 sec 1.68 GBytes 14.4 Gbits/sec 0 1.20 MBytes [ 5] 29.00-30.00 sec 1.67 GBytes 14.4 Gbits/sec 0 1.38 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 44.8 GBytes 12.8 Gbits/sec 4477 sender [ 5] 0.00-30.01 sec 44.8 GBytes 12.8 Gbits/sec receiver After a few runs: [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [ 5] local 192.168.10.31 port 52152 connected to 192.168.10.30 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.91 GBytes 16.4 Gbits/sec 67 606 KBytes [ 5] 1.00-2.00 sec 1.78 GBytes 15.3 Gbits/sec 0 1.07 MBytes [ 5] 2.00-3.00 sec 1.60 GBytes 13.7 Gbits/sec 0 1.54 MBytes [ 5] 3.00-4.00 sec 1.61 GBytes 13.8 Gbits/sec 0 1.61 MBytes [ 5] 4.00-5.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.61 MBytes [ 5] 5.00-6.00 sec 1.67 GBytes 14.3 Gbits/sec 0 1.61 MBytes [ 5] 6.00-7.00 sec 1.65 GBytes 14.1 Gbits/sec 0 1.61 MBytes [ 5] 7.00-8.00 sec 1.70 GBytes 14.6 Gbits/sec 0 1.61 MBytes [ 5] 8.00-9.00 sec 1.72 GBytes 14.8 Gbits/sec 0 1.61 MBytes [ 5] 9.00-10.00 sec 1.85 GBytes 15.9 Gbits/sec 0 1.61 MBytes [ 5] 10.00-11.00 sec 1.81 GBytes 15.5 Gbits/sec 0 1.61 MBytes [ 5] 11.00-12.00 sec 1.67 GBytes 14.3 Gbits/sec 0 1.61 MBytes [ 5] 12.00-13.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.61 MBytes [ 5] 13.00-14.00 sec 1.83 GBytes 15.7 Gbits/sec 0 1.61 MBytes [ 5] 14.00-15.00 sec 1.18 GBytes 10.1 Gbits/sec 0 794 KBytes [ 5] 15.00-16.00 sec 1.67 GBytes 14.4 Gbits/sec 0 1.60 MBytes [ 5] 16.00-17.00 sec 1.73 GBytes 14.8 Gbits/sec 0 1.60 MBytes [ 5] 17.00-18.00 sec 1.73 GBytes 14.9 Gbits/sec 0 1.60 MBytes [ 5] 18.00-19.00 sec 1.83 GBytes 15.7 Gbits/sec 0 1.61 MBytes [ 5] 19.00-20.00 sec 1.77 GBytes 15.2 Gbits/sec 0 1.61 MBytes [ 5] 20.00-21.00 sec 1.80 GBytes 15.5 Gbits/sec 0 1.61 MBytes [ 5] 21.00-22.00 sec 2.03 GBytes 17.4 Gbits/sec 0 1.61 MBytes [ 5] 22.00-23.00 sec 1.88 GBytes 16.1 Gbits/sec 0 1.61 MBytes [ 5] 23.00-24.00 sec 1.80 GBytes 15.5 Gbits/sec 0 1.61 MBytes [ 5] 24.00-25.01 sec 1.59 GBytes 13.6 Gbits/sec 0 1.61 MBytes [ 5] 25.01-26.00 sec 1.33 GBytes 11.4 Gbits/sec 0 1.61 MBytes [ 5] 26.00-27.00 sec 1.71 GBytes 14.7 Gbits/sec 0 1.61 MBytes [ 5] 27.00-28.00 sec 1.71 GBytes 14.7 Gbits/sec 97 1.01 MBytes [ 5] 28.00-29.00 sec 719 MBytes 6.03 Gbits/sec 0 1.01 MBytes [ 5] 29.00-30.00 sec 727 MBytes 6.10 Gbits/sec 0 1.01 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 49.3 GBytes 14.1 Gbits/sec 164 sender [ 5] 0.00-30.00 sec 49.3 GBytes 14.1 Gbits/sec receiver The CPU usage is rather low, the NIC is in the OCP port, so im sure thats designed accordingly and the NIC is bound to numa0. CPU: 0.0% user, 0.0% nice, 0.5% system, 0.7% interrupt, 98.8% idle PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 2195 root 1 52 0 17M 6884K select 83 0:14 27.99% iperf3 # vmstat -i -w1|grep mlx5 irq671: mlx5_core0 49969 47008 (this drops to about 14k with HW LRO enabled) The dump is rather large, i dont think i can attach it in the mailing list, but if you wish to see it i can upload it somewhere. Thank You. ---- On Mon, 13 Jun 2022 16:42:59 -0400 Hans Petter Selasky <mailto:hps@selasky.org> wrote ---- Some ideas: Try to disable "rxpause,txpause" when setting the media. Keep HW LRO off for now, it doesn't work for large number of connections. What is the CPU usage during test? Is iperf3 running on a CPU-core which has direct access to the NIC's numa domain? Is the NIC installed in the "correct" PCI high-performance slot? There are some sysctl knobs which may tell where the problem is, if it's PCI backpressure or something else. sysctl -a | grep diag_pci_enable sysctl -a | grep diag_general_enable Set these two to 1, then run some traffic and dump all mce sysctls: sysctl -a | grep mce > dump.txt --HPS Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use of the addressee(s) and may contain information that is privileged or confidential. If you are not the intended recipient, or responsible for delivering the information to the intended recipient, you are hereby notified that any dissemination, distribution, printing or copying of this e-mail and any attachments is strictly prohibited. If this e-mail and any attachments were received in error, please notify the sender by reply e-mail and delete the original message. Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use of the addressee(s) and may contain information that is privileged or confidential. If you are not the intended recipient, or responsible for delivering the information to the intended recipient, you are hereby notified that any dissemination, distribution, printing or copying of this e-mail and any attachments is strictly prohibited. If this e-mail and any attachments were received in error, please notify the sender by reply e-mail and delete the original message. [-- Attachment #2 --] <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt;"><div>Actually, i believe its the disabling to HW LRO that makes the difference (i disabled it and rx/tx pause previously). With rx/tx pause on and LRO off i get similar results. The throughput is still very sporadic though.<br></div><div><br></div><div>Connecting to host db-01, port 5201 <br></div><div>[ 5] local 192.168.10.31 port 59055 connected to 192.168.10.30 port 5201<br></div><div>[ ID] Interval Transfer Bitrate Retr Cwnd<br></div><div>[ 5] 0.00-1.00 sec 1.83 GBytes 15.7 Gbits/sec 0 1.11 MBytes<br></div><div>[ 5] 1.00-2.00 sec 1.83 GBytes 15.7 Gbits/sec 0 1.11 MBytes<br></div><div>[ 5] 2.00-3.00 sec 2.06 GBytes 17.7 Gbits/sec 0 1.12 MBytes<br></div><div>[ 5] 3.00-4.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.12 MBytes<br></div><div>[ 5] 4.00-5.00 sec 2.15 GBytes 18.4 Gbits/sec 0 1.12 MBytes<br></div><div>[ 5] 5.00-6.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.13 MBytes<br></div><div>[ 5] 6.00-7.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.13 MBytes<br></div><div>[ 5] 7.00-8.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.13 MBytes<br></div><div>[ 5] 8.00-9.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.13 MBytes<br></div><div>[ 5] 9.00-10.00 sec 2.15 GBytes 18.5 Gbits/sec 0 1.13 MBytes<br></div><div>[ 5] 10.00-11.00 sec 1.72 GBytes 14.7 Gbits/sec 0 1.14 MBytes<br></div><div>[ 5] 11.00-12.00 sec 1.18 GBytes 10.1 Gbits/sec 131 932 KBytes<br></div><div>[ 5] 12.00-13.00 sec 739 MBytes 6.20 Gbits/sec 352 913 KBytes<br></div><div>[ 5] 13.00-14.00 sec 724 MBytes 6.07 Gbits/sec 142 636 KBytes<br></div><div>[ 5] 14.00-15.00 sec 759 MBytes 6.37 Gbits/sec 246 748 KBytes<br></div><div>[ 5] 15.00-16.00 sec 748 MBytes 6.27 Gbits/sec 22 962 KBytes<br></div><div>[ 5] 16.00-17.00 sec 753 MBytes 6.32 Gbits/sec 170 711 KBytes<br></div><div>[ 5] 17.00-18.01 sec 690 MBytes 5.76 Gbits/sec 389 981 KBytes<br></div><div>[ 5] 18.01-19.00 sec 771 MBytes 6.50 Gbits/sec 145 896 KBytes<br></div><div>[ 5] 19.00-20.00 sec 741 MBytes 6.22 Gbits/sec 13 952 KBytes<br></div><div>[ 5] 20.00-21.00 sec 743 MBytes 6.23 Gbits/sec 86 985 KBytes<br></div><div>[ 5] 21.00-22.00 sec 751 MBytes 6.30 Gbits/sec 245 944 KBytes<br></div><div>[ 5] 22.00-23.00 sec 724 MBytes 6.07 Gbits/sec 178 774 KBytes<br></div><div>[ 5] 23.00-24.00 sec 728 MBytes 6.11 Gbits/sec 348 546 KBytes<br></div><div>[ 5] 24.00-25.01 sec 622 MBytes 5.18 Gbits/sec 215 842 KBytes<br></div><div>[ 5] 25.01-26.00 sec 920 MBytes 7.78 Gbits/sec 359 1.03 MBytes<br></div><div>[ 5] 26.00-27.00 sec 1.02 GBytes 8.80 Gbits/sec 385 1011 KBytes<br></div><div>[ 5] 27.00-28.00 sec 999 MBytes 8.38 Gbits/sec 161 757 KBytes<br></div><div>[ 5] 28.00-29.00 sec 1.68 GBytes 14.4 Gbits/sec 0 1.03 MBytes<br></div><div>[ 5] 29.00-30.00 sec 1.65 GBytes 14.2 Gbits/sec 0 1.51 MBytes<br></div><div>- - - - - - - - - - - - - - - - - - - - - - - - -<br></div><div>[ ID] Interval Transfer Bitrate Retr<br></div><div>[ 5] 0.00-30.00 sec 39.2 GBytes 11.2 Gbits/sec 3587 sender<br></div><div>[ 5] 0.00-30.00 sec 39.2 GBytes 11.2 Gbits/sec receiver<br></div><div><br></div><div><br></div><div>[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 <br></div><div>Connecting to host db-01, port 5201<br></div><div>[ 5] local 192.168.10.31 port 60931 connected to 192.168.10.30 port 5201<br></div><div>[ ID] Interval Transfer Bitrate Retr Cwnd<br></div><div>[ 5] 0.00-1.00 sec 931 MBytes 7.81 Gbits/sec 146 689 KBytes<br></div><div>[ 5] 1.00-2.00 sec 674 MBytes 5.66 Gbits/sec 267 575 KBytes<br></div><div>[ 5] 2.00-3.00 sec 641 MBytes 5.38 Gbits/sec 270 485 KBytes<br></div><div>[ 5] 3.00-4.00 sec 668 MBytes 5.60 Gbits/sec 221 801 KBytes<br></div><div>[ 5] 4.00-5.01 sec 532 MBytes 4.40 Gbits/sec 208 981 KBytes<br></div><div>[ 5] 5.01-6.00 sec 838 MBytes 7.13 Gbits/sec 291 949 KBytes<br></div><div>[ 5] 6.00-7.00 sec 703 MBytes 5.90 Gbits/sec 178 628 KBytes<br></div><div>[ 5] 7.00-8.00 sec 696 MBytes 5.84 Gbits/sec 85 928 KBytes<br></div><div>[ 5] 8.00-9.00 sec 1.06 GBytes 9.14 Gbits/sec 66 565 KBytes<br></div><div>[ 5] 9.00-10.00 sec 975 MBytes 8.18 Gbits/sec 0 801 KBytes<br></div><div>[ 5] 10.00-11.00 sec 679 MBytes 5.70 Gbits/sec 0 859 KBytes<br></div><div>[ 5] 11.00-12.01 sec 628 MBytes 5.23 Gbits/sec 123 1.12 MBytes<br></div><div>[ 5] 12.01-13.00 sec 731 MBytes 6.16 Gbits/sec 311 864 KBytes<br></div><div>[ 5] 13.00-14.00 sec 696 MBytes 5.86 Gbits/sec 39 798 KBytes<br></div><div>[ 5] 14.00-15.00 sec 914 MBytes 7.66 Gbits/sec 65 1024 KBytes<br></div><div>[ 5] 15.00-16.00 sec 1.66 GBytes 14.2 Gbits/sec 0 1.47 MBytes<br></div><div>[ 5] 16.00-17.00 sec 955 MBytes 8.01 Gbits/sec 222 260 KBytes<br></div><div>[ 5] 17.00-18.00 sec 769 MBytes 6.45 Gbits/sec 0 520 KBytes<br></div><div>[ 5] 18.00-19.00 sec 781 MBytes 6.55 Gbits/sec 119 432 KBytes<br></div><div>[ 5] 19.00-20.00 sec 779 MBytes 6.54 Gbits/sec 148 816 KBytes<br></div><div>[ 5] 20.00-21.00 sec 771 MBytes 6.47 Gbits/sec 127 965 KBytes<br></div><div>[ 5] 21.00-22.00 sec 745 MBytes 6.25 Gbits/sec 337 927 KBytes<br></div><div>[ 5] 22.00-23.00 sec 637 MBytes 5.35 Gbits/sec 0 938 KBytes<br></div><div>[ 5] 23.00-24.00 sec 626 MBytes 5.25 Gbits/sec 0 938 KBytes<br></div><div>[ 5] 24.00-25.00 sec 1.96 GBytes 16.8 Gbits/sec 185 862 KBytes<br></div><div>[ 5] 25.00-26.00 sec 1.71 GBytes 14.7 Gbits/sec 22 945 KBytes<br></div><div>[ 5] 26.00-27.00 sec 2.14 GBytes 18.3 Gbits/sec 365 593 KBytes<br></div><div>[ 5] 27.00-28.00 sec 1.56 GBytes 13.4 Gbits/sec 33 1.23 MBytes<br></div><div>[ 5] 28.00-29.00 sec 1.67 GBytes 14.4 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 29.00-30.00 sec 1.65 GBytes 14.2 Gbits/sec 35 1.14 MBytes<br></div><div>- - - - - - - - - - - - - - - - - - - - - - - - -<br></div><div>[ ID] Interval Transfer Bitrate Retr<br></div><div>[ 5] 0.00-30.00 sec 29.4 GBytes 8.42 Gbits/sec 3863 sender<br></div><div>[ 5] 0.00-30.00 sec 29.4 GBytes 8.42 Gbits/sec receiver<br></div><div><br></div><div><br></div><div><br></div><div class="zmail_extra_hr" style="border-top: 1px solid rgb(204, 204, 204); height: 0px; margin-top: 10px; margin-bottom: 10px; line-height: 0px;"><br></div><div class="zmail_extra" data-zbluepencil-ignore="true"><div><br></div><div id="Zm-_Id_-Sgn1">---- On Tue, 14 Jun 2022 10:21:51 -0400 <b>Mike Jakubik <mike.jakubik@swiftsmsgateway.com></b> wrote ----<br></div><div><br></div><blockquote style="margin: 0px;"><div><div style="font-family: Verdana, Arial, Helvetica, sans-serif;font-size: 10.0pt;"><div>Disabling rx/tx pause seems to produce higher peaks.<br></div><div><br></div><div>[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 <br></div><div>Connecting to host db-01, port 5201<br></div><div>[ 5] local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5201<br></div><div>[ ID] Interval Transfer Bitrate Retr Cwnd<br></div><div>[ 5] 0.00-1.00 sec 1.89 GBytes 16.2 Gbits/sec 0 1.10 MBytes<br></div><div>[ 5] 1.00-2.00 sec 1.86 GBytes 15.9 Gbits/sec 0 1.10 MBytes<br></div><div>[ 5] 2.00-3.00 sec 2.05 GBytes 17.6 Gbits/sec 0 1.11 MBytes<br></div><div>[ 5] 3.00-4.00 sec 859 MBytes 7.20 Gbits/sec 21 938 KBytes<br></div><div>[ 5] 4.00-5.00 sec 652 MBytes 5.47 Gbits/sec 0 1.01 MBytes<br></div><div>[ 5] 5.00-6.00 sec 659 MBytes 5.53 Gbits/sec 0 1.03 MBytes<br></div><div>[ 5] 6.00-7.00 sec 666 MBytes 5.59 Gbits/sec 0 1.05 MBytes<br></div><div>[ 5] 7.00-8.00 sec 657 MBytes 5.51 Gbits/sec 98 989 KBytes<br></div><div>[ 5] 8.00-9.00 sec 665 MBytes 5.58 Gbits/sec 139 712 KBytes<br></div><div>[ 5] 9.00-10.00 sec 647 MBytes 5.43 Gbits/sec 0 1.02 MBytes<br></div><div>[ 5] 10.00-11.00 sec 650 MBytes 5.45 Gbits/sec 4 606 KBytes<br></div><div>[ 5] 11.00-12.00 sec 1.53 GBytes 13.1 Gbits/sec 358 1.07 MBytes<br></div><div>[ 5] 12.00-13.00 sec 2.10 GBytes 18.1 Gbits/sec 162 837 KBytes<br></div><div>[ 5] 13.00-14.00 sec 2.09 GBytes 18.0 Gbits/sec 332 838 KBytes<br></div><div>[ 5] 14.00-15.00 sec 2.43 GBytes 20.9 Gbits/sec 639 747 KBytes<br></div><div>[ 5] 15.00-16.00 sec 2.38 GBytes 20.4 Gbits/sec 612 1.02 MBytes<br></div><div>[ 5] 16.00-17.00 sec 2.25 GBytes 19.3 Gbits/sec 535 1.24 MBytes<br></div><div>[ 5] 17.00-18.00 sec 2.52 GBytes 21.6 Gbits/sec 818 423 KBytes<br></div><div>[ 5] 18.00-19.00 sec 2.29 GBytes 19.7 Gbits/sec 218 444 KBytes<br></div><div>[ 5] 19.00-20.00 sec 2.29 GBytes 19.7 Gbits/sec 114 859 KBytes<br></div><div>[ 5] 20.00-21.00 sec 1.65 GBytes 14.1 Gbits/sec 100 541 KBytes<br></div><div>[ 5] 21.00-22.00 sec 1.01 GBytes 8.67 Gbits/sec 0 639 KBytes<br></div><div>[ 5] 22.00-23.00 sec 625 MBytes 5.24 Gbits/sec 0 648 KBytes<br></div><div>[ 5] 23.00-24.00 sec 630 MBytes 5.28 Gbits/sec 0 648 KBytes<br></div><div>[ 5] 24.00-25.00 sec 1.56 GBytes 13.4 Gbits/sec 0 702 KBytes<br></div><div>[ 5] 25.00-26.00 sec 1.78 GBytes 15.3 Gbits/sec 118 406 KBytes<br></div><div>[ 5] 26.00-27.00 sec 1.37 GBytes 11.8 Gbits/sec 105 890 KBytes<br></div><div>[ 5] 27.00-28.00 sec 1.82 GBytes 15.6 Gbits/sec 104 963 KBytes<br></div><div>[ 5] 28.00-29.00 sec 1.68 GBytes 14.4 Gbits/sec 0 1.20 MBytes<br></div><div>[ 5] 29.00-30.00 sec 1.67 GBytes 14.4 Gbits/sec 0 1.38 MBytes<br></div><div>- - - - - - - - - - - - - - - - - - - - - - - - - <br></div><div>[ ID] Interval Transfer Bitrate Retr<br></div><div>[ 5] 0.00-30.00 sec 44.8 GBytes 12.8 Gbits/sec 4477 sender<br></div><div>[ 5] 0.00-30.01 sec 44.8 GBytes 12.8 Gbits/sec receiver<br></div><div><br></div><div>After a few runs:<br></div><div><br></div><div>[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 <br></div><div>Connecting to host db-01, port 5201<br></div><div>[ 5] local 192.168.10.31 port 52152 connected to 192.168.10.30 port 5201<br></div><div>[ ID] Interval Transfer Bitrate Retr Cwnd<br></div><div>[ 5] 0.00-1.00 sec 1.91 GBytes 16.4 Gbits/sec 67 606 KBytes<br></div><div>[ 5] 1.00-2.00 sec 1.78 GBytes 15.3 Gbits/sec 0 1.07 MBytes<br></div><div>[ 5] 2.00-3.00 sec 1.60 GBytes 13.7 Gbits/sec 0 1.54 MBytes<br></div><div>[ 5] 3.00-4.00 sec 1.61 GBytes 13.8 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 4.00-5.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 5.00-6.00 sec 1.67 GBytes 14.3 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 6.00-7.00 sec 1.65 GBytes 14.1 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 7.00-8.00 sec 1.70 GBytes 14.6 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 8.00-9.00 sec 1.72 GBytes 14.8 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 9.00-10.00 sec 1.85 GBytes 15.9 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 10.00-11.00 sec 1.81 GBytes 15.5 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 11.00-12.00 sec 1.67 GBytes 14.3 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 12.00-13.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 13.00-14.00 sec 1.83 GBytes 15.7 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 14.00-15.00 sec 1.18 GBytes 10.1 Gbits/sec 0 794 KBytes<br></div><div>[ 5] 15.00-16.00 sec 1.67 GBytes 14.4 Gbits/sec 0 1.60 MBytes<br></div><div>[ 5] 16.00-17.00 sec 1.73 GBytes 14.8 Gbits/sec 0 1.60 MBytes<br></div><div>[ 5] 17.00-18.00 sec 1.73 GBytes 14.9 Gbits/sec 0 1.60 MBytes<br></div><div>[ 5] 18.00-19.00 sec 1.83 GBytes 15.7 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 19.00-20.00 sec 1.77 GBytes 15.2 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 20.00-21.00 sec 1.80 GBytes 15.5 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 21.00-22.00 sec 2.03 GBytes 17.4 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 22.00-23.00 sec 1.88 GBytes 16.1 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 23.00-24.00 sec 1.80 GBytes 15.5 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 24.00-25.01 sec 1.59 GBytes 13.6 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 25.01-26.00 sec 1.33 GBytes 11.4 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 26.00-27.00 sec 1.71 GBytes 14.7 Gbits/sec 0 1.61 MBytes<br></div><div>[ 5] 27.00-28.00 sec 1.71 GBytes 14.7 Gbits/sec 97 1.01 MBytes<br></div><div>[ 5] 28.00-29.00 sec 719 MBytes 6.03 Gbits/sec 0 1.01 MBytes<br></div><div>[ 5] 29.00-30.00 sec 727 MBytes 6.10 Gbits/sec 0 1.01 MBytes<br></div><div>- - - - - - - - - - - - - - - - - - - - - - - - -<br></div><div>[ ID] Interval Transfer Bitrate Retr<br></div><div>[ 5] 0.00-30.00 sec 49.3 GBytes 14.1 Gbits/sec 164 sender<br></div><div>[ 5] 0.00-30.00 sec 49.3 GBytes 14.1 Gbits/sec receiver<br></div><div><br></div><div><br></div><div><br></div><div>The CPU usage is rather low, the NIC is in the OCP port, so im sure thats designed accordingly and the NIC is bound to numa0.<br></div><div><br></div><div>CPU: 0.0% user, 0.0% nice, 0.5% system, 0.7% interrupt, 98.8% idle<br></div><div><br></div><div> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND <br></div><div>2195 root 1 52 0 17M 6884K select 83 0:14 27.99% iperf3<br></div><div><br></div><div># vmstat -i -w1|grep mlx5<br></div><div>irq671: mlx5_core0 49969 47008<br></div><div><br></div><div>(this drops to about 14k with HW LRO enabled)<br></div><div><br></div><div>The dump is rather large, i dont think i can attach it in the mailing list, but if you wish to see it i can upload it somewhere.<br></div><div><br></div><div>Thank You.<br></div><div><br></div><div><br></div><div class="x_-614958649zmail_extra_hr" style="border-top: 1.0px solid rgb(204,204,204);min-height: 0.0px;margin-top: 10.0px;margin-bottom: 10.0px;line-height: 0.0px;"><br></div><div class="x_-614958649zmail_extra"><div><br></div><div id="x_-614958649Zm-_Id_-Sgn1">---- On Mon, 13 Jun 2022 16:42:59 -0400 <b>Hans Petter Selasky <<a href="mailto:hps@selasky.org" target="_blank">hps@selasky.org</a>></b> wrote ----<br></div><div><br></div><blockquote style="margin: 0.0px;"><div><br>Some ideas: <br> <br>Try to disable "rxpause,txpause" when setting the media. <br> <br>Keep HW LRO off for now, it doesn't work for large number of connections. <br> <br>What is the CPU usage during test? Is iperf3 running on a CPU-core which <br>has direct access to the NIC's numa domain? <br> <br>Is the NIC installed in the "correct" PCI high-performance slot? <br> <br>There are some sysctl knobs which may tell where the problem is, if it's <br>PCI backpressure or something else. <br> <br>sysctl -a | grep diag_pci_enable <br>sysctl -a | grep diag_general_enable <br> <br>Set these two to 1, then run some traffic and dump all mce sysctls: <br> <br>sysctl -a | grep mce > dump.txt <br> <br>--HPS <br> <br></div></blockquote></div><div><br></div><div class="x_-614958649zmail_signature_below"><div id="x_-614958649Zm-_Id_-Sgn"><div>Mike Jakubik<br></div><div><a target="_blank" href="https://www.swiftsmsgateway.com/">https://www.swiftsmsgateway.com/</a><br></div><div><br></div><div><span class="size" style="font-size:10px">Disclaimer: This e-mail and any attachments are intended only for the use of the addressee(s) and may contain information that is privileged or confidential. If you are not the intended recipient, or responsible for delivering the information to the intended recipient, you are hereby notified that any dissemination, distribution, printing or copying of this e-mail and any attachments is strictly prohibited. If this e-mail and any attachments were received in error, please notify the sender by reply e-mail and delete the original message.</span><br></div></div></div><div><br></div></div><br></div></blockquote></div><div><br></div><div class="zmail_signature_below"><div id="Zm-_Id_-Sgn" data-zbluepencil-ignore="true" data-sigid="7886077000000096004"><div>Mike Jakubik<br></div><div><a target="_blank" href="https://www.swiftsmsgateway.com/">https://www.swiftsmsgateway.com/</a><br></div><div><br></div><div><span class="size" style="font-size:10px">Disclaimer: This e-mail and any attachments are intended only for the use of the addressee(s) and may contain information that is privileged or confidential. If you are not the intended recipient, or responsible for delivering the information to the intended recipient, you are hereby notified that any dissemination, distribution, printing or copying of this e-mail and any attachments is strictly prohibited. If this e-mail and any attachments were received in error, please notify the sender by reply e-mail and delete the original message.</span><br></div></div></div><div><br></div></div><br></body></html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?18162a4a3f6.10a1a03d11472072.3783895140221599214>