Skip site navigation (1)Skip section navigation (2) 
Date:      Thu, 16 Jun 2022 16:48:50 -0400
From:      Mike Jakubik <mike.jakubik@swiftsmsgateway.com>
To:        "freebsd-net" <freebsd-net@FreeBSD.org>
Subject:   Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)
Message-ID:  <1816e469bdf.126cdb81b2139485.369352368493375815@swiftsmsgateway.com>
In-Reply-To: <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com>
References:  <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com>
next in thread | previous in thread | raw e-mail | index | archive | help 

[-- Attachment #1 --]
After multiple tests and tweaks i believe the issue is not with the HW or Numa related (Infinity fabric should do around 32GB) but rather with FreeBSD TCP/IP stack. It's like it cant figure itself out properly for the speed that the HW can do, i keep getting widely varying results when testing. Below is an example of two tests, with about a 15 second break in between the two.



[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 49155 connected to 192.168.10.30 port 5201

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd

[  5]   0.00-1.00   sec   991 MBytes  8.32 Gbits/sec  268    579 KBytes

[  5]   1.00-2.00   sec   945 MBytes  7.93 Gbits/sec  369    777 KBytes

[  5]   2.00-3.00   sec   793 MBytes  6.65 Gbits/sec   60   1.03 MBytes

[  5]   3.00-4.00   sec   666 MBytes  5.59 Gbits/sec  203    976 KBytes

[  5]   4.00-5.01   sec   575 MBytes  4.78 Gbits/sec  202   1.13 MBytes

[  5]   5.01-6.00   sec   169 MBytes  1.43 Gbits/sec  134    699 KBytes

[  5]   6.00-7.00   sec  1.21 GBytes  10.4 Gbits/sec  383   1.08 MBytes

[  5]   7.00-8.00   sec  1.21 GBytes  10.4 Gbits/sec    0   1.16 MBytes

[  5]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec  124    780 KBytes

[  5]   9.00-10.00  sec   690 MBytes  5.79 Gbits/sec  316    605 KBytes

[  5]  10.00-11.00  sec   685 MBytes  5.75 Gbits/sec   97    854 KBytes

[  5]  11.00-12.00  sec  1.08 GBytes  9.30 Gbits/sec  383    538 KBytes

[  5]  12.00-13.00  sec   682 MBytes  5.72 Gbits/sec   88    870 KBytes

[  5]  13.00-14.00  sec   678 MBytes  5.69 Gbits/sec  123    964 KBytes

[  5]  14.00-15.00  sec   670 MBytes  5.62 Gbits/sec  290    763 KBytes

[  5]  15.00-16.00  sec  1.01 GBytes  8.71 Gbits/sec  228   1.08 MBytes

[  5]  16.00-17.00  sec   886 MBytes  7.44 Gbits/sec  118    615 KBytes

[  5]  17.00-18.00  sec   734 MBytes  6.16 Gbits/sec  291    902 KBytes

[  5]  18.00-19.00  sec  1.04 GBytes  8.96 Gbits/sec  212    323 KBytes

[  5]  19.00-20.00  sec   710 MBytes  5.96 Gbits/sec  193    547 KBytes

[  5]  20.00-21.00  sec   693 MBytes  5.82 Gbits/sec  370    942 KBytes

[  5]  21.00-22.00  sec   704 MBytes  5.91 Gbits/sec   80   1022 KBytes

[  5]  22.00-23.00  sec  1.26 GBytes  10.8 Gbits/sec  262    965 KBytes

[  5]  23.00-24.00  sec   828 MBytes  6.94 Gbits/sec  202    763 KBytes

[  5]  24.00-25.00  sec   774 MBytes  6.49 Gbits/sec  227    581 KBytes

[  5]  25.00-26.00  sec   734 MBytes  6.15 Gbits/sec  256    664 KBytes

[  5]  26.00-27.00  sec   753 MBytes  6.32 Gbits/sec  331    540 KBytes

[  5]  27.00-28.00  sec   764 MBytes  6.41 Gbits/sec  298    823 KBytes

[  5]  28.00-29.00  sec   757 MBytes  6.35 Gbits/sec  123    850 KBytes

[  5]  29.00-30.00  sec   754 MBytes  6.32 Gbits/sec   74    970 KBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Retr

[  5]   0.00-30.00  sec  24.4 GBytes  6.98 Gbits/sec  6305             sender

[  5]   0.00-30.00  sec  24.4 GBytes  6.98 Gbits/sec                  receiver



iperf Done.



[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 25061 connected to 192.168.10.30 port 5201

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd

[  5]   0.00-1.00   sec  1.81 GBytes  15.5 Gbits/sec    0   1.11 MBytes

[  5]   1.00-2.00   sec  1.83 GBytes  15.7 Gbits/sec    0   1.11 MBytes

[  5]   2.00-3.00   sec  1.98 GBytes  17.0 Gbits/sec    0   1.11 MBytes

[  5]   3.00-4.00   sec  2.11 GBytes  18.1 Gbits/sec    0   1.11 MBytes

[  5]   4.00-5.00   sec  2.12 GBytes  18.2 Gbits/sec    0   1.11 MBytes

[  5]   5.00-6.00   sec  2.16 GBytes  18.5 Gbits/sec    0   1.11 MBytes

[  5]   6.00-7.00   sec  1.90 GBytes  16.3 Gbits/sec    0   1.12 MBytes

[  5]   7.00-8.02   sec  1.28 GBytes  10.8 Gbits/sec    0   1.12 MBytes

[  5]   8.02-9.00   sec  1.83 GBytes  16.0 Gbits/sec    0   1.17 MBytes

[  5]   9.00-10.00  sec  1.91 GBytes  16.4 Gbits/sec    0   1.20 MBytes

[  5]  10.00-11.00  sec  1.79 GBytes  15.3 Gbits/sec    0   1.60 MBytes

[  5]  11.00-12.00  sec  1.77 GBytes  15.2 Gbits/sec    0   1.60 MBytes

[  5]  12.00-13.00  sec  1.69 GBytes  14.5 Gbits/sec    0   1.61 MBytes

[  5]  13.00-14.00  sec  1.57 GBytes  13.5 Gbits/sec    0   1.61 MBytes

[  5]  14.00-15.00  sec  1.60 GBytes  13.8 Gbits/sec    0   1.61 MBytes

[  5]  15.00-16.00  sec  1.89 GBytes  16.2 Gbits/sec    0   1.61 MBytes

[  5]  16.00-17.00  sec  1.76 GBytes  15.1 Gbits/sec    0   1.61 MBytes

[  5]  17.00-18.00  sec  1.93 GBytes  16.6 Gbits/sec    0   1.61 MBytes

[  5]  18.00-19.00  sec  1.77 GBytes  15.2 Gbits/sec    0   1.61 MBytes

[  5]  19.00-20.00  sec  1.68 GBytes  14.5 Gbits/sec    0   1.61 MBytes

[  5]  20.00-21.00  sec  1.66 GBytes  14.3 Gbits/sec    0   1.61 MBytes

[  5]  21.00-22.00  sec  1.75 GBytes  15.1 Gbits/sec    0   1.61 MBytes

[  5]  22.00-23.00  sec  1.76 GBytes  15.1 Gbits/sec    0   1.61 MBytes

[  5]  23.00-24.00  sec  2.05 GBytes  17.7 Gbits/sec    0   1.61 MBytes

[  5]  24.00-25.00  sec  1.79 GBytes  15.4 Gbits/sec    0   1.61 MBytes

[  5]  25.00-26.00  sec  1.71 GBytes  14.7 Gbits/sec    0   1.61 MBytes

[  5]  26.00-27.00  sec  2.08 GBytes  17.9 Gbits/sec    0   1.61 MBytes

[  5]  27.00-28.00  sec  2.09 GBytes  18.0 Gbits/sec    0   1.61 MBytes

[  5]  28.00-29.00  sec  2.05 GBytes  17.6 Gbits/sec    0   1.61 MBytes

[  5]  29.00-30.00  sec  1.81 GBytes  15.6 Gbits/sec    0   1.61 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval           Transfer     Bitrate         Retr

[  5]   0.00-30.00  sec  55.2 GBytes  15.8 Gbits/sec    0             sender

[  5]   0.00-30.00  sec  55.2 GBytes  15.8 Gbits/sec                  receiver



iperf Done.
[-- Attachment #2 --]
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt;"><div>After multiple tests and tweaks i believe the issue is not with the HW or Numa related (Infinity fabric should do around 32GB) but rather with FreeBSD TCP/IP stack. It's like it cant figure itself out properly for the speed that the HW can do, i keep getting widely varying results when testing. Below is an example of two tests, with about a 15 second break in between the two.<br></div><div><br></div><div>[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 <br></div><div>Connecting to host db-01, port 5201<br></div><div>[&nbsp; 5] local 192.168.10.31 port 49155 connected to 192.168.10.30 port 5201<br></div><div>[ ID] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Transfer&nbsp;&nbsp;&nbsp;&nbsp; Bitrate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retr&nbsp; Cwnd<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-1.00&nbsp;&nbsp; sec&nbsp;&nbsp; 991 MBytes&nbsp; 8.32 Gbits/sec&nbsp; 268&nbsp;&nbsp;&nbsp; 579 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 1.00-2.00&nbsp;&nbsp; sec&nbsp;&nbsp; 945 MBytes&nbsp; 7.93 Gbits/sec&nbsp; 369&nbsp;&nbsp;&nbsp; 777 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 2.00-3.00&nbsp;&nbsp; sec&nbsp;&nbsp; 793 MBytes&nbsp; 6.65 Gbits/sec&nbsp;&nbsp; 60&nbsp;&nbsp; 1.03 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 3.00-4.00&nbsp;&nbsp; sec&nbsp;&nbsp; 666 MBytes&nbsp; 5.59 Gbits/sec&nbsp; 203&nbsp;&nbsp;&nbsp; 976 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 4.00-5.01&nbsp;&nbsp; sec&nbsp;&nbsp; 575 MBytes&nbsp; 4.78 Gbits/sec&nbsp; 202&nbsp;&nbsp; 1.13 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 5.01-6.00&nbsp;&nbsp; sec&nbsp;&nbsp; 169 MBytes&nbsp; 1.43 Gbits/sec&nbsp; 134&nbsp;&nbsp;&nbsp; 699 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 6.00-7.00&nbsp;&nbsp; sec&nbsp; 1.21 GBytes&nbsp; 10.4 Gbits/sec&nbsp; 383&nbsp;&nbsp; 1.08 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 7.00-8.00&nbsp;&nbsp; sec&nbsp; 1.21 GBytes&nbsp; 10.4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.16 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 8.00-9.00&nbsp;&nbsp; sec&nbsp; 1.32 GBytes&nbsp; 11.3 Gbits/sec&nbsp; 124&nbsp;&nbsp;&nbsp; 780 KBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 9.00-10.00&nbsp; sec&nbsp;&nbsp; 690 MBytes&nbsp; 5.79 Gbits/sec&nbsp; 316&nbsp;&nbsp;&nbsp; 605 KBytes<br></div><div>[&nbsp; 5]&nbsp; 10.00-11.00&nbsp; sec&nbsp;&nbsp; 685 MBytes&nbsp; 5.75 Gbits/sec&nbsp;&nbsp; 97&nbsp;&nbsp;&nbsp; 854 KBytes<br></div><div>[&nbsp; 5]&nbsp; 11.00-12.00&nbsp; sec&nbsp; 1.08 GBytes&nbsp; 9.30 Gbits/sec&nbsp; 383&nbsp;&nbsp;&nbsp; 538 KBytes<br></div><div>[&nbsp; 5]&nbsp; 12.00-13.00&nbsp; sec&nbsp;&nbsp; 682 MBytes&nbsp; 5.72 Gbits/sec&nbsp;&nbsp; 88&nbsp;&nbsp;&nbsp; 870 KBytes<br></div><div>[&nbsp; 5]&nbsp; 13.00-14.00&nbsp; sec&nbsp;&nbsp; 678 MBytes&nbsp; 5.69 Gbits/sec&nbsp; 123&nbsp;&nbsp;&nbsp; 964 KBytes<br></div><div>[&nbsp; 5]&nbsp; 14.00-15.00&nbsp; sec&nbsp;&nbsp; 670 MBytes&nbsp; 5.62 Gbits/sec&nbsp; 290&nbsp;&nbsp;&nbsp; 763 KBytes<br></div><div>[&nbsp; 5]&nbsp; 15.00-16.00&nbsp; sec&nbsp; 1.01 GBytes&nbsp; 8.71 Gbits/sec&nbsp; 228&nbsp;&nbsp; 1.08 MBytes<br></div><div>[&nbsp; 5]&nbsp; 16.00-17.00&nbsp; sec&nbsp;&nbsp; 886 MBytes&nbsp; 7.44 Gbits/sec&nbsp; 118&nbsp;&nbsp;&nbsp; 615 KBytes<br></div><div>[&nbsp; 5]&nbsp; 17.00-18.00&nbsp; sec&nbsp;&nbsp; 734 MBytes&nbsp; 6.16 Gbits/sec&nbsp; 291&nbsp;&nbsp;&nbsp; 902 KBytes<br></div><div>[&nbsp; 5]&nbsp; 18.00-19.00&nbsp; sec&nbsp; 1.04 GBytes&nbsp; 8.96 Gbits/sec&nbsp; 212&nbsp;&nbsp;&nbsp; 323 KBytes<br></div><div>[&nbsp; 5]&nbsp; 19.00-20.00&nbsp; sec&nbsp;&nbsp; 710 MBytes&nbsp; 5.96 Gbits/sec&nbsp; 193&nbsp;&nbsp;&nbsp; 547 KBytes<br></div><div>[&nbsp; 5]&nbsp; 20.00-21.00&nbsp; sec&nbsp;&nbsp; 693 MBytes&nbsp; 5.82 Gbits/sec&nbsp; 370&nbsp;&nbsp;&nbsp; 942 KBytes<br></div><div>[&nbsp; 5]&nbsp; 21.00-22.00&nbsp; sec&nbsp;&nbsp; 704 MBytes&nbsp; 5.91 Gbits/sec&nbsp;&nbsp; 80&nbsp;&nbsp; 1022 KBytes<br></div><div>[&nbsp; 5]&nbsp; 22.00-23.00&nbsp; sec&nbsp; 1.26 GBytes&nbsp; 10.8 Gbits/sec&nbsp; 262&nbsp;&nbsp;&nbsp; 965 KBytes<br></div><div>[&nbsp; 5]&nbsp; 23.00-24.00&nbsp; sec&nbsp;&nbsp; 828 MBytes&nbsp; 6.94 Gbits/sec&nbsp; 202&nbsp;&nbsp;&nbsp; 763 KBytes<br></div><div>[&nbsp; 5]&nbsp; 24.00-25.00&nbsp; sec&nbsp;&nbsp; 774 MBytes&nbsp; 6.49 Gbits/sec&nbsp; 227&nbsp;&nbsp;&nbsp; 581 KBytes<br></div><div>[&nbsp; 5]&nbsp; 25.00-26.00&nbsp; sec&nbsp;&nbsp; 734 MBytes&nbsp; 6.15 Gbits/sec&nbsp; 256&nbsp;&nbsp;&nbsp; 664 KBytes<br></div><div>[&nbsp; 5]&nbsp; 26.00-27.00&nbsp; sec&nbsp;&nbsp; 753 MBytes&nbsp; 6.32 Gbits/sec&nbsp; 331&nbsp;&nbsp;&nbsp; 540 KBytes<br></div><div>[&nbsp; 5]&nbsp; 27.00-28.00&nbsp; sec&nbsp;&nbsp; 764 MBytes&nbsp; 6.41 Gbits/sec&nbsp; 298&nbsp;&nbsp;&nbsp; 823 KBytes<br></div><div>[&nbsp; 5]&nbsp; 28.00-29.00&nbsp; sec&nbsp;&nbsp; 757 MBytes&nbsp; 6.35 Gbits/sec&nbsp; 123&nbsp;&nbsp;&nbsp; 850 KBytes<br></div><div>[&nbsp; 5]&nbsp; 29.00-30.00&nbsp; sec&nbsp;&nbsp; 754 MBytes&nbsp; 6.32 Gbits/sec&nbsp;&nbsp; 74&nbsp;&nbsp;&nbsp; 970 KBytes<br></div><div>- - - - - - - - - - - - - - - - - - - - - - - - -<br></div><div>[ ID] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Transfer&nbsp;&nbsp;&nbsp;&nbsp; Bitrate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retr<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-30.00&nbsp; sec&nbsp; 24.4 GBytes&nbsp; 6.98 Gbits/sec&nbsp; 6305&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sender<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-30.00&nbsp; sec&nbsp; 24.4 GBytes&nbsp; 6.98 Gbits/sec&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; receiver<br></div><div><br></div><div>iperf Done.<br></div><div><br></div><div>[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01<br></div><div>Connecting to host db-01, port 5201<br></div><div>[&nbsp; 5] local 192.168.10.31 port 25061 connected to 192.168.10.30 port 5201<br></div><div>[ ID] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Transfer&nbsp;&nbsp;&nbsp;&nbsp; Bitrate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retr&nbsp; Cwnd<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-1.00&nbsp;&nbsp; sec&nbsp; 1.81 GBytes&nbsp; 15.5 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.11 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 1.00-2.00&nbsp;&nbsp; sec&nbsp; 1.83 GBytes&nbsp; 15.7 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.11 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 2.00-3.00&nbsp;&nbsp; sec&nbsp; 1.98 GBytes&nbsp; 17.0 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.11 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 3.00-4.00&nbsp;&nbsp; sec&nbsp; 2.11 GBytes&nbsp; 18.1 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.11 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 4.00-5.00&nbsp;&nbsp; sec&nbsp; 2.12 GBytes&nbsp; 18.2 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.11 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 5.00-6.00&nbsp;&nbsp; sec&nbsp; 2.16 GBytes&nbsp; 18.5 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.11 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 6.00-7.00&nbsp;&nbsp; sec&nbsp; 1.90 GBytes&nbsp; 16.3 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.12 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 7.00-8.02&nbsp;&nbsp; sec&nbsp; 1.28 GBytes&nbsp; 10.8 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.12 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 8.02-9.00&nbsp;&nbsp; sec&nbsp; 1.83 GBytes&nbsp; 16.0 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.17 MBytes<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 9.00-10.00&nbsp; sec&nbsp; 1.91 GBytes&nbsp; 16.4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.20 MBytes<br></div><div>[&nbsp; 5]&nbsp; 10.00-11.00&nbsp; sec&nbsp; 1.79 GBytes&nbsp; 15.3 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.60 MBytes<br></div><div>[&nbsp; 5]&nbsp; 11.00-12.00&nbsp; sec&nbsp; 1.77 GBytes&nbsp; 15.2 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.60 MBytes<br></div><div>[&nbsp; 5]&nbsp; 12.00-13.00&nbsp; sec&nbsp; 1.69 GBytes&nbsp; 14.5 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 13.00-14.00&nbsp; sec&nbsp; 1.57 GBytes&nbsp; 13.5 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 14.00-15.00&nbsp; sec&nbsp; 1.60 GBytes&nbsp; 13.8 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 15.00-16.00&nbsp; sec&nbsp; 1.89 GBytes&nbsp; 16.2 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 16.00-17.00&nbsp; sec&nbsp; 1.76 GBytes&nbsp; 15.1 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 17.00-18.00&nbsp; sec&nbsp; 1.93 GBytes&nbsp; 16.6 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 18.00-19.00&nbsp; sec&nbsp; 1.77 GBytes&nbsp; 15.2 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 19.00-20.00&nbsp; sec&nbsp; 1.68 GBytes&nbsp; 14.5 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 20.00-21.00&nbsp; sec&nbsp; 1.66 GBytes&nbsp; 14.3 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 21.00-22.00&nbsp; sec&nbsp; 1.75 GBytes&nbsp; 15.1 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 22.00-23.00&nbsp; sec&nbsp; 1.76 GBytes&nbsp; 15.1 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 23.00-24.00&nbsp; sec&nbsp; 2.05 GBytes&nbsp; 17.7 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 24.00-25.00&nbsp; sec&nbsp; 1.79 GBytes&nbsp; 15.4 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 25.00-26.00&nbsp; sec&nbsp; 1.71 GBytes&nbsp; 14.7 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 26.00-27.00&nbsp; sec&nbsp; 2.08 GBytes&nbsp; 17.9 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 27.00-28.00&nbsp; sec&nbsp; 2.09 GBytes&nbsp; 18.0 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 28.00-29.00&nbsp; sec&nbsp; 2.05 GBytes&nbsp; 17.6 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>[&nbsp; 5]&nbsp; 29.00-30.00&nbsp; sec&nbsp; 1.81 GBytes&nbsp; 15.6 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp; 1.61 MBytes<br></div><div>- - - - - - - - - - - - - - - - - - - - - - - - -<br></div><div>[ ID] Interval&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Transfer&nbsp;&nbsp;&nbsp;&nbsp; Bitrate&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Retr<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-30.00&nbsp; sec&nbsp; 55.2 GBytes&nbsp; 15.8 Gbits/sec&nbsp;&nbsp;&nbsp; 0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; sender<br></div><div>[&nbsp; 5]&nbsp;&nbsp; 0.00-30.00&nbsp; sec&nbsp; 55.2 GBytes&nbsp; 15.8 Gbits/sec&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; receiver<br></div><div><br></div><div>iperf Done.<br></div></div><br></body></html>

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1816e469bdf.126cdb81b2139485.369352368493375815>