From nobody Mon Jun 13 18:25:36 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B431B84084C for ; Mon, 13 Jun 2022 18:25:48 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) Received: from sender4-op-o13.zoho.com (sender4-op-o13.zoho.com [136.143.188.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LMKkg3gTQz3NnF for ; Mon, 13 Jun 2022 18:25:47 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) ARC-Seal: i=1; a=rsa-sha256; t=1655144738; cv=none; d=zohomail.com; s=zohoarc; b=YVcUwwhSqb6ad+2RcJlfPeTsISsZPI9HkGcIwDXCb7AHO/leJ0ES95KN8LJKPNxL+nfNfya9OJ8gwzSUI126T0kwFXV/SRdsZMElNuuVMrJsJQV+UE7NWExLdx1rPU4F5xoRJKTN5j6UeUIMWqUCpx5e5leJ0IEkVcIKtggTx/0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1655144738; h=Content-Type:Date:From:MIME-Version:Message-ID:Reply-To:Subject:To; bh=bzZUj7oud+MXF4Uqky3S+bFLZ+d/eWZ+cQFv11UpP4o=; b=iVnxq2fyGcGBZTivO1yb2ehRuToOmpdflImGrGaHhs7KUERVDDHHE3auBsxaglvroGFeGtFfnS+kWk+G0qC/fCT1fgu3+Ylm8qb1Sbav95o5HpRAnJjpwgkCDaXUeQdkcy1MGdOJFnYJB2En3eluAM6hX3vKbSldLqKz4CLYdjY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=swiftsmsgateway.com; spf=pass smtp.mailfrom=michal.jakubik@zoho.com; dmarc=pass header.from= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:reply-to:to:message-id:subject:mime-version:content-type:user-agent; b=RVSehU8o8Yihqm5C1CpzUs9+R1RjoLs4UBdI3OcrCLCtz/y+vuKk/fxcTKP5C8YTHXo7JY7GYw9a kQibi0+VM+3IQjFT47mI9j/haq6fOmdHAjyeA3ia6I3p9bxLoMCN DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1655144738; s=zoho; d=swiftsmsgateway.com; i=mike.jakubik@swiftsmsgateway.com; h=Date:Date:From:From:Reply-To:Reply-To:To:To:Message-Id:Message-Id:Subject:Subject:MIME-Version:Content-Type:Cc; bh=bzZUj7oud+MXF4Uqky3S+bFLZ+d/eWZ+cQFv11UpP4o=; b=Iimnu/elYZnjHTUzwmovQMEdITWzVSifPeWIUTZgfGKLFAuRdShx78kLbMO7InQF C+QsHodVcQ3t9Bc7nLetcKTxk7E/zDCxaRIGejl36inmMgO0kmAJkhTGZ04+dAQp5qC tkwfri39PrAHcicY+IvwpmZyeMoYw3lPUHvXt0RU= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1655144736922692.4432627025342; Mon, 13 Jun 2022 11:25:36 -0700 (PDT) Date: Mon, 13 Jun 2022 14:25:36 -0400 From: Mike Jakubik Reply-To: mike.jakubik@swiftsmsgateway.com To: "freebsd-net" Message-Id: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> Subject: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_3768786_1051015825.1655144736888" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Rspamd-Queue-Id: 4LMKkg3gTQz3NnF X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=swiftsmsgateway.com header.s=zoho header.b="Iimnu/el"; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (mx1.freebsd.org: domain of michal.jakubik@zoho.com designates 136.143.188.13 as permitted sender) smtp.mailfrom=michal.jakubik@zoho.com X-Spamd-Result: default: False [-3.97 / 15.00]; HAS_REPLYTO(0.00)[mike.jakubik@swiftsmsgateway.com]; XM_UA_NO_VERSION(0.01)[]; R_SPF_ALLOW(-0.20)[+ip4:136.143.188.0/24]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[swiftsmsgateway.com:+]; NEURAL_HAM_SHORT(-0.84)[-0.845]; FORGED_SENDER(0.30)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_ALLOW(-1.00)[zohomail.com:s=zohoarc:i=1]; ASN(0.00)[asn:2639, ipnet:136.143.188.0/23, country:US]; FROM_NEQ_ENVFROM(0.00)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; NEURAL_HAM_MEDIUM(-0.93)[-0.931]; FREEMAIL_ENVFROM(0.00)[zoho.com]; R_DKIM_ALLOW(-0.20)[swiftsmsgateway.com:s=zoho]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[swiftsmsgateway.com]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[136.143.188.13:from]; MLMMJ_DEST(0.00)[freebsd-net]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N ------=_Part_3768786_1051015825.1655144736888 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, how= ever, I am unable to get much more than 6Gb/s when testing with iperf3. The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 GB R= AM, Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter) They are connected to a Dell=C2=A0N3224PX-ON switch. Both servers are idle = and not in use, with a fresh install of=C2=A0stable/13-ebea872f8, nothing r= unning on them except ssh, sendmail, etc. When i test with iperf3 I am unable to get a higher avg than about 6Gb/s. I= have tried just about every knob listed in=C2=A0https://calomel.org/freebs= d_network_tuning.html=C2=A0with little impact on the performance. The netwo= rk cards have HW LRO enabled as per the driver documentation (though this o= nly seems to lower IRQ usage with no impact on actual throughput). The same exact servers tested on Linux (fedora 34) produced nearly 3x the p= erformance (see attached screenshots), i was able to get a steady 14.6Gb/s = rate with nearly 0 retries shown in iperf, the performance on FreeBSD seems= to avg at around 6Gbs but it is very sporadic during the iperf run. I have run out of ideas, any suggestions are welcome. Considering Netflix u= ses very similar HW and they push 400 Gb/s tells me there is something real= ly wrong here or Netflix isnt sharing all their secret sauce. # ifconfig mce0 mce0: flags=3D8863 metric 0 mtu 150= 0=20 options=3Dffed07bb =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ether b8:ce:f6:81:df:6a =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 inet 192.168.10.31 netmask 0xfff= fff00 broadcast 192.168.10.255 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 media: Ethernet 25GBase-CR =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: active =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 nd6 options=3D29 [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 64695 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 930 MBytes=C2= =A0 7.80 Gbits/sec=C2=A0=C2=A0 62=C2=A0=C2=A0=C2=A0 789 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 942 MBytes=C2= =A0 7.90 Gbits/sec=C2=A0 164=C2=A0=C2=A0=C2=A0 824 KBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 1.00 GBytes=C2=A0 8.= 61 Gbits/sec=C2=A0 402=C2=A0=C2=A0=C2=A0 879 KBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 761 MBytes=C2= =A0 6.39 Gbits/sec=C2=A0=C2=A0 61=C2=A0=C2=A0=C2=A0 588 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 724 MBytes=C2= =A0 6.07 Gbits/sec=C2=A0 220=C2=A0=C2=A0=C2=A0 497 KBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 723 MBytes=C2= =A0 6.07 Gbits/sec=C2=A0=C2=A0 54=C2=A0=C2=A0=C2=A0 364 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 716 MBytes=C2= =A0 6.01 Gbits/sec=C2=A0 187=C2=A0=C2=A0=C2=A0 682 KBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 728 MBytes=C2= =A0 6.11 Gbits/sec=C2=A0=C2=A0 86=C2=A0=C2=A0=C2=A0 568 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 761 MBytes=C2= =A0 6.39 Gbits/sec=C2=A0=C2=A0 37=C2=A0=C2=A0=C2=A0 418 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 733 MBytes=C2=A0 6.= 15 Gbits/sec=C2=A0=C2=A0=C2=A0 8=C2=A0=C2=A0=C2=A0 617 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 734 MBytes=C2=A0 6.16 Gb= its/sec=C2=A0 238=C2=A0=C2=A0=C2=A0 474 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0=C2=A0 736 MBytes=C2=A0 6.17 Gb= its/sec=C2=A0 164=C2=A0=C2=A0=C2=A0 757 KBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0=C2=A0 610 MBytes=C2=A0 5.12 Gb= its/sec=C2=A0 118=C2=A0=C2=A0=C2=A0 579 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 1.02 GBytes=C2=A0 8.75 Gbits/s= ec=C2=A0 447=C2=A0=C2=A0=C2=A0 449 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0=C2=A0 728 MBytes=C2=A0 6.11 Gb= its/sec=C2=A0 132=C2=A0=C2=A0=C2=A0 719 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0=C2=A0 724 MBytes=C2=A0 6.07 Gb= its/sec=C2=A0 185=C2=A0=C2=A0=C2=A0 649 KBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0=C2=A0 597 MBytes=C2=A0 5.01 Gb= its/sec=C2=A0 142=C2=A0=C2=A0=C2=A0 570 KBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0=C2=A0 733 MBytes=C2=A0 6.15 Gb= its/sec=C2=A0 102=C2=A0=C2=A0=C2=A0 484 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0=C2=A0 726 MBytes=C2=A0 6.09 Gb= its/sec=C2=A0=C2=A0 15=C2=A0=C2=A0=C2=A0 569 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0=C2=A0 733 MBytes=C2=A0 6.15 Gb= its/sec=C2=A0 181=C2=A0=C2=A0=C2=A0 527 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0=C2=A0 729 MBytes=C2=A0 6.12 Gb= its/sec=C2=A0 118=C2=A0=C2=A0=C2=A0 430 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0=C2=A0 733 MBytes=C2=A0 6.15 Gb= its/sec=C2=A0 116=C2=A0=C2=A0=C2=A0 641 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0=C2=A0 728 MBytes=C2=A0 6.10 Gb= its/sec=C2=A0 182=C2=A0=C2=A0=C2=A0 598 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 743 MBytes=C2=A0 6.24 Gb= its/sec=C2=A0 209=C2=A0=C2=A0=C2=A0 614 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0=C2=A0 746 MBytes=C2=A0 6.26 Gb= its/sec=C2=A0=C2=A0 72=C2=A0=C2=A0=C2=A0 758 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0=C2=A0 742 MBytes=C2=A0 6.23 Gb= its/sec=C2=A0 199=C2=A0=C2=A0=C2=A0 675 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0=C2=A0 799 MBytes=C2=A0 6.70 Gb= its/sec=C2=A0 183=C2=A0=C2=A0=C2=A0 542 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0=C2=A0 908 MBytes=C2=A0 7.61 Gb= its/sec=C2=A0=C2=A0=C2=A0 7=C2=A0=C2=A0 1.19 MBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.37 GBytes=C2=A0 11.7 Gbits/s= ec=C2=A0 606=C2=A0=C2=A0 1013 KBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.31 GBytes=C2=A0 11.3 Gbits/s= ec=C2=A0=C2=A0 74=C2=A0=C2=A0 1.02 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 23.7 GBytes=C2=A0 6.79 Gb= its/sec=C2=A0 4771=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 23.7 GBytes=C2=A0 6.79 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver I have even tried changing to the RACK TCP stack, only to get slightly bett= er results, however with RACK the amount of retries is nearly 0. [root@db-02 ~]# sysctl net.inet.tcp.functions_default=3Drack=20 net.inet.tcp.functions_default: rack -> rack [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 51894 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 761 MBytes=C2= =A0 6.38 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 737 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 859 MBytes=C2= =A0 7.21 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 761 KBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 880 MBytes=C2= =A0 7.38 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 785 KBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 734 MBytes=C2= =A0 6.16 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 804 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 777 MBytes=C2= =A0 6.52 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 824 KBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 719 MBytes=C2= =A0 6.03 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 841 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 865 MBytes=C2= =A0 7.26 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 862 KBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 880 MBytes=C2= =A0 7.38 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 882 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 906 MBytes=C2= =A0 7.60 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 904 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 749 MBytes=C2=A0 6.= 29 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 921 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 798 MBytes=C2=A0 6.69 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 938 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0=C2=A0 746 MBytes=C2=A0 6.26 Gb= its/sec=C2=A0 209=C2=A0=C2=A0=C2=A0 772 KBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0=C2=A0 768 MBytes=C2=A0 6.44 Gb= its/sec=C2=A0=C2=A0 35=C2=A0=C2=A0=C2=A0 644 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0=C2=A0 948 MBytes=C2=A0 7.95 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 673 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 1.23 GBytes=C2=A0 10.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 711 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.32 GBytes=C2=A0 11.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 748 KBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 1.31 GBytes=C2=A0 11.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 785 KBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 819 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 1.30 GBytes=C2=A0 11.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 852 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 1.34 GBytes=C2=A0 11.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 883 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.29 GBytes=C2=A0 11.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 914 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 1.36 GBytes=C2=A0 11.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 944 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 1.33 GBytes=C2=A0 11.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 974 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 1.31 GBytes=C2=A0 11.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1003 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 1.30 GBytes=C2=A0 11.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.00 MBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 1.34 GBytes=C2=A0 11.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.03 MBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.32 GBytes=C2=A0 11.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.06 MBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0=C2=A0 957 MBytes=C2=A0 8.03 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.07 MBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0=C2=A0 837 MBytes=C2=A0 7.02 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.09 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0=C2=A0 729 MBytes=C2=A0 6.11 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.10 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 30.6 GBytes=C2=A0 8.77 Gb= its/sec=C2=A0 244=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 30.6 GBytes=C2=A0 8.77 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver More data can be found @=C2=A0https://forums.freebsd.org/threads/poor-perfo= rmance-with-stable-13-and-mellanox-connectx-6-mlx5.85460/ Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use o= f the addressee(s) and may contain information that is privileged or confid= ential. If you are not the intended recipient, or responsible for deliverin= g the information to the intended recipient, you are hereby notified that a= ny dissemination, distribution, printing or copying of this e-mail and any = attachments is strictly prohibited. If this e-mail and any attachments were= received in error, please notify the sender by reply e-mail and delete the= original message. ------=_Part_3768786_1051015825.1655144736888 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
Hello,

I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, ho= wever, I am unable to get much more than 6Gb/s when testing with iperf3.

The servers are Lenovo SR665 (2 x AMD EPYC 7443 = 24-Core Processor, 256 GB RAM, Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port= OCP Ethernet Adapter)

They are connected to a= Dell N3224PX-ON switch. Both servers are idle and not in use, with a = fresh install of stable/13-ebea872f8, nothing running on them except s= sh, sendmail, etc.


The same exact servers tested on Linux (fedora 34) produced nearly 3x the = performance (see attached screenshots), i was able to get a steady 14.6Gb/s= rate with nearly 0 retries shown in iperf, the performance on FreeBSD seem= s to avg at around 6Gbs but it is very sporadic during the iperf run.

I have run out of ideas, any suggestions are welcom= e. Considering Netflix uses very similar HW and they push 400 Gb/s tells me= there is something really wrong here or Netflix isnt sharing all their sec= ret sauce.


# ifconfig mce0
<= /div>
mce0: flags=3D8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> = metric 0 mtu 1500
options=3Dffed07bb<RXCSUM,TXCSUM,VLAN_M= TU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HW= TSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,TXRTLMT,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6= ,VXLAN_HWCSUM,VXLAN_HWTSO,TXTLS_RTLMT>
   &= nbsp;    ether b8:ce:f6:81:df:6a
  &= nbsp;     inet 192.168.10.31 netmask 0xffffff00 broadca= st 192.168.10.255
        = media: Ethernet 25GBase-CR <full-duplex,rxpause,txpause>
        status: active
&= nbsp;       nd6 options=3D29<PERFORMNUD,IF= DISABLED,AUTO_LINKLOCAL>


[ro= ot@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host d= b-01, port 5201
[  5] local 192.168.10.31 port 64695 con= nected to 192.168.10.30 port 5201
[ ID] Interval  &= nbsp;        Transfer   &= nbsp; Bitrate         Retr  Cw= nd
[  5]   0.00-1.00   sec &nbs= p; 930 MBytes  7.80 Gbits/sec   62    789 KBy= tes
[  5]   1.00-2.00   sec &nb= sp; 942 MBytes  7.90 Gbits/sec  164    824 KBytes<= br>
[  5]   2.00-3.00   sec  1.00 G= Bytes  8.61 Gbits/sec  402    879 KBytes
=
[  5]   3.00-4.00   sec   761 MByte= s  6.39 Gbits/sec   61    588 KBytes
[  5]   4.00-5.00   sec   724 MByt= es  6.07 Gbits/sec  220    497 KBytes
[  5]   5.00-6.00   sec   723 MBytes&n= bsp; 6.07 Gbits/sec   54    364 KBytes
[  5]   6.00-7.00   sec   716 MBytes&= nbsp; 6.01 Gbits/sec  187    682 KBytes
[  5]   9.00-10.00  sec   733 MBytes  6= .15 Gbits/sec    8    617 KBytes
[  5]  10.00-11.00  sec   734 MBytes  6.16 = Gbits/sec  238    474 KBytes
[  5]&n= bsp; 11.00-12.00  sec   736 MBytes  6.17 Gbits/sec = ; 164    757 KBytes
[  5]  12.00-13.= 00  sec   610 MBytes  5.12 Gbits/sec  118 &nb= sp;  579 KBytes
[  5]  13.00-14.00  sec&n= bsp; 1.02 GBytes  8.75 Gbits/sec  447    449 KByte= s
[  5]  14.00-15.00  sec   728 MByt= es  6.11 Gbits/sec  132    719 KBytes
[  5]  15.00-16.00  sec   724 MBytes  6.07 = Gbits/sec  185    649 KBytes
[  5]&n= bsp; 16.00-17.00  sec   597 MBytes  5.01 Gbits/sec = ; 142    570 KBytes
[  5]  17.00-18.= 00  sec   733 MBytes  6.15 Gbits/sec  102 &nb= sp;  484 KBytes
[  5]  18.00-19.00  sec&n= bsp;  726 MBytes  6.09 Gbits/sec   15   = 569 KBytes
[  5]  19.00-20.00  sec  = ; 733 MBytes  6.15 Gbits/sec  181    527 KBytes
[  5]  20.00-21.00  sec   729 MBytes&n= bsp; 6.12 Gbits/sec  118    430 KBytes
[&= nbsp; 5]  21.00-22.00  sec   733 MBytes  6.15 Gbit= s/sec  116    641 KBytes
[  5] = 22.00-23.00  sec   728 MBytes  6.10 Gbits/sec  18= 2    598 KBytes
[  5]  23.00-24.00&n= bsp; sec   743 MBytes  6.24 Gbits/sec  209  &= nbsp; 614 KBytes
[  5]  24.00-25.00  sec =   746 MBytes  6.26 Gbits/sec   72    758= KBytes
[  5]  25.00-26.00  sec   74= 2 MBytes  6.23 Gbits/sec  199    675 KBytes
[  5]  26.00-27.00  sec   799 MBytes = 6.70 Gbits/sec  183    542 KBytes
[ = ; 5]  27.00-28.00  sec   908 MBytes  7.61 Gbits/se= c    7   1.19 MBytes
[  5] = ; 28.00-29.00  sec  1.37 GBytes  11.7 Gbits/sec  606&nb= sp;  1013 KBytes
[  5]  29.00-30.00  sec&= nbsp; 1.31 GBytes  11.3 Gbits/sec   74   1.02 MByt= es
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval         &= nbsp; Transfer     Bitrate     = ;    Retr
[  5]   0.00-30.00&nb= sp; sec  23.7 GBytes  6.79 Gbits/sec  4771   =           sender
[  5]   0.00-30.00  sec  23.7 GBytes  6.79 G= bits/sec           &= nbsp;      receiver

<= br>
I have even tried changing to the RACK TCP stack, only to get= slightly better results, however with RACK the amount of retries is nearly= 0.

[root@db-02 ~]# sysctl net.inet.tcp.functi= ons_default=3Drack
net.inet.tcp.functions_default: rack ->= ; rack
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to h= ost db-01, port 5201
[  5] local 192.168.10.31 port 5189= 4 connected to 192.168.10.30 port 5201
[ ID] Interval &n= bsp;         Transfer  &n= bsp;  Bitrate         Retr&nbs= p; Cwnd
[  5]   0.00-1.00   sec = ;  761 MBytes  6.38 Gbits/sec    0  &nbs= p; 737 KBytes
[  5]   1.00-2.00   se= c   859 MBytes  7.21 Gbits/sec    0 &nbs= p;  761 KBytes
[  5]   2.00-3.00 &nb= sp; sec   880 MBytes  7.38 Gbits/sec    0&nbs= p;   785 KBytes
[  5]   3.00-4.00&nb= sp;  sec   734 MBytes  6.16 Gbits/sec   = 0    804 KBytes
[  5]   4.00-5= .00   sec   777 MBytes  6.52 Gbits/sec  =   0    824 KBytes
[  5]   = 5.00-6.00   sec   719 MBytes  6.03 Gbits/sec =    0    841 KBytes
[  5] &= nbsp; 6.00-7.00   sec   865 MBytes  7.26 Gbits/sec=     0    862 KBytes
[  5]&= nbsp;  7.00-8.00   sec   880 MBytes  7.38 Gbi= ts/sec    0    882 KBytes
[&nbs= p; 5]   8.00-9.00   sec   906 MBytes  7.= 60 Gbits/sec    0    904 KBytes
[  5]   9.00-10.00  sec   749 MBytes  6= .29 Gbits/sec    0    921 KBytes
[  5]  10.00-11.00  sec   798 MBytes  6.69 = Gbits/sec    0    938 KBytes
[&= nbsp; 5]  11.00-12.00  sec   746 MBytes  6.26 Gbit= s/sec  209    772 KBytes
[  5] = 12.00-13.00  sec   768 MBytes  6.44 Gbits/sec &nb= sp; 35    644 KBytes
[  5]  13.00-14= .00  sec   948 MBytes  7.95 Gbits/sec   = 0    673 KBytes
[  5]  14.00-15.00&= nbsp; sec  1.23 GBytes  10.5 Gbits/sec    0 &= nbsp;  711 KBytes
[  5]  15.00-16.00  sec=   1.32 GBytes  11.4 Gbits/sec    0  &nbs= p; 748 KBytes
[  5]  16.00-17.00  sec  1.= 31 GBytes  11.2 Gbits/sec    0    785 KB= ytes
[  5]  17.00-18.00  sec  1.29 GBytes=   11.1 Gbits/sec    0    819 KBytes
<= /div>
[  5]  18.00-19.00  sec  1.30 GBytes  11= .2 Gbits/sec    0    852 KBytes
[  5]  19.00-20.00  sec  1.34 GBytes  11.5 Gbits/= sec    0    883 KBytes
[  = 5]  20.00-21.00  sec  1.29 GBytes  11.1 Gbits/sec =    0    914 KBytes
[  5]  = 21.00-22.00  sec  1.36 GBytes  11.7 Gbits/sec  &nb= sp; 0    944 KBytes
[  5]  22.00-23.= 00  sec  1.33 GBytes  11.4 Gbits/sec    0&nbs= p;   974 KBytes
[  5]  23.00-24.00  = sec  1.31 GBytes  11.2 Gbits/sec    0   = 1003 KBytes
[  5]  24.00-25.00  sec  1.30= GBytes  11.2 Gbits/sec    0   1.00 MBytes
[  5]  25.00-26.00  sec  1.34 GBytes  = 11.5 Gbits/sec    0   1.03 MBytes
[&= nbsp; 5]  26.00-27.00  sec  1.32 GBytes  11.3 Gbits/sec=     0   1.06 MBytes
[  5] = 27.00-28.00  sec   957 MBytes  8.03 Gbits/sec &nb= sp;  0   1.07 MBytes
[  5]  28.00-29= .00  sec   837 MBytes  7.02 Gbits/sec   = 0   1.09 MBytes
[  5]  29.00-30.00 = sec   729 MBytes  6.11 Gbits/sec    0 &= nbsp; 1.10 MBytes
- - - - - - - - - - - - - - - - - - - - - -= - - -
[ ID] Interval      &nbs= p;    Transfer     Bitrate  &n= bsp;      Retr
[  5]  = ; 0.00-30.00  sec  30.6 GBytes  8.77 Gbits/sec  244&nbs= p;            sender=
[  5]   0.00-30.00  sec  30.6 GByte= s  8.77 Gbits/sec         = ;         receiver





Mike Jakubik

Disclaimer: This e-mail and any attachme= nts are intended only for the use of the addressee(s) and may contain infor= mation that is privileged or confidential. If you are not the intended reci= pient, or responsible for delivering the information to the intended recipi= ent, you are hereby notified that any dissemination, distribution, printing= or copying of this e-mail and any attachments is strictly prohibited. If t= his e-mail and any attachments were received in error, please notify the se= nder by reply e-mail and delete the original message.




------=_Part_3768786_1051015825.1655144736888--