From nobody Thu Jun 16 20:48:50 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E752F85B541 for ; Thu, 16 Jun 2022 20:49:00 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) Received: from sender4-op-o13.zoho.com (sender4-op-o13.zoho.com [136.143.188.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LPDmW4J32z3NN8 for ; Thu, 16 Jun 2022 20:48:59 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) ARC-Seal: i=1; a=rsa-sha256; t=1655412531; cv=none; d=zohomail.com; s=zohoarc; b=Nox/mLGUJcadUVVHLYDYjdwCfiQjOpS3TeGeIgfJ5u63NzFW8oW44gJS1e6hl0jaU3FGNxBqrW/Lasq+YWaYROhXN91WI4WfmO4+y2OQvf3Gs+mU04NGoUjh4pfdXKVPGO0ln7YyUdwzbwbT19wr1U6gxHszwsgXVssuh8iH67Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1655412531; h=Content-Type:Date:From:In-Reply-To:MIME-Version:Message-ID:Reply-To:References:Subject:To; bh=Y9kh4Kz8vhTiovgiSmWm6JhceF9Tr1lE7g2U7FupvHg=; b=V5s7VC7pZ4+ihZvisF7OVKKl6ADIDycdQkZw6b5CGctBjd43+x/UGbsGFE10Iae8VdGaDo07QUxHQNdMuBoih/gQtkIjQFaSAzgolZwTVRonS6Kg+aV2CEvtMfHNEJcECNtRKd1lJyGdJiTJs+srpnbwoXyQlhf8rAGU1w23c5s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=swiftsmsgateway.com; spf=pass smtp.mailfrom=michal.jakubik@zoho.com; dmarc=pass header.from= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:reply-to:to:message-id:in-reply-to:references:subject:mime-version:content-type:user-agent; b=aLQdJMQ7jvJTpsfyxlubKyZTtn9VPgo49rrwKyI0tIu1/MIuIlvWoDWlyjy2SWpLFfmX3Z7yxvCv 6qrQVSRxzp7J9EwBEtnaJwTGkNWIq0ZVHPNJ25RPNZ2nPgnnu8Nt DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1655412531; s=zoho; d=swiftsmsgateway.com; i=mike.jakubik@swiftsmsgateway.com; h=Date:Date:From:From:Reply-To:Reply-To:To:To:Message-Id:Message-Id:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Cc; bh=Y9kh4Kz8vhTiovgiSmWm6JhceF9Tr1lE7g2U7FupvHg=; b=nwcURqlYwgp6qbZVCVQsVCXIl7nk/AFolr0oZPMnLRVLQO+HvnmPRqRyceS9tmju tNpfLJF2g58IJWKbekG76iqabEQQfDZdSTvweCG4KDSp3GFOT0R8RkIru8X60tsQ7Bd x1VGiOg2l7gK7o6aSq1RUQN+kDUIJXWv7wqC+Ryw= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 165541253017282.94121660521466; Thu, 16 Jun 2022 13:48:50 -0700 (PDT) Date: Thu, 16 Jun 2022 16:48:50 -0400 From: Mike Jakubik Reply-To: mike.jakubik@swiftsmsgateway.com To: "freebsd-net" Message-Id: <1816e469bdf.126cdb81b2139485.369352368493375815@swiftsmsgateway.com> In-Reply-To: <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_6787798_67531831.1655412530143" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Rspamd-Queue-Id: 4LPDmW4J32z3NN8 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=swiftsmsgateway.com header.s=zoho header.b=nwcURqlY; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (mx1.freebsd.org: domain of michal.jakubik@zoho.com designates 136.143.188.13 as permitted sender) smtp.mailfrom=michal.jakubik@zoho.com X-Spamd-Result: default: False [-3.02 / 15.00]; HAS_REPLYTO(0.00)[mike.jakubik@swiftsmsgateway.com]; XM_UA_NO_VERSION(0.01)[]; R_SPF_ALLOW(-0.20)[+ip4:136.143.188.0/24]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[swiftsmsgateway.com:+]; NEURAL_HAM_SHORT(-0.01)[-0.010]; FORGED_SENDER(0.30)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_ALLOW(-1.00)[zohomail.com:s=zohoarc:i=1]; ASN(0.00)[asn:2639, ipnet:136.143.188.0/23, country:US]; FROM_NEQ_ENVFROM(0.00)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; NEURAL_HAM_MEDIUM(-0.82)[-0.824]; FREEMAIL_ENVFROM(0.00)[zoho.com]; R_DKIM_ALLOW(-0.20)[swiftsmsgateway.com:s=zoho]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[swiftsmsgateway.com]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[136.143.188.13:from]; MLMMJ_DEST(0.00)[freebsd-net]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N ------=_Part_6787798_67531831.1655412530143 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable After multiple tests and tweaks i believe the issue is not with the HW or N= uma related (Infinity fabric should do around 32GB) but rather with FreeBSD= TCP/IP stack. It's like it cant figure itself out properly for the speed t= hat the HW can do, i keep getting widely varying results when testing. Belo= w is an example of two tests, with about a 15 second break in between the t= wo. [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 49155 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 991 MBytes=C2= =A0 8.32 Gbits/sec=C2=A0 268=C2=A0=C2=A0=C2=A0 579 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 945 MBytes=C2= =A0 7.93 Gbits/sec=C2=A0 369=C2=A0=C2=A0=C2=A0 777 KBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 793 MBytes=C2= =A0 6.65 Gbits/sec=C2=A0=C2=A0 60=C2=A0=C2=A0 1.03 MBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 666 MBytes=C2= =A0 5.59 Gbits/sec=C2=A0 203=C2=A0=C2=A0=C2=A0 976 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.01=C2=A0=C2=A0 sec=C2=A0=C2=A0 575 MBytes=C2= =A0 4.78 Gbits/sec=C2=A0 202=C2=A0=C2=A0 1.13 MBytes [=C2=A0 5]=C2=A0=C2=A0 5.01-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 169 MBytes=C2= =A0 1.43 Gbits/sec=C2=A0 134=C2=A0=C2=A0=C2=A0 699 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 1.21 GBytes=C2=A0 10= .4 Gbits/sec=C2=A0 383=C2=A0=C2=A0 1.08 MBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 1.21 GBytes=C2=A0 10= .4 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.16 MBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 1.32 GBytes=C2=A0 11= .3 Gbits/sec=C2=A0 124=C2=A0=C2=A0=C2=A0 780 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 690 MBytes=C2=A0 5.= 79 Gbits/sec=C2=A0 316=C2=A0=C2=A0=C2=A0 605 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 685 MBytes=C2=A0 5.75 Gb= its/sec=C2=A0=C2=A0 97=C2=A0=C2=A0=C2=A0 854 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.08 GBytes=C2=A0 9.30 Gbits/s= ec=C2=A0 383=C2=A0=C2=A0=C2=A0 538 KBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0=C2=A0 682 MBytes=C2=A0 5.72 Gb= its/sec=C2=A0=C2=A0 88=C2=A0=C2=A0=C2=A0 870 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0=C2=A0 678 MBytes=C2=A0 5.69 Gb= its/sec=C2=A0 123=C2=A0=C2=A0=C2=A0 964 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0=C2=A0 670 MBytes=C2=A0 5.62 Gb= its/sec=C2=A0 290=C2=A0=C2=A0=C2=A0 763 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.01 GBytes=C2=A0 8.71 Gbits/s= ec=C2=A0 228=C2=A0=C2=A0 1.08 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0=C2=A0 886 MBytes=C2=A0 7.44 Gb= its/sec=C2=A0 118=C2=A0=C2=A0=C2=A0 615 KBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0=C2=A0 734 MBytes=C2=A0 6.16 Gb= its/sec=C2=A0 291=C2=A0=C2=A0=C2=A0 902 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 1.04 GBytes=C2=A0 8.96 Gbits/s= ec=C2=A0 212=C2=A0=C2=A0=C2=A0 323 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0=C2=A0 710 MBytes=C2=A0 5.96 Gb= its/sec=C2=A0 193=C2=A0=C2=A0=C2=A0 547 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0=C2=A0 693 MBytes=C2=A0 5.82 Gb= its/sec=C2=A0 370=C2=A0=C2=A0=C2=A0 942 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0=C2=A0 704 MBytes=C2=A0 5.91 Gb= its/sec=C2=A0=C2=A0 80=C2=A0=C2=A0 1022 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 1.26 GBytes=C2=A0 10.8 Gbits/s= ec=C2=A0 262=C2=A0=C2=A0=C2=A0 965 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 828 MBytes=C2=A0 6.94 Gb= its/sec=C2=A0 202=C2=A0=C2=A0=C2=A0 763 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0=C2=A0 774 MBytes=C2=A0 6.49 Gb= its/sec=C2=A0 227=C2=A0=C2=A0=C2=A0 581 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0=C2=A0 734 MBytes=C2=A0 6.15 Gb= its/sec=C2=A0 256=C2=A0=C2=A0=C2=A0 664 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0=C2=A0 753 MBytes=C2=A0 6.32 Gb= its/sec=C2=A0 331=C2=A0=C2=A0=C2=A0 540 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0=C2=A0 764 MBytes=C2=A0 6.41 Gb= its/sec=C2=A0 298=C2=A0=C2=A0=C2=A0 823 KBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0=C2=A0 757 MBytes=C2=A0 6.35 Gb= its/sec=C2=A0 123=C2=A0=C2=A0=C2=A0 850 KBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0=C2=A0 754 MBytes=C2=A0 6.32 Gb= its/sec=C2=A0=C2=A0 74=C2=A0=C2=A0=C2=A0 970 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 24.4 GBytes=C2=A0 6.98 Gb= its/sec=C2=A0 6305=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 24.4 GBytes=C2=A0 6.98 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver iperf Done. [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 25061 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.81 GBytes=C2=A0 15= .5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15= .7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 1.98 GBytes=C2=A0 17= .0 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 2.11 GBytes=C2=A0 18= .1 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 2.12 GBytes=C2=A0 18= .2 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 2.16 GBytes=C2=A0 18= .5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 1.90 GBytes=C2=A0 16= .3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.12 MBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.02=C2=A0=C2=A0 sec=C2=A0 1.28 GBytes=C2=A0 10= .8 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.12 MBytes [=C2=A0 5]=C2=A0=C2=A0 8.02-9.00=C2=A0=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 16= .0 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.17 MBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 1.91 GBytes=C2=A0 16.4 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.20 MBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 1.79 GBytes=C2=A0 15.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.77 GBytes=C2=A0 15.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 1.69 GBytes=C2=A0 14.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 1.57 GBytes=C2=A0 13.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 1.60 GBytes=C2=A0 13.8 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.89 GBytes=C2=A0 16.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 1.76 GBytes=C2=A0 15.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 1.93 GBytes=C2=A0 16.6 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 1.77 GBytes=C2=A0 15.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 1.68 GBytes=C2=A0 14.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 1.75 GBytes=C2=A0 15.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 1.76 GBytes=C2=A0 15.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 2.05 GBytes=C2=A0 17.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 1.79 GBytes=C2=A0 15.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 2.08 GBytes=C2=A0 17.9 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 2.09 GBytes=C2=A0 18.0 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 2.05 GBytes=C2=A0 17.6 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.81 GBytes=C2=A0 15.6 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 55.2 GBytes=C2=A0 15.8 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 55.2 GBytes=C2=A0 15.8 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver iperf Done. ------=_Part_6787798_67531831.1655412530143 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
After multiple tests and tweaks i believe the issu= e is not with the HW or Numa related (Infinity fabric should do around 32GB= ) but rather with FreeBSD TCP/IP stack. It's like it cant figure itself out= properly for the speed that the HW can do, i keep getting widely varying r= esults when testing. Below is an example of two tests, with about a 15 seco= nd break in between the two.

[root@db-02 ~]# i= perf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 520= 1
[  5] local 192.168.10.31 port 49155 connected to 192.= 168.10.30 port 5201
[ ID] Interval    &nb= sp;      Transfer     Bitrate&= nbsp;        Retr  Cwnd
[  5]   0.00-1.00   sec   991 MBytes&= nbsp; 8.32 Gbits/sec  268    579 KBytes
[=   5]   1.00-2.00   sec   945 MBytes = ; 7.93 Gbits/sec  369    777 KBytes
[&nbs= p; 5]   2.00-3.00   sec   793 MBytes  6.= 65 Gbits/sec   60   1.03 MBytes
[  5= ]   3.00-4.00   sec   666 MBytes  5.59 G= bits/sec  203    976 KBytes
[  5]&nb= sp;  4.00-5.01   sec   575 MBytes  4.78 Gbits= /sec  202   1.13 MBytes
[  5]  = 5.01-6.00   sec   169 MBytes  1.43 Gbits/sec = ; 134    699 KBytes
[  5]   6.0= 0-7.00   sec  1.21 GBytes  10.4 Gbits/sec  383&nbs= p;  1.08 MBytes
[  5]   7.00-8.00 &n= bsp; sec  1.21 GBytes  10.4 Gbits/sec    0 &n= bsp; 1.16 MBytes
[  5]   8.00-9.00  = sec  1.32 GBytes  11.3 Gbits/sec  124    780= KBytes
[  5]   9.00-10.00  sec &nbs= p; 690 MBytes  5.79 Gbits/sec  316    605 KBytes
[  5]  10.00-11.00  sec   685 MBytes&= nbsp; 5.75 Gbits/sec   97    854 KBytes
<= div>[  5]  11.00-12.00  sec  1.08 GBytes  9.30 Gbi= ts/sec  383    538 KBytes
[  5] = ; 12.00-13.00  sec   682 MBytes  5.72 Gbits/sec &n= bsp; 88    870 KBytes
[  5]  13.00-1= 4.00  sec   678 MBytes  5.69 Gbits/sec  123 &= nbsp;  964 KBytes
[  5]  14.00-15.00  sec=    670 MBytes  5.62 Gbits/sec  290    76= 3 KBytes
[  5]  15.00-16.00  sec  1.01 GB= ytes  8.71 Gbits/sec  228   1.08 MBytes
[=   5]  16.00-17.00  sec   886 MBytes  7.44 Gbi= ts/sec  118    615 KBytes
[  5] = ; 17.00-18.00  sec   734 MBytes  6.16 Gbits/sec  2= 91    902 KBytes
[  5]  18.00-19.00&= nbsp; sec  1.04 GBytes  8.96 Gbits/sec  212   = ; 323 KBytes
[  5]  19.00-20.00  sec &nbs= p; 710 MBytes  5.96 Gbits/sec  193    547 KBytes
[  5]  20.00-21.00  sec   693 MBytes&= nbsp; 5.82 Gbits/sec  370    942 KBytes
[=   5]  21.00-22.00  sec   704 MBytes  5.91 Gbi= ts/sec   80   1022 KBytes
[  5] = ; 22.00-23.00  sec  1.26 GBytes  10.8 Gbits/sec  262&nb= sp;   965 KBytes
[  5]  23.00-24.00 = sec   828 MBytes  6.94 Gbits/sec  202   = ; 763 KBytes
[  5]  24.00-25.00  sec &nbs= p; 774 MBytes  6.49 Gbits/sec  227    581 KBytes
[  5]  25.00-26.00  sec   734 MBytes&= nbsp; 6.15 Gbits/sec  256    664 KBytes
[=   5]  26.00-27.00  sec   753 MBytes  6.32 Gbi= ts/sec  331    540 KBytes
[  5] = ; 27.00-28.00  sec   764 MBytes  6.41 Gbits/sec  2= 98    823 KBytes
[  5]  28.00-29.00&= nbsp; sec   757 MBytes  6.35 Gbits/sec  123  =   850 KBytes
[  5]  29.00-30.00  sec = ;  754 MBytes  6.32 Gbits/sec   74    97= 0 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval        &= nbsp;  Transfer     Bitrate    = ;     Retr
[  5]   0.00-30= .00  sec  24.4 GBytes  6.98 Gbits/sec  6305  =            sender
[  5]   0.00-30.00  sec  24.4 GBytes  = 6.98 Gbits/sec          &= nbsp;       receiver

=
iperf Done.

[root@db-02 ~]# iperf3 -i 1 -= t 30 -c db-01
Connecting to host db-01, port 5201
[  5] local 192.168.10.31 port 25061 connected to 192.168.10.30 por= t 5201
[ ID] Interval      &nbs= p;    Transfer     Bitrate  &n= bsp;      Retr  Cwnd
[  5]=    0.00-1.00   sec  1.81 GBytes  15.5 Gbits/s= ec    0   1.11 MBytes
[  5]&nbs= p;  1.00-2.00   sec  1.83 GBytes  15.7 Gbits/sec&n= bsp;   0   1.11 MBytes
[  5] &n= bsp; 2.00-3.00   sec  1.98 GBytes  17.0 Gbits/sec =    0   1.11 MBytes
[  5]  = 3.00-4.00   sec  2.11 GBytes  18.1 Gbits/sec &nbs= p;  0   1.11 MBytes
[  5]   4.0= 0-5.00   sec  2.12 GBytes  18.2 Gbits/sec  &n= bsp; 0   1.11 MBytes
[  5]   5.00-6.= 00   sec  2.16 GBytes  18.5 Gbits/sec   = 0   1.11 MBytes
[  5]   6.00-7.00&n= bsp;  sec  1.90 GBytes  16.3 Gbits/sec    0&n= bsp;  1.12 MBytes
[  5]   7.00-8.02 =   sec  1.28 GBytes  10.8 Gbits/sec    0 =   1.12 MBytes
[  5]   8.02-9.00 &nbs= p; sec  1.83 GBytes  16.0 Gbits/sec    0 &nbs= p; 1.17 MBytes
[  5]   9.00-10.00  sec&nb= sp; 1.91 GBytes  16.4 Gbits/sec    0   1.20 M= Bytes
[  5]  10.00-11.00  sec  1.79 GByte= s  15.3 Gbits/sec    0   1.60 MBytes
[  5]  11.00-12.00  sec  1.77 GBytes  15.2 G= bits/sec    0   1.60 MBytes
[  = 5]  12.00-13.00  sec  1.69 GBytes  14.5 Gbits/sec =    0   1.61 MBytes
[  5]  13.00= -14.00  sec  1.57 GBytes  13.5 Gbits/sec    0=    1.61 MBytes
[  5]  14.00-15.00  s= ec  1.60 GBytes  13.8 Gbits/sec    0   1= .61 MBytes
[  5]  15.00-16.00  sec  1.89 = GBytes  16.2 Gbits/sec    0   1.61 MBytes
=
[  5]  16.00-17.00  sec  1.76 GBytes  1= 5.1 Gbits/sec    0   1.61 MBytes
[&n= bsp; 5]  17.00-18.00  sec  1.93 GBytes  16.6 Gbits/sec&= nbsp;   0   1.61 MBytes
[  5]  = 18.00-19.00  sec  1.77 GBytes  15.2 Gbits/sec  &nb= sp; 0   1.61 MBytes
[  5]  19.00-20.00&nb= sp; sec  1.68 GBytes  14.5 Gbits/sec    0 &nb= sp; 1.61 MBytes
[  5]  20.00-21.00  sec  = 1.66 GBytes  14.3 Gbits/sec    0   1.61 MByte= s
[  5]  21.00-22.00  sec  1.75 GBytes&nb= sp; 15.1 Gbits/sec    0   1.61 MBytes
[  5]  22.00-23.00  sec  1.76 GBytes  15.1 Gbits= /sec    0   1.61 MBytes
[  5]&n= bsp; 23.00-24.00  sec  2.05 GBytes  17.7 Gbits/sec &nbs= p;  0   1.61 MBytes
[  5]  24.00-25.= 00  sec  1.79 GBytes  15.4 Gbits/sec    0&nbs= p;  1.61 MBytes
[  5]  25.00-26.00  sec&n= bsp; 1.71 GBytes  14.7 Gbits/sec    0   1.61 = MBytes
[  5]  26.00-27.00  sec  2.08 GByt= es  17.9 Gbits/sec    0   1.61 MBytes
[  5]  27.00-28.00  sec  2.09 GBytes  18.0 = Gbits/sec    0   1.61 MBytes
[ = 5]  28.00-29.00  sec  2.05 GBytes  17.6 Gbits/sec = ;   0   1.61 MBytes
[  5]  29.0= 0-30.00  sec  1.81 GBytes  15.6 Gbits/sec    = 0   1.61 MBytes
- - - - - - - - - - - - - - - - - -= - - - - - - -
[ ID] Interval     &n= bsp;     Transfer     Bitrate =         Retr
[  5]&nb= sp;  0.00-30.00  sec  55.2 GBytes  15.8 Gbits/sec =    0          &= nbsp;  sender
[  5]   0.00-30.00  se= c  55.2 GBytes  15.8 Gbits/sec      = ;            receive= r

iperf Done.

------=_Part_6787798_67531831.1655412530143--