From nobody Thu Jun 16 21:00:25 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5C7DE85D329 for ; Thu, 16 Jun 2022 21:00:38 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward501o.mail.yandex.net (forward501o.mail.yandex.net [37.140.190.203]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LPF1w53yhz3Q4g for ; Thu, 16 Jun 2022 21:00:36 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from myt5-1892386aa303.qloud-c.yandex.net (myt5-1892386aa303.qloud-c.yandex.net [IPv6:2a02:6b8:c12:4323:0:640:1892:386a]) by forward501o.mail.yandex.net (Yandex) with ESMTP id 2A20C45C56B3; Fri, 17 Jun 2022 00:00:28 +0300 (MSK) Received: from myt5-89cdf5c4a3a5.qloud-c.yandex.net (myt5-89cdf5c4a3a5.qloud-c.yandex.net [2a02:6b8:c12:289b:0:640:89cd:f5c4]) by myt5-1892386aa303.qloud-c.yandex.net (mxback/Yandex) with ESMTP id s8DhG4rbOg-0RfioHhe; Fri, 17 Jun 2022 00:00:28 +0300 X-Yandex-Fwd: 2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ipfw.ru; s=mail; t=1655413228; bh=CvpYoeJw0pesrUteTJsO05vXQq/NA69z8n6ai0peVHs=; h=References:Date:In-Reply-To:Subject:Cc:From:Message-Id:To; b=QaJIwmw0X3uNLGOCQF20P89d3PbV/UquGjjZug+GGH+VtwWerGsWc4htGDrW2wXqu Bo5Koz9llLy57RbRsiMrMchmIFbbV7lpOyxKyaOo5B3COqlOgQOio9D6mobUVQsxXS mMMR+GkePabyg8N/6siN/ASdgrW5JJjiUSlq2eOo= Received: by myt5-89cdf5c4a3a5.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id SQrJqVjF3M-0QNWkwbX; Fri, 17 Jun 2022 00:00:27 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) Content-Type: text/plain; charset=utf-8 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 15.0 \(3693.60.0.1.1\)) Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) From: "Alexander V. Chernikov" In-Reply-To: <1816e469bdf.126cdb81b2139485.369352368493375815@swiftsmsgateway.com> Date: Thu, 16 Jun 2022 22:00:25 +0100 Cc: freebsd-net Content-Transfer-Encoding: quoted-printable Message-Id: References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> <1816e469bdf.126cdb81b2139485.369352368493375815@swiftsmsgateway.com> To: mike.jakubik@swiftsmsgateway.com X-Mailer: Apple Mail (2.3693.60.0.1.1) X-Rspamd-Queue-Id: 4LPF1w53yhz3Q4g X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ipfw.ru header.s=mail header.b=QaJIwmw0; dmarc=none; spf=pass (mx1.freebsd.org: domain of melifaro@ipfw.ru designates 37.140.190.203 as permitted sender) smtp.mailfrom=melifaro@ipfw.ru X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[ipfw.ru:s=mail]; FREEFALL_USER(0.00)[melifaro]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:37.140.128.0/18]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[ipfw.ru]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[ipfw.ru:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[37.140.190.203:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-net]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:13238, ipnet:37.140.128.0/18, country:RU]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N > On 16 Jun 2022, at 21:48, Mike Jakubik = wrote: >=20 > After multiple tests and tweaks i believe the issue is not with the HW = or Numa related (Infinity fabric should do around 32GB) but rather with = FreeBSD TCP/IP stack. It's like it cant figure itself out properly for = the speed that the HW can do, i keep getting widely varying results when = testing. Below is an example of two tests, with about a 15 second break = in between the two. Does pinning iperf to a specific CPU core (or range) address the variety = part? e.g. cpuset -l 1 perf =E2=80=A6 The output you shared above shows CPU#83 as the core iperf is running = on. Just wondering whether the scheduler migrates iperf too often, = thrashing the caches. >=20 > [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 > Connecting to host db-01, port 5201 > [ 5] local 192.168.10.31 port 49155 connected to 192.168.10.30 port = 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 991 MBytes 8.32 Gbits/sec 268 579 = KBytes > [ 5] 1.00-2.00 sec 945 MBytes 7.93 Gbits/sec 369 777 = KBytes > [ 5] 2.00-3.00 sec 793 MBytes 6.65 Gbits/sec 60 1.03 = MBytes > [ 5] 3.00-4.00 sec 666 MBytes 5.59 Gbits/sec 203 976 = KBytes > [ 5] 4.00-5.01 sec 575 MBytes 4.78 Gbits/sec 202 1.13 = MBytes > [ 5] 5.01-6.00 sec 169 MBytes 1.43 Gbits/sec 134 699 = KBytes > [ 5] 6.00-7.00 sec 1.21 GBytes 10.4 Gbits/sec 383 1.08 = MBytes > [ 5] 7.00-8.00 sec 1.21 GBytes 10.4 Gbits/sec 0 1.16 = MBytes > [ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec 124 780 = KBytes > [ 5] 9.00-10.00 sec 690 MBytes 5.79 Gbits/sec 316 605 = KBytes > [ 5] 10.00-11.00 sec 685 MBytes 5.75 Gbits/sec 97 854 = KBytes > [ 5] 11.00-12.00 sec 1.08 GBytes 9.30 Gbits/sec 383 538 = KBytes > [ 5] 12.00-13.00 sec 682 MBytes 5.72 Gbits/sec 88 870 = KBytes > [ 5] 13.00-14.00 sec 678 MBytes 5.69 Gbits/sec 123 964 = KBytes > [ 5] 14.00-15.00 sec 670 MBytes 5.62 Gbits/sec 290 763 = KBytes > [ 5] 15.00-16.00 sec 1.01 GBytes 8.71 Gbits/sec 228 1.08 = MBytes > [ 5] 16.00-17.00 sec 886 MBytes 7.44 Gbits/sec 118 615 = KBytes > [ 5] 17.00-18.00 sec 734 MBytes 6.16 Gbits/sec 291 902 = KBytes > [ 5] 18.00-19.00 sec 1.04 GBytes 8.96 Gbits/sec 212 323 = KBytes > [ 5] 19.00-20.00 sec 710 MBytes 5.96 Gbits/sec 193 547 = KBytes > [ 5] 20.00-21.00 sec 693 MBytes 5.82 Gbits/sec 370 942 = KBytes > [ 5] 21.00-22.00 sec 704 MBytes 5.91 Gbits/sec 80 1022 = KBytes > [ 5] 22.00-23.00 sec 1.26 GBytes 10.8 Gbits/sec 262 965 = KBytes > [ 5] 23.00-24.00 sec 828 MBytes 6.94 Gbits/sec 202 763 = KBytes > [ 5] 24.00-25.00 sec 774 MBytes 6.49 Gbits/sec 227 581 = KBytes > [ 5] 25.00-26.00 sec 734 MBytes 6.15 Gbits/sec 256 664 = KBytes > [ 5] 26.00-27.00 sec 753 MBytes 6.32 Gbits/sec 331 540 = KBytes > [ 5] 27.00-28.00 sec 764 MBytes 6.41 Gbits/sec 298 823 = KBytes > [ 5] 28.00-29.00 sec 757 MBytes 6.35 Gbits/sec 123 850 = KBytes > [ 5] 29.00-30.00 sec 754 MBytes 6.32 Gbits/sec 74 970 = KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-30.00 sec 24.4 GBytes 6.98 Gbits/sec 6305 = sender > [ 5] 0.00-30.00 sec 24.4 GBytes 6.98 Gbits/sec = receiver >=20 > iperf Done. >=20 > [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 > Connecting to host db-01, port 5201 > [ 5] local 192.168.10.31 port 25061 connected to 192.168.10.30 port = 5201 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 1.81 GBytes 15.5 Gbits/sec 0 1.11 = MBytes > [ 5] 1.00-2.00 sec 1.83 GBytes 15.7 Gbits/sec 0 1.11 = MBytes > [ 5] 2.00-3.00 sec 1.98 GBytes 17.0 Gbits/sec 0 1.11 = MBytes > [ 5] 3.00-4.00 sec 2.11 GBytes 18.1 Gbits/sec 0 1.11 = MBytes > [ 5] 4.00-5.00 sec 2.12 GBytes 18.2 Gbits/sec 0 1.11 = MBytes > [ 5] 5.00-6.00 sec 2.16 GBytes 18.5 Gbits/sec 0 1.11 = MBytes > [ 5] 6.00-7.00 sec 1.90 GBytes 16.3 Gbits/sec 0 1.12 = MBytes > [ 5] 7.00-8.02 sec 1.28 GBytes 10.8 Gbits/sec 0 1.12 = MBytes > [ 5] 8.02-9.00 sec 1.83 GBytes 16.0 Gbits/sec 0 1.17 = MBytes > [ 5] 9.00-10.00 sec 1.91 GBytes 16.4 Gbits/sec 0 1.20 = MBytes > [ 5] 10.00-11.00 sec 1.79 GBytes 15.3 Gbits/sec 0 1.60 = MBytes > [ 5] 11.00-12.00 sec 1.77 GBytes 15.2 Gbits/sec 0 1.60 = MBytes > [ 5] 12.00-13.00 sec 1.69 GBytes 14.5 Gbits/sec 0 1.61 = MBytes > [ 5] 13.00-14.00 sec 1.57 GBytes 13.5 Gbits/sec 0 1.61 = MBytes > [ 5] 14.00-15.00 sec 1.60 GBytes 13.8 Gbits/sec 0 1.61 = MBytes > [ 5] 15.00-16.00 sec 1.89 GBytes 16.2 Gbits/sec 0 1.61 = MBytes > [ 5] 16.00-17.00 sec 1.76 GBytes 15.1 Gbits/sec 0 1.61 = MBytes > [ 5] 17.00-18.00 sec 1.93 GBytes 16.6 Gbits/sec 0 1.61 = MBytes > [ 5] 18.00-19.00 sec 1.77 GBytes 15.2 Gbits/sec 0 1.61 = MBytes > [ 5] 19.00-20.00 sec 1.68 GBytes 14.5 Gbits/sec 0 1.61 = MBytes > [ 5] 20.00-21.00 sec 1.66 GBytes 14.3 Gbits/sec 0 1.61 = MBytes > [ 5] 21.00-22.00 sec 1.75 GBytes 15.1 Gbits/sec 0 1.61 = MBytes > [ 5] 22.00-23.00 sec 1.76 GBytes 15.1 Gbits/sec 0 1.61 = MBytes > [ 5] 23.00-24.00 sec 2.05 GBytes 17.7 Gbits/sec 0 1.61 = MBytes > [ 5] 24.00-25.00 sec 1.79 GBytes 15.4 Gbits/sec 0 1.61 = MBytes > [ 5] 25.00-26.00 sec 1.71 GBytes 14.7 Gbits/sec 0 1.61 = MBytes > [ 5] 26.00-27.00 sec 2.08 GBytes 17.9 Gbits/sec 0 1.61 = MBytes > [ 5] 27.00-28.00 sec 2.09 GBytes 18.0 Gbits/sec 0 1.61 = MBytes > [ 5] 28.00-29.00 sec 2.05 GBytes 17.6 Gbits/sec 0 1.61 = MBytes > [ 5] 29.00-30.00 sec 1.81 GBytes 15.6 Gbits/sec 0 1.61 = MBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-30.00 sec 55.2 GBytes 15.8 Gbits/sec 0 = sender > [ 5] 0.00-30.00 sec 55.2 GBytes 15.8 Gbits/sec = receiver >=20 > iperf Done. >=20