From nobody Tue Jun 14 14:21:51 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A5FC485056E for ; Tue, 14 Jun 2022 14:21:59 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) Received: from sender4-op-o13.zoho.com (sender4-op-o13.zoho.com [136.143.188.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LMrGt651Yz4d6X for ; Tue, 14 Jun 2022 14:21:58 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) ARC-Seal: i=1; a=rsa-sha256; t=1655216514; cv=none; d=zohomail.com; s=zohoarc; b=PmKSLaAOTEcJ6HqpK4IxcraodmKFP+SijGHfshSmlWTbZIZr9jNPvnmH3c05tzn4DunkFkBqT2h0qdipbpMIPnbNI9np9V2x0H0fk+HJUY5sV8Vzlyz/bMerwWezZvvh5W4k0ql7zjNfzQzRLVZtKWUtqa5L8UrmBGJKD25ZW2U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1655216514; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:Reply-To:References:Subject:To; bh=wUjluDi/UrGig2DYMYxs1sLdKaMeAJ7cUZufg6wMxdE=; b=b9rOf29TuAIig2fXq+GPabQSMQa3nr4wt609W+dJjGMXYyRaGvIslUYavOuJAvAH8SezcQPbLhmschbuswt//lR7LSz/GtgADq3NXPnZRhtPzMphY6+i9EK9hXyb5lRfXsGYTdzmXnh/KFsjvFvsmvJT/o/aW+3oZObp2Z8S+Yc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=swiftsmsgateway.com; spf=pass smtp.mailfrom=michal.jakubik@zoho.com; dmarc=pass header.from= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:reply-to:to:cc:message-id:in-reply-to:references:subject:mime-version:content-type:user-agent; b=aOO4toCuEXTQKjwQQoux1uDGPbljY2H3np2O0g7Y25vsJ0CIHAV9eWmjb9bEdgen+GxSQXXjgheW d535HYpUoahVY5h85xCKz2mDUSpD9Ai6qsJcC00dvG97VlJuHVFY DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1655216514; s=zoho; d=swiftsmsgateway.com; i=mike.jakubik@swiftsmsgateway.com; h=Date:Date:From:From:Reply-To:Reply-To:To:To:Cc:Cc:Message-Id:Message-Id:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type; bh=wUjluDi/UrGig2DYMYxs1sLdKaMeAJ7cUZufg6wMxdE=; b=ZIHs2jqUQUpgManNWuXj7/osA+LtCkyd4b8xJTxBGIK588e97T85NUBqE8mxIlpf Dyu3VgN5hnH1uylPC6XXrZ3DkbqxD0DXiA4Q6JQUx6fwBanhd+GrgeN+THkP40hKOis DvTMG+izB9R3xuYg6cyVsdKb1yt8CR7uD/e1+7FE= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1655216511664900.5162833295063; Tue, 14 Jun 2022 07:21:51 -0700 (PDT) Date: Tue, 14 Jun 2022 10:21:51 -0400 From: Mike Jakubik Reply-To: mike.jakubik@swiftsmsgateway.com To: "Hans Petter Selasky" Cc: "freebsd-net" Message-Id: <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> In-Reply-To: <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_4634877_126180685.1655216511632" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Rspamd-Queue-Id: 4LMrGt651Yz4d6X X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=swiftsmsgateway.com header.s=zoho header.b=ZIHs2jqU; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (mx1.freebsd.org: domain of michal.jakubik@zoho.com designates 136.143.188.13 as permitted sender) smtp.mailfrom=michal.jakubik@zoho.com X-Spamd-Result: default: False [-2.71 / 15.00]; HAS_REPLYTO(0.00)[mike.jakubik@swiftsmsgateway.com]; XM_UA_NO_VERSION(0.01)[]; R_SPF_ALLOW(-0.20)[+ip4:136.143.188.0/24]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[swiftsmsgateway.com:+]; RCPT_COUNT_TWO(0.00)[2]; FORGED_SENDER(0.30)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[zoho.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; ASN(0.00)[asn:2639, ipnet:136.143.188.0/23, country:US]; NEURAL_HAM_MEDIUM(-0.86)[-0.863]; R_DKIM_ALLOW(-0.20)[swiftsmsgateway.com:s=zoho]; FROM_HAS_DN(0.00)[]; ARC_ALLOW(-1.00)[zohomail.com:s=zohoarc:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[swiftsmsgateway.com]; NEURAL_SPAM_SHORT(0.35)[0.346]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[136.143.188.13:from]; MLMMJ_DEST(0.00)[freebsd-net]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N ------=_Part_4634877_126180685.1655216511632 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Disabling rx/tx pause seems to produce higher peaks. [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.89 GBytes=C2=A0 16= .2 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.10 MBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.86 GBytes=C2=A0 15= .9 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.10 MBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 2.05 GBytes=C2=A0 17= .6 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 859 MBytes=C2= =A0 7.20 Gbits/sec=C2=A0=C2=A0 21=C2=A0=C2=A0=C2=A0 938 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 652 MBytes=C2= =A0 5.47 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 659 MBytes=C2= =A0 5.53 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.03 MBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 666 MBytes=C2= =A0 5.59 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.05 MBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 657 MBytes=C2= =A0 5.51 Gbits/sec=C2=A0=C2=A0 98=C2=A0=C2=A0=C2=A0 989 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 665 MBytes=C2= =A0 5.58 Gbits/sec=C2=A0 139=C2=A0=C2=A0=C2=A0 712 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 647 MBytes=C2=A0 5.= 43 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.02 MBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 650 MBytes=C2=A0 5.45 Gb= its/sec=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0=C2=A0 606 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.53 GBytes=C2=A0 13.1 Gbits/s= ec=C2=A0 358=C2=A0=C2=A0 1.07 MBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 2.10 GBytes=C2=A0 18.1 Gbits/s= ec=C2=A0 162=C2=A0=C2=A0=C2=A0 837 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 2.09 GBytes=C2=A0 18.0 Gbits/s= ec=C2=A0 332=C2=A0=C2=A0=C2=A0 838 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 2.43 GBytes=C2=A0 20.9 Gbits/s= ec=C2=A0 639=C2=A0=C2=A0=C2=A0 747 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 2.38 GBytes=C2=A0 20.4 Gbits/s= ec=C2=A0 612=C2=A0=C2=A0 1.02 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 2.25 GBytes=C2=A0 19.3 Gbits/s= ec=C2=A0 535=C2=A0=C2=A0 1.24 MBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 2.52 GBytes=C2=A0 21.6 Gbits/s= ec=C2=A0 818=C2=A0=C2=A0=C2=A0 423 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0 218=C2=A0=C2=A0=C2=A0 444 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0 114=C2=A0=C2=A0=C2=A0 859 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14.1 Gbits/s= ec=C2=A0 100=C2=A0=C2=A0=C2=A0 541 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 1.01 GBytes=C2=A0 8.67 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 639 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0=C2=A0 625 MBytes=C2=A0 5.24 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 648 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 630 MBytes=C2=A0 5.28 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 648 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 1.56 GBytes=C2=A0 13.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 702 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 1.78 GBytes=C2=A0 15.3 Gbits/s= ec=C2=A0 118=C2=A0=C2=A0=C2=A0 406 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.37 GBytes=C2=A0 11.8 Gbits/s= ec=C2=A0 105=C2=A0=C2=A0=C2=A0 890 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.82 GBytes=C2=A0 15.6 Gbits/s= ec=C2=A0 104=C2=A0=C2=A0=C2=A0 963 KBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.68 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.20 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.38 MBytes - - - - - - - - - - - - - - - - - - - - - - - - -=20 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 44.8 GBytes=C2=A0 12.8 Gb= its/sec=C2=A0 4477=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.01=C2=A0 sec=C2=A0 44.8 GBytes=C2=A0 12.8 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver After a few runs: [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 52152 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.91 GBytes=C2=A0 16= .4 Gbits/sec=C2=A0=C2=A0 67=C2=A0=C2=A0=C2=A0 606 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.78 GBytes=C2=A0 15= .3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.07 MBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 1.60 GBytes=C2=A0 13= .7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.54 MBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 1.61 GBytes=C2=A0 13= .8 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14= .3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14= .3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14= .1 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 1.70 GBytes=C2=A0 14= .6 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 1.72 GBytes=C2=A0 14= .8 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 1.85 GBytes=C2=A0 15.9 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 1.81 GBytes=C2=A0 15.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 1.18 GBytes=C2=A0 10.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 794 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 1.73 GBytes=C2=A0 14.8 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 1.73 GBytes=C2=A0 14.9 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 1.77 GBytes=C2=A0 15.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.80 GBytes=C2=A0 15.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 2.03 GBytes=C2=A0 17.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 1.88 GBytes=C2=A0 16.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 1.80 GBytes=C2=A0 15.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 24.00-25.01=C2=A0 sec=C2=A0 1.59 GBytes=C2=A0 13.6 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 25.01-26.00=C2=A0 sec=C2=A0 1.33 GBytes=C2=A0 11.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s= ec=C2=A0=C2=A0 97=C2=A0=C2=A0 1.01 MBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0=C2=A0 719 MBytes=C2=A0 6.03 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0=C2=A0 727 MBytes=C2=A0 6.10 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 49.3 GBytes=C2=A0 14.1 Gb= its/sec=C2=A0 164=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 49.3 GBytes=C2=A0 14.1 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver The CPU usage is rather low, the NIC is in the OCP port, so im sure thats d= esigned accordingly and the NIC is bound to numa0. CPU:=C2=A0 0.0% user,=C2=A0 0.0% nice,=C2=A0 0.5% system,=C2=A0 0.7% interr= upt, 98.8% idle =C2=A0 PID USERNAME=C2=A0=C2=A0=C2=A0 THR PRI NICE=C2=A0=C2=A0 SIZE=C2=A0= =C2=A0=C2=A0 RES STATE=C2=A0=C2=A0=C2=A0 C=C2=A0=C2=A0 TIME=C2=A0=C2=A0=C2= =A0 WCPU COMMAND=20 2195 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0 52= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 17M=C2=A0 6884K select=C2=A0 83=C2= =A0=C2=A0 0:14=C2=A0 27.99% iperf3 # vmstat -i -w1|grep mlx5 irq671: mlx5_core0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49969=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 47008 (this drops to about 14k with HW LRO enabled) The dump is rather large, i dont think i can attach it in the mailing list,= but if you wish to see it i can upload it somewhere. Thank You. ---- On Mon, 13 Jun 2022 16:42:59 -0400 Hans Petter Selasky wrote ---- Some ideas:=20 =20 Try to disable "rxpause,txpause" when setting the media.=20 =20 Keep HW LRO off for now, it doesn't work for large number of connections.= =20 =20 What is the CPU usage during test? Is iperf3 running on a CPU-core which=20 has direct access to the NIC's numa domain?=20 =20 Is the NIC installed in the "correct" PCI high-performance slot?=20 =20 There are some sysctl knobs which may tell where the problem is, if it's=20 PCI backpressure or something else.=20 =20 sysctl -a | grep diag_pci_enable=20 sysctl -a | grep diag_general_enable=20 =20 Set these two to 1, then run some traffic and dump all mce sysctls:=20 =20 sysctl -a | grep mce > dump.txt=20 =20 --HPS=20 =20 Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use o= f the addressee(s) and may contain information that is privileged or confid= ential. If you are not the intended recipient, or responsible for deliverin= g the information to the intended recipient, you are hereby notified that a= ny dissemination, distribution, printing or copying of this e-mail and any = attachments is strictly prohibited. If this e-mail and any attachments were= received in error, please notify the sender by reply e-mail and delete the= original message. ------=_Part_4634877_126180685.1655216511632 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
Disabling rx/tx pause seems to produce higher peak= s.

[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 =
Connecting to host db-01, port 5201
[  5]= local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5201
[ ID] Interval         = ;  Transfer     Bitrate    &nb= sp;    Retr  Cwnd
[  5]   = 0.00-1.00   sec  1.89 GBytes  16.2 Gbits/sec  = ;  0   1.10 MBytes
[  5]   1.00= -2.00   sec  1.86 GBytes  15.9 Gbits/sec  &nb= sp; 0   1.10 MBytes
[  5]   2.00-3.0= 0   sec  2.05 GBytes  17.6 Gbits/sec    = 0   1.11 MBytes
[  5]   3.00-4.00&nb= sp;  sec   859 MBytes  7.20 Gbits/sec   21&nb= sp;   938 KBytes
[  5]   4.00-5.00&n= bsp;  sec   652 MBytes  5.47 Gbits/sec   = ; 0   1.01 MBytes
[  5]   5.00-6.00&= nbsp;  sec   659 MBytes  5.53 Gbits/sec  &nbs= p; 0   1.03 MBytes
[  5]   6.00-7.00=    sec   666 MBytes  5.59 Gbits/sec  &nb= sp; 0   1.05 MBytes
[  5]   7.00-8.0= 0   sec   657 MBytes  5.51 Gbits/sec   9= 8    989 KBytes
[  5]   8.00-9.= 00   sec   665 MBytes  5.58 Gbits/sec  139&nb= sp;   712 KBytes
[  5]   9.00-10.00&= nbsp; sec   647 MBytes  5.43 Gbits/sec    0&n= bsp;  1.02 MBytes
[  5]  10.00-11.00  sec=    650 MBytes  5.45 Gbits/sec    4  = ;  606 KBytes
[  5]  11.00-12.00  sec&nbs= p; 1.53 GBytes  13.1 Gbits/sec  358   1.07 MBytes
[  5]  12.00-13.00  sec  2.10 GBytes  18.= 1 Gbits/sec  162    837 KBytes
[  5]=   13.00-14.00  sec  2.09 GBytes  18.0 Gbits/sec  3= 32    838 KBytes
[  5]  14.00-15.00&= nbsp; sec  2.43 GBytes  20.9 Gbits/sec  639   = ; 747 KBytes
[  5]  15.00-16.00  sec  2.3= 8 GBytes  20.4 Gbits/sec  612   1.02 MBytes
[  5]  16.00-17.00  sec  2.25 GBytes  19.3 Gbit= s/sec  535   1.24 MBytes
[  5]  17.0= 0-18.00  sec  2.52 GBytes  21.6 Gbits/sec  818 &nb= sp;  423 KBytes
[  5]  18.00-19.00  sec&n= bsp; 2.29 GBytes  19.7 Gbits/sec  218    444 KByte= s
[  5]  19.00-20.00  sec  2.29 GBytes&nb= sp; 19.7 Gbits/sec  114    859 KBytes
[&n= bsp; 5]  20.00-21.00  sec  1.65 GBytes  14.1 Gbits/sec&= nbsp; 100    541 KBytes
[  5]  21.00= -22.00  sec  1.01 GBytes  8.67 Gbits/sec    0=     639 KBytes
[  5]  22.00-23.00&nb= sp; sec   625 MBytes  5.24 Gbits/sec    0&nbs= p;   648 KBytes
[  5]  23.00-24.00  = sec   630 MBytes  5.28 Gbits/sec    0 &n= bsp;  648 KBytes
[  5]  24.00-25.00  sec&= nbsp; 1.56 GBytes  13.4 Gbits/sec    0   = ; 702 KBytes
[  5]  25.00-26.00  sec  1.7= 8 GBytes  15.3 Gbits/sec  118    406 KBytes
[  5]  26.00-27.00  sec  1.37 GBytes  11.8= Gbits/sec  105    890 KBytes
[  5]&= nbsp; 27.00-28.00  sec  1.82 GBytes  15.6 Gbits/sec  10= 4    963 KBytes
[  5]  28.00-29.00&n= bsp; sec  1.68 GBytes  14.4 Gbits/sec    0 &n= bsp; 1.20 MBytes
[  5]  29.00-30.00  sec = 1.67 GBytes  14.4 Gbits/sec    0   1.38 MByt= es
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval         =   Transfer     Bitrate    &nbs= p;    Retr
[  5]   0.00-30.00&n= bsp; sec  44.8 GBytes  12.8 Gbits/sec  4477   = ;          sender
[  5]   0.00-30.01  sec  44.8 GBytes  12.8 = Gbits/sec           =        receiver

= After a few runs:

[root@db-02 ~]# iperf3 -i 1 = -t 30 -c db-01
Connecting to host db-01, port 5201
=
[  5] local 192.168.10.31 port 52152 connected to 192.168.10.30 p= ort 5201
[ ID] Interval      &n= bsp;    Transfer     Bitrate  =        Retr  Cwnd
[  = 5]   0.00-1.00   sec  1.91 GBytes  16.4 Gbits= /sec   67    606 KBytes
[  5]&n= bsp;  1.00-2.00   sec  1.78 GBytes  15.3 Gbits/sec=     0   1.07 MBytes
[  5] =   2.00-3.00   sec  1.60 GBytes  13.7 Gbits/sec&nbs= p;   0   1.54 MBytes
[  5] &nbs= p; 3.00-4.00   sec  1.61 GBytes  13.8 Gbits/sec &n= bsp;  0   1.61 MBytes
[  5]   4= .00-5.00   sec  1.66 GBytes  14.3 Gbits/sec  =   0   1.61 MBytes
[  5]   5.00-= 6.00   sec  1.67 GBytes  14.3 Gbits/sec  &nbs= p; 0   1.61 MBytes
[  5]   6.00-7.00=    sec  1.65 GBytes  14.1 Gbits/sec    0=    1.61 MBytes
[  5]   7.00-8.00&nbs= p;  sec  1.70 GBytes  14.6 Gbits/sec    0&nbs= p;  1.61 MBytes
[  5]   8.00-9.00 &n= bsp; sec  1.72 GBytes  14.8 Gbits/sec    0 &n= bsp; 1.61 MBytes
[  5]   9.00-10.00  sec&= nbsp; 1.85 GBytes  15.9 Gbits/sec    0   1.61= MBytes
[  5]  10.00-11.00  sec  1.81 GBy= tes  15.5 Gbits/sec    0   1.61 MBytes
[  5]  11.00-12.00  sec  1.67 GBytes  14.3= Gbits/sec    0   1.61 MBytes
[ = ; 5]  12.00-13.00  sec  1.66 GBytes  14.3 Gbits/sec&nbs= p;   0   1.61 MBytes
[  5]  13.= 00-14.00  sec  1.83 GBytes  15.7 Gbits/sec   = 0   1.61 MBytes
[  5]  14.00-15.00 = sec  1.18 GBytes  10.1 Gbits/sec    0  =   794 KBytes
[  5]  15.00-16.00  sec = ; 1.67 GBytes  14.4 Gbits/sec    0   1.60 MBy= tes
[  5]  16.00-17.00  sec  1.73 GBytes&= nbsp; 14.8 Gbits/sec    0   1.60 MBytes
<= div>[  5]  17.00-18.00  sec  1.73 GBytes  14.9 Gbi= ts/sec    0   1.60 MBytes
[  5]=   18.00-19.00  sec  1.83 GBytes  15.7 Gbits/sec &n= bsp;  0   1.61 MBytes
[  5]  19.00-2= 0.00  sec  1.77 GBytes  15.2 Gbits/sec    0&n= bsp;  1.61 MBytes
[  5]  20.00-21.00  sec=   1.80 GBytes  15.5 Gbits/sec    0   1.6= 1 MBytes
[  5]  21.00-22.00  sec  2.03 GB= ytes  17.4 Gbits/sec    0   1.61 MBytes
[  5]  22.00-23.00  sec  1.88 GBytes  16.= 1 Gbits/sec    0   1.61 MBytes
[&nbs= p; 5]  23.00-24.00  sec  1.80 GBytes  15.5 Gbits/sec&nb= sp;   0   1.61 MBytes
[  5]  24= .00-25.01  sec  1.59 GBytes  13.6 Gbits/sec   = ; 0   1.61 MBytes
[  5]  25.01-26.00 = ; sec  1.33 GBytes  11.4 Gbits/sec    0  = ; 1.61 MBytes
[  5]  26.00-27.00  sec  1.= 71 GBytes  14.7 Gbits/sec    0   1.61 MBytes<= br>
[  5]  27.00-28.00  sec  1.71 GBytes = ; 14.7 Gbits/sec   97   1.01 MBytes
[&nbs= p; 5]  28.00-29.00  sec   719 MBytes  6.03 Gbits/s= ec    0   1.01 MBytes
[  5]&nbs= p; 29.00-30.00  sec   727 MBytes  6.10 Gbits/sec &= nbsp;  0   1.01 MBytes
- - - - - - - - - - - -= - - - - - - - - - - - - -
[ ID] Interval   &n= bsp;       Transfer     B= itrate         Retr
[=   5]   0.00-30.00  sec  49.3 GBytes  14.1 Gbi= ts/sec  164          = ;   sender
[  5]   0.00-30.00  = sec  49.3 GBytes  14.1 Gbits/sec     &nb= sp;            recei= ver



The CPU usag= e is rather low, the NIC is in the OCP port, so im sure thats designed acco= rdingly and the NIC is bound to numa0.

CPU:&nb= sp; 0.0% user,  0.0% nice,  0.5% system,  0.7% interrupt, 98= .8% idle

  PID USERNAME   = THR PRI NICE   SIZE    RES STATE   = ; C   TIME    WCPU COMMAND
2195 roo= t          1  52 &nb= sp;  0    17M  6884K select  83   0= :14  27.99% iperf3

# vmstat -i -w1|grep m= lx5
irq671: mlx5_core0      &nb= sp;          49969  =     47008

(this drops to about = 14k with HW LRO enabled)

The dump is rather la= rge, i dont think i can attach it in the mailing list, but if you wish to s= ee it i can upload it somewhere.

Thank You.




---- = On Mon, 13 Jun 2022 16:42:59 -0400 Hans Petter Selasky <hps@selasky.o= rg> wrote ----


Some ideas:

Try to disable "rxpause,txpause" when s= etting the media.

Keep HW LRO off for now, it doesn't work for lar= ge number of connections.

What is the CPU usage during test? Is ip= erf3 running on a CPU-core which
has direct access to the NIC's numa do= main?

Is the NIC installed in the "correct" PCI high-performance s= lot?

There are some sysctl knobs which may tell where the problem = is, if it's
PCI backpressure or something else.

sysctl -a | gr= ep diag_pci_enable
sysctl -a | grep diag_general_enable

Set th= ese two to 1, then run some traffic and dump all mce sysctls:

sysc= tl -a | grep mce > dump.txt

--HPS

<= /div>

<= div>Mike Jakubik

<= /div>
Disclaimer: This e-= mail and any attachments are intended only for the use of the addressee(s) = and may contain information that is privileged or confidential. If you are = not the intended recipient, or responsible for delivering the information t= o the intended recipient, you are hereby notified that any dissemination, d= istribution, printing or copying of this e-mail and any attachments is stri= ctly prohibited. If this e-mail and any attachments were received in error,= please notify the sender by reply e-mail and delete the original message.<= /span>


------=_Part_4634877_126180685.1655216511632--