From nobody Tue Jun 14 14:36:06 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6D5E185319F for ; Tue, 14 Jun 2022 14:36:19 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) Received: from sender4-op-o13.zoho.com (sender4-op-o13.zoho.com [136.143.188.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LMrbQ4vpmz4gHY for ; Tue, 14 Jun 2022 14:36:18 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) ARC-Seal: i=1; a=rsa-sha256; t=1655217369; cv=none; d=zohomail.com; s=zohoarc; b=Qbd5f/xYcIaaB/STGrfRTFyCBGQuPo9IGjEDx7+bZ3JljIaD8S/Tm5UBKOBdUTYwpRK6jOuR0rM9MRxLyPvMK9ewqbr3OfkQRZju5DTrEzHK/KU49jTcukhh9g1roegYEZ4ppw11+dEcoIW4+zOMO3IKpO/trQ/eH3M4g9ihUfs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1655217369; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:Reply-To:References:Subject:To; bh=qv7oRWvakCE2o44k1CSUeClDrcIOQM5PYYN2izjWJKw=; b=mTd6v3yU/ndxaxrF/4BsuSSdNk1fv+qpTH7slCWME+ZITVTSbM1eviqu2Pdvr61kK1RXSV87wSS6WZoV1vg8u9HNPd56SS26vwX6xmiYTN1uN+3ofID4IthnpXpk0Z+OjuQPNLwDcB8e9JhibfclzS2N68RcrZQPo/UGIUJyXLE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=swiftsmsgateway.com; spf=pass smtp.mailfrom=michal.jakubik@zoho.com; dmarc=pass header.from= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:reply-to:to:cc:message-id:in-reply-to:references:subject:mime-version:content-type:user-agent; b=hqcE2hvaFXj24SniIYNiy4IF32cuuSC8IWGlDEa5fZgkKRRxn/R/6Us34HtnDFKjtB6W9re5vvdK e55blGH07busebOTqpUJAgrJ9YuNPmy6QRlClMLAdUQo2bzQDVNG DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1655217369; s=zoho; d=swiftsmsgateway.com; i=mike.jakubik@swiftsmsgateway.com; h=Date:Date:From:From:Reply-To:Reply-To:To:To:Cc:Cc:Message-Id:Message-Id:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type; bh=qv7oRWvakCE2o44k1CSUeClDrcIOQM5PYYN2izjWJKw=; b=oeyDisALODEQs8f141ZXPivbZ9w2Wb2WAwFaIL3YWXe0yrHvK5xJOi/UkowWCw1d oAEqiXcGtPFZAAUMHarR1YsQ9Ahlz7uOaWFCKm8zefiFfWLAEZXCwcDSHTCm63BOMNL TpBh9wH4GlKzNg9dczQdTDkZsfSCv6jC9KSRnD+w= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1655217366047690.7081880418361; Tue, 14 Jun 2022 07:36:06 -0700 (PDT) Date: Tue, 14 Jun 2022 10:36:06 -0400 From: Mike Jakubik Reply-To: mike.jakubik@swiftsmsgateway.com To: "mikejakubik" Cc: "Hans Petter Selasky" , "freebsd-net" Message-Id: <18162a4a3f6.10a1a03d11472072.3783895140221599214@swiftsmsgateway.com> In-Reply-To: <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> References: <1815e506878.cf301a5a1195924.6506017618978817828@swiftsmsgateway.com> <63396d47-3d0b-fd83-7b2e-ae5c02eeae2e@selasky.org> <18162979a8f.e81f383a1466900.9104319828015733292@swiftsmsgateway.com> Subject: Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5) List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_4650864_1970601159.1655217366007" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Rspamd-Queue-Id: 4LMrbQ4vpmz4gHY X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=swiftsmsgateway.com header.s=zoho header.b=oeyDisAL; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (mx1.freebsd.org: domain of michal.jakubik@zoho.com designates 136.143.188.13 as permitted sender) smtp.mailfrom=michal.jakubik@zoho.com X-Spamd-Result: default: False [-3.28 / 15.00]; HAS_REPLYTO(0.00)[mike.jakubik@swiftsmsgateway.com]; XM_UA_NO_VERSION(0.01)[]; R_SPF_ALLOW(-0.20)[+ip4:136.143.188.0/24]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[swiftsmsgateway.com:+]; NEURAL_HAM_SHORT(-0.90)[-0.901]; FORGED_SENDER(0.30)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[zoho.com]; MID_RHS_MATCH_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; ASN(0.00)[asn:2639, ipnet:136.143.188.0/23, country:US]; NEURAL_HAM_MEDIUM(-0.19)[-0.187]; R_DKIM_ALLOW(-0.20)[swiftsmsgateway.com:s=zoho]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; ARC_ALLOW(-1.00)[zohomail.com:s=zohoarc:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[swiftsmsgateway.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[136.143.188.13:from]; MLMMJ_DEST(0.00)[freebsd-net]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N ------=_Part_4650864_1970601159.1655217366007 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Actually, i believe its the disabling to HW LRO that makes the difference (= i disabled it and rx/tx pause previously). With rx/tx pause on and LRO off = i get similar results. The throughput is still very sporadic though. Connecting to host db-01, port 5201=20 [=C2=A0 5] local 192.168.10.31 port 59055 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15= .7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15= .7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 2.06 GBytes=C2=A0 17= .7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.12 MBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18= .5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.12 MBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18= .4 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.12 MBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18= .5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.13 MBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18= .5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.13 MBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18= .5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.13 MBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18= .5 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.13 MBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 2.15 GBytes=C2=A0 18.5 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.13 MBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 1.72 GBytes=C2=A0 14.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.14 MBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.18 GBytes=C2=A0 10.1 Gbits/s= ec=C2=A0 131=C2=A0=C2=A0=C2=A0 932 KBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0=C2=A0 739 MBytes=C2=A0 6.20 Gb= its/sec=C2=A0 352=C2=A0=C2=A0=C2=A0 913 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0=C2=A0 724 MBytes=C2=A0 6.07 Gb= its/sec=C2=A0 142=C2=A0=C2=A0=C2=A0 636 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0=C2=A0 759 MBytes=C2=A0 6.37 Gb= its/sec=C2=A0 246=C2=A0=C2=A0=C2=A0 748 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0=C2=A0 748 MBytes=C2=A0 6.27 Gb= its/sec=C2=A0=C2=A0 22=C2=A0=C2=A0=C2=A0 962 KBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0=C2=A0 753 MBytes=C2=A0 6.32 Gb= its/sec=C2=A0 170=C2=A0=C2=A0=C2=A0 711 KBytes [=C2=A0 5]=C2=A0 17.00-18.01=C2=A0 sec=C2=A0=C2=A0 690 MBytes=C2=A0 5.76 Gb= its/sec=C2=A0 389=C2=A0=C2=A0=C2=A0 981 KBytes [=C2=A0 5]=C2=A0 18.01-19.00=C2=A0 sec=C2=A0=C2=A0 771 MBytes=C2=A0 6.50 Gb= its/sec=C2=A0 145=C2=A0=C2=A0=C2=A0 896 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0=C2=A0 741 MBytes=C2=A0 6.22 Gb= its/sec=C2=A0=C2=A0 13=C2=A0=C2=A0=C2=A0 952 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0=C2=A0 743 MBytes=C2=A0 6.23 Gb= its/sec=C2=A0=C2=A0 86=C2=A0=C2=A0=C2=A0 985 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0=C2=A0 751 MBytes=C2=A0 6.30 Gb= its/sec=C2=A0 245=C2=A0=C2=A0=C2=A0 944 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0=C2=A0 724 MBytes=C2=A0 6.07 Gb= its/sec=C2=A0 178=C2=A0=C2=A0=C2=A0 774 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 728 MBytes=C2=A0 6.11 Gb= its/sec=C2=A0 348=C2=A0=C2=A0=C2=A0 546 KBytes [=C2=A0 5]=C2=A0 24.00-25.01=C2=A0 sec=C2=A0=C2=A0 622 MBytes=C2=A0 5.18 Gb= its/sec=C2=A0 215=C2=A0=C2=A0=C2=A0 842 KBytes [=C2=A0 5]=C2=A0 25.01-26.00=C2=A0 sec=C2=A0=C2=A0 920 MBytes=C2=A0 7.78 Gb= its/sec=C2=A0 359=C2=A0=C2=A0 1.03 MBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.02 GBytes=C2=A0 8.80 Gbits/s= ec=C2=A0 385=C2=A0=C2=A0 1011 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0=C2=A0 999 MBytes=C2=A0 8.38 Gb= its/sec=C2=A0 161=C2=A0=C2=A0=C2=A0 757 KBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.68 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.03 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.51 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 39.2 GBytes=C2=A0 11.2 Gb= its/sec=C2=A0 3587=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 39.2 GBytes=C2=A0 11.2 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 60931 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 931 MBytes=C2= =A0 7.81 Gbits/sec=C2=A0 146=C2=A0=C2=A0=C2=A0 689 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 674 MBytes=C2= =A0 5.66 Gbits/sec=C2=A0 267=C2=A0=C2=A0=C2=A0 575 KBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 641 MBytes=C2= =A0 5.38 Gbits/sec=C2=A0 270=C2=A0=C2=A0=C2=A0 485 KBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 668 MBytes=C2= =A0 5.60 Gbits/sec=C2=A0 221=C2=A0=C2=A0=C2=A0 801 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.01=C2=A0=C2=A0 sec=C2=A0=C2=A0 532 MBytes=C2= =A0 4.40 Gbits/sec=C2=A0 208=C2=A0=C2=A0=C2=A0 981 KBytes [=C2=A0 5]=C2=A0=C2=A0 5.01-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 838 MBytes=C2= =A0 7.13 Gbits/sec=C2=A0 291=C2=A0=C2=A0=C2=A0 949 KBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 703 MBytes=C2= =A0 5.90 Gbits/sec=C2=A0 178=C2=A0=C2=A0=C2=A0 628 KBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 696 MBytes=C2= =A0 5.84 Gbits/sec=C2=A0=C2=A0 85=C2=A0=C2=A0=C2=A0 928 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 1.06 GBytes=C2=A0 9.= 14 Gbits/sec=C2=A0=C2=A0 66=C2=A0=C2=A0=C2=A0 565 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 975 MBytes=C2=A0 8.= 18 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 801 KBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 679 MBytes=C2=A0 5.70 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 859 KBytes [=C2=A0 5]=C2=A0 11.00-12.01=C2=A0 sec=C2=A0=C2=A0 628 MBytes=C2=A0 5.23 Gb= its/sec=C2=A0 123=C2=A0=C2=A0 1.12 MBytes [=C2=A0 5]=C2=A0 12.01-13.00=C2=A0 sec=C2=A0=C2=A0 731 MBytes=C2=A0 6.16 Gb= its/sec=C2=A0 311=C2=A0=C2=A0=C2=A0 864 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0=C2=A0 696 MBytes=C2=A0 5.86 Gb= its/sec=C2=A0=C2=A0 39=C2=A0=C2=A0=C2=A0 798 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0=C2=A0 914 MBytes=C2=A0 7.66 Gb= its/sec=C2=A0=C2=A0 65=C2=A0=C2=A0 1024 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.47 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0=C2=A0 955 MBytes=C2=A0 8.01 Gb= its/sec=C2=A0 222=C2=A0=C2=A0=C2=A0 260 KBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0=C2=A0 769 MBytes=C2=A0 6.45 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 520 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0=C2=A0 781 MBytes=C2=A0 6.55 Gb= its/sec=C2=A0 119=C2=A0=C2=A0=C2=A0 432 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0=C2=A0 779 MBytes=C2=A0 6.54 Gb= its/sec=C2=A0 148=C2=A0=C2=A0=C2=A0 816 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0=C2=A0 771 MBytes=C2=A0 6.47 Gb= its/sec=C2=A0 127=C2=A0=C2=A0=C2=A0 965 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0=C2=A0 745 MBytes=C2=A0 6.25 Gb= its/sec=C2=A0 337=C2=A0=C2=A0=C2=A0 927 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0=C2=A0 637 MBytes=C2=A0 5.35 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 938 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 626 MBytes=C2=A0 5.25 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 938 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 1.96 GBytes=C2=A0 16.8 Gbits/s= ec=C2=A0 185=C2=A0=C2=A0=C2=A0 862 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s= ec=C2=A0=C2=A0 22=C2=A0=C2=A0=C2=A0 945 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 2.14 GBytes=C2=A0 18.3 Gbits/s= ec=C2=A0 365=C2=A0=C2=A0=C2=A0 593 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.56 GBytes=C2=A0 13.4 Gbits/s= ec=C2=A0=C2=A0 33=C2=A0=C2=A0 1.23 MBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14.2 Gbits/s= ec=C2=A0=C2=A0 35=C2=A0=C2=A0 1.14 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 29.4 GBytes=C2=A0 8.42 Gb= its/sec=C2=A0 3863=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 29.4 GBytes=C2=A0 8.42 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver ---- On Tue, 14 Jun 2022 10:21:51 -0400 Mike Jakubik wrote ---- Disabling rx/tx pause seems to produce higher peaks. [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.89 GBytes=C2=A0 16= .2 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.10 MBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.86 GBytes=C2=A0 15= .9 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.10 MBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 2.05 GBytes=C2=A0 17= .6 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.11 MBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 859 MBytes=C2= =A0 7.20 Gbits/sec=C2=A0=C2=A0 21=C2=A0=C2=A0=C2=A0 938 KBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 652 MBytes=C2= =A0 5.47 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 659 MBytes=C2= =A0 5.53 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.03 MBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 666 MBytes=C2= =A0 5.59 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.05 MBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 657 MBytes=C2= =A0 5.51 Gbits/sec=C2=A0=C2=A0 98=C2=A0=C2=A0=C2=A0 989 KBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0=C2=A0 665 MBytes=C2= =A0 5.58 Gbits/sec=C2=A0 139=C2=A0=C2=A0=C2=A0 712 KBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0=C2=A0 647 MBytes=C2=A0 5.= 43 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.02 MBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0=C2=A0 650 MBytes=C2=A0 5.45 Gb= its/sec=C2=A0=C2=A0=C2=A0 4=C2=A0=C2=A0=C2=A0 606 KBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.53 GBytes=C2=A0 13.1 Gbits/s= ec=C2=A0 358=C2=A0=C2=A0 1.07 MBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 2.10 GBytes=C2=A0 18.1 Gbits/s= ec=C2=A0 162=C2=A0=C2=A0=C2=A0 837 KBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 2.09 GBytes=C2=A0 18.0 Gbits/s= ec=C2=A0 332=C2=A0=C2=A0=C2=A0 838 KBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 2.43 GBytes=C2=A0 20.9 Gbits/s= ec=C2=A0 639=C2=A0=C2=A0=C2=A0 747 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 2.38 GBytes=C2=A0 20.4 Gbits/s= ec=C2=A0 612=C2=A0=C2=A0 1.02 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 2.25 GBytes=C2=A0 19.3 Gbits/s= ec=C2=A0 535=C2=A0=C2=A0 1.24 MBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 2.52 GBytes=C2=A0 21.6 Gbits/s= ec=C2=A0 818=C2=A0=C2=A0=C2=A0 423 KBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0 218=C2=A0=C2=A0=C2=A0 444 KBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 2.29 GBytes=C2=A0 19.7 Gbits/s= ec=C2=A0 114=C2=A0=C2=A0=C2=A0 859 KBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14.1 Gbits/s= ec=C2=A0 100=C2=A0=C2=A0=C2=A0 541 KBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 1.01 GBytes=C2=A0 8.67 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 639 KBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0=C2=A0 625 MBytes=C2=A0 5.24 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 648 KBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0=C2=A0 630 MBytes=C2=A0 5.28 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 648 KBytes [=C2=A0 5]=C2=A0 24.00-25.00=C2=A0 sec=C2=A0 1.56 GBytes=C2=A0 13.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 702 KBytes [=C2=A0 5]=C2=A0 25.00-26.00=C2=A0 sec=C2=A0 1.78 GBytes=C2=A0 15.3 Gbits/s= ec=C2=A0 118=C2=A0=C2=A0=C2=A0 406 KBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.37 GBytes=C2=A0 11.8 Gbits/s= ec=C2=A0 105=C2=A0=C2=A0=C2=A0 890 KBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.82 GBytes=C2=A0 15.6 Gbits/s= ec=C2=A0 104=C2=A0=C2=A0=C2=A0 963 KBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0 1.68 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.20 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.38 MBytes - - - - - - - - - - - - - - - - - - - - - - - - -=20 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 44.8 GBytes=C2=A0 12.8 Gb= its/sec=C2=A0 4477=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.01=C2=A0 sec=C2=A0 44.8 GBytes=C2=A0 12.8 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver After a few runs: [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01=20 Connecting to host db-01, port 5201 [=C2=A0 5] local 192.168.10.31 port 52152 connected to 192.168.10.30 port 5= 201 [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr=C2=A0 Cwnd [=C2=A0 5]=C2=A0=C2=A0 0.00-1.00=C2=A0=C2=A0 sec=C2=A0 1.91 GBytes=C2=A0 16= .4 Gbits/sec=C2=A0=C2=A0 67=C2=A0=C2=A0=C2=A0 606 KBytes [=C2=A0 5]=C2=A0=C2=A0 1.00-2.00=C2=A0=C2=A0 sec=C2=A0 1.78 GBytes=C2=A0 15= .3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.07 MBytes [=C2=A0 5]=C2=A0=C2=A0 2.00-3.00=C2=A0=C2=A0 sec=C2=A0 1.60 GBytes=C2=A0 13= .7 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.54 MBytes [=C2=A0 5]=C2=A0=C2=A0 3.00-4.00=C2=A0=C2=A0 sec=C2=A0 1.61 GBytes=C2=A0 13= .8 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 4.00-5.00=C2=A0=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14= .3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 5.00-6.00=C2=A0=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14= .3 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 6.00-7.00=C2=A0=C2=A0 sec=C2=A0 1.65 GBytes=C2=A0 14= .1 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 7.00-8.00=C2=A0=C2=A0 sec=C2=A0 1.70 GBytes=C2=A0 14= .6 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 8.00-9.00=C2=A0=C2=A0 sec=C2=A0 1.72 GBytes=C2=A0 14= .8 Gbits/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0=C2=A0 9.00-10.00=C2=A0 sec=C2=A0 1.85 GBytes=C2=A0 15.9 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 10.00-11.00=C2=A0 sec=C2=A0 1.81 GBytes=C2=A0 15.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 11.00-12.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 12.00-13.00=C2=A0 sec=C2=A0 1.66 GBytes=C2=A0 14.3 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 13.00-14.00=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 14.00-15.00=C2=A0 sec=C2=A0 1.18 GBytes=C2=A0 10.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 794 KBytes [=C2=A0 5]=C2=A0 15.00-16.00=C2=A0 sec=C2=A0 1.67 GBytes=C2=A0 14.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 16.00-17.00=C2=A0 sec=C2=A0 1.73 GBytes=C2=A0 14.8 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 17.00-18.00=C2=A0 sec=C2=A0 1.73 GBytes=C2=A0 14.9 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.60 MBytes [=C2=A0 5]=C2=A0 18.00-19.00=C2=A0 sec=C2=A0 1.83 GBytes=C2=A0 15.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 19.00-20.00=C2=A0 sec=C2=A0 1.77 GBytes=C2=A0 15.2 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 20.00-21.00=C2=A0 sec=C2=A0 1.80 GBytes=C2=A0 15.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 21.00-22.00=C2=A0 sec=C2=A0 2.03 GBytes=C2=A0 17.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 22.00-23.00=C2=A0 sec=C2=A0 1.88 GBytes=C2=A0 16.1 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 23.00-24.00=C2=A0 sec=C2=A0 1.80 GBytes=C2=A0 15.5 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 24.00-25.01=C2=A0 sec=C2=A0 1.59 GBytes=C2=A0 13.6 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 25.01-26.00=C2=A0 sec=C2=A0 1.33 GBytes=C2=A0 11.4 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 26.00-27.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s= ec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.61 MBytes [=C2=A0 5]=C2=A0 27.00-28.00=C2=A0 sec=C2=A0 1.71 GBytes=C2=A0 14.7 Gbits/s= ec=C2=A0=C2=A0 97=C2=A0=C2=A0 1.01 MBytes [=C2=A0 5]=C2=A0 28.00-29.00=C2=A0 sec=C2=A0=C2=A0 719 MBytes=C2=A0 6.03 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes [=C2=A0 5]=C2=A0 29.00-30.00=C2=A0 sec=C2=A0=C2=A0 727 MBytes=C2=A0 6.10 Gb= its/sec=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0 1.01 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = Transfer=C2=A0=C2=A0=C2=A0=C2=A0 Bitrate=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 Retr [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 49.3 GBytes=C2=A0 14.1 Gb= its/sec=C2=A0 164=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 sender [=C2=A0 5]=C2=A0=C2=A0 0.00-30.00=C2=A0 sec=C2=A0 49.3 GBytes=C2=A0 14.1 Gb= its/sec=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 receiver The CPU usage is rather low, the NIC is in the OCP port, so im sure thats d= esigned accordingly and the NIC is bound to numa0. CPU:=C2=A0 0.0% user,=C2=A0 0.0% nice,=C2=A0 0.5% system,=C2=A0 0.7% interr= upt, 98.8% idle =C2=A0 PID USERNAME=C2=A0=C2=A0=C2=A0 THR PRI NICE=C2=A0=C2=A0 SIZE=C2=A0= =C2=A0=C2=A0 RES STATE=C2=A0=C2=A0=C2=A0 C=C2=A0=C2=A0 TIME=C2=A0=C2=A0=C2= =A0 WCPU COMMAND=20 2195 root=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1=C2=A0 52= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0 17M=C2=A0 6884K select=C2=A0 83=C2= =A0=C2=A0 0:14=C2=A0 27.99% iperf3 # vmstat -i -w1|grep mlx5 irq671: mlx5_core0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 49969=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 47008 (this drops to about 14k with HW LRO enabled) The dump is rather large, i dont think i can attach it in the mailing list,= but if you wish to see it i can upload it somewhere. Thank You. ---- On Mon, 13 Jun 2022 16:42:59 -0400 Hans Petter Selasky wrote ---- Some ideas:=20 =20 Try to disable "rxpause,txpause" when setting the media.=20 =20 Keep HW LRO off for now, it doesn't work for large number of connections.= =20 =20 What is the CPU usage during test? Is iperf3 running on a CPU-core which=20 has direct access to the NIC's numa domain?=20 =20 Is the NIC installed in the "correct" PCI high-performance slot?=20 =20 There are some sysctl knobs which may tell where the problem is, if it's=20 PCI backpressure or something else.=20 =20 sysctl -a | grep diag_pci_enable=20 sysctl -a | grep diag_general_enable=20 =20 Set these two to 1, then run some traffic and dump all mce sysctls:=20 =20 sysctl -a | grep mce > dump.txt=20 =20 --HPS=20 =20 Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use o= f the addressee(s) and may contain information that is privileged or confid= ential. If you are not the intended recipient, or responsible for deliverin= g the information to the intended recipient, you are hereby notified that a= ny dissemination, distribution, printing or copying of this e-mail and any = attachments is strictly prohibited. If this e-mail and any attachments were= received in error, please notify the sender by reply e-mail and delete the= original message. Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use o= f the addressee(s) and may contain information that is privileged or confid= ential. If you are not the intended recipient, or responsible for deliverin= g the information to the intended recipient, you are hereby notified that a= ny dissemination, distribution, printing or copying of this e-mail and any = attachments is strictly prohibited. If this e-mail and any attachments were= received in error, please notify the sender by reply e-mail and delete the= original message. ------=_Part_4650864_1970601159.1655217366007 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
Actually, i believe its the disabling to HW LRO th= at makes the difference (i disabled it and rx/tx pause previously). With rx= /tx pause on and LRO off i get similar results. The throughput is still ver= y sporadic though.

Connecting to host db-01, p= ort 5201
[  5] local 192.168.10.31 port 59055 connected= to 192.168.10.30 port 5201
[ ID] Interval   &= nbsp;       Transfer     = Bitrate         Retr  Cwnd
=
[  5]   0.00-1.00   sec  1.83 GByt= es  15.7 Gbits/sec    0   1.11 MBytes
[  5]   1.00-2.00   sec  1.83 GBytes&n= bsp; 15.7 Gbits/sec    0   1.11 MBytes
[  5]   2.00-3.00   sec  2.06 GBytes = 17.7 Gbits/sec    0   1.12 MBytes
[=   5]   3.00-4.00   sec  2.15 GBytes  18.= 5 Gbits/sec    0   1.12 MBytes
[&nbs= p; 5]   4.00-5.00   sec  2.15 GBytes  18.4 Gb= its/sec    0   1.12 MBytes
[  5= ]   5.00-6.00   sec  2.15 GBytes  18.5 Gbits/= sec    0   1.13 MBytes
[  5]&nb= sp;  6.00-7.00   sec  2.15 GBytes  18.5 Gbits/sec&= nbsp;   0   1.13 MBytes
[  5] &= nbsp; 7.00-8.00   sec  2.15 GBytes  18.5 Gbits/sec = ;   0   1.13 MBytes
[  5]  = ; 8.00-9.00   sec  2.15 GBytes  18.5 Gbits/sec &nb= sp;  0   1.13 MBytes
[  5]   9.= 00-10.00  sec  2.15 GBytes  18.5 Gbits/sec   = 0   1.13 MBytes
[  5]  10.00-11.00 = sec  1.72 GBytes  14.7 Gbits/sec    0  = 1.14 MBytes
[  5]  11.00-12.00  sec  1.1= 8 GBytes  10.1 Gbits/sec  131    932 KBytes
[  5]  12.00-13.00  sec   739 MBytes = 6.20 Gbits/sec  352    913 KBytes
[ = ; 5]  13.00-14.00  sec   724 MBytes  6.07 Gbits/se= c  142    636 KBytes
[  5]  14.= 00-15.00  sec   759 MBytes  6.37 Gbits/sec  246&nb= sp;   748 KBytes
[  5]  15.00-16.00 = sec   748 MBytes  6.27 Gbits/sec   22  =   962 KBytes
[  5]  16.00-17.00  sec = ;  753 MBytes  6.32 Gbits/sec  170    711 KBy= tes
[  5]  17.00-18.01  sec   690 MB= ytes  5.76 Gbits/sec  389    981 KBytes
<= div>[  5]  18.01-19.00  sec   771 MBytes  6.5= 0 Gbits/sec  145    896 KBytes
[  5]=   19.00-20.00  sec   741 MBytes  6.22 Gbits/sec&nb= sp;  13    952 KBytes
[  5]  20= .00-21.00  sec   743 MBytes  6.23 Gbits/sec  = 86    985 KBytes
[  5]  21.00-22.00=   sec   751 MBytes  6.30 Gbits/sec  245  = ;  944 KBytes
[  5]  22.00-23.00  sec&nbs= p;  724 MBytes  6.07 Gbits/sec  178    774 KB= ytes
[  5]  23.00-24.00  sec   728 M= Bytes  6.11 Gbits/sec  348    546 KBytes
=
[  5]  24.00-25.01  sec   622 MBytes  5.= 18 Gbits/sec  215    842 KBytes
[  5= ]  25.01-26.00  sec   920 MBytes  7.78 Gbits/sec&n= bsp; 359   1.03 MBytes
[  5]  26.00-27.00=   sec  1.02 GBytes  8.80 Gbits/sec  385   101= 1 KBytes
[  5]  27.00-28.00  sec   9= 99 MBytes  8.38 Gbits/sec  161    757 KBytes
[  5]  28.00-29.00  sec  1.68 GBytes  14.= 4 Gbits/sec    0   1.03 MBytes
[&nbs= p; 5]  29.00-30.00  sec  1.65 GBytes  14.2 Gbits/sec&nb= sp;   0   1.51 MBytes
- - - - - - - - - -= - - - - - - - - - - - - - - -
[ ID] Interval  &nbs= p;        Transfer   &nbs= p; Bitrate         Retr
[  5]   0.00-30.00  sec  39.2 GBytes  11.2= Gbits/sec  3587         =     sender
[  5]   0.00-30.00&n= bsp; sec  39.2 GBytes  11.2 Gbits/sec    &nbs= p;             = receiver


[root@db-02 ~]# iperf3= -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
=
[  5] local 192.168.10.31 port 60931 connected to 192.168.1= 0.30 port 5201
[ ID] Interval     &n= bsp;     Transfer     Bitrate =         Retr  Cwnd
[&= nbsp; 5]   0.00-1.00   sec   931 MBytes = 7.81 Gbits/sec  146    689 KBytes
[ = ; 5]   1.00-2.00   sec   674 MBytes  5.6= 6 Gbits/sec  267    575 KBytes
[  5]=    2.00-3.00   sec   641 MBytes  5.38 Gb= its/sec  270    485 KBytes
[  5]&nbs= p;  3.00-4.00   sec   668 MBytes  5.60 Gbits/= sec  221    801 KBytes
[  5] &n= bsp; 4.00-5.01   sec   532 MBytes  4.40 Gbits/sec&= nbsp; 208    981 KBytes
[  5]  = 5.01-6.00   sec   838 MBytes  7.13 Gbits/sec = ; 291    949 KBytes
[  5]   6.0= 0-7.00   sec   703 MBytes  5.90 Gbits/sec  17= 8    628 KBytes
[  5]   7.00-8.= 00   sec   696 MBytes  5.84 Gbits/sec   = 85    928 KBytes
[  5]   8.00-9= .00   sec  1.06 GBytes  9.14 Gbits/sec   66&n= bsp;   565 KBytes
[  5]   9.00-10.00=   sec   975 MBytes  8.18 Gbits/sec    0&= nbsp;   801 KBytes
[  5]  10.00-11.00&nbs= p; sec   679 MBytes  5.70 Gbits/sec    0 = ;   859 KBytes
[  5]  11.00-12.01  s= ec   628 MBytes  5.23 Gbits/sec  123   1.12 M= Bytes
[  5]  12.01-13.00  sec   731 = MBytes  6.16 Gbits/sec  311    864 KBytes
[  5]  13.00-14.00  sec   696 MBytes  5= .86 Gbits/sec   39    798 KBytes
[&n= bsp; 5]  14.00-15.00  sec   914 MBytes  7.66 Gbits= /sec   65   1024 KBytes
[  5]  = 15.00-16.00  sec  1.66 GBytes  14.2 Gbits/sec  &nb= sp; 0   1.47 MBytes
[  5]  16.00-17.00&nb= sp; sec   955 MBytes  8.01 Gbits/sec  222  &n= bsp; 260 KBytes
[  5]  17.00-18.00  sec &= nbsp; 769 MBytes  6.45 Gbits/sec    0   = 520 KBytes
[  5]  18.00-19.00  sec  = ; 781 MBytes  6.55 Gbits/sec  119    432 KBytes
[  5]  19.00-20.00  sec   779 MBytes&n= bsp; 6.54 Gbits/sec  148    816 KBytes
[&= nbsp; 5]  20.00-21.00  sec   771 MBytes  6.47 Gbit= s/sec  127    965 KBytes
[  5] = 21.00-22.00  sec   745 MBytes  6.25 Gbits/sec  33= 7    927 KBytes
[  5]  22.00-23.00&n= bsp; sec   637 MBytes  5.35 Gbits/sec    0&nb= sp;   938 KBytes
[  5]  23.00-24.00 = sec   626 MBytes  5.25 Gbits/sec    0 &= nbsp;  938 KBytes
[  5]  24.00-25.00  sec=   1.96 GBytes  16.8 Gbits/sec  185    862 KBy= tes
[  5]  25.00-26.00  sec  1.71 GBytes&= nbsp; 14.7 Gbits/sec   22    945 KBytes
<= div>[  5]  26.00-27.00  sec  2.14 GBytes  18.3 Gbi= ts/sec  365    593 KBytes
[  5] = ; 27.00-28.00  sec  1.56 GBytes  13.4 Gbits/sec   = 33   1.23 MBytes
[  5]  28.00-29.00 = sec  1.67 GBytes  14.4 Gbits/sec    0  = 1.61 MBytes
[  5]  29.00-30.00  sec  1.6= 5 GBytes  14.2 Gbits/sec   35   1.14 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ I= D] Interval           Tra= nsfer     Bitrate      &n= bsp;  Retr
[  5]   0.00-30.00  sec&n= bsp; 29.4 GBytes  8.42 Gbits/sec  3863    &nb= sp;        sender
[  = 5]   0.00-30.00  sec  29.4 GBytes  8.42 Gbits/sec&= nbsp;           &nbs= p;     receiver


=



---- On Tue, 14 Jun 202= 2 10:21:51 -0400 Mike Jakubik <mike.jakubik@swiftsmsgateway.com> wrote ----

Disabling rx/tx pause seems to produce higher peaks.
<= /div>

[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
Connecting to host db-01, port 5201
[  5] local = 192.168.10.31 port 10146 connected to 192.168.10.30 port 5201
[ ID] Interval          = Transfer     Bitrate     &nbs= p;   Retr  Cwnd
[  5]   0.00-1.= 00   sec  1.89 GBytes  16.2 Gbits/sec   = 0   1.10 MBytes
[  5]   1.00-2.00&n= bsp;  sec  1.86 GBytes  15.9 Gbits/sec    0&n= bsp;  1.10 MBytes
[  5]   2.00-3.00 =   sec  2.05 GBytes  17.6 Gbits/sec    0 =   1.11 MBytes
[  5]   3.00-4.00 &nbs= p; sec   859 MBytes  7.20 Gbits/sec   21 &nbs= p;  938 KBytes
[  5]   4.00-5.00 &nb= sp; sec   652 MBytes  5.47 Gbits/sec    0&nbs= p;  1.01 MBytes
[  5]   5.00-6.00 &n= bsp; sec   659 MBytes  5.53 Gbits/sec    0&nb= sp;  1.03 MBytes
[  5]   6.00-7.00 &= nbsp; sec   666 MBytes  5.59 Gbits/sec    0&n= bsp;  1.05 MBytes
[  5]   7.00-8.00 =   sec   657 MBytes  5.51 Gbits/sec   98 =    989 KBytes
[  5]   8.00-9.00 = ;  sec   665 MBytes  5.58 Gbits/sec  139 &nbs= p;  712 KBytes
[  5]   9.00-10.00  s= ec   647 MBytes  5.43 Gbits/sec    0 &nb= sp; 1.02 MBytes
[  5]  10.00-11.00  sec &= nbsp; 650 MBytes  5.45 Gbits/sec    4   = 606 KBytes
[  5]  11.00-12.00  sec  1.53= GBytes  13.1 Gbits/sec  358   1.07 MBytes
[  5]  12.00-13.00  sec  2.10 GBytes  18.1 Gbits= /sec  162    837 KBytes
[  5]  = 13.00-14.00  sec  2.09 GBytes  18.0 Gbits/sec  332 = ;   838 KBytes
[  5]  14.00-15.00  s= ec  2.43 GBytes  20.9 Gbits/sec  639    747 K= Bytes
[  5]  15.00-16.00  sec  2.38 GByte= s  20.4 Gbits/sec  612   1.02 MBytes
[&nb= sp; 5]  16.00-17.00  sec  2.25 GBytes  19.3 Gbits/sec&n= bsp; 535   1.24 MBytes
[  5]  17.00-18.00=   sec  2.52 GBytes  21.6 Gbits/sec  818  &nbs= p; 423 KBytes
[  5]  18.00-19.00  sec  2.= 29 GBytes  19.7 Gbits/sec  218    444 KBytes
[  5]  19.00-20.00  sec  2.29 GBytes  19.= 7 Gbits/sec  114    859 KBytes
[  5]=   20.00-21.00  sec  1.65 GBytes  14.1 Gbits/sec  1= 00    541 KBytes
[  5]  21.00-22.00&= nbsp; sec  1.01 GBytes  8.67 Gbits/sec    0 &= nbsp;  639 KBytes
[  5]  22.00-23.00  sec=    625 MBytes  5.24 Gbits/sec    0  = ;  648 KBytes
[  5]  23.00-24.00  sec&nbs= p;  630 MBytes  5.28 Gbits/sec    0  &nb= sp; 648 KBytes
[  5]  24.00-25.00  sec  1= .56 GBytes  13.4 Gbits/sec    0    702 K= Bytes
[  5]  25.00-26.00  sec  1.78 GByte= s  15.3 Gbits/sec  118    406 KBytes
[  5]  26.00-27.00  sec  1.37 GBytes  11.8 Gbits/= sec  105    890 KBytes
[  5]  2= 7.00-28.00  sec  1.82 GBytes  15.6 Gbits/sec  104 =    963 KBytes
[  5]  28.00-29.00  se= c  1.68 GBytes  14.4 Gbits/sec    0   1.= 20 MBytes
[  5]  29.00-30.00  sec  1.67 G= Bytes  14.4 Gbits/sec    0   1.38 MBytes
<= /div>
- - - - - - - - - - - - - - - - - - - - - - - - -
= [ ID] Interval           = Transfer     Bitrate      = ;   Retr
[  5]   0.00-30.00  se= c  44.8 GBytes  12.8 Gbits/sec  4477    =          sender
[&nbs= p; 5]   0.00-30.01  sec  44.8 GBytes  12.8 Gbits/s= ec            &= nbsp;     receiver

After a= few runs:

[root@db-02 ~]# iperf3 -i 1 -t 30 -= c db-01
Connecting to host db-01, port 5201
[&= nbsp; 5] local 192.168.10.31 port 52152 connected to 192.168.10.30 port 520= 1
[ ID] Interval       &nb= sp;   Transfer     Bitrate   &= nbsp;     Retr  Cwnd
[  5] = ;  0.00-1.00   sec  1.91 GBytes  16.4 Gbits/sec&nb= sp;  67    606 KBytes
[  5] &nb= sp; 1.00-2.00   sec  1.78 GBytes  15.3 Gbits/sec &= nbsp;  0   1.07 MBytes
[  5]   = 2.00-3.00   sec  1.60 GBytes  13.7 Gbits/sec  = ;  0   1.54 MBytes
[  5]   3.00= -4.00   sec  1.61 GBytes  13.8 Gbits/sec  &nb= sp; 0   1.61 MBytes
[  5]   4.00-5.0= 0   sec  1.66 GBytes  14.3 Gbits/sec    = 0   1.61 MBytes
[  5]   5.00-6.00&nb= sp;  sec  1.67 GBytes  14.3 Gbits/sec    0&nb= sp;  1.61 MBytes
[  5]   6.00-7.00 &= nbsp; sec  1.65 GBytes  14.1 Gbits/sec    0 &= nbsp; 1.61 MBytes
[  5]   7.00-8.00  = ; sec  1.70 GBytes  14.6 Gbits/sec    0  = ; 1.61 MBytes
[  5]   8.00-9.00   se= c  1.72 GBytes  14.8 Gbits/sec    0   1.= 61 MBytes
[  5]   9.00-10.00  sec  1= .85 GBytes  15.9 Gbits/sec    0   1.61 MBytes=
[  5]  10.00-11.00  sec  1.81 GBytes&nbs= p; 15.5 Gbits/sec    0   1.61 MBytes
[  5]  11.00-12.00  sec  1.67 GBytes  14.3 Gbits/= sec    0   1.61 MBytes
[  5]&nb= sp; 12.00-13.00  sec  1.66 GBytes  14.3 Gbits/sec  = ;  0   1.61 MBytes
[  5]  13.00-14.0= 0  sec  1.83 GBytes  15.7 Gbits/sec    0 = ;  1.61 MBytes
[  5]  14.00-15.00  sec&nb= sp; 1.18 GBytes  10.1 Gbits/sec    0    = 794 KBytes
[  5]  15.00-16.00  sec  1.67 = GBytes  14.4 Gbits/sec    0   1.60 MBytes
=
[  5]  16.00-17.00  sec  1.73 GBytes  1= 4.8 Gbits/sec    0   1.60 MBytes
[&n= bsp; 5]  17.00-18.00  sec  1.73 GBytes  14.9 Gbits/sec&= nbsp;   0   1.60 MBytes
[  5]  = 18.00-19.00  sec  1.83 GBytes  15.7 Gbits/sec  &nb= sp; 0   1.61 MBytes
[  5]  19.00-20.00&nb= sp; sec  1.77 GBytes  15.2 Gbits/sec    0 &nb= sp; 1.61 MBytes
[  5]  20.00-21.00  sec  = 1.80 GBytes  15.5 Gbits/sec    0   1.61 MByte= s
[  5]  21.00-22.00  sec  2.03 GBytes&nb= sp; 17.4 Gbits/sec    0   1.61 MBytes
[  5]  22.00-23.00  sec  1.88 GBytes  16.1 Gbits= /sec    0   1.61 MBytes
[  5]&n= bsp; 23.00-24.00  sec  1.80 GBytes  15.5 Gbits/sec &nbs= p;  0   1.61 MBytes
[  5]  24.00-25.= 01  sec  1.59 GBytes  13.6 Gbits/sec    0&nbs= p;  1.61 MBytes
[  5]  25.01-26.00  sec&n= bsp; 1.33 GBytes  11.4 Gbits/sec    0   1.61 = MBytes
[  5]  26.00-27.00  sec  1.71 GByt= es  14.7 Gbits/sec    0   1.61 MBytes
[  5]  27.00-28.00  sec  1.71 GBytes  14.7 = Gbits/sec   97   1.01 MBytes
[  5]&n= bsp; 28.00-29.00  sec   719 MBytes  6.03 Gbits/sec = ;   0   1.01 MBytes
[  5]  29.0= 0-30.00  sec   727 MBytes  6.10 Gbits/sec  &n= bsp; 0   1.01 MBytes
- - - - - - - - - - - - - - - = - - - - - - - - - -
[ ID] Interval    &nb= sp;      Transfer     Bitrate&= nbsp;        Retr
[  = 5]   0.00-30.00  sec  49.3 GBytes  14.1 Gbits/sec&= nbsp; 164           =   sender
[  5]   0.00-30.00  sec&nbs= p; 49.3 GBytes  14.1 Gbits/sec      &nbs= p;           receiver
=



The CPU usage is ra= ther low, the NIC is in the OCP port, so im sure thats designed accordingly= and the NIC is bound to numa0.

CPU:  0.0= % user,  0.0% nice,  0.5% system,  0.7% interrupt, 98.8% idl= e

  PID USERNAME    THR PR= I NICE   SIZE    RES STATE    C&nbs= p;  TIME    WCPU COMMAND
2195 root =          1  52  &nbs= p; 0    17M  6884K select  83   0:14&nbs= p; 27.99% iperf3

# vmstat -i -w1|grep mlx5
=
irq671: mlx5_core0       &nbs= p;         49969   &= nbsp;  47008

(this drops to about 14k wit= h HW LRO enabled)

The dump is rather large, i = dont think i can attach it in the mailing list, but if you wish to see it i= can upload it somewhere.

Thank You.
=




---- On Mon, 13 Jun 2022 16:42:59 -0400 Hans Petter Selasky <hps@selasky.org>= wrote ----


Some ideas:

Try to disable "rxpause,txpause" when setting t= he media.

Keep HW LRO off for now, it doesn't work for large numbe= r of connections.

What is the CPU usage during test? Is iperf3 run= ning on a CPU-core which
has direct access to the NIC's numa domain?
Is the NIC installed in the "correct" PCI high-performance slot?
There are some sysctl knobs which may tell where the problem is, if i= t's
PCI backpressure or something else.

sysctl -a | grep diag_= pci_enable
sysctl -a | grep diag_general_enable

Set these two = to 1, then run some traffic and dump all mce sysctls:

sysctl -a | = grep mce > dump.txt

--HPS


Mike Jakubik



Mike Jakubik



<= /body> ------=_Part_4650864_1970601159.1655217366007--