From nobody Wed Aug 24 23:22:51 2022 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MChwR38hnz4b6HG for ; Wed, 24 Aug 2022 23:23:03 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) Received: from sender4-op-o13.zoho.com (sender4-op-o13.zoho.com [136.143.188.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4MChwQ28ZJz3j9Q; Wed, 24 Aug 2022 23:23:02 +0000 (UTC) (envelope-from michal.jakubik@zoho.com) ARC-Seal: i=1; a=rsa-sha256; t=1661383373; cv=none; d=zohomail.com; s=zohoarc; b=e3fiiAlAcNWEa7Hv1R89anFzoxWRMmF7VzjNTkGPIwJ5E0YjZZgwmiTnyYdHf2IDXd+u1uj7d5lsYMUnYjp415AHwcxd0ftT2UqSRBI1Mos8uCBybWepMhWp/MgZTxdjxwR0a0fCm+Bq+lskMrN6mYSc+kg0Xg7iISztF0FrmA0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1661383373; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:Reply-To:References:Subject:To; bh=Bq1PAL4ZJzjrnp/jtJlV/E0SZxWVxW4SpZa/gHIrLiI=; b=TkCGZfyl3bmbKZ56pMRXH0pKIEM0hloA05KZr6KV13M8AO3ojb5TxuIK/l7NU//GXZIK4bCrsZV4dMUK2c5v6L5EMmBgzSl2myBNAHT7m3MJ3hHJdsQiSOBuJ0tdI8qSOndok13H7w317jOQEwYwXfY0ziJb/uQSVCOSnmXmft4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=swiftsmsgateway.com; spf=pass smtp.mailfrom=michal.jakubik@zoho.com; dmarc=pass header.from= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=date:from:reply-to:to:cc:message-id:in-reply-to:references:subject:mime-version:content-type:user-agent; b=VkazYFyBvgSf3BdF7RZWcIi4VskeZ6N0VULdKLM0eO/mRlVVunRnxqzGZ/F0Xbr5WXmqrq07itgw UYqo5QLEvqAl+MDUWTALRrwPe8LlXr+2tYyE9j3Oo7fTku7GrAup DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1661383373; s=zoho; d=swiftsmsgateway.com; i=mike.jakubik@swiftsmsgateway.com; h=Date:Date:From:From:Reply-To:Reply-To:To:To:Cc:Cc:Message-Id:Message-Id:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type; bh=Bq1PAL4ZJzjrnp/jtJlV/E0SZxWVxW4SpZa/gHIrLiI=; b=qXLySd97Qu6ZAANa9BSI64V60f82wBKOxg+yapcsTYqCMp+sD3k7tMLxiBHat69D VAIMU2hPmZU34VPbTOpPXAbLRNBO+k05MvopUKpeOQCVz9QneV9QkLtyzy441BU8Ljj 7DdAiDwUgxE1Gm4vURwf2Me/6AZO+DVCPs61E+1s= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1661383371917459.93992893508926; Wed, 24 Aug 2022 16:22:51 -0700 (PDT) Date: Wed, 24 Aug 2022 19:22:51 -0400 From: Mike Jakubik Reply-To: mike.jakubik@swiftsmsgateway.com To: "mike tancsa" Cc: "pieper, jeffrey e" , "jim king" , "stable@freebsd.org" , "kbowling@freebsd.org" Message-Id: <182d22a6c6d.1119560c11283607.2998737705092721009@swiftsmsgateway.com> In-Reply-To: <879b9239-2b9a-f0ae-4173-4a226c84cd85@sentex.net> References: <59b9cec0-d8c2-ce72-b5e9-99d1a1e807f8@sentex.net> <86995d10-af63-d053-972e-dd233029f3bf@jimking.net> <3d874f65-8ce2-8f06-f19a-14cd550166e3@sentex.net> <879b9239-2b9a-f0ae-4173-4a226c84cd85@sentex.net> Subject: Re: igc problems with heavy traffic List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_3994682_1854819823.1661383371885" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Rspamd-Queue-Id: 4MChwQ28ZJz3j9Q X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=swiftsmsgateway.com header.s=zoho header.b=qXLySd97; arc=pass ("zohomail.com:s=zohoarc:i=1"); dmarc=none; spf=pass (mx1.freebsd.org: domain of michal.jakubik@zoho.com designates 136.143.188.13 as permitted sender) smtp.mailfrom=michal.jakubik@zoho.com X-Spamd-Result: default: False [-4.06 / 15.00]; ARC_ALLOW(-1.00)[zohomail.com:s=zohoarc:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.996]; NEURAL_HAM_LONG(-0.88)[-0.876]; FORGED_SENDER(0.30)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; R_SPF_ALLOW(-0.20)[+ip4:136.143.188.0/24]; R_DKIM_ALLOW(-0.20)[swiftsmsgateway.com:s=zoho]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; HAS_REPLYTO(0.00)[mike.jakubik@swiftsmsgateway.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[swiftsmsgateway.com]; RCVD_IN_DNSWL_NONE(0.00)[136.143.188.13:from]; ASN(0.00)[asn:2639, ipnet:136.143.188.0/23, country:US]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_SOME(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[mike.jakubik@swiftsmsgateway.com,michal.jakubik@zoho.com]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[swiftsmsgateway.com:+]; FREEMAIL_ENVFROM(0.00)[zoho.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N ------=_Part_3994682_1854819823.1661383371885 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable What kind of HW are you running on? Im assuming some sort of fairly modern = x86 CPU with at least 4 cores.. Is it multiple CPUs with Numa nodes perhaps= ? In any case, if you are testing with iperf3, try using cpuset on iperf3 t= o bind it to specific cores. I had a performance issue on a modern Epyc ser= ver with a Mellanox 25Gb card. It turns out the issue was with the schedule= r and how it was bouncing the processes around diff cores/CPU caches. See "= Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)" on the free= bsd-net mailing list for details. P.S. I also use a number of igc (Intel i225 @ 2.5Gb) cards at home and have= had no issues with them. ---- On Fri, 12 Aug 2022 11:04:19 -0400 mike tancsa wrote= --- On 8/10/2022 3:53 PM, mike tancsa wrote:=20 > On 8/10/2022 1:47 PM, Pieper, Jeffrey E wrote:=20 >>=20 >> You could try disabling EEE (Energy Efficient Ethernet). Something=20 >> like: sysctl dev.igc.0.eee_control=3D0.=20 >=20 >=20 > It does not seem to make a difference. If I have the FC as default, I=20 > get the link bounce on the 2.5G xover (cat 6 cable) maybe 2-3 min in=20 > running iper3 tests.=C2=A0 However, if I disable all flow control=20 >=20 > dev.igc.0.fc=3D0=20 > dev.igc.1.fc=3D0=20 > dev.igc.2.fc=3D0=20 > dev.igc.3.fc=3D0=20 >=20 > It *seems* to be less frequent but still happens.=C2=A0 I ordered a 2.5 G= =20 > switch so I can try and at least see which side is dropping the link.=20 > Should have it Friday to continue testing=20 >=20 =20 OK, I repeated the tests with a 2.5G unmanaged switch in between the two=20 units rather than xover. It looks like its the server that is sending=20 the majority of the packets that drops the link, not the receiver.=20 =20 One other test I did was to up hw.igc.max_interrupt_rate=3D13000 from the= =20 default of 8000. That seems to make the problem MUCH more acute.=20 =20 Here is the before and after of the link drop.=20 =20 =C2=A0dev.igc.1.wake: 0=20 =C2=A0dev.igc.1.interrupts.rx_desc_min_thresh: 0=20 -dev.igc.1.interrupts.asserts: 65=20 +dev.igc.1.interrupts.asserts: 4879479=20 =C2=A0dev.igc.1.mac_stats.tso_txd: 0=20 -dev.igc.1.mac_stats.tx_frames_1024_1522: 3=20 -dev.igc.1.mac_stats.tx_frames_512_1023: 1=20 -dev.igc.1.mac_stats.tx_frames_256_511: 2=20 -dev.igc.1.mac_stats.tx_frames_128_255: 15=20 -dev.igc.1.mac_stats.tx_frames_65_127: 2=20 +dev.igc.1.mac_stats.tx_frames_1024_1522: 12973065=20 +dev.igc.1.mac_stats.tx_frames_512_1023: 58=20 +dev.igc.1.mac_stats.tx_frames_256_511: 107=20 +dev.igc.1.mac_stats.tx_frames_128_255: 1215725=20 +dev.igc.1.mac_stats.tx_frames_65_127: 192=20 =C2=A0dev.igc.1.mac_stats.tx_frames_64: 1=20 =C2=A0dev.igc.1.mac_stats.mcast_pkts_txd: 0=20 =C2=A0dev.igc.1.mac_stats.bcast_pkts_txd: 1=20 -dev.igc.1.mac_stats.good_pkts_txd: 24=20 -dev.igc.1.mac_stats.total_pkts_txd: 24=20 -dev.igc.1.mac_stats.good_octets_txd: 7674=20 -dev.igc.1.mac_stats.good_octets_recvd: 6492=20 -dev.igc.1.mac_stats.rx_frames_1024_1522: 2=20 -dev.igc.1.mac_stats.rx_frames_512_1023: 1=20 -dev.igc.1.mac_stats.rx_frames_256_511: 2=20 -dev.igc.1.mac_stats.rx_frames_128_255: 15=20 -dev.igc.1.mac_stats.rx_frames_65_127: 2=20 +dev.igc.1.mac_stats.good_pkts_txd: 14189148=20 +dev.igc.1.mac_stats.total_pkts_txd: 14189148=20 +dev.igc.1.mac_stats.good_octets_txd: 19450753554=20 +dev.igc.1.mac_stats.good_octets_recvd: 14933399426=20 +dev.igc.1.mac_stats.rx_frames_1024_1522: 9823228=20 +dev.igc.1.mac_stats.rx_frames_512_1023: 3=20 +dev.igc.1.mac_stats.rx_frames_256_511: 62=20 +dev.igc.1.mac_stats.rx_frames_128_255: 2365665=20 +dev.igc.1.mac_stats.rx_frames_65_127: 213=20 =C2=A0dev.igc.1.mac_stats.rx_frames_64: 1=20 =C2=A0dev.igc.1.mac_stats.mcast_pkts_recvd: 0=20 =C2=A0dev.igc.1.mac_stats.bcast_pkts_recvd: 0=20 -dev.igc.1.mac_stats.good_pkts_recvd: 23=20 -dev.igc.1.mac_stats.total_pkts_recvd: 23=20 +dev.igc.1.mac_stats.good_pkts_recvd: 12189172=20 +dev.igc.1.mac_stats.total_pkts_recvd: 12189172=20 =C2=A0dev.igc.1.mac_stats.xoff_txd: 0=20 =C2=A0dev.igc.1.mac_stats.xoff_recvd: 0=20 =C2=A0dev.igc.1.mac_stats.xon_txd: 0=20 =C2=A0dev.igc.1.mac_stats.single_coll: 0=20 =C2=A0dev.igc.1.mac_stats.excess_coll: 0=20 =C2=A0dev.igc.1.queue_rx_3.rx_irq: 0=20 -dev.igc.1.queue_rx_3.rxd_tail: 21=20 -dev.igc.1.queue_rx_3.rxd_head: 22=20 +dev.igc.1.queue_rx_3.rxd_tail: 498=20 +dev.igc.1.queue_rx_3.rxd_head: 499=20 =C2=A0dev.igc.1.queue_rx_2.rx_irq: 0=20 =C2=A0dev.igc.1.queue_rx_2.rxd_tail: 128=20 =C2=A0dev.igc.1.queue_rx_2.rxd_head: 0=20 =C2=A0dev.igc.1.queue_rx_0.rxd_tail: 0=20 =C2=A0dev.igc.1.queue_rx_0.rxd_head: 1=20 =C2=A0dev.igc.1.queue_tx_3.tx_irq: 0=20 -dev.igc.1.queue_tx_3.txd_tail: 0=20 -dev.igc.1.queue_tx_3.txd_head: 0=20 +dev.igc.1.queue_tx_3.txd_tail: 746=20 +dev.igc.1.queue_tx_3.txd_head: 746=20 =C2=A0dev.igc.1.queue_tx_2.tx_irq: 0=20 -dev.igc.1.queue_tx_2.txd_tail: 0=20 -dev.igc.1.queue_tx_2.txd_head: 0=20 +dev.igc.1.queue_tx_2.txd_tail: 186=20 +dev.igc.1.queue_tx_2.txd_head: 186=20 =C2=A0dev.igc.1.queue_tx_1.tx_irq: 0=20 -dev.igc.1.queue_tx_1.txd_tail: 0=20 -dev.igc.1.queue_tx_1.txd_head: 0=20 +dev.igc.1.queue_tx_1.txd_tail: 520=20 +dev.igc.1.queue_tx_1.txd_head: 520=20 =C2=A0dev.igc.1.queue_tx_0.tx_irq: 0=20 -dev.igc.1.queue_tx_0.txd_tail: 45=20 -dev.igc.1.queue_tx_0.txd_head: 45=20 +dev.igc.1.queue_tx_0.txd_tail: 777=20 +dev.igc.1.queue_tx_0.txd_head: 777=20 =C2=A0dev.igc.1.fc_low_water: 32752=20 =C2=A0dev.igc.1.fc_high_water: 32768=20 =C2=A0dev.igc.1.rx_control: 71335938=20 =C2=A0dev.igc.1.device_control: 404489793=20 =C2=A0dev.igc.1.watchdog_timeouts: 0=20 =C2=A0dev.igc.1.rx_overruns: 0=20 -dev.igc.1.link_irq: 2=20 +dev.igc.1.link_irq: 4=20 =C2=A0dev.igc.1.dropped: 0=20 =C2=A0dev.igc.1.eee_control: 0=20 =C2=A0dev.igc.1.itr: 488=20 =C2=A0dev.igc.1.nvm: -1=20 =C2=A0dev.igc.1.iflib.rxq3.rxq_fl0.buf_size: 2048=20 =C2=A0dev.igc.1.iflib.rxq3.rxq_fl0.credits: 1023=20 -dev.igc.1.iflib.rxq3.rxq_fl0.cidx: 22=20 -dev.igc.1.iflib.rxq3.rxq_fl0.pidx: 21=20 +dev.igc.1.iflib.rxq3.rxq_fl0.cidx: 499=20 +dev.igc.1.iflib.rxq3.rxq_fl0.pidx: 498=20 =C2=A0dev.igc.1.iflib.rxq3.cpu: 3=20 =C2=A0dev.igc.1.iflib.rxq2.rxq_fl0.buf_size: 2048=20 =C2=A0dev.igc.1.iflib.rxq2.rxq_fl0.credits: 128=20 =C2=A0dev.igc.1.iflib.txq3.r_abdications: 0=20 =C2=A0dev.igc.1.iflib.txq3.r_restarts: 0=20 =C2=A0dev.igc.1.iflib.txq3.r_stalls: 0=20 -dev.igc.1.iflib.txq3.r_starts: 0=20 +dev.igc.1.iflib.txq3.r_starts: 6175093=20 =C2=A0dev.igc.1.iflib.txq3.r_drops: 0=20 -dev.igc.1.iflib.txq3.r_enqueues: 0=20 -dev.igc.1.iflib.txq3.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx:=20 0000 state: IDLE=20 -dev.igc.1.iflib.txq3.txq_cleaned: 0=20 -dev.igc.1.iflib.txq3.txq_processed: 0=20 -dev.igc.1.iflib.txq3.txq_in_use: 0=20 -dev.igc.1.iflib.txq3.txq_cidx_processed: 0=20 -dev.igc.1.iflib.txq3.txq_cidx: 0=20 -dev.igc.1.iflib.txq3.txq_pidx: 0=20 +dev.igc.1.iflib.txq3.r_enqueues: 6175093=20 +dev.igc.1.iflib.txq3.ring_state: pidx_head: 0373 pidx_tail: 0373 cidx:=20 0373 state: IDLE=20 +dev.igc.1.iflib.txq3.txq_cleaned: 12350144=20 +dev.igc.1.iflib.txq3.txq_processed: 12350184=20 +dev.igc.1.iflib.txq3.txq_in_use: 42=20 +dev.igc.1.iflib.txq3.txq_cidx_processed: 744=20 +dev.igc.1.iflib.txq3.txq_cidx: 704=20 +dev.igc.1.iflib.txq3.txq_pidx: 746=20 =C2=A0dev.igc.1.iflib.txq3.no_tx_dma_setup: 0=20 =C2=A0dev.igc.1.iflib.txq3.txd_encap_efbig: 0=20 =C2=A0dev.igc.1.iflib.txq3.tx_map_failed: 0=20 =C2=A0dev.igc.1.iflib.txq2.r_abdications: 0=20 =C2=A0dev.igc.1.iflib.txq2.r_restarts: 0=20 =C2=A0dev.igc.1.iflib.txq2.r_stalls: 0=20 -dev.igc.1.iflib.txq2.r_starts: 0=20 +dev.igc.1.iflib.txq2.r_starts: 3421789=20 =C2=A0dev.igc.1.iflib.txq2.r_drops: 0=20 -dev.igc.1.iflib.txq2.r_enqueues: 0=20 -dev.igc.1.iflib.txq2.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx:=20 0000 state: IDLE=20 -dev.igc.1.iflib.txq2.txq_cleaned: 0=20 -dev.igc.1.iflib.txq2.txq_processed: 0=20 -dev.igc.1.iflib.txq2.txq_in_use: 0=20 -dev.igc.1.iflib.txq2.txq_cidx_processed: 0=20 -dev.igc.1.iflib.txq2.txq_cidx: 0=20 -dev.igc.1.iflib.txq2.txq_pidx: 0=20 +dev.igc.1.iflib.txq2.r_enqueues: 3421789=20 +dev.igc.1.iflib.txq2.ring_state: pidx_head: 1629 pidx_tail: 1629 cidx:=20 1629 state: IDLE=20 +dev.igc.1.iflib.txq2.txq_cleaned: 6843536=20 +dev.igc.1.iflib.txq2.txq_processed: 6843576=20 +dev.igc.1.iflib.txq2.txq_in_use: 42=20 +dev.igc.1.iflib.txq2.txq_cidx_processed: 184=20 +dev.igc.1.iflib.txq2.txq_cidx: 144=20 +dev.igc.1.iflib.txq2.txq_pidx: 186=20 =C2=A0dev.igc.1.iflib.txq2.no_tx_dma_setup: 0=20 =C2=A0dev.igc.1.iflib.txq2.txd_encap_efbig: 0=20 =C2=A0dev.igc.1.iflib.txq2.tx_map_failed: 0=20 =C2=A0dev.igc.1.iflib.txq1.r_abdications: 0=20 =C2=A0dev.igc.1.iflib.txq1.r_restarts: 0=20 =C2=A0dev.igc.1.iflib.txq1.r_stalls: 0=20 -dev.igc.1.iflib.txq1.r_starts: 0=20 +dev.igc.1.iflib.txq1.r_starts: 2734852=20 =C2=A0dev.igc.1.iflib.txq1.r_drops: 0=20 -dev.igc.1.iflib.txq1.r_enqueues: 0=20 -dev.igc.1.iflib.txq1.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx:=20 0000 state: IDLE=20 -dev.igc.1.iflib.txq1.txq_cleaned: 0=20 -dev.igc.1.iflib.txq1.txq_processed: 0=20 -dev.igc.1.iflib.txq1.txq_in_use: 0=20 -dev.igc.1.iflib.txq1.txq_cidx_processed: 0=20 -dev.igc.1.iflib.txq1.txq_cidx: 0=20 -dev.igc.1.iflib.txq1.txq_pidx: 0=20 +dev.igc.1.iflib.txq1.r_enqueues: 2734852=20 +dev.igc.1.iflib.txq1.ring_state: pidx_head: 0772 pidx_tail: 0772 cidx:=20 0772 state: IDLE=20 +dev.igc.1.iflib.txq1.txq_cleaned: 5469662=20 +dev.igc.1.iflib.txq1.txq_processed: 5469702=20 +dev.igc.1.iflib.txq1.txq_in_use: 42=20 +dev.igc.1.iflib.txq1.txq_cidx_processed: 518=20 +dev.igc.1.iflib.txq1.txq_cidx: 478=20 +dev.igc.1.iflib.txq1.txq_pidx: 520=20 =C2=A0dev.igc.1.iflib.txq1.no_tx_dma_setup: 0=20 =C2=A0dev.igc.1.iflib.txq1.txd_encap_efbig: 0=20 =C2=A0dev.igc.1.iflib.txq1.tx_map_failed: 0=20 =C2=A0dev.igc.1.iflib.txq0.r_abdications: 0=20 =C2=A0dev.igc.1.iflib.txq0.r_restarts: 0=20 =C2=A0dev.igc.1.iflib.txq0.r_stalls: 0=20 -dev.igc.1.iflib.txq0.r_starts: 24=20 +dev.igc.1.iflib.txq0.r_starts: 1857414=20 =C2=A0dev.igc.1.iflib.txq0.r_drops: 0=20 -dev.igc.1.iflib.txq0.r_enqueues: 24=20 -dev.igc.1.iflib.txq0.ring_state: pidx_head: 0024 pidx_tail: 0024 cidx:=20 0024 state: IDLE=20 -dev.igc.1.iflib.txq0.txq_cleaned: 3=20 -dev.igc.1.iflib.txq0.txq_processed: 43=20 +dev.igc.1.iflib.txq0.r_enqueues: 1857414=20 +dev.igc.1.iflib.txq0.ring_state: pidx_head: 1926 pidx_tail: 1926 cidx:=20 1926 state: IDLE=20 +dev.igc.1.iflib.txq0.txq_cleaned: 3714783=20 +dev.igc.1.iflib.txq0.txq_processed: 3714823=20 =C2=A0dev.igc.1.iflib.txq0.txq_in_use: 42=20 -dev.igc.1.iflib.txq0.txq_cidx_processed: 43=20 -dev.igc.1.iflib.txq0.txq_cidx: 3=20 -dev.igc.1.iflib.txq0.txq_pidx: 45=20 +dev.igc.1.iflib.txq0.txq_cidx_processed: 775=20 +dev.igc.1.iflib.txq0.txq_cidx: 735=20 +dev.igc.1.iflib.txq0.txq_pidx: 777=20 =C2=A0dev.igc.1.iflib.txq0.no_tx_dma_setup: 0=20 =C2=A0dev.igc.1.iflib.txq0.txd_encap_efbig: 0=20 =C2=A0dev.igc.1.iflib.txq0.tx_map_failed: 0=20 =C2=A0dev.igc.1.%desc: Intel(R) Ethernet Controller I225-V=20 =20 Interface is RUNNING and ACTIVE=20 igc1: TX Queue 0 ------=20 igc1: hw tdh =3D 777, hw tdt =3D 777=20 igc1: TX Queue 1 ------=20 igc1: hw tdh =3D 520, hw tdt =3D 520=20 igc1: TX Queue 2 ------=20 igc1: hw tdh =3D 186, hw tdt =3D 186=20 igc1: TX Queue 3 ------=20 igc1: hw tdh =3D 746, hw tdt =3D 746=20 igc1: RX Queue 0 ------=20 igc1: hw rdh =3D 1, hw rdt =3D 0=20 igc1: RX Queue 1 ------=20 igc1: hw rdh =3D 0, hw rdt =3D 128=20 igc1: RX Queue 2 ------=20 igc1: hw rdh =3D 0, hw rdt =3D 128=20 igc1: RX Queue 3 ------=20 igc1: hw rdh =3D 499, hw rdt =3D 498=20 =20 =20 =20 =20 Mike Jakubik https://www.swiftsmsgateway.com/ Disclaimer: This e-mail and any attachments are intended only for the use o= f the addressee(s) and may contain information that is privileged or confid= ential. If you are not the intended recipient, or responsible for deliverin= g the information to the intended recipient, you are hereby notified that a= ny dissemination, distribution, printing or copying of this e-mail and any = attachments is strictly prohibited. If this e-mail and any attachments were= received in error, please notify the sender by reply e-mail and delete the= original message. ------=_Part_3994682_1854819823.1661383371885 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
What kind of HW are you running on? Im assuming so= me sort of fairly modern x86 CPU with at least 4 cores.. Is it multiple CPU= s with Numa nodes perhaps? In any case, if you are testing with iperf3, try= using cpuset on iperf3 to bind it to specific cores. I had a performance i= ssue on a modern Epyc server with a Mellanox 25Gb card. It turns out the is= sue was with the scheduler and how it was bouncing the processes around dif= f cores/CPU caches. See "Poor performance with stable/13 and Mellanox Conne= ctX-6 (mlx5)" on the freebsd-net mailing list for details.
P.S. I also use a number of igc (Intel i225 @ 2.5Gb) cards at = home and have had no issues with them.



---- On Fri, 12 Aug 2022 11:04:19 -0400 mike = tancsa <mike@sentex.net> wrote ---


On 8/10/2022 3:53 PM, mike tancsa wro= te:
> On 8/10/2022 1:47 PM, Pieper, Jeffrey E wrote:
>> >> You could try disabling EEE (Energy Efficient Ethernet). Somethi= ng
>> like: sysctl dev.igc.0.eee_control=3D0.
>
> <= br>> It does not seem to make a difference. If I have the FC as default,= I
> get the link bounce on the 2.5G xover (cat 6 cable) maybe 2-3 m= in in
> running iper3 tests.  However, if I disable all flow co= ntrol
>
> dev.igc.0.fc=3D0
> dev.igc.1.fc=3D0
>= dev.igc.2.fc=3D0
> dev.igc.3.fc=3D0
>
> It *seems* to= be less frequent but still happens.  I ordered a 2.5 G
> switc= h so I can try and at least see which side is dropping the link.
> S= hould have it Friday to continue testing
>

OK, I repeated t= he tests with a 2.5G unmanaged switch in between the two
units rather t= han xover. It looks like its the server that is sending
the majority of= the packets that drops the link, not the receiver.

One other test= I did was to up hw.igc.max_interrupt_rate=3D13000 from the
default of = 8000. That seems to make the problem MUCH more acute.

Here is the = before and after of the link drop.

 dev.igc.1.wake: 0
&= nbsp;dev.igc.1.interrupts.rx_desc_min_thresh: 0
-dev.igc.1.interrupts.a= sserts: 65
+dev.igc.1.interrupts.asserts: 4879479
 dev.igc.1.= mac_stats.tso_txd: 0
-dev.igc.1.mac_stats.tx_frames_1024_1522: 3
-d= ev.igc.1.mac_stats.tx_frames_512_1023: 1
-dev.igc.1.mac_stats.tx_frames= _256_511: 2
-dev.igc.1.mac_stats.tx_frames_128_255: 15
-dev.igc.1.m= ac_stats.tx_frames_65_127: 2
+dev.igc.1.mac_stats.tx_frames_1024_1522: = 12973065
+dev.igc.1.mac_stats.tx_frames_512_1023: 58
+dev.igc.1.mac= _stats.tx_frames_256_511: 107
+dev.igc.1.mac_stats.tx_frames_128_255: 1= 215725
+dev.igc.1.mac_stats.tx_frames_65_127: 192
 dev.igc.1.= mac_stats.tx_frames_64: 1
 dev.igc.1.mac_stats.mcast_pkts_txd: 0 =
 dev.igc.1.mac_stats.bcast_pkts_txd: 1
-dev.igc.1.mac_stats.g= ood_pkts_txd: 24
-dev.igc.1.mac_stats.total_pkts_txd: 24
-dev.igc.1= .mac_stats.good_octets_txd: 7674
-dev.igc.1.mac_stats.good_octets_recvd= : 6492
-dev.igc.1.mac_stats.rx_frames_1024_1522: 2
-dev.igc.1.mac_s= tats.rx_frames_512_1023: 1
-dev.igc.1.mac_stats.rx_frames_256_511: 2 -dev.igc.1.mac_stats.rx_frames_128_255: 15
-dev.igc.1.mac_stats.rx_fr= ames_65_127: 2
+dev.igc.1.mac_stats.good_pkts_txd: 14189148
+dev.ig= c.1.mac_stats.total_pkts_txd: 14189148
+dev.igc.1.mac_stats.good_octets= _txd: 19450753554
+dev.igc.1.mac_stats.good_octets_recvd: 14933399426 <= br>+dev.igc.1.mac_stats.rx_frames_1024_1522: 9823228
+dev.igc.1.mac_sta= ts.rx_frames_512_1023: 3
+dev.igc.1.mac_stats.rx_frames_256_511: 62 +dev.igc.1.mac_stats.rx_frames_128_255: 2365665
+dev.igc.1.mac_stats.r= x_frames_65_127: 213
 dev.igc.1.mac_stats.rx_frames_64: 1
&n= bsp;dev.igc.1.mac_stats.mcast_pkts_recvd: 0
 dev.igc.1.mac_stats.= bcast_pkts_recvd: 0
-dev.igc.1.mac_stats.good_pkts_recvd: 23
-dev.i= gc.1.mac_stats.total_pkts_recvd: 23
+dev.igc.1.mac_stats.good_pkts_recv= d: 12189172
+dev.igc.1.mac_stats.total_pkts_recvd: 12189172
 = dev.igc.1.mac_stats.xoff_txd: 0
 dev.igc.1.mac_stats.xoff_recvd: = 0
 dev.igc.1.mac_stats.xon_txd: 0
 dev.igc.1.mac_stats.= single_coll: 0
 dev.igc.1.mac_stats.excess_coll: 0
 dev= .igc.1.queue_rx_3.rx_irq: 0
-dev.igc.1.queue_rx_3.rxd_tail: 21
-dev= .igc.1.queue_rx_3.rxd_head: 22
+dev.igc.1.queue_rx_3.rxd_tail: 498
= +dev.igc.1.queue_rx_3.rxd_head: 499
 dev.igc.1.queue_rx_2.rx_irq:= 0
 dev.igc.1.queue_rx_2.rxd_tail: 128
 dev.igc.1.queue= _rx_2.rxd_head: 0
 dev.igc.1.queue_rx_0.rxd_tail: 0
 de= v.igc.1.queue_rx_0.rxd_head: 1
 dev.igc.1.queue_tx_3.tx_irq: 0 -dev.igc.1.queue_tx_3.txd_tail: 0
-dev.igc.1.queue_tx_3.txd_head: 0 <= br>+dev.igc.1.queue_tx_3.txd_tail: 746
+dev.igc.1.queue_tx_3.txd_head: = 746
 dev.igc.1.queue_tx_2.tx_irq: 0
-dev.igc.1.queue_tx_2.txd= _tail: 0
-dev.igc.1.queue_tx_2.txd_head: 0
+dev.igc.1.queue_tx_2.tx= d_tail: 186
+dev.igc.1.queue_tx_2.txd_head: 186
 dev.igc.1.qu= eue_tx_1.tx_irq: 0
-dev.igc.1.queue_tx_1.txd_tail: 0
-dev.igc.1.que= ue_tx_1.txd_head: 0
+dev.igc.1.queue_tx_1.txd_tail: 520
+dev.igc.1.= queue_tx_1.txd_head: 520
 dev.igc.1.queue_tx_0.tx_irq: 0
-dev= .igc.1.queue_tx_0.txd_tail: 45
-dev.igc.1.queue_tx_0.txd_head: 45
+= dev.igc.1.queue_tx_0.txd_tail: 777
+dev.igc.1.queue_tx_0.txd_head: 777 =
 dev.igc.1.fc_low_water: 32752
 dev.igc.1.fc_high_water= : 32768
 dev.igc.1.rx_control: 71335938
 dev.igc.1.devi= ce_control: 404489793
 dev.igc.1.watchdog_timeouts: 0
 = dev.igc.1.rx_overruns: 0
-dev.igc.1.link_irq: 2
+dev.igc.1.link_irq= : 4
 dev.igc.1.dropped: 0
 dev.igc.1.eee_control: 0  dev.igc.1.itr: 488
 dev.igc.1.nvm: -1
 dev.igc.= 1.iflib.rxq3.rxq_fl0.buf_size: 2048
 dev.igc.1.iflib.rxq3.rxq_fl0= .credits: 1023
-dev.igc.1.iflib.rxq3.rxq_fl0.cidx: 22
-dev.igc.1.if= lib.rxq3.rxq_fl0.pidx: 21
+dev.igc.1.iflib.rxq3.rxq_fl0.cidx: 499
+= dev.igc.1.iflib.rxq3.rxq_fl0.pidx: 498
 dev.igc.1.iflib.rxq3.cpu:= 3
 dev.igc.1.iflib.rxq2.rxq_fl0.buf_size: 2048
 dev.ig= c.1.iflib.rxq2.rxq_fl0.credits: 128
 dev.igc.1.iflib.txq3.r_abdic= ations: 0
 dev.igc.1.iflib.txq3.r_restarts: 0
 dev.igc.= 1.iflib.txq3.r_stalls: 0
-dev.igc.1.iflib.txq3.r_starts: 0
+dev.igc= .1.iflib.txq3.r_starts: 6175093
 dev.igc.1.iflib.txq3.r_drops: 0 =
-dev.igc.1.iflib.txq3.r_enqueues: 0
-dev.igc.1.iflib.txq3.ring_stat= e: pidx_head: 0000 pidx_tail: 0000 cidx:
0000 state: IDLE
-dev.igc.= 1.iflib.txq3.txq_cleaned: 0
-dev.igc.1.iflib.txq3.txq_processed: 0
= -dev.igc.1.iflib.txq3.txq_in_use: 0
-dev.igc.1.iflib.txq3.txq_cidx_proc= essed: 0
-dev.igc.1.iflib.txq3.txq_cidx: 0
-dev.igc.1.iflib.txq3.tx= q_pidx: 0
+dev.igc.1.iflib.txq3.r_enqueues: 6175093
+dev.igc.1.ifli= b.txq3.ring_state: pidx_head: 0373 pidx_tail: 0373 cidx:
0373 state: ID= LE
+dev.igc.1.iflib.txq3.txq_cleaned: 12350144
+dev.igc.1.iflib.txq= 3.txq_processed: 12350184
+dev.igc.1.iflib.txq3.txq_in_use: 42
+dev= .igc.1.iflib.txq3.txq_cidx_processed: 744
+dev.igc.1.iflib.txq3.txq_cid= x: 704
+dev.igc.1.iflib.txq3.txq_pidx: 746
 dev.igc.1.iflib.t= xq3.no_tx_dma_setup: 0
 dev.igc.1.iflib.txq3.txd_encap_efbig: 0 <= br>  dev.igc.1.iflib.txq3.tx_map_failed: 0
 dev.igc.1.iflib.= txq2.r_abdications: 0
 dev.igc.1.iflib.txq2.r_restarts: 0
&n= bsp;dev.igc.1.iflib.txq2.r_stalls: 0
-dev.igc.1.iflib.txq2.r_starts: 0 =
+dev.igc.1.iflib.txq2.r_starts: 3421789
 dev.igc.1.iflib.txq2= .r_drops: 0
-dev.igc.1.iflib.txq2.r_enqueues: 0
-dev.igc.1.iflib.tx= q2.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx:
0000 state: IDLE <= br>-dev.igc.1.iflib.txq2.txq_cleaned: 0
-dev.igc.1.iflib.txq2.txq_proce= ssed: 0
-dev.igc.1.iflib.txq2.txq_in_use: 0
-dev.igc.1.iflib.txq2.t= xq_cidx_processed: 0
-dev.igc.1.iflib.txq2.txq_cidx: 0
-dev.igc.1.i= flib.txq2.txq_pidx: 0
+dev.igc.1.iflib.txq2.r_enqueues: 3421789
+de= v.igc.1.iflib.txq2.ring_state: pidx_head: 1629 pidx_tail: 1629 cidx:
16= 29 state: IDLE
+dev.igc.1.iflib.txq2.txq_cleaned: 6843536
+dev.igc.= 1.iflib.txq2.txq_processed: 6843576
+dev.igc.1.iflib.txq2.txq_in_use: 4= 2
+dev.igc.1.iflib.txq2.txq_cidx_processed: 184
+dev.igc.1.iflib.tx= q2.txq_cidx: 144
+dev.igc.1.iflib.txq2.txq_pidx: 186
 dev.igc= .1.iflib.txq2.no_tx_dma_setup: 0
 dev.igc.1.iflib.txq2.txd_encap_= efbig: 0
 dev.igc.1.iflib.txq2.tx_map_failed: 0
 dev.ig= c.1.iflib.txq1.r_abdications: 0
 dev.igc.1.iflib.txq1.r_restarts:= 0
 dev.igc.1.iflib.txq1.r_stalls: 0
-dev.igc.1.iflib.txq1.r_= starts: 0
+dev.igc.1.iflib.txq1.r_starts: 2734852
 dev.igc.1.= iflib.txq1.r_drops: 0
-dev.igc.1.iflib.txq1.r_enqueues: 0
-dev.igc.= 1.iflib.txq1.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx:
0000 sta= te: IDLE
-dev.igc.1.iflib.txq1.txq_cleaned: 0
-dev.igc.1.iflib.txq1= .txq_processed: 0
-dev.igc.1.iflib.txq1.txq_in_use: 0
-dev.igc.1.if= lib.txq1.txq_cidx_processed: 0
-dev.igc.1.iflib.txq1.txq_cidx: 0
-d= ev.igc.1.iflib.txq1.txq_pidx: 0
+dev.igc.1.iflib.txq1.r_enqueues: 27348= 52
+dev.igc.1.iflib.txq1.ring_state: pidx_head: 0772 pidx_tail: 0772 ci= dx:
0772 state: IDLE
+dev.igc.1.iflib.txq1.txq_cleaned: 5469662 +dev.igc.1.iflib.txq1.txq_processed: 5469702
+dev.igc.1.iflib.txq1.txq= _in_use: 42
+dev.igc.1.iflib.txq1.txq_cidx_processed: 518
+dev.igc.= 1.iflib.txq1.txq_cidx: 478
+dev.igc.1.iflib.txq1.txq_pidx: 520
&nb= sp;dev.igc.1.iflib.txq1.no_tx_dma_setup: 0
 dev.igc.1.iflib.txq1.= txd_encap_efbig: 0
 dev.igc.1.iflib.txq1.tx_map_failed: 0
&n= bsp;dev.igc.1.iflib.txq0.r_abdications: 0
 dev.igc.1.iflib.txq0.r= _restarts: 0
 dev.igc.1.iflib.txq0.r_stalls: 0
-dev.igc.1.ifl= ib.txq0.r_starts: 24
+dev.igc.1.iflib.txq0.r_starts: 1857414
 = ;dev.igc.1.iflib.txq0.r_drops: 0
-dev.igc.1.iflib.txq0.r_enqueues: 24 <= br>-dev.igc.1.iflib.txq0.ring_state: pidx_head: 0024 pidx_tail: 0024 cidx: =
0024 state: IDLE
-dev.igc.1.iflib.txq0.txq_cleaned: 3
-dev.igc.= 1.iflib.txq0.txq_processed: 43
+dev.igc.1.iflib.txq0.r_enqueues: 185741= 4
+dev.igc.1.iflib.txq0.ring_state: pidx_head: 1926 pidx_tail: 1926 cid= x:
1926 state: IDLE
+dev.igc.1.iflib.txq0.txq_cleaned: 3714783
= +dev.igc.1.iflib.txq0.txq_processed: 3714823
 dev.igc.1.iflib.txq= 0.txq_in_use: 42
-dev.igc.1.iflib.txq0.txq_cidx_processed: 43
-dev.= igc.1.iflib.txq0.txq_cidx: 3
-dev.igc.1.iflib.txq0.txq_pidx: 45
+de= v.igc.1.iflib.txq0.txq_cidx_processed: 775
+dev.igc.1.iflib.txq0.txq_ci= dx: 735
+dev.igc.1.iflib.txq0.txq_pidx: 777
 dev.igc.1.iflib.= txq0.no_tx_dma_setup: 0
 dev.igc.1.iflib.txq0.txd_encap_efbig: 0 =
 dev.igc.1.iflib.txq0.tx_map_failed: 0
 dev.igc.1.%desc= : Intel(R) Ethernet Controller I225-V

Interface is RUNNING and ACT= IVE
igc1: TX Queue 0 ------
igc1: hw tdh =3D 777, hw tdt =3D 777 igc1: TX Queue 1 ------
igc1: hw tdh =3D 520, hw tdt =3D 520
igc1= : TX Queue 2 ------
igc1: hw tdh =3D 186, hw tdt =3D 186
igc1: TX Q= ueue 3 ------
igc1: hw tdh =3D 746, hw tdt =3D 746
igc1: RX Queue 0= ------
igc1: hw rdh =3D 1, hw rdt =3D 0
igc1: RX Queue 1 ------ igc1: hw rdh =3D 0, hw rdt =3D 128
igc1: RX Queue 2 ------
igc1: = hw rdh =3D 0, hw rdt =3D 128
igc1: RX Queue 3 ------
igc1: hw rdh = =3D 499, hw rdt =3D 498





Mik= e Jakubik

Disclaimer: This e-mail an= d any attachments are intended only for the use of the addressee(s) and may= contain information that is privileged or confidential. If you are not the= intended recipient, or responsible for delivering the information to the i= ntended recipient, you are hereby notified that any dissemination, distribu= tion, printing or copying of this e-mail and any attachments is strictly pr= ohibited. If this e-mail and any attachments were received in error, please= notify the sender by reply e-mail and delete the original message.<= br>


------=_Part_3994682_1854819823.1661383371885--