From nobody Wed Sep 14 15:03:54 2022 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4MSNry5zyHz4cKx0 for ; Wed, 14 Sep 2022 15:04:02 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4MSNrx6Z0kz46FH; Wed, 14 Sep 2022 15:04:01 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 28EF3rlg057740 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 14 Sep 2022 11:03:53 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [192.168.43.29] ([192.168.43.29]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 28EF3qkx029757 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 14 Sep 2022 11:03:53 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: Date: Wed, 14 Sep 2022 11:03:54 -0400 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.2.2 Subject: Re: igc problems with heavy traffic (update) Content-Language: en-US From: mike tancsa To: "Pieper, Jeffrey E" , Jim King , "stable@freebsd.org" , "kbowling@FreeBSD.org" References: <59b9cec0-d8c2-ce72-b5e9-99d1a1e807f8@sentex.net> <86995d10-af63-d053-972e-dd233029f3bf@jimking.net> <3d874f65-8ce2-8f06-f19a-14cd550166e3@sentex.net> <879b9239-2b9a-f0ae-4173-4a226c84cd85@sentex.net> In-Reply-To: <879b9239-2b9a-f0ae-4173-4a226c84cd85@sentex.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Rspamd-Queue-Id: 4MSNrx6Z0kz46FH X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[]; FREEFALL_USER(0.00)[mike]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[sentex.net]; TO_DN_EQ_ADDR_SOME(0.00)[] X-ThisMailContainsUnwantedMimeParts: N OK, an update hence the top post. I got a new pair of boxes which use a different Jasper Lake chipset and have i226-V vs the i225 of the previous box. dev.igc.0.%parent: pci2 dev.igc.0.%pnpinfo: vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000 class=0x020000 dev.igc.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 handle=\_SB_.PC00.RP05.PXSX dev.igc.0.%driver: igc dev.igc.0.%desc: Intel(R) Ethernet Controller I226-V dev.igc.%parent: WIth a default RELENG_13, out of the box with no tweaks, I am NOT able to cause the transmitting nic to bounce with heave traffic. I used the same test script (a constant stream of iperf3 alternating in direction) maxing out the NIC's bandwidth and all seems fine running the test for some 18hrs.  Maybe something different about the i225 version of this NIC that needs some different driver defaults ?     ---Mike On 8/12/2022 11:04 AM, mike tancsa wrote: > > On 8/10/2022 3:53 PM, mike tancsa wrote: >> On 8/10/2022 1:47 PM, Pieper, Jeffrey E wrote: >>> >>> You could try disabling EEE (Energy Efficient Ethernet). Something >>> like: sysctl dev.igc.0.eee_control=0. >> >> >> It does not seem to make a difference. If I have the FC as default, I >> get the link bounce on the 2.5G xover (cat 6 cable) maybe 2-3 min in >> running iper3 tests.  However, if I disable all flow control >> >> dev.igc.0.fc=0 >> dev.igc.1.fc=0 >> dev.igc.2.fc=0 >> dev.igc.3.fc=0 >> >> It *seems* to be less frequent but still happens.  I ordered a 2.5 G >> switch so I can try and at least see which side is dropping the link. >> Should have it Friday to continue testing >> > > OK, I repeated the tests with a 2.5G unmanaged switch in between the > two units rather than xover. It looks like its the server that is > sending the majority of the packets that drops the link, not the > receiver. > > One other test I did was to up hw.igc.max_interrupt_rate=13000 from > the default of 8000. That seems to make the problem MUCH more acute. > > Here is the before and after of the link drop. > >  dev.igc.1.wake: 0 >  dev.igc.1.interrupts.rx_desc_min_thresh: 0 > -dev.igc.1.interrupts.asserts: 65 > +dev.igc.1.interrupts.asserts: 4879479 >  dev.igc.1.mac_stats.tso_txd: 0 > -dev.igc.1.mac_stats.tx_frames_1024_1522: 3 > -dev.igc.1.mac_stats.tx_frames_512_1023: 1 > -dev.igc.1.mac_stats.tx_frames_256_511: 2 > -dev.igc.1.mac_stats.tx_frames_128_255: 15 > -dev.igc.1.mac_stats.tx_frames_65_127: 2 > +dev.igc.1.mac_stats.tx_frames_1024_1522: 12973065 > +dev.igc.1.mac_stats.tx_frames_512_1023: 58 > +dev.igc.1.mac_stats.tx_frames_256_511: 107 > +dev.igc.1.mac_stats.tx_frames_128_255: 1215725 > +dev.igc.1.mac_stats.tx_frames_65_127: 192 >  dev.igc.1.mac_stats.tx_frames_64: 1 >  dev.igc.1.mac_stats.mcast_pkts_txd: 0 >  dev.igc.1.mac_stats.bcast_pkts_txd: 1 > -dev.igc.1.mac_stats.good_pkts_txd: 24 > -dev.igc.1.mac_stats.total_pkts_txd: 24 > -dev.igc.1.mac_stats.good_octets_txd: 7674 > -dev.igc.1.mac_stats.good_octets_recvd: 6492 > -dev.igc.1.mac_stats.rx_frames_1024_1522: 2 > -dev.igc.1.mac_stats.rx_frames_512_1023: 1 > -dev.igc.1.mac_stats.rx_frames_256_511: 2 > -dev.igc.1.mac_stats.rx_frames_128_255: 15 > -dev.igc.1.mac_stats.rx_frames_65_127: 2 > +dev.igc.1.mac_stats.good_pkts_txd: 14189148 > +dev.igc.1.mac_stats.total_pkts_txd: 14189148 > +dev.igc.1.mac_stats.good_octets_txd: 19450753554 > +dev.igc.1.mac_stats.good_octets_recvd: 14933399426 > +dev.igc.1.mac_stats.rx_frames_1024_1522: 9823228 > +dev.igc.1.mac_stats.rx_frames_512_1023: 3 > +dev.igc.1.mac_stats.rx_frames_256_511: 62 > +dev.igc.1.mac_stats.rx_frames_128_255: 2365665 > +dev.igc.1.mac_stats.rx_frames_65_127: 213 >  dev.igc.1.mac_stats.rx_frames_64: 1 >  dev.igc.1.mac_stats.mcast_pkts_recvd: 0 >  dev.igc.1.mac_stats.bcast_pkts_recvd: 0 > -dev.igc.1.mac_stats.good_pkts_recvd: 23 > -dev.igc.1.mac_stats.total_pkts_recvd: 23 > +dev.igc.1.mac_stats.good_pkts_recvd: 12189172 > +dev.igc.1.mac_stats.total_pkts_recvd: 12189172 >  dev.igc.1.mac_stats.xoff_txd: 0 >  dev.igc.1.mac_stats.xoff_recvd: 0 >  dev.igc.1.mac_stats.xon_txd: 0 >  dev.igc.1.mac_stats.single_coll: 0 >  dev.igc.1.mac_stats.excess_coll: 0 >  dev.igc.1.queue_rx_3.rx_irq: 0 > -dev.igc.1.queue_rx_3.rxd_tail: 21 > -dev.igc.1.queue_rx_3.rxd_head: 22 > +dev.igc.1.queue_rx_3.rxd_tail: 498 > +dev.igc.1.queue_rx_3.rxd_head: 499 >  dev.igc.1.queue_rx_2.rx_irq: 0 >  dev.igc.1.queue_rx_2.rxd_tail: 128 >  dev.igc.1.queue_rx_2.rxd_head: 0 >  dev.igc.1.queue_rx_0.rxd_tail: 0 >  dev.igc.1.queue_rx_0.rxd_head: 1 >  dev.igc.1.queue_tx_3.tx_irq: 0 > -dev.igc.1.queue_tx_3.txd_tail: 0 > -dev.igc.1.queue_tx_3.txd_head: 0 > +dev.igc.1.queue_tx_3.txd_tail: 746 > +dev.igc.1.queue_tx_3.txd_head: 746 >  dev.igc.1.queue_tx_2.tx_irq: 0 > -dev.igc.1.queue_tx_2.txd_tail: 0 > -dev.igc.1.queue_tx_2.txd_head: 0 > +dev.igc.1.queue_tx_2.txd_tail: 186 > +dev.igc.1.queue_tx_2.txd_head: 186 >  dev.igc.1.queue_tx_1.tx_irq: 0 > -dev.igc.1.queue_tx_1.txd_tail: 0 > -dev.igc.1.queue_tx_1.txd_head: 0 > +dev.igc.1.queue_tx_1.txd_tail: 520 > +dev.igc.1.queue_tx_1.txd_head: 520 >  dev.igc.1.queue_tx_0.tx_irq: 0 > -dev.igc.1.queue_tx_0.txd_tail: 45 > -dev.igc.1.queue_tx_0.txd_head: 45 > +dev.igc.1.queue_tx_0.txd_tail: 777 > +dev.igc.1.queue_tx_0.txd_head: 777 >  dev.igc.1.fc_low_water: 32752 >  dev.igc.1.fc_high_water: 32768 >  dev.igc.1.rx_control: 71335938 >  dev.igc.1.device_control: 404489793 >  dev.igc.1.watchdog_timeouts: 0 >  dev.igc.1.rx_overruns: 0 > -dev.igc.1.link_irq: 2 > +dev.igc.1.link_irq: 4 >  dev.igc.1.dropped: 0 >  dev.igc.1.eee_control: 0 >  dev.igc.1.itr: 488 >  dev.igc.1.nvm: -1 >  dev.igc.1.iflib.rxq3.rxq_fl0.buf_size: 2048 >  dev.igc.1.iflib.rxq3.rxq_fl0.credits: 1023 > -dev.igc.1.iflib.rxq3.rxq_fl0.cidx: 22 > -dev.igc.1.iflib.rxq3.rxq_fl0.pidx: 21 > +dev.igc.1.iflib.rxq3.rxq_fl0.cidx: 499 > +dev.igc.1.iflib.rxq3.rxq_fl0.pidx: 498 >  dev.igc.1.iflib.rxq3.cpu: 3 >  dev.igc.1.iflib.rxq2.rxq_fl0.buf_size: 2048 >  dev.igc.1.iflib.rxq2.rxq_fl0.credits: 128 >  dev.igc.1.iflib.txq3.r_abdications: 0 >  dev.igc.1.iflib.txq3.r_restarts: 0 >  dev.igc.1.iflib.txq3.r_stalls: 0 > -dev.igc.1.iflib.txq3.r_starts: 0 > +dev.igc.1.iflib.txq3.r_starts: 6175093 >  dev.igc.1.iflib.txq3.r_drops: 0 > -dev.igc.1.iflib.txq3.r_enqueues: 0 > -dev.igc.1.iflib.txq3.ring_state: pidx_head: 0000 pidx_tail: 0000 > cidx: 0000 state: IDLE > -dev.igc.1.iflib.txq3.txq_cleaned: 0 > -dev.igc.1.iflib.txq3.txq_processed: 0 > -dev.igc.1.iflib.txq3.txq_in_use: 0 > -dev.igc.1.iflib.txq3.txq_cidx_processed: 0 > -dev.igc.1.iflib.txq3.txq_cidx: 0 > -dev.igc.1.iflib.txq3.txq_pidx: 0 > +dev.igc.1.iflib.txq3.r_enqueues: 6175093 > +dev.igc.1.iflib.txq3.ring_state: pidx_head: 0373 pidx_tail: 0373 > cidx: 0373 state: IDLE > +dev.igc.1.iflib.txq3.txq_cleaned: 12350144 > +dev.igc.1.iflib.txq3.txq_processed: 12350184 > +dev.igc.1.iflib.txq3.txq_in_use: 42 > +dev.igc.1.iflib.txq3.txq_cidx_processed: 744 > +dev.igc.1.iflib.txq3.txq_cidx: 704 > +dev.igc.1.iflib.txq3.txq_pidx: 746 >  dev.igc.1.iflib.txq3.no_tx_dma_setup: 0 >  dev.igc.1.iflib.txq3.txd_encap_efbig: 0 >  dev.igc.1.iflib.txq3.tx_map_failed: 0 >  dev.igc.1.iflib.txq2.r_abdications: 0 >  dev.igc.1.iflib.txq2.r_restarts: 0 >  dev.igc.1.iflib.txq2.r_stalls: 0 > -dev.igc.1.iflib.txq2.r_starts: 0 > +dev.igc.1.iflib.txq2.r_starts: 3421789 >  dev.igc.1.iflib.txq2.r_drops: 0 > -dev.igc.1.iflib.txq2.r_enqueues: 0 > -dev.igc.1.iflib.txq2.ring_state: pidx_head: 0000 pidx_tail: 0000 > cidx: 0000 state: IDLE > -dev.igc.1.iflib.txq2.txq_cleaned: 0 > -dev.igc.1.iflib.txq2.txq_processed: 0 > -dev.igc.1.iflib.txq2.txq_in_use: 0 > -dev.igc.1.iflib.txq2.txq_cidx_processed: 0 > -dev.igc.1.iflib.txq2.txq_cidx: 0 > -dev.igc.1.iflib.txq2.txq_pidx: 0 > +dev.igc.1.iflib.txq2.r_enqueues: 3421789 > +dev.igc.1.iflib.txq2.ring_state: pidx_head: 1629 pidx_tail: 1629 > cidx: 1629 state: IDLE > +dev.igc.1.iflib.txq2.txq_cleaned: 6843536 > +dev.igc.1.iflib.txq2.txq_processed: 6843576 > +dev.igc.1.iflib.txq2.txq_in_use: 42 > +dev.igc.1.iflib.txq2.txq_cidx_processed: 184 > +dev.igc.1.iflib.txq2.txq_cidx: 144 > +dev.igc.1.iflib.txq2.txq_pidx: 186 >  dev.igc.1.iflib.txq2.no_tx_dma_setup: 0 >  dev.igc.1.iflib.txq2.txd_encap_efbig: 0 >  dev.igc.1.iflib.txq2.tx_map_failed: 0 >  dev.igc.1.iflib.txq1.r_abdications: 0 >  dev.igc.1.iflib.txq1.r_restarts: 0 >  dev.igc.1.iflib.txq1.r_stalls: 0 > -dev.igc.1.iflib.txq1.r_starts: 0 > +dev.igc.1.iflib.txq1.r_starts: 2734852 >  dev.igc.1.iflib.txq1.r_drops: 0 > -dev.igc.1.iflib.txq1.r_enqueues: 0 > -dev.igc.1.iflib.txq1.ring_state: pidx_head: 0000 pidx_tail: 0000 > cidx: 0000 state: IDLE > -dev.igc.1.iflib.txq1.txq_cleaned: 0 > -dev.igc.1.iflib.txq1.txq_processed: 0 > -dev.igc.1.iflib.txq1.txq_in_use: 0 > -dev.igc.1.iflib.txq1.txq_cidx_processed: 0 > -dev.igc.1.iflib.txq1.txq_cidx: 0 > -dev.igc.1.iflib.txq1.txq_pidx: 0 > +dev.igc.1.iflib.txq1.r_enqueues: 2734852 > +dev.igc.1.iflib.txq1.ring_state: pidx_head: 0772 pidx_tail: 0772 > cidx: 0772 state: IDLE > +dev.igc.1.iflib.txq1.txq_cleaned: 5469662 > +dev.igc.1.iflib.txq1.txq_processed: 5469702 > +dev.igc.1.iflib.txq1.txq_in_use: 42 > +dev.igc.1.iflib.txq1.txq_cidx_processed: 518 > +dev.igc.1.iflib.txq1.txq_cidx: 478 > +dev.igc.1.iflib.txq1.txq_pidx: 520 >  dev.igc.1.iflib.txq1.no_tx_dma_setup: 0 >  dev.igc.1.iflib.txq1.txd_encap_efbig: 0 >  dev.igc.1.iflib.txq1.tx_map_failed: 0 >  dev.igc.1.iflib.txq0.r_abdications: 0 >  dev.igc.1.iflib.txq0.r_restarts: 0 >  dev.igc.1.iflib.txq0.r_stalls: 0 > -dev.igc.1.iflib.txq0.r_starts: 24 > +dev.igc.1.iflib.txq0.r_starts: 1857414 >  dev.igc.1.iflib.txq0.r_drops: 0 > -dev.igc.1.iflib.txq0.r_enqueues: 24 > -dev.igc.1.iflib.txq0.ring_state: pidx_head: 0024 pidx_tail: 0024 > cidx: 0024 state: IDLE > -dev.igc.1.iflib.txq0.txq_cleaned: 3 > -dev.igc.1.iflib.txq0.txq_processed: 43 > +dev.igc.1.iflib.txq0.r_enqueues: 1857414 > +dev.igc.1.iflib.txq0.ring_state: pidx_head: 1926 pidx_tail: 1926 > cidx: 1926 state: IDLE > +dev.igc.1.iflib.txq0.txq_cleaned: 3714783 > +dev.igc.1.iflib.txq0.txq_processed: 3714823 >  dev.igc.1.iflib.txq0.txq_in_use: 42 > -dev.igc.1.iflib.txq0.txq_cidx_processed: 43 > -dev.igc.1.iflib.txq0.txq_cidx: 3 > -dev.igc.1.iflib.txq0.txq_pidx: 45 > +dev.igc.1.iflib.txq0.txq_cidx_processed: 775 > +dev.igc.1.iflib.txq0.txq_cidx: 735 > +dev.igc.1.iflib.txq0.txq_pidx: 777 >  dev.igc.1.iflib.txq0.no_tx_dma_setup: 0 >  dev.igc.1.iflib.txq0.txd_encap_efbig: 0 >  dev.igc.1.iflib.txq0.tx_map_failed: 0 >  dev.igc.1.%desc: Intel(R) Ethernet Controller I225-V > > Interface is RUNNING and ACTIVE > igc1: TX Queue 0 ------ > igc1: hw tdh = 777, hw tdt = 777 > igc1: TX Queue 1 ------ > igc1: hw tdh = 520, hw tdt = 520 > igc1: TX Queue 2 ------ > igc1: hw tdh = 186, hw tdt = 186 > igc1: TX Queue 3 ------ > igc1: hw tdh = 746, hw tdt = 746 > igc1: RX Queue 0 ------ > igc1: hw rdh = 1, hw rdt = 0 > igc1: RX Queue 1 ------ > igc1: hw rdh = 0, hw rdt = 128 > igc1: RX Queue 2 ------ > igc1: hw rdh = 0, hw rdt = 128 > igc1: RX Queue 3 ------ > igc1: hw rdh = 499, hw rdt = 498 > > >