From nobody Wed Aug 10 19:53:00 2022 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M30wX57LCz4Y2xZ for ; Wed, 10 Aug 2022 19:53:00 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M30wX2936z3n6F; Wed, 10 Aug 2022 19:53:00 +0000 (UTC) (envelope-from mike@sentex.net) Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.16.1/8.16.1) with ESMTPS id 27AJqwLv001807 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 10 Aug 2022 15:52:59 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:50c8:2c61:7c85:d1a1] ([IPv6:2607:f3e0:0:4:50c8:2c61:7c85:d1a1]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 27AJqwhI092726 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Wed, 10 Aug 2022 15:52:58 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: Date: Wed, 10 Aug 2022 15:53:00 -0400 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.1.1 Subject: Re: igc problems with heavy traffic Content-Language: en-US To: "Pieper, Jeffrey E" , Jim King , "stable@freebsd.org" , "kbowling@FreeBSD.org" References: <59b9cec0-d8c2-ce72-b5e9-99d1a1e807f8@sentex.net> <86995d10-af63-d053-972e-dd233029f3bf@jimking.net> <3d874f65-8ce2-8f06-f19a-14cd550166e3@sentex.net> From: mike tancsa In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Rspamd-Queue-Id: 4M30wX2936z3n6F X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of mike@sentex.net designates 2607:f3e0:0:1::12 as permitted sender) smtp.mailfrom=mike@sentex.net X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+ip6:2607:f3e0::/32]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[199.212.134.19:received]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[stable@freebsd.org]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROM(0.00)[]; FREEFALL_USER(0.00)[mike]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[sentex.net]; TO_DN_EQ_ADDR_SOME(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 8/10/2022 1:47 PM, Pieper, Jeffrey E wrote: > > You could try disabling EEE (Energy Efficient Ethernet). Something like: sysctl dev.igc.0.eee_control=0. It does not seem to make a difference. If I have the FC as default, I get the link bounce on the 2.5G xover (cat 6 cable) maybe 2-3 min in running iper3 tests.  However, if I disable all flow control dev.igc.0.fc=0 dev.igc.1.fc=0 dev.igc.2.fc=0 dev.igc.3.fc=0 It *seems* to be less frequent but still happens.  I ordered a 2.5 G switch so I can try and at least see which side is dropping the link. Should have it Friday to continue testing sysctl before and after the link drop --- 1660160715a 2022-08-10 15:46:17.919982000 -0400 +++ 1660160715c 2022-08-10 15:48:20.046391000 -0400 @@ -1,35 +1,35 @@  dev.igc.1.wake: 0  dev.igc.1.interrupts.rx_desc_min_thresh: 0 -dev.igc.1.interrupts.asserts: 480070 -dev.igc.1.mac_stats.tso_txd: 3467154 -dev.igc.1.mac_stats.tx_frames_1024_1522: 12191901 -dev.igc.1.mac_stats.tx_frames_512_1023: 1 -dev.igc.1.mac_stats.tx_frames_256_511: 1 -dev.igc.1.mac_stats.tx_frames_128_255: 1 -dev.igc.1.mac_stats.tx_frames_65_127: 15 +dev.igc.1.interrupts.asserts: 2321605 +dev.igc.1.mac_stats.tso_txd: 8123778 +dev.igc.1.mac_stats.tx_frames_1024_1522: 24385207 +dev.igc.1.mac_stats.tx_frames_512_1023: 2 +dev.igc.1.mac_stats.tx_frames_256_511: 5 +dev.igc.1.mac_stats.tx_frames_128_255: 3 +dev.igc.1.mac_stats.tx_frames_65_127: 5112192  dev.igc.1.mac_stats.tx_frames_64: 1  dev.igc.1.mac_stats.mcast_pkts_txd: 0  dev.igc.1.mac_stats.bcast_pkts_txd: 1 -dev.igc.1.mac_stats.good_pkts_txd: 12191920 -dev.igc.1.mac_stats.total_pkts_txd: 12191920 -dev.igc.1.mac_stats.good_octets_txd: 18507307410 -dev.igc.1.mac_stats.good_octets_recvd: 426707988 -dev.igc.1.mac_stats.rx_frames_1024_1522: 0 -dev.igc.1.mac_stats.rx_frames_512_1023: 0 -dev.igc.1.mac_stats.rx_frames_256_511: 1 -dev.igc.1.mac_stats.rx_frames_128_255: 0 -dev.igc.1.mac_stats.rx_frames_65_127: 6095576 -dev.igc.1.mac_stats.rx_frames_64: 270 +dev.igc.1.mac_stats.good_pkts_txd: 29497410 +dev.igc.1.mac_stats.total_pkts_txd: 29497410 +dev.igc.1.mac_stats.good_octets_txd: 37374600999 +dev.igc.1.mac_stats.good_octets_recvd: 16320101159 +dev.igc.1.mac_stats.rx_frames_1024_1522: 10174497 +dev.igc.1.mac_stats.rx_frames_512_1023: 26747 +dev.igc.1.mac_stats.rx_frames_256_511: 11521 +dev.igc.1.mac_stats.rx_frames_128_255: 10027 +dev.igc.1.mac_stats.rx_frames_65_127: 12193703 +dev.igc.1.mac_stats.rx_frames_64: 438  dev.igc.1.mac_stats.mcast_pkts_recvd: 0  dev.igc.1.mac_stats.bcast_pkts_recvd: 0 -dev.igc.1.mac_stats.good_pkts_recvd: 6095847 -dev.igc.1.mac_stats.total_pkts_recvd: 6095847 +dev.igc.1.mac_stats.good_pkts_recvd: 22416935 +dev.igc.1.mac_stats.total_pkts_recvd: 22416950  dev.igc.1.mac_stats.xoff_txd: 0  dev.igc.1.mac_stats.xoff_recvd: 0  dev.igc.1.mac_stats.xon_txd: 0  dev.igc.1.mac_stats.xon_recvd: 0  dev.igc.1.mac_stats.alignment_errs: 0 -dev.igc.1.mac_stats.crc_errs: 0 +dev.igc.1.mac_stats.crc_errs: 1  dev.igc.1.mac_stats.recv_errs: 0  dev.igc.1.mac_stats.recv_jabber: 0  dev.igc.1.mac_stats.recv_oversize: 0 @@ -40,7 +40,7 @@  dev.igc.1.mac_stats.defer_count: 0  dev.igc.1.mac_stats.sequence_errors: 0  dev.igc.1.mac_stats.symbol_errors: 0 -dev.igc.1.mac_stats.collision_count: 0 +dev.igc.1.mac_stats.collision_count: 1  dev.igc.1.mac_stats.late_coll: 0  dev.igc.1.mac_stats.multiple_coll: 0  dev.igc.1.mac_stats.single_coll: 0 @@ -49,34 +49,34 @@  dev.igc.1.queue_rx_3.rxd_tail: 128  dev.igc.1.queue_rx_3.rxd_head: 0  dev.igc.1.queue_rx_2.rx_irq: 0 -dev.igc.1.queue_rx_2.rxd_tail: 12 -dev.igc.1.queue_rx_2.rxd_head: 13 +dev.igc.1.queue_rx_2.rxd_tail: 456 +dev.igc.1.queue_rx_2.rxd_head: 457  dev.igc.1.queue_rx_1.rx_irq: 0  dev.igc.1.queue_rx_1.rxd_tail: 128  dev.igc.1.queue_rx_1.rxd_head: 0  dev.igc.1.queue_rx_0.rx_irq: 0 -dev.igc.1.queue_rx_0.rxd_tail: 985 -dev.igc.1.queue_rx_0.rxd_head: 986 +dev.igc.1.queue_rx_0.rxd_tail: 6 +dev.igc.1.queue_rx_0.rxd_head: 7  dev.igc.1.queue_tx_3.tx_irq: 0  dev.igc.1.queue_tx_3.txd_tail: 0  dev.igc.1.queue_tx_3.txd_head: 0  dev.igc.1.queue_tx_2.tx_irq: 0 -dev.igc.1.queue_tx_2.txd_tail: 28 -dev.igc.1.queue_tx_2.txd_head: 28 +dev.igc.1.queue_tx_2.txd_tail: 460 +dev.igc.1.queue_tx_2.txd_head: 460  dev.igc.1.queue_tx_1.tx_irq: 0  dev.igc.1.queue_tx_1.txd_tail: 0  dev.igc.1.queue_tx_1.txd_head: 0  dev.igc.1.queue_tx_0.tx_irq: 0 -dev.igc.1.queue_tx_0.txd_tail: 69 -dev.igc.1.queue_tx_0.txd_head: 69 +dev.igc.1.queue_tx_0.txd_tail: 15 +dev.igc.1.queue_tx_0.txd_head: 15  dev.igc.1.fc_low_water: 32752  dev.igc.1.fc_high_water: 32768 -dev.igc.1.rx_control: 71335942 +dev.igc.1.rx_control: 71335966  dev.igc.1.device_control: 404489793  dev.igc.1.watchdog_timeouts: 0  dev.igc.1.rx_overruns: 0 -dev.igc.1.link_irq: 4 -dev.igc.1.dropped: 0 +dev.igc.1.link_irq: 6 +dev.igc.1.dropped: 1  dev.igc.1.eee_control: 0  dev.igc.1.itr: 488  dev.igc.1.tx_abs_int_delay: 66 @@ -93,10 +93,10 @@         ICR      00000000  RX Registers -       RCTL     04408006 +       RCTL     0440801e         RDLEN    00004000 -       RDH      000003da -       RDT      000003d9 +       RDH      00000007 +       RDT      00000006         RXDCTL   02040808         RDBAL    04544000         RDBAH    00000000 @@ -106,8 +106,8 @@         TDBAL    04506000         TDBAH    00000000         TDLEN    00004000 -       TDH      00000045 -       TDT      00000045 +       TDH      0000000f +       TDT      0000000f         TXDCTL   0201011f         TDFH     00000000         TDFT     00000000 @@ -125,8 +125,8 @@  dev.igc.1.iflib.rxq3.cpu: 3  dev.igc.1.iflib.rxq2.rxq_fl0.buf_size: 2048  dev.igc.1.iflib.rxq2.rxq_fl0.credits: 1023 -dev.igc.1.iflib.rxq2.rxq_fl0.cidx: 13 -dev.igc.1.iflib.rxq2.rxq_fl0.pidx: 12 +dev.igc.1.iflib.rxq2.rxq_fl0.cidx: 457 +dev.igc.1.iflib.rxq2.rxq_fl0.pidx: 456  dev.igc.1.iflib.rxq2.cpu: 2  dev.igc.1.iflib.rxq1.rxq_fl0.buf_size: 2048  dev.igc.1.iflib.rxq1.rxq_fl0.credits: 128 @@ -135,8 +135,8 @@  dev.igc.1.iflib.rxq1.cpu: 1  dev.igc.1.iflib.rxq0.rxq_fl0.buf_size: 2048  dev.igc.1.iflib.rxq0.rxq_fl0.credits: 1023 -dev.igc.1.iflib.rxq0.rxq_fl0.cidx: 986 -dev.igc.1.iflib.rxq0.rxq_fl0.pidx: 985 +dev.igc.1.iflib.rxq0.rxq_fl0.cidx: 7 +dev.igc.1.iflib.rxq0.rxq_fl0.pidx: 6  dev.igc.1.iflib.rxq0.cpu: 0  dev.igc.1.iflib.txq3.r_abdications: 0  dev.igc.1.iflib.txq3.r_restarts: 0 @@ -144,7 +144,7 @@  dev.igc.1.iflib.txq3.r_starts: 0  dev.igc.1.iflib.txq3.r_drops: 0  dev.igc.1.iflib.txq3.r_enqueues: 0 -dev.igc.1.iflib.txq3.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE +dev.igc.1.iflib.txq3.ring_state: pidx_head: 0306 pidx_tail: 0306 cidx: 0306 state: IDLE  dev.igc.1.iflib.txq3.txq_cleaned: 0  dev.igc.1.iflib.txq3.txq_processed: 0  dev.igc.1.iflib.txq3.txq_in_use: 0 @@ -162,16 +162,16 @@  dev.igc.1.iflib.txq2.r_abdications: 0  dev.igc.1.iflib.txq2.r_restarts: 0  dev.igc.1.iflib.txq2.r_stalls: 0 -dev.igc.1.iflib.txq2.r_starts: 13 +dev.igc.1.iflib.txq2.r_starts: 4550374  dev.igc.1.iflib.txq2.r_drops: 0 -dev.igc.1.iflib.txq2.r_enqueues: 13 -dev.igc.1.iflib.txq2.ring_state: pidx_head: 0013 pidx_tail: 0013 cidx: 0013 state: IDLE -dev.igc.1.iflib.txq2.txq_cleaned: 0 -dev.igc.1.iflib.txq2.txq_processed: 24 -dev.igc.1.iflib.txq2.txq_in_use: 28 -dev.igc.1.iflib.txq2.txq_cidx_processed: 24 -dev.igc.1.iflib.txq2.txq_cidx: 0 -dev.igc.1.iflib.txq2.txq_pidx: 28 +dev.igc.1.iflib.txq2.r_enqueues: 4550374 +dev.igc.1.iflib.txq2.ring_state: pidx_head: 1224 pidx_tail: 1224 cidx: 1224 state: IDLE +dev.igc.1.iflib.txq2.txq_cleaned: 9100706 +dev.igc.1.iflib.txq2.txq_processed: 9100746 +dev.igc.1.iflib.txq2.txq_in_use: 42 +dev.igc.1.iflib.txq2.txq_cidx_processed: 458 +dev.igc.1.iflib.txq2.txq_cidx: 418 +dev.igc.1.iflib.txq2.txq_pidx: 460  dev.igc.1.iflib.txq2.no_tx_dma_setup: 0  dev.igc.1.iflib.txq2.txd_encap_efbig: 0  dev.igc.1.iflib.txq2.tx_map_failed: 0 @@ -186,7 +186,7 @@  dev.igc.1.iflib.txq1.r_starts: 0  dev.igc.1.iflib.txq1.r_drops: 0  dev.igc.1.iflib.txq1.r_enqueues: 0 -dev.igc.1.iflib.txq1.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE +dev.igc.1.iflib.txq1.ring_state: pidx_head: 0958 pidx_tail: 0958 cidx: 0958 state: IDLE  dev.igc.1.iflib.txq1.txq_cleaned: 0  dev.igc.1.iflib.txq1.txq_processed: 0  dev.igc.1.iflib.txq1.txq_in_use: 0 @@ -202,18 +202,18 @@  dev.igc.1.iflib.txq1.mbuf_defrag: 0  dev.igc.1.iflib.txq1.cpu: 1  dev.igc.1.iflib.txq0.r_abdications: 0 -dev.igc.1.iflib.txq0.r_restarts: 324218 -dev.igc.1.iflib.txq0.r_stalls: 324218 -dev.igc.1.iflib.txq0.r_starts: 95287 +dev.igc.1.iflib.txq0.r_restarts: 0 +dev.igc.1.iflib.txq0.r_stalls: 0 +dev.igc.1.iflib.txq0.r_starts: 7  dev.igc.1.iflib.txq0.r_drops: 0 -dev.igc.1.iflib.txq0.r_enqueues: 4031110 -dev.igc.1.iflib.txq0.ring_state: pidx_head: 0646 pidx_tail: 0646 cidx: 0646 state: IDLE -dev.igc.1.iflib.txq0.txq_cleaned: 17253402 -dev.igc.1.iflib.txq0.txq_processed: 17253442 -dev.igc.1.iflib.txq0.txq_in_use: 43 -dev.igc.1.iflib.txq0.txq_cidx_processed: 66 -dev.igc.1.iflib.txq0.txq_cidx: 26 -dev.igc.1.iflib.txq0.txq_pidx: 69 +dev.igc.1.iflib.txq0.r_enqueues: 7 +dev.igc.1.iflib.txq0.ring_state: pidx_head: 0005 pidx_tail: 0005 cidx: 0005 state: IDLE +dev.igc.1.iflib.txq0.txq_cleaned: 0 +dev.igc.1.iflib.txq0.txq_processed: 11 +dev.igc.1.iflib.txq0.txq_in_use: 15 +dev.igc.1.iflib.txq0.txq_cidx_processed: 11 +dev.igc.1.iflib.txq0.txq_cidx: 0 +dev.igc.1.iflib.txq0.txq_pidx: 15  dev.igc.1.iflib.txq0.no_tx_dma_setup: 0  dev.igc.1.iflib.txq0.txd_encap_efbig: 0  dev.igc.1.iflib.txq0.tx_map_failed: 0