Date: Wed, 10 Aug 2022 15:53:00 -0400 From: mike tancsa <mike@sentex.net> To: "Pieper, Jeffrey E" <jeffrey.e.pieper@intel.com>, Jim King <jim@jimking.net>, "stable@freebsd.org" <stable@freebsd.org>, "kbowling@FreeBSD.org" <kbowling@FreeBSD.org> Subject: Re: igc problems with heavy traffic Message-ID: <a9935ba0-9cb2-5a41-ca73-b6962fef5e4d@sentex.net> In-Reply-To: <MW4PR11MB5890493674ADD1757BB47075D0659@MW4PR11MB5890.namprd11.prod.outlook.com> References: <fc256428-3ff1-68ba-cfcc-a00ca427e85b@jimking.net> <59b9cec0-d8c2-ce72-b5e9-99d1a1e807f8@sentex.net> <e714cd76-0aaa-3ea0-3c31-5e61badffa18@sentex.net> <86995d10-af63-d053-972e-dd233029f3bf@jimking.net> <3d874f65-8ce2-8f06-f19a-14cd550166e3@sentex.net> <a8192d60-2970-edb5-ce1a-c17ea875bf07@jimking.net> <fd1e825b-c306-64b1-f9ef-fec0344a9c95@sentex.net> <a4ddc96a-3dd5-4fee-8003-05f228d10858@jimking.net> <MW4PR11MB5890493674ADD1757BB47075D0659@MW4PR11MB5890.namprd11.prod.outlook.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8/10/2022 1:47 PM, Pieper, Jeffrey E wrote: > > You could try disabling EEE (Energy Efficient Ethernet). Something like: sysctl dev.igc.0.eee_control=0. It does not seem to make a difference. If I have the FC as default, I get the link bounce on the 2.5G xover (cat 6 cable) maybe 2-3 min in running iper3 tests. However, if I disable all flow control dev.igc.0.fc=0 dev.igc.1.fc=0 dev.igc.2.fc=0 dev.igc.3.fc=0 It *seems* to be less frequent but still happens. I ordered a 2.5 G switch so I can try and at least see which side is dropping the link. Should have it Friday to continue testing sysctl before and after the link drop --- 1660160715a 2022-08-10 15:46:17.919982000 -0400 +++ 1660160715c 2022-08-10 15:48:20.046391000 -0400 @@ -1,35 +1,35 @@ dev.igc.1.wake: 0 dev.igc.1.interrupts.rx_desc_min_thresh: 0 -dev.igc.1.interrupts.asserts: 480070 -dev.igc.1.mac_stats.tso_txd: 3467154 -dev.igc.1.mac_stats.tx_frames_1024_1522: 12191901 -dev.igc.1.mac_stats.tx_frames_512_1023: 1 -dev.igc.1.mac_stats.tx_frames_256_511: 1 -dev.igc.1.mac_stats.tx_frames_128_255: 1 -dev.igc.1.mac_stats.tx_frames_65_127: 15 +dev.igc.1.interrupts.asserts: 2321605 +dev.igc.1.mac_stats.tso_txd: 8123778 +dev.igc.1.mac_stats.tx_frames_1024_1522: 24385207 +dev.igc.1.mac_stats.tx_frames_512_1023: 2 +dev.igc.1.mac_stats.tx_frames_256_511: 5 +dev.igc.1.mac_stats.tx_frames_128_255: 3 +dev.igc.1.mac_stats.tx_frames_65_127: 5112192 dev.igc.1.mac_stats.tx_frames_64: 1 dev.igc.1.mac_stats.mcast_pkts_txd: 0 dev.igc.1.mac_stats.bcast_pkts_txd: 1 -dev.igc.1.mac_stats.good_pkts_txd: 12191920 -dev.igc.1.mac_stats.total_pkts_txd: 12191920 -dev.igc.1.mac_stats.good_octets_txd: 18507307410 -dev.igc.1.mac_stats.good_octets_recvd: 426707988 -dev.igc.1.mac_stats.rx_frames_1024_1522: 0 -dev.igc.1.mac_stats.rx_frames_512_1023: 0 -dev.igc.1.mac_stats.rx_frames_256_511: 1 -dev.igc.1.mac_stats.rx_frames_128_255: 0 -dev.igc.1.mac_stats.rx_frames_65_127: 6095576 -dev.igc.1.mac_stats.rx_frames_64: 270 +dev.igc.1.mac_stats.good_pkts_txd: 29497410 +dev.igc.1.mac_stats.total_pkts_txd: 29497410 +dev.igc.1.mac_stats.good_octets_txd: 37374600999 +dev.igc.1.mac_stats.good_octets_recvd: 16320101159 +dev.igc.1.mac_stats.rx_frames_1024_1522: 10174497 +dev.igc.1.mac_stats.rx_frames_512_1023: 26747 +dev.igc.1.mac_stats.rx_frames_256_511: 11521 +dev.igc.1.mac_stats.rx_frames_128_255: 10027 +dev.igc.1.mac_stats.rx_frames_65_127: 12193703 +dev.igc.1.mac_stats.rx_frames_64: 438 dev.igc.1.mac_stats.mcast_pkts_recvd: 0 dev.igc.1.mac_stats.bcast_pkts_recvd: 0 -dev.igc.1.mac_stats.good_pkts_recvd: 6095847 -dev.igc.1.mac_stats.total_pkts_recvd: 6095847 +dev.igc.1.mac_stats.good_pkts_recvd: 22416935 +dev.igc.1.mac_stats.total_pkts_recvd: 22416950 dev.igc.1.mac_stats.xoff_txd: 0 dev.igc.1.mac_stats.xoff_recvd: 0 dev.igc.1.mac_stats.xon_txd: 0 dev.igc.1.mac_stats.xon_recvd: 0 dev.igc.1.mac_stats.alignment_errs: 0 -dev.igc.1.mac_stats.crc_errs: 0 +dev.igc.1.mac_stats.crc_errs: 1 dev.igc.1.mac_stats.recv_errs: 0 dev.igc.1.mac_stats.recv_jabber: 0 dev.igc.1.mac_stats.recv_oversize: 0 @@ -40,7 +40,7 @@ dev.igc.1.mac_stats.defer_count: 0 dev.igc.1.mac_stats.sequence_errors: 0 dev.igc.1.mac_stats.symbol_errors: 0 -dev.igc.1.mac_stats.collision_count: 0 +dev.igc.1.mac_stats.collision_count: 1 dev.igc.1.mac_stats.late_coll: 0 dev.igc.1.mac_stats.multiple_coll: 0 dev.igc.1.mac_stats.single_coll: 0 @@ -49,34 +49,34 @@ dev.igc.1.queue_rx_3.rxd_tail: 128 dev.igc.1.queue_rx_3.rxd_head: 0 dev.igc.1.queue_rx_2.rx_irq: 0 -dev.igc.1.queue_rx_2.rxd_tail: 12 -dev.igc.1.queue_rx_2.rxd_head: 13 +dev.igc.1.queue_rx_2.rxd_tail: 456 +dev.igc.1.queue_rx_2.rxd_head: 457 dev.igc.1.queue_rx_1.rx_irq: 0 dev.igc.1.queue_rx_1.rxd_tail: 128 dev.igc.1.queue_rx_1.rxd_head: 0 dev.igc.1.queue_rx_0.rx_irq: 0 -dev.igc.1.queue_rx_0.rxd_tail: 985 -dev.igc.1.queue_rx_0.rxd_head: 986 +dev.igc.1.queue_rx_0.rxd_tail: 6 +dev.igc.1.queue_rx_0.rxd_head: 7 dev.igc.1.queue_tx_3.tx_irq: 0 dev.igc.1.queue_tx_3.txd_tail: 0 dev.igc.1.queue_tx_3.txd_head: 0 dev.igc.1.queue_tx_2.tx_irq: 0 -dev.igc.1.queue_tx_2.txd_tail: 28 -dev.igc.1.queue_tx_2.txd_head: 28 +dev.igc.1.queue_tx_2.txd_tail: 460 +dev.igc.1.queue_tx_2.txd_head: 460 dev.igc.1.queue_tx_1.tx_irq: 0 dev.igc.1.queue_tx_1.txd_tail: 0 dev.igc.1.queue_tx_1.txd_head: 0 dev.igc.1.queue_tx_0.tx_irq: 0 -dev.igc.1.queue_tx_0.txd_tail: 69 -dev.igc.1.queue_tx_0.txd_head: 69 +dev.igc.1.queue_tx_0.txd_tail: 15 +dev.igc.1.queue_tx_0.txd_head: 15 dev.igc.1.fc_low_water: 32752 dev.igc.1.fc_high_water: 32768 -dev.igc.1.rx_control: 71335942 +dev.igc.1.rx_control: 71335966 dev.igc.1.device_control: 404489793 dev.igc.1.watchdog_timeouts: 0 dev.igc.1.rx_overruns: 0 -dev.igc.1.link_irq: 4 -dev.igc.1.dropped: 0 +dev.igc.1.link_irq: 6 +dev.igc.1.dropped: 1 dev.igc.1.eee_control: 0 dev.igc.1.itr: 488 dev.igc.1.tx_abs_int_delay: 66 @@ -93,10 +93,10 @@ ICR 00000000 RX Registers - RCTL 04408006 + RCTL 0440801e RDLEN 00004000 - RDH 000003da - RDT 000003d9 + RDH 00000007 + RDT 00000006 RXDCTL 02040808 RDBAL 04544000 RDBAH 00000000 @@ -106,8 +106,8 @@ TDBAL 04506000 TDBAH 00000000 TDLEN 00004000 - TDH 00000045 - TDT 00000045 + TDH 0000000f + TDT 0000000f TXDCTL 0201011f TDFH 00000000 TDFT 00000000 @@ -125,8 +125,8 @@ dev.igc.1.iflib.rxq3.cpu: 3 dev.igc.1.iflib.rxq2.rxq_fl0.buf_size: 2048 dev.igc.1.iflib.rxq2.rxq_fl0.credits: 1023 -dev.igc.1.iflib.rxq2.rxq_fl0.cidx: 13 -dev.igc.1.iflib.rxq2.rxq_fl0.pidx: 12 +dev.igc.1.iflib.rxq2.rxq_fl0.cidx: 457 +dev.igc.1.iflib.rxq2.rxq_fl0.pidx: 456 dev.igc.1.iflib.rxq2.cpu: 2 dev.igc.1.iflib.rxq1.rxq_fl0.buf_size: 2048 dev.igc.1.iflib.rxq1.rxq_fl0.credits: 128 @@ -135,8 +135,8 @@ dev.igc.1.iflib.rxq1.cpu: 1 dev.igc.1.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.igc.1.iflib.rxq0.rxq_fl0.credits: 1023 -dev.igc.1.iflib.rxq0.rxq_fl0.cidx: 986 -dev.igc.1.iflib.rxq0.rxq_fl0.pidx: 985 +dev.igc.1.iflib.rxq0.rxq_fl0.cidx: 7 +dev.igc.1.iflib.rxq0.rxq_fl0.pidx: 6 dev.igc.1.iflib.rxq0.cpu: 0 dev.igc.1.iflib.txq3.r_abdications: 0 dev.igc.1.iflib.txq3.r_restarts: 0 @@ -144,7 +144,7 @@ dev.igc.1.iflib.txq3.r_starts: 0 dev.igc.1.iflib.txq3.r_drops: 0 dev.igc.1.iflib.txq3.r_enqueues: 0 -dev.igc.1.iflib.txq3.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE +dev.igc.1.iflib.txq3.ring_state: pidx_head: 0306 pidx_tail: 0306 cidx: 0306 state: IDLE dev.igc.1.iflib.txq3.txq_cleaned: 0 dev.igc.1.iflib.txq3.txq_processed: 0 dev.igc.1.iflib.txq3.txq_in_use: 0 @@ -162,16 +162,16 @@ dev.igc.1.iflib.txq2.r_abdications: 0 dev.igc.1.iflib.txq2.r_restarts: 0 dev.igc.1.iflib.txq2.r_stalls: 0 -dev.igc.1.iflib.txq2.r_starts: 13 +dev.igc.1.iflib.txq2.r_starts: 4550374 dev.igc.1.iflib.txq2.r_drops: 0 -dev.igc.1.iflib.txq2.r_enqueues: 13 -dev.igc.1.iflib.txq2.ring_state: pidx_head: 0013 pidx_tail: 0013 cidx: 0013 state: IDLE -dev.igc.1.iflib.txq2.txq_cleaned: 0 -dev.igc.1.iflib.txq2.txq_processed: 24 -dev.igc.1.iflib.txq2.txq_in_use: 28 -dev.igc.1.iflib.txq2.txq_cidx_processed: 24 -dev.igc.1.iflib.txq2.txq_cidx: 0 -dev.igc.1.iflib.txq2.txq_pidx: 28 +dev.igc.1.iflib.txq2.r_enqueues: 4550374 +dev.igc.1.iflib.txq2.ring_state: pidx_head: 1224 pidx_tail: 1224 cidx: 1224 state: IDLE +dev.igc.1.iflib.txq2.txq_cleaned: 9100706 +dev.igc.1.iflib.txq2.txq_processed: 9100746 +dev.igc.1.iflib.txq2.txq_in_use: 42 +dev.igc.1.iflib.txq2.txq_cidx_processed: 458 +dev.igc.1.iflib.txq2.txq_cidx: 418 +dev.igc.1.iflib.txq2.txq_pidx: 460 dev.igc.1.iflib.txq2.no_tx_dma_setup: 0 dev.igc.1.iflib.txq2.txd_encap_efbig: 0 dev.igc.1.iflib.txq2.tx_map_failed: 0 @@ -186,7 +186,7 @@ dev.igc.1.iflib.txq1.r_starts: 0 dev.igc.1.iflib.txq1.r_drops: 0 dev.igc.1.iflib.txq1.r_enqueues: 0 -dev.igc.1.iflib.txq1.ring_state: pidx_head: 0000 pidx_tail: 0000 cidx: 0000 state: IDLE +dev.igc.1.iflib.txq1.ring_state: pidx_head: 0958 pidx_tail: 0958 cidx: 0958 state: IDLE dev.igc.1.iflib.txq1.txq_cleaned: 0 dev.igc.1.iflib.txq1.txq_processed: 0 dev.igc.1.iflib.txq1.txq_in_use: 0 @@ -202,18 +202,18 @@ dev.igc.1.iflib.txq1.mbuf_defrag: 0 dev.igc.1.iflib.txq1.cpu: 1 dev.igc.1.iflib.txq0.r_abdications: 0 -dev.igc.1.iflib.txq0.r_restarts: 324218 -dev.igc.1.iflib.txq0.r_stalls: 324218 -dev.igc.1.iflib.txq0.r_starts: 95287 +dev.igc.1.iflib.txq0.r_restarts: 0 +dev.igc.1.iflib.txq0.r_stalls: 0 +dev.igc.1.iflib.txq0.r_starts: 7 dev.igc.1.iflib.txq0.r_drops: 0 -dev.igc.1.iflib.txq0.r_enqueues: 4031110 -dev.igc.1.iflib.txq0.ring_state: pidx_head: 0646 pidx_tail: 0646 cidx: 0646 state: IDLE -dev.igc.1.iflib.txq0.txq_cleaned: 17253402 -dev.igc.1.iflib.txq0.txq_processed: 17253442 -dev.igc.1.iflib.txq0.txq_in_use: 43 -dev.igc.1.iflib.txq0.txq_cidx_processed: 66 -dev.igc.1.iflib.txq0.txq_cidx: 26 -dev.igc.1.iflib.txq0.txq_pidx: 69 +dev.igc.1.iflib.txq0.r_enqueues: 7 +dev.igc.1.iflib.txq0.ring_state: pidx_head: 0005 pidx_tail: 0005 cidx: 0005 state: IDLE +dev.igc.1.iflib.txq0.txq_cleaned: 0 +dev.igc.1.iflib.txq0.txq_processed: 11 +dev.igc.1.iflib.txq0.txq_in_use: 15 +dev.igc.1.iflib.txq0.txq_cidx_processed: 11 +dev.igc.1.iflib.txq0.txq_cidx: 0 +dev.igc.1.iflib.txq0.txq_pidx: 15 dev.igc.1.iflib.txq0.no_tx_dma_setup: 0 dev.igc.1.iflib.txq0.txd_encap_efbig: 0 dev.igc.1.iflib.txq0.tx_map_failed: 0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a9935ba0-9cb2-5a41-ca73-b6962fef5e4d>