From nobody Sun Aug 2 04:27:38 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hCRcC2TgXz6n6Bd for ; Sun, 02 Aug 2026 04:27:39 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hCRcC04n2z4FTk for ; Sun, 02 Aug 2026 04:27:39 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1785644859; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=qGQP13dPfiE8LvojIfMrSis8X5DVCB/8bTwZOK8NTAY=; b=EXhwyfzMkV97BrTYxAt67tsZGhiHjoSmRG1t8mLfXQnRL8VQJRhOy9WQrBXBC00XW8NBvE 7TOUmXeHOPsXyMzcjM8NqBE6Wnxe6g+TSltZhYojjey19d/PXQqTfqiGxY3PeKvtYGZhdS DlpFoPR5RsUN2L3LLllgWBHR7D4kpq7BtvmIoL5JJSvZHBuXWAaWMI78BMSwxulrO1a23F SmqOzdOWNRgl4k9W8pl7RB5x5oRiMoaBMvGAdKz0QzOYasHc2mkiqAOt+V8SIflHZQsxsl RP4NplW+sT0prZrE7psMD87Pg2EexXphqOz8vkemTq4paAJb0UwFPfRERX6D/w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1785644859; a=rsa-sha256; cv=none; b=jhHnl8hjrBTp+pKRoddhtpLpeobolH+IDzkVY3ahPS1EQv75u4txdV9VWkQ8nlyrvGaBmO f4klgw04be4xXkKgv/xTXoOATU8N0CxO93HzlQieYZg0moeu9DpaLvQ1K/IQvkZj8NYTmx yYiLW5OhD+D8t5ogdam2nkRBqak62xvnyBjSwAa5sZ/tFZmAHhIxwSeDn2PNR/QTUJbEb/ rIB3he0eHzTGYTxRycM0xUPuw486qI9/2sWhuL9F3yMbrRl4Gw4J4EYsy7UZMD54vcy3MT 5Fs0joiQRsffhGzrFACKhld3yBRN+OmxE0N1rlHCBJU6XQKFDJdFg/z+jq2ZXA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1785644859; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=qGQP13dPfiE8LvojIfMrSis8X5DVCB/8bTwZOK8NTAY=; b=WNaTUp7gvIhzwTsQ3C6z8NQe+mLuf7fX116E/DsEkoFK5ilV8AMa0QDxjAZzgn9H0Fda4s myRGOlf/tPBtoy9ZibqvWh3UnWr66uS1n76Gq1YSB61diUhjdrLUNgAqF1IvrSaug6CNrE 5zDh3ITyiselKK2Z+dT6YWs/M4zLueqEkm/9EdGrH3z1apprQCezNJkLy7mN6tMkt/XZRX G6C4vZ4rFns/765Amc/q3D9fiTPuOFjaLcvTrOqe2iYsYuuWuzo0D22SRvujpv43op5sZQ xiMdXQUPQXYp8jPTu5I199Ml91rPYQ0if3oQ82QygMs8eBWyQ0uPJiJlP3hLNA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hCRcB67Nxz1DKw for ; Sun, 02 Aug 2026 04:27:38 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 3c30e by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Sun, 02 Aug 2026 04:27:38 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Kevin Bowling Subject: git: 7db1936ca0df - stable/14 - igc: make AIM counter sampling coherent List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 7db1936ca0df93ed2db2fc42e9619e33222202a8 Auto-Submitted: auto-generated Date: Sun, 02 Aug 2026 04:27:38 +0000 Message-Id: <6a6ec73a.3c30e.2364b596@gitrepo.freebsd.org> The branch stable/14 has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=7db1936ca0df93ed2db2fc42e9619e33222202a8 commit 7db1936ca0df93ed2db2fc42e9619e33222202a8 Author: Kevin Bowling AuthorDate: 2026-07-25 23:57:42 +0000 Commit: Kevin Bowling CommitDate: 2026-08-02 04:26:34 +0000 igc: make AIM counter sampling coherent Sample free-running counters by delta instead of clearing them from the interrupt filter, which can race their producers. Publish byte and packet counts together at the TX and RX doorbells so each sample is coherent. Aggregate every TX ring assigned to the interrupt vector so unequal RX and TX queue counts are safe. Count RX bytes only after a frame is accepted. (cherry picked from commit 2290ea7f4311e899019fe77bf7c7775033af6b24) --- sys/dev/igc/if_igc.c | 97 +++++++++++++++++++++++++++++++++++++------------- sys/dev/igc/if_igc.h | 45 ++++++++++++++++++++--- sys/dev/igc/igc_txrx.c | 4 ++- 3 files changed, 116 insertions(+), 30 deletions(-) diff --git a/sys/dev/igc/if_igc.c b/sys/dev/igc/if_igc.c index 8c9ec0566a38..78a97660649c 100644 --- a/sys/dev/igc/if_igc.c +++ b/sys/dev/igc/if_igc.c @@ -144,7 +144,7 @@ static int igc_get_rs(SYSCTL_HANDLER_ARGS); static void igc_print_debug_info(struct igc_softc *); static int igc_is_valid_ether_addr(u8 *); static void igc_neweitr(struct igc_softc *, struct igc_rx_queue *, - struct tx_ring *, struct rx_ring *); + struct rx_ring *); static int igc_sysctl_tso_tcp_flags_mask(SYSCTL_HANDLER_ARGS); /* Management and WOL Support */ static void igc_get_hw_control(struct igc_softc *); @@ -893,6 +893,47 @@ igc_if_init(if_ctx_t ctx) igc_set_eee_i225(&sc->hw, true, true, true); } +/* + * RX publishes its byte and packet counters as one snapshot when iflib + * returns descriptors to hardware. This also covers watchdog-driven RX + * processing, which can run while the interrupt vector is unmasked. + */ +static __inline void +igc_aim_rx_delta(struct rx_ring *rxr, u32 *bytes, u32 *packets) +{ + uint64_t snapshot; + u32 now_bytes, now_packets; + + snapshot = atomic_load_acq_64(&rxr->rx_aim_snapshot); + now_bytes = snapshot >> 32; + now_packets = (u32)snapshot; + *bytes = now_bytes - rxr->rx_bytes_last; + *packets = now_packets - rxr->rx_packets_last; + rxr->rx_bytes_last = now_bytes; + rxr->rx_packets_last = now_packets; +} + +/* + * TX publishes its byte and packet counters as one snapshot at the doorbell, + * because encapsulation can overlap the interrupt filter. The two halves + * remain independent free running u32 counters, so their deltas are correct + * across wrap. + */ +static __inline void +igc_aim_tx_delta(struct tx_ring *txr, u32 *bytes, u32 *packets) +{ + uint64_t snapshot; + u32 now_bytes, now_packets; + + snapshot = atomic_load_acq_64(&txr->tx_aim_snapshot); + now_bytes = snapshot >> 32; + now_packets = (u32)snapshot; + *bytes = now_bytes - txr->tx_bytes_last; + *packets = now_packets - txr->tx_packets_last; + txr->tx_bytes_last = now_bytes; + txr->tx_packets_last = now_packets; +} + enum eitr_latency_target { eitr_latency_disabled = 0, eitr_latency_lowest = 1, @@ -906,16 +947,32 @@ enum eitr_latency_target { *********************************************************************/ static void igc_neweitr(struct igc_softc *sc, struct igc_rx_queue *que, - struct tx_ring *txr, struct rx_ring *rxr) + struct rx_ring *rxr) { struct igc_hw *hw = &sc->hw; - unsigned long bytes, bytes_per_packet, packets; - unsigned long rxbytes, rxpackets, txbytes, txpackets; + struct igc_tx_queue *tx_que; + u32 bytes, bytes_per_packet, packets; + u32 ringbytes, ringpackets, rxbytes, rxpackets, txbytes, txpackets; u32 neweitr; u8 nextlatency; + int i; - rxbytes = atomic_load_long(&rxr->rx_bytes); - txbytes = atomic_load_long(&txr->tx_bytes); + igc_aim_rx_delta(rxr, &rxbytes, &rxpackets); + + /* + * A vector can service more than one TX ring when iflib is configured + * with unequal RX and TX queue counts. Sample every ring routed to + * this vector rather than treating the vector as a TX queue index. + */ + txbytes = txpackets = 0; + for (i = 0; i < sc->tx_num_queues; i++) { + tx_que = &sc->tx_queues[i]; + if (tx_que->msix != que->msix) + continue; + igc_aim_tx_delta(&tx_que->txr, &ringbytes, &ringpackets); + txbytes += ringbytes; + txpackets += ringpackets; + } /* Idle, do nothing */ if (txbytes == 0 && rxbytes == 0) @@ -938,15 +995,13 @@ igc_neweitr(struct igc_softc *sc, struct igc_rx_queue *que, goto igc_set_next_eitr; } - bytes = bytes_per_packet = 0; + bytes = bytes_per_packet = packets = 0; /* Get largest values from the associated tx and rx ring */ - txpackets = atomic_load_long(&txr->tx_packets); if (txpackets != 0) { bytes = txbytes; bytes_per_packet = txbytes / txpackets; packets = txpackets; } - rxpackets = atomic_load_long(&rxr->rx_packets); if (rxpackets != 0) { bytes = lmax(bytes, rxbytes); bytes_per_packet = @@ -1048,7 +1103,6 @@ igc_intr(void *arg) struct igc_softc *sc = arg; struct igc_hw *hw = &sc->hw; struct igc_rx_queue *que = &sc->rx_queues[0]; - struct tx_ring *txr = &sc->tx_queues[0].txr; struct rx_ring *rxr = &que->rxr; if_ctx_t ctx = sc->ctx; u32 reg_icr; @@ -1081,13 +1135,7 @@ igc_intr(void *arg) if (reg_icr & IGC_ICR_RXO) sc->rx_overruns++; - igc_neweitr(sc, que, txr, rxr); - - /* Reset state */ - txr->tx_bytes = 0; - txr->tx_packets = 0; - rxr->rx_bytes = 0; - rxr->rx_packets = 0; + igc_neweitr(sc, que, rxr); return (FILTER_SCHEDULE_THREAD); } @@ -1122,18 +1170,11 @@ igc_msix_que(void *arg) { struct igc_rx_queue *que = arg; struct igc_softc *sc = que->sc; - struct tx_ring *txr = &sc->tx_queues[que->msix].txr; struct rx_ring *rxr = &que->rxr; ++que->irqs; - igc_neweitr(sc, que, txr, rxr); - - /* Reset state */ - txr->tx_bytes = 0; - txr->tx_packets = 0; - rxr->rx_bytes = 0; - rxr->rx_packets = 0; + igc_neweitr(sc, que, rxr); return (FILTER_SCHEDULE_THREAD); } @@ -2027,6 +2068,9 @@ igc_if_tx_queues_alloc(if_ctx_t ctx, caddr_t *vaddrs, uint64_t *paddrs, /* Set up some basics */ struct tx_ring *txr = &que->txr; + KASSERT(__is_aligned(&txr->tx_aim_snapshot, sizeof(uint64_t)), + ("%s: misaligned TX AIM snapshot %p", __func__, + &txr->tx_aim_snapshot)); txr->sc = que->sc = sc; que->me = txr->me = i; @@ -2079,6 +2123,9 @@ igc_if_rx_queues_alloc(if_ctx_t ctx, caddr_t *vaddrs, uint64_t *paddrs, for (i = 0, que = sc->rx_queues; i < nrxqsets; i++, que++) { /* Set up some basics */ struct rx_ring *rxr = &que->rxr; + KASSERT(__is_aligned(&rxr->rx_aim_snapshot, sizeof(uint64_t)), + ("%s: misaligned RX AIM snapshot %p", __func__, + &rxr->rx_aim_snapshot)); rxr->sc = que->sc = sc; rxr->que = que; que->me = rxr->me = i; diff --git a/sys/dev/igc/if_igc.h b/sys/dev/igc/if_igc.h index 236a16c4add3..d8d5d5b251fa 100644 --- a/sys/dev/igc/if_igc.h +++ b/sys/dev/igc/if_igc.h @@ -232,9 +232,18 @@ struct tx_ring { /* Soft stats */ unsigned long tx_irq; - unsigned long tx_packets; - unsigned long tx_bytes; + /* + * Free running AIM counters. The producer updates these while + * encapsulating packets, then publishes both together at the TX + * doorbell. The interrupt handler samples only the published value, + * so it cannot observe one counter without the other. + */ + u32 tx_packets; + u32 tx_bytes; + uint64_t tx_aim_snapshot __aligned(8); + u32 tx_packets_last; + u32 tx_bytes_last; /* Saved csum offloading context information */ int csum_flags; @@ -249,6 +258,15 @@ struct tx_ring { uint32_t csum_txd_lower; /* last field */ }; +static __inline void +igc_aim_publish(struct tx_ring *txr) +{ + uint64_t snapshot; + + snapshot = ((uint64_t)txr->tx_bytes << 32) | txr->tx_packets; + atomic_store_rel_64(&txr->tx_aim_snapshot, snapshot); +} + /* * The Receive ring, one per rx queue */ @@ -267,13 +285,32 @@ struct rx_ring { /* Soft stats */ unsigned long rx_irq; unsigned long rx_discarded; - unsigned long rx_packets; - unsigned long rx_bytes; + + /* + * Free running AIM counters. RX publishes both together when iflib + * returns descriptors to hardware. The interrupt handler samples only + * the published value, so watchdog-driven RX processing cannot expose + * one counter without the other. + */ + u32 rx_packets; + u32 rx_bytes; + uint64_t rx_aim_snapshot __aligned(8); + u32 rx_packets_last; + u32 rx_bytes_last; /* Next requested EITR latency */ u8 rx_nextlatency; }; +static __inline void +igc_aim_publish_rx(struct rx_ring *rxr) +{ + uint64_t snapshot; + + snapshot = ((uint64_t)rxr->rx_bytes << 32) | rxr->rx_packets; + atomic_store_rel_64(&rxr->rx_aim_snapshot, snapshot); +} + struct igc_tx_queue { struct igc_softc *sc; u32 msix; diff --git a/sys/dev/igc/igc_txrx.c b/sys/dev/igc/igc_txrx.c index d6d383b40127..905b859e843f 100644 --- a/sys/dev/igc/igc_txrx.c +++ b/sys/dev/igc/igc_txrx.c @@ -334,6 +334,7 @@ igc_isc_txd_flush(void *arg, uint16_t txqid, qidx_t pidx) struct tx_ring *txr = &que->txr; IGC_WRITE_REG(&sc->hw, IGC_TDT(txr->me), pidx); + igc_aim_publish(txr); } static int @@ -425,6 +426,7 @@ igc_isc_rxd_flush(void *arg, uint16_t rxqid, uint8_t flid __unused, struct rx_ring *rxr = &que->rxr; IGC_WRITE_REG(&sc->hw, IGC_RDT(rxr->me), pidx); + igc_aim_publish_rx(rxr); } static int @@ -488,7 +490,6 @@ igc_isc_rxd_pkt_get(void *arg, if_rxd_info_t ri) le32toh(rxd->wb.lower.lo_dword.data) & IGC_PKTTYPE_MASK; ri->iri_len += len; - rxr->rx_bytes += len; rxd->wb.upper.status_error = 0; eop = ((staterr & IGC_RXD_STAT_EOP) == IGC_RXD_STAT_EOP); @@ -516,6 +517,7 @@ igc_isc_rxd_pkt_get(void *arg, if_rxd_info_t ri) i++; } while (!eop); + rxr->rx_bytes += ri->iri_len; rxr->rx_packets++; if ((scctx->isc_capenable & IFCAP_RXCSUM) != 0)