From nobody Sun Aug 2 04:22:57 2026 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hCRVp1H4cz6n62K for ; Sun, 02 Aug 2026 04:22:58 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hCRVn5t7cz49LC for ; Sun, 02 Aug 2026 04:22:57 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1785644577; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=HGVLA0Yi38ttZJXiVhxK8ApTopoI8zVh7QXFd2vSWQ8=; b=yljrstE8P6rUiiBI4pShYfd0CLoa3NsGCtz9Nkr+e2OfrxKJvvpVZ7Iwv7puYkqA7lyNxw 5DBPINZYu4Ft9QKF5XjbuXLNlMWGtSCDA26zO8KfnVtei5GdQ9u338J0Wlkv4qZCpF2glx EQ6oj2nN1fY9l7UDEWwLZZ8e5y9XOCTx++l95Pzxj5V8MacsFAfXQLeJSXdYIyoDgt6yqA pwcxey3S7oe5YJGR1ZYAC+2DVzXDYw3/YrBaOanuurQ6DbHvP3voM8MX3hoA6H3qTBv3Vr BmA0A00NLqFbPCWrljGEtd67n1iRgB5qgZAi8FWaNC8dz1eFDw3eFkHLbqX4HQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1785644577; a=rsa-sha256; cv=none; b=aLTAOZHP0I7vJq2AKMJIoQYkJS9uaGVoWSgMeixaelYohxzik642Stp358DehikBdY36hA pymsnUcnZc4CzI0F0ceckx5mjO7NUuYDGQgbae+vUWBEgPlP+CtCxNlznrGUb1USGSM58m mwCoHyJxzaPcGXC+KHMbW18m3tJ87GINl+OH6Qfrh/EY6oeFzyo+90D0xeeZj4hsOYWBV+ QE5kpuvzsjxLyk6woZeb3rlcBgrgTW//PdqD5xxyb5eZcbTFyR1QelnLYiDFVDQ9AgTGBG 1losFp1U7UwcDY5GvHG1vA0Jz+QjstLIO/fzKVrCRCmA0iAc2mhsRyPWakQ3xw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1785644577; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=HGVLA0Yi38ttZJXiVhxK8ApTopoI8zVh7QXFd2vSWQ8=; b=p5bgXvursB1WkvUnMeh3bFw22EzTZWYdqIGE6HcSUcMK85Xf94ae4j9q3UcV3hkrPpBY0C LdEvP/j5W4KTzHCt4Oum8Y/VGxfmjU4TPxCmpWhhGCZwrd0+Q407sUnbUy6wQBEBG11i7u 4KV7Ih6IZwaGn54aYK1Z9+OOMHNPqCxDUwCEzx8Irix8c0e4WojniNR9n2TRbHhhfdAd8x nyCrkkPWsSyfV2QDF55zq/8oh1BZuLI85a4STAuFbH5U4HjDHyQj2K7W297w/5Mq0TGSFN pzNWBnDlelMxMuuWJb6VF8lVSUfjzRN1hy1I0sR7daMMiS2Iij/XwUsyxCoYbA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hCRVn3vyyz1DTw for ; Sun, 02 Aug 2026 04:22:57 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 3c89e by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Sun, 02 Aug 2026 04:22:57 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Kevin Bowling Subject: git: e8cf1bfe07b0 - stable/15 - e1000: make AIM counter sampling coherent List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kbowling X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: e8cf1bfe07b0743adf6c9aa54309efde64a9dfa2 Auto-Submitted: auto-generated Date: Sun, 02 Aug 2026 04:22:57 +0000 Message-Id: <6a6ec621.3c89e.86da4fe@gitrepo.freebsd.org> The branch stable/15 has been updated by kbowling: URL: https://cgit.FreeBSD.org/src/commit/?id=e8cf1bfe07b0743adf6c9aa54309efde64a9dfa2 commit e8cf1bfe07b0743adf6c9aa54309efde64a9dfa2 Author: Kevin Bowling AuthorDate: 2026-07-25 23:48:15 +0000 Commit: Kevin Bowling CommitDate: 2026-08-02 04:19:46 +0000 e1000: make AIM counter sampling coherent Sample free-running counters by delta instead of clearing them from the interrupt filter, which can race their producers. Publish byte and packet counts together at the TX and RX doorbells so each sample is coherent. Aggregate every TX ring assigned to the interrupt vector so unequal RX and TX queue counts are safe. Count RX bytes only after a frame is accepted. (cherry picked from commit bc5e7b0cbbb555ffebc7d73b273c421f9ee24c23) --- sys/dev/e1000/em_txrx.c | 8 +++- sys/dev/e1000/if_em.c | 97 +++++++++++++++++++++++++++++++++++------------- sys/dev/e1000/if_em.h | 46 +++++++++++++++++++++-- sys/dev/e1000/igb_txrx.c | 4 +- 4 files changed, 123 insertions(+), 32 deletions(-) diff --git a/sys/dev/e1000/em_txrx.c b/sys/dev/e1000/em_txrx.c index cfeb43036563..889c36c3fb84 100644 --- a/sys/dev/e1000/em_txrx.c +++ b/sys/dev/e1000/em_txrx.c @@ -478,6 +478,8 @@ em_isc_txd_flush(void *arg, uint16_t txqid, qidx_t pidx) struct tx_ring *txr = &que->txr; E1000_WRITE_REG(&sc->hw, E1000_TDT(txr->me), pidx); + if (sc->hw.mac.type >= e1000_82540) + em_aim_publish(txr); } static int @@ -603,6 +605,8 @@ em_isc_rxd_flush(void *arg, uint16_t rxqid, uint8_t flid __unused, struct rx_ring *rxr = &que->rxr; E1000_WRITE_REG(&sc->hw, E1000_RDT(rxr->me), pidx); + if (sc->hw.mac.type >= e1000_82540) + em_aim_publish_rx(rxr); } static int @@ -681,7 +685,6 @@ lem_isc_rxd_pkt_get(void *arg, if_rxd_info_t ri) len = le16toh(rxd->length); ri->iri_len += len; - rxr->rx_bytes += len; eop = (status & E1000_RXD_STAT_EOP) != 0; @@ -703,6 +706,7 @@ lem_isc_rxd_pkt_get(void *arg, if_rxd_info_t ri) i++; } while (!eop); + rxr->rx_bytes += ri->iri_len; rxr->rx_packets++; if (scctx->isc_capenable & IFCAP_RXCSUM) @@ -747,7 +751,6 @@ em_isc_rxd_pkt_get(void *arg, if_rxd_info_t ri) len = le16toh(rxd->wb.upper.length); ri->iri_len += len; - rxr->rx_bytes += len; eop = (staterr & E1000_RXD_STAT_EOP) != 0; @@ -768,6 +771,7 @@ em_isc_rxd_pkt_get(void *arg, if_rxd_info_t ri) i++; } while (!eop); + rxr->rx_bytes += ri->iri_len; rxr->rx_packets++; if (scctx->isc_capenable & IFCAP_RXCSUM) diff --git a/sys/dev/e1000/if_em.c b/sys/dev/e1000/if_em.c index 238d089c9018..fbf02c92e7a2 100644 --- a/sys/dev/e1000/if_em.c +++ b/sys/dev/e1000/if_em.c @@ -462,7 +462,7 @@ static int em_get_rs(SYSCTL_HANDLER_ARGS); static void em_print_debug_info(struct e1000_softc *); static int em_is_valid_ether_addr(u8 *); static void em_newitr(struct e1000_softc *, struct em_rx_queue *, - struct tx_ring *, struct rx_ring *); + struct rx_ring *); static bool em_automask_tso(if_ctx_t); static int em_sysctl_tso_tcp_flags_mask(SYSCTL_HANDLER_ARGS); static int em_sysctl_int_delay(SYSCTL_HANDLER_ARGS); @@ -1641,6 +1641,47 @@ em_if_init(if_ctx_t ctx) } } +/* + * RX publishes its byte and packet counters as one snapshot when iflib + * returns descriptors to hardware. This also covers watchdog-driven RX + * processing, which can run while the interrupt vector is unmasked. + */ +static __inline void +em_aim_rx_delta(struct rx_ring *rxr, u32 *bytes, u32 *packets) +{ + uint64_t snapshot; + u32 now_bytes, now_packets; + + snapshot = atomic_load_acq_64(&rxr->rx_aim_snapshot); + now_bytes = snapshot >> 32; + now_packets = (u32)snapshot; + *bytes = now_bytes - rxr->rx_bytes_last; + *packets = now_packets - rxr->rx_packets_last; + rxr->rx_bytes_last = now_bytes; + rxr->rx_packets_last = now_packets; +} + +/* + * TX publishes its byte and packet counters as one snapshot at the doorbell, + * because encapsulation can overlap the interrupt filter. The two halves + * remain independent free running u32 counters, so their deltas are correct + * across wrap. + */ +static __inline void +em_aim_tx_delta(struct tx_ring *txr, u32 *bytes, u32 *packets) +{ + uint64_t snapshot; + u32 now_bytes, now_packets; + + snapshot = atomic_load_acq_64(&txr->tx_aim_snapshot); + now_bytes = snapshot >> 32; + now_packets = (u32)snapshot; + *bytes = now_bytes - txr->tx_bytes_last; + *packets = now_packets - txr->tx_packets_last; + txr->tx_bytes_last = now_bytes; + txr->tx_packets_last = now_packets; +} + enum itr_latency_target { itr_latency_disabled = 0, itr_latency_lowest = 1, @@ -1654,16 +1695,32 @@ enum itr_latency_target { *********************************************************************/ static void em_newitr(struct e1000_softc *sc, struct em_rx_queue *que, - struct tx_ring *txr, struct rx_ring *rxr) + struct rx_ring *rxr) { struct e1000_hw *hw = &sc->hw; - unsigned long bytes, bytes_per_packet, packets; - unsigned long rxbytes, rxpackets, txbytes, txpackets; + struct em_tx_queue *tx_que; + u32 bytes, bytes_per_packet, packets; + u32 ringbytes, ringpackets, rxbytes, rxpackets, txbytes, txpackets; u32 newitr; u8 nextlatency; + int i; - rxbytes = atomic_load_long(&rxr->rx_bytes); - txbytes = atomic_load_long(&txr->tx_bytes); + em_aim_rx_delta(rxr, &rxbytes, &rxpackets); + + /* + * A vector can service more than one TX ring when iflib is configured + * with unequal RX and TX queue counts. Sample every ring routed to + * this vector rather than treating the vector as a TX queue index. + */ + txbytes = txpackets = 0; + for (i = 0; i < sc->tx_num_queues; i++) { + tx_que = &sc->tx_queues[i]; + if (tx_que->msix != que->msix) + continue; + em_aim_tx_delta(&tx_que->txr, &ringbytes, &ringpackets); + txbytes += ringbytes; + txpackets += ringpackets; + } /* Idle, do nothing */ if (txbytes == 0 && rxbytes == 0) @@ -1686,15 +1743,13 @@ em_newitr(struct e1000_softc *sc, struct em_rx_queue *que, goto em_set_next_itr; } - bytes = bytes_per_packet = 0; + bytes = bytes_per_packet = packets = 0; /* Get largest values from the associated tx and rx ring */ - txpackets = atomic_load_long(&txr->tx_packets); if (txpackets != 0) { bytes = txbytes; bytes_per_packet = txbytes / txpackets; packets = txpackets; } - rxpackets = atomic_load_long(&rxr->rx_packets); if (rxpackets != 0) { bytes = lmax(bytes, rxbytes); bytes_per_packet = @@ -1816,7 +1871,6 @@ em_intr(void *arg) struct e1000_softc *sc = arg; struct e1000_hw *hw = &sc->hw; struct em_rx_queue *que = &sc->rx_queues[0]; - struct tx_ring *txr = &sc->tx_queues[0].txr; struct rx_ring *rxr = &que->rxr; if_ctx_t ctx = sc->ctx; u32 reg_icr; @@ -1855,13 +1909,7 @@ em_intr(void *arg) sc->rx_overruns++; if (hw->mac.type >= e1000_82540) - em_newitr(sc, que, txr, rxr); - - /* Reset state */ - txr->tx_bytes = 0; - txr->tx_packets = 0; - rxr->rx_bytes = 0; - rxr->rx_packets = 0; + em_newitr(sc, que, rxr); return (FILTER_SCHEDULE_THREAD); } @@ -1916,18 +1964,11 @@ em_msix_que(void *arg) { struct em_rx_queue *que = arg; struct e1000_softc *sc = que->sc; - struct tx_ring *txr = &sc->tx_queues[que->msix].txr; struct rx_ring *rxr = &que->rxr; ++que->irqs; - em_newitr(sc, que, txr, rxr); - - /* Reset state */ - txr->tx_bytes = 0; - txr->tx_packets = 0; - rxr->rx_bytes = 0; - rxr->rx_packets = 0; + em_newitr(sc, que, rxr); return (FILTER_SCHEDULE_THREAD); } @@ -3547,6 +3588,9 @@ em_if_tx_queues_alloc(if_ctx_t ctx, caddr_t *vaddrs, uint64_t *paddrs, /* Set up some basics */ struct tx_ring *txr = &que->txr; + KASSERT(__is_aligned(&txr->tx_aim_snapshot, sizeof(uint64_t)), + ("%s: misaligned TX AIM snapshot %p", __func__, + &txr->tx_aim_snapshot)); txr->sc = que->sc = sc; que->me = txr->me = i; @@ -3600,6 +3644,9 @@ em_if_rx_queues_alloc(if_ctx_t ctx, caddr_t *vaddrs, uint64_t *paddrs, for (i = 0, que = sc->rx_queues; i < nrxqsets; i++, que++) { /* Set up some basics */ struct rx_ring *rxr = &que->rxr; + KASSERT(__is_aligned(&rxr->rx_aim_snapshot, sizeof(uint64_t)), + ("%s: misaligned RX AIM snapshot %p", __func__, + &rxr->rx_aim_snapshot)); rxr->sc = que->sc = sc; rxr->que = que; que->me = rxr->me = i; diff --git a/sys/dev/e1000/if_em.h b/sys/dev/e1000/if_em.h index 59d3acca33c9..1ed797b2c0c2 100644 --- a/sys/dev/e1000/if_em.h +++ b/sys/dev/e1000/if_em.h @@ -410,8 +410,18 @@ struct tx_ring { /* Soft stats */ unsigned long tx_irq; - unsigned long tx_packets; - unsigned long tx_bytes; + + /* + * Free running AIM counters. The producer updates these while + * encapsulating packets, then publishes both together at the TX + * doorbell. The interrupt handler samples only the published value, + * so it cannot observe one counter without the other. + */ + u32 tx_packets; + u32 tx_bytes; + uint64_t tx_aim_snapshot __aligned(8); + u32 tx_packets_last; + u32 tx_bytes_last; /* Saved csum offloading context information */ int csum_flags; @@ -426,6 +436,15 @@ struct tx_ring { uint32_t csum_txd_lower; /* last field */ }; +static __inline void +em_aim_publish(struct tx_ring *txr) +{ + uint64_t snapshot; + + snapshot = ((uint64_t)txr->tx_bytes << 32) | txr->tx_packets; + atomic_store_rel_64(&txr->tx_aim_snapshot, snapshot); +} + /* * The Receive ring, one per rx queue */ @@ -445,13 +464,32 @@ struct rx_ring { /* Soft stats */ unsigned long rx_irq; unsigned long rx_discarded; - unsigned long rx_packets; - unsigned long rx_bytes; + + /* + * Free running AIM counters. RX publishes both together when iflib + * returns descriptors to hardware. The interrupt handler samples only + * the published value, so watchdog-driven RX processing cannot expose + * one counter without the other. + */ + u32 rx_packets; + u32 rx_bytes; + uint64_t rx_aim_snapshot __aligned(8); + u32 rx_packets_last; + u32 rx_bytes_last; /* Next requested ITR latency */ u8 rx_nextlatency; }; +static __inline void +em_aim_publish_rx(struct rx_ring *rxr) +{ + uint64_t snapshot; + + snapshot = ((uint64_t)rxr->rx_bytes << 32) | rxr->rx_packets; + atomic_store_rel_64(&rxr->rx_aim_snapshot, snapshot); +} + struct em_tx_queue { struct e1000_softc *sc; u32 msix; diff --git a/sys/dev/e1000/igb_txrx.c b/sys/dev/e1000/igb_txrx.c index 19b365bf020a..62cafff4010a 100644 --- a/sys/dev/e1000/igb_txrx.c +++ b/sys/dev/e1000/igb_txrx.c @@ -306,6 +306,7 @@ igb_isc_txd_flush(void *arg, uint16_t txqid, qidx_t pidx) struct tx_ring *txr = &que->txr; E1000_WRITE_REG(&sc->hw, E1000_TDT(txr->me), pidx); + em_aim_publish(txr); } static int @@ -397,6 +398,7 @@ igb_isc_rxd_flush(void *arg, uint16_t rxqid, uint8_t flid __unused, struct rx_ring *rxr = &que->rxr; E1000_WRITE_REG(&sc->hw, E1000_RDT(rxr->me), pidx); + em_aim_publish_rx(rxr); } static int @@ -460,7 +462,6 @@ igb_isc_rxd_pkt_get(void *arg, if_rxd_info_t ri) le32toh(rxd->wb.lower.lo_dword.data) & IGB_PKTTYPE_MASK; ri->iri_len += len; - rxr->rx_bytes += len; rxd->wb.upper.status_error = 0; eop = ((staterr & E1000_RXD_STAT_EOP) == E1000_RXD_STAT_EOP); @@ -489,6 +490,7 @@ igb_isc_rxd_pkt_get(void *arg, if_rxd_info_t ri) i++; } while (!eop); + rxr->rx_bytes += ri->iri_len; rxr->rx_packets++; if ((scctx->isc_capenable & IFCAP_RXCSUM) != 0)