From nobody Sat Aug 1 14:20:29 2026 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hC4pk2705z6n7C0 for ; Sat, 01 Aug 2026 14:20:30 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hC4pk0fSwz3S1D for ; Sat, 01 Aug 2026 14:20:30 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1785594030; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ImvFj8ErbOvH17SQCZUvhF1WKR+n6s2pnbaiFgGNIds=; b=ERZOOq5LxIBXKFWw7rydcd1C6fUJ8IG5N23QHwn2wRrurh0RinWVXnFR0fAs99urbwZYJ1 Xyk7oNOg1WNE2sypoAXp6ghWjWHhQIgDkVToEsdlZx6Y4JQN7fmXSvex/3nt3Z/Q8B2tm/ 3j0KNnM46SJZ+MrnUAdA9fLtg1LUDkl99EuIfnE+JehwDYdf56ySjhsi0ZP+MCmYbgt+JO ViK+9nlbFARQMP43/gq+DEmrd3SEE1g1k258LTXBTTfcf1T2bQ6ugjq7Fh5eD0mkPeBvfk J7nrI6FmpVmlVcGYxxpSSa/d9mxIzKhJaYGSjHPFWaFiNghHLgBWYhql37FRVQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1785594030; a=rsa-sha256; cv=none; b=yK2NEDFWGQd+pNnk/ejM48qbfYOQtSIn+f8RwivGDU2kh9Hnwum6aTwRqcloChZ1jmfH+T NCIkdTaMPXJSUbXcHhlzEY9ZyzE4PdxBnjQQ9g0xeWQEDk9a0Kiii7y+ozeAfHzoCiIfoz SIE/XFfsplTYRvHlBMseovYLWXjmH5vmps7gSEf94rqEpBhFHVraTJI9otAyuAjpR++zlm Wtpsqg/slBNi/3m0lFDnNElyEjKfPDQxasDYR0g29Ae9cy6tTkV49+ELv9Iw+LEbNfBWKM BWkgm8s+unczkMgQSKGAnbzbZTPPVip4Y6M6+tHsuDoDtqJkbC+T0EpaHkLhSg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1785594030; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=ImvFj8ErbOvH17SQCZUvhF1WKR+n6s2pnbaiFgGNIds=; b=B4eHHq93cMA+ALhBNoVULZCAFcGRXXVqJJDe9DsvcrZIaHBVRC9nA1b1wgealrkVSAzyeW yKWMPdq7clAqJtwEz2bhCKX4b+BPIHqvOso01hLlqD0f7vDvbcneh9QKzI5pA9IloxvlTO lr9E3qItvmUPYgn43XauqFWjbkwUzQ7oY4tyUhxOx/DU9SE4c75T1ieJMlmAFWK8azQKJz 2StVEa1RwLIE6v9JJKq4J1mY4/vhNfdHpttmjVY3QJ7Jy2GwImtZZMdVIPa8iJ5HK8PsCP DWKu3C77RMDbEzXLEtyFkk3TDcPsIvKquQhw/odOGkV+/pu+GJrQzSIKhlIVCQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hC4pj6byyzn93 for ; Sat, 01 Aug 2026 14:20:29 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 30dfe by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Sat, 01 Aug 2026 14:20:29 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org Cc: Gilad Ben Yakov From: Arthur Kiyanovski Subject: git: ff519eadf222 - stable/15 - ena: Fix false 'missing TX completions' warnings due to timestamp race List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: akiyano X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: ff519eadf2228d97818be080ee18faec95e606c4 Auto-Submitted: auto-generated Date: Sat, 01 Aug 2026 14:20:29 +0000 Message-Id: <6a6e00ad.30dfe.7767010b@gitrepo.freebsd.org> The branch stable/15 has been updated by akiyano: URL: https://cgit.FreeBSD.org/src/commit/?id=ff519eadf2228d97818be080ee18faec95e606c4 commit ff519eadf2228d97818be080ee18faec95e606c4 Author: Gilad Ben Yakov AuthorDate: 2026-06-21 12:45:38 +0000 Commit: Arthur Kiyanovski CommitDate: 2026-08-01 14:06:36 +0000 ena: Fix false 'missing TX completions' warnings due to timestamp race Sporadic 'Found a Tx that wasn't completed on time' warnings appear under sustained TX load, always reporting '1 msecs since last cleanup' despite the 5-second timeout threshold. The per-packet TX timestamp uses struct bintime (128 bits: two 64-bit fields sec and frac) which is read and written non-atomically. A race exists between the missing TX completion check (check_missing_comp_in_tx_queue reading the timestamp) and the TX submit path or cleanup path writing it on another CPU. Since the two fields are not updated atomically, the check can observe a partially written timestamp - one field from the old value and one from the new. This can produce a timestamp with {sec=0, frac=valid}, causing the check to compute a time offset equal to system uptime and falsely exceeding the 5-second timeout. Confirmed by instrumentation showing all occurrences had sec=0 with valid frac/mbuf, cleanup_running=0, and ticks==last_cleanup_ticks. Replace struct bintime with sbintime_t (a single 64-bit value) for tx_buf->timestamp. An aligned 64-bit store/load cannot be torn on 64-bit architectures. Additionally, snapshot the timestamp into a local variable in the check path to prevent a read-then-read race where the timestamp could be zeroed between the zero-check and the offset calculation. Testing: On m6i.large (FreeBSD 15.0-RELEASE-p6 amd64, 2 IO queues), two instances with MTU 1500. Ran iperf -P 20 -u -b 320kpps (CPU saturated at ~7 Gbps aggregate). Without the fix: 8 warnings in 6 hours (first at ~72 min). With the fix: 0 warnings after 20+ hours under identical conditions. Fixes: 9b8d05b8ac78 ("Add support for Amazon Elastic Network Adapter (ENA) NIC") Submitted by: Gilad Ben Yakov MFC after: 2 weeks Sponsored by: Amazon, Inc. Reviewed by: cperciva Differential Revision: https://reviews.freebsd.org/D58241 (cherry picked from commit 74bcb1151bb94279a4269fc842aa2be00800545e) --- sys/dev/ena/ena.c | 13 +++++++------ sys/dev/ena/ena.h | 2 +- sys/dev/ena/ena_datapath.c | 4 ++-- 3 files changed, 10 insertions(+), 9 deletions(-) diff --git a/sys/dev/ena/ena.c b/sys/dev/ena/ena.c index 486fafa7c045..8a3bb8f7960e 100644 --- a/sys/dev/ena/ena.c +++ b/sys/dev/ena/ena.c @@ -3213,7 +3213,7 @@ check_missing_comp_in_tx_queue(struct ena_adapter *adapter, { uint32_t missed_tx = 0, new_missed_tx = 0; device_t pdev = adapter->pdev; - struct bintime curtime, time; + sbintime_t curtime; struct ena_tx_buffer *tx_buf; int time_since_last_cleanup; int missing_tx_comp_to; @@ -3222,17 +3222,18 @@ check_missing_comp_in_tx_queue(struct ena_adapter *adapter, enum ena_regs_reset_reason_types reset_reason = ENA_REGS_RESET_MISS_TX_CMPL; bool cleanup_scheduled, cleanup_running; - getbinuptime(&curtime); + curtime = getsbinuptime(); for (i = 0; i < tx_ring->ring_size; i++) { + sbintime_t ts; + tx_buf = &tx_ring->tx_buffer_info[i]; - if (bintime_isset(&tx_buf->timestamp) == 0) + ts = atomic_load_64(&tx_buf->timestamp); + if (ts == 0) continue; - time = curtime; - bintime_sub(&time, &tx_buf->timestamp); - time_offset = bttosbt(time); + time_offset = curtime - ts; if (unlikely(!atomic_load_8(&tx_ring->first_interrupt) && time_offset > 2 * adapter->missing_tx_timeout)) { diff --git a/sys/dev/ena/ena.h b/sys/dev/ena/ena.h index ca5abe494e04..77cb8730627f 100644 --- a/sys/dev/ena/ena.h +++ b/sys/dev/ena/ena.h @@ -263,7 +263,7 @@ struct ena_tx_buffer { bus_dmamap_t dmamap; /* Used to detect missing tx packets */ - struct bintime timestamp; + sbintime_t timestamp; bool print_once; #ifdef DEV_NETMAP diff --git a/sys/dev/ena/ena_datapath.c b/sys/dev/ena/ena_datapath.c index 1ce5c2ae3e65..1fe1209a77d4 100644 --- a/sys/dev/ena/ena_datapath.c +++ b/sys/dev/ena/ena_datapath.c @@ -276,7 +276,7 @@ ena_tx_cleanup(struct ena_ring *tx_ring) mbuf = tx_info->mbuf; tx_info->mbuf = NULL; - bintime_clear(&tx_info->timestamp); + atomic_store_64(&tx_info->timestamp, 0); bus_dmamap_sync(adapter->tx_buf_tag, tx_info->dmamap, BUS_DMASYNC_POSTWRITE); @@ -1059,7 +1059,7 @@ ena_xmit_mbuf(struct ena_ring *tx_ring, struct mbuf **mbuf) counter_exit(); tx_info->tx_descs = nb_hw_desc; - getbinuptime(&tx_info->timestamp); + atomic_store_64(&tx_info->timestamp, getsbinuptime()); tx_info->print_once = true; tx_ring->next_to_use = ENA_TX_RING_IDX_NEXT(next_to_use,