From nobody Fri Oct 3 00:04:49 2025 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cd86p290Wz6BMyj; Fri, 03 Oct 2025 00:04:50 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cd86n4SzJz3t2F; Fri, 03 Oct 2025 00:04:49 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1759449889; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=RKJspgOhnGskwo4qSnV14URjlD4/nnGQhf9JeWrObDs=; b=CZvwsdPVmsLwdRK1gIKL5lpvOzj0V3zxudQIedj23Oh86ZIzXv1i/LFfRFREwy72O+40AM b7PyE7yYvWNIHzFRAUPoWj4epmoNrMoVZJySOpI8B7LJKIGG7nYe0SHjtu1Qpz6Ty8crQg e0CVrdh+HbmtKwqLsajA+gXdXeAzWtpAUt+OpGkgo3tMjNOE+1pV5/Q9qiLeLnrN4tAI+c XciguJqkd3OlXKC8I0QCjhPEK8+a5CIV+V3gQxr7nEDJgNd0WmBmovU6F0z4t0Mc8fyNHx ydksiurHK0RGd3AwQm6Pe7C+Hhai7BfIVqAMzRZFzmXCoKj3SxPvHXcKZ9GG3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1759449889; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=RKJspgOhnGskwo4qSnV14URjlD4/nnGQhf9JeWrObDs=; b=H5+mAVmdEb7Cwn+0rGp4fAWM3kespa1NM8jdA7N7HFtsIL+4F/XGGEu0P0GxTw0H/ehhxd 9KWSx6GXk8kpNFfbfe10SmvR2n3+/YROhCNkKypDGx1IZ+Jx1WbvLcVZ8o16T82+a2oCKa dmxGSlxxMgQxcAu9ICJUOacSqy4Eb9qORLrqk8HRc813RPYGNZyTPIO2MN7EHLRplASdQw A74Ox48ltQqYzl++RJq4BY0fovxwvP5FOA2wcr4p80LCMNkvx5JS7IuapWJoj90+Kfelg9 3SobSipTnB7e0OYM6QuFYSqa1EuLwPYhVfpvnN2pXdtwZ9KRvE9IyXGlMWRTBQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1759449889; a=rsa-sha256; cv=none; b=AgfM/tmV+suLR2G8Q6Cq9gZNIEmp/sTY5qVxR47eauayVEbvZ5TJjWZXhKmq4iO1PBWytL mR3R7afrPu7rvGT5UK8bFQeo5llOGlot6535jEPuaX9Rt97V8E0fw6KTCEtii0agyD9fFK QLxTqBDAiq7WH2QA6KmtbEeWnheW2S+jzlXYQzseiLYH8qEaEoXNDGUdkMLaV8HdDrCGQN 3jkqABenx5wBtUKudue0JrhwnfvOQEnAwkWUPJ4Yn2PvihiMzkst9MUzoNfkAUsJ0BDou5 oGhqU5cxaW8nmrT0e2E+hEMvRBXBSWlfB4nMjMZ0a2U+9lefgKsdfIj0z1EkLQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cd86n3pNszVcv; Fri, 03 Oct 2025 00:04:49 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 59304nLM057625; Fri, 3 Oct 2025 00:04:49 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 59304nR8057622; Fri, 3 Oct 2025 00:04:49 GMT (envelope-from git) Date: Fri, 3 Oct 2025 00:04:49 GMT Message-Id: <202510030004.59304nR8057622@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Navdeep Parhar Subject: git: 48656dc9c52d - stable/15 - cxgbe mp_ring: Add mp_ring_enqueue_only List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: np X-Git-Repository: src X-Git-Refname: refs/heads/stable/15 X-Git-Reftype: branch X-Git-Commit: 48656dc9c52d8136df546893906d09e57b0958de Auto-Submitted: auto-generated The branch stable/15 has been updated by np: URL: https://cgit.FreeBSD.org/src/commit/?id=48656dc9c52d8136df546893906d09e57b0958de commit 48656dc9c52d8136df546893906d09e57b0958de Author: John Baldwin AuthorDate: 2025-09-29 15:10:42 +0000 Commit: Navdeep Parhar CommitDate: 2025-10-02 22:09:08 +0000 cxgbe mp_ring: Add mp_ring_enqueue_only Unlike mp_ring_enqueue, this function is meant to be called from within a drain routine to enqueue more packets to the queue currently being drained. To avoid recursion, it only enqueues additional items without becoming a consumer. Sponsored by: Chelsio Communications (cherry picked from commit 96054b97b418d1ff2b781e9e23cc86a79d91909a) --- sys/dev/cxgbe/t4_mp_ring.c | 81 +++++++++++++++++++++++++++++++++++++++++++++- sys/dev/cxgbe/t4_mp_ring.h | 1 + 2 files changed, 81 insertions(+), 1 deletion(-) diff --git a/sys/dev/cxgbe/t4_mp_ring.c b/sys/dev/cxgbe/t4_mp_ring.c index 531fd356728e..916c363a0c2a 100644 --- a/sys/dev/cxgbe/t4_mp_ring.c +++ b/sys/dev/cxgbe/t4_mp_ring.c @@ -305,7 +305,6 @@ failed: } void - mp_ring_free(struct mp_ring *r) { int i; @@ -472,6 +471,86 @@ mp_ring_enqueue(struct mp_ring *r, void **items, int n, int budget) return (0); } +/* + * Enqueue n items but never drain the ring. Can be called + * to enqueue new items while draining the ring. + * + * Returns an errno. + */ +int +mp_ring_enqueue_only(struct mp_ring *r, void **items, int n) +{ + union ring_state os, ns; + uint16_t pidx_start, pidx_stop; + int i; + + MPASS(items != NULL); + MPASS(n > 0); + + /* + * Reserve room for the new items. Our reservation, if successful, is + * from 'pidx_start' to 'pidx_stop'. + */ + os.state = atomic_load_64(&r->state); + + /* Should only be used from the drain callback. */ + MPASS(os.flags == BUSY || os.flags == TOO_BUSY || + os.flags == TAKING_OVER); + + for (;;) { + if (__predict_false(space_available(r, os) < n)) { + /* Not enough room in the ring. */ + counter_u64_add(r->dropped, n); + return (ENOBUFS); + } + + /* There is room in the ring. */ + + ns.state = os.state; + ns.pidx_head = increment_idx(r, os.pidx_head, n); + critical_enter(); + if (atomic_fcmpset_64(&r->state, &os.state, ns.state)) + break; + critical_exit(); + cpu_spinwait(); + }; + + pidx_start = os.pidx_head; + pidx_stop = ns.pidx_head; + + /* + * Wait for other producers who got in ahead of us to enqueue their + * items, one producer at a time. It is our turn when the ring's + * pidx_tail reaches the beginning of our reservation (pidx_start). + */ + while (ns.pidx_tail != pidx_start) { + cpu_spinwait(); + ns.state = atomic_load_64(&r->state); + } + + /* Now it is our turn to fill up the area we reserved earlier. */ + i = pidx_start; + do { + r->items[i] = *items++; + if (__predict_false(++i == r->size)) + i = 0; + } while (i != pidx_stop); + + /* + * Update the ring's pidx_tail. The release style atomic guarantees + * that the items are visible to any thread that sees the updated pidx. + */ + os.state = atomic_load_64(&r->state); + do { + ns.state = os.state; + ns.pidx_tail = pidx_stop; + } while (atomic_fcmpset_rel_64(&r->state, &os.state, ns.state) == 0); + critical_exit(); + + counter_u64_add(r->not_consumer, 1); + return (0); +} + void mp_ring_check_drainage(struct mp_ring *r, int budget) { diff --git a/sys/dev/cxgbe/t4_mp_ring.h b/sys/dev/cxgbe/t4_mp_ring.h index 949174b9056d..07b15906cd43 100644 --- a/sys/dev/cxgbe/t4_mp_ring.h +++ b/sys/dev/cxgbe/t4_mp_ring.h @@ -62,6 +62,7 @@ int mp_ring_alloc(struct mp_ring **, int, void *, ring_drain_t, ring_can_drain_t, struct malloc_type *, struct mtx *, int); void mp_ring_free(struct mp_ring *); int mp_ring_enqueue(struct mp_ring *, void **, int, int); +int mp_ring_enqueue_only(struct mp_ring *, void **, int); void mp_ring_check_drainage(struct mp_ring *, int); void mp_ring_reset_stats(struct mp_ring *); bool mp_ring_is_idle(struct mp_ring *);