From nobody Mon Sep 2 08:50:55 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Wy2Xc2xWKz5Mntn; Mon, 02 Sep 2024 08:50:56 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Wy2Xc06lnz42my; Mon, 2 Sep 2024 08:50:56 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725267056; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=s0nqLoe2cv9W6NRXOvb9hMHqHPvanWKX4/kPjk1jgDs=; b=Tk1pGjcDagf+NKe1rZCwuKq1uXK/Kc80BqHHqYVbxjgVLP5l/cX6PBQ9gFxPOZao8Ady14 UBJCMbTM96M+sN4vULyNI98YQw4e2A8uxSQesdlFyY8x6S0h28uvW9cBHL6jdpi+5H64or a8Su1MwrOluVME/Bt25H6Xhsf0+KJR3X+U8u8xxSV4faj9Qb1XdiIdSGkI2VT7KkH8abd/ JPE9YjoSt+yJUrTfaObfpsWf8Wb+s4VfSXEbLgVo4EYzIdp4rY/ysUr6jaLN6lGynj07R4 OzJzys8GLcIC4Us1GUoATUxywErDrWu2z9n7G63oaT1I8KSzQQnZs7D8MRB66A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725267056; a=rsa-sha256; cv=none; b=DXbgPMMVeKRw1+WJXRiy2KXtMn/wS0kUJBqPXWOsy0poAjnqYTFaLkCtipaqdmRiW9EoXf EQf0fXhs6IzpHrNs2I1BgBda6KsssPwMkYWKmV8ADN6ZTPTcqlcye+FR3St4VoN1/kB2hs poB8wPdJWI+qWT5hE1Q++4vzut9gT5yeqCCuCWrfJ3JsFfhM/1FVQg5VUBVqnB8MQumsrY D+KfkduhS27uXZm2lkIK1jyT1EwQiwrcHabPl72s5imFREIfSO8ZlZSBuvSeVqTiw48zsX F/ToU76rxyBA5/GcG9kuyo7mpkeXS/zFFR/5WFRFABkGdziEZBdrkSkfuAKWHw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725267056; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=s0nqLoe2cv9W6NRXOvb9hMHqHPvanWKX4/kPjk1jgDs=; b=tReBp6e72H7kun+0FTxnuS+Gq7AHL4XjUeWh2LTiQBZdT2jxNX9RHsvk/jhoHzBHgz3pgj 9dqSULsKRdl9/Ip4pTylIvZTWN7lIwZue/S1KfJygL71FNXhPsoiV1qzoAj0nDQzWu36vH X5ZLeraCFJWnZ0HktVLB/yVr8Yzie90ia7Lzy7XHszMy7CbiGYAmQN2DQXbNNiD0rK2qUG 5bOJ9n6aAcPDE2jP0nperzxSpqDaAtDf6HtEV6GRVQqQSUsdqXnOuNXsX2+2NMeRLzrKHK VqoJaRLFothJMKUuXLn59EPaUFAjmxmGD+zBVzEtADkMHaI0jHiV422s237gkw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Wy2Xb6mJqz156h; Mon, 2 Sep 2024 08:50:55 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 4828otVC007698; Mon, 2 Sep 2024 08:50:55 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 4828otAI007695; Mon, 2 Sep 2024 08:50:55 GMT (envelope-from git) Date: Mon, 2 Sep 2024 08:50:55 GMT Message-Id: <202409020850.4828otAI007695@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Andrew Turner Subject: git: bfd435bfe8e1 - stable/14 - buf_ring: Use atomic operations with br_prod_tail List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: andrew X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: bfd435bfe8e1b740d932355831b8a3cc644c14ed Auto-Submitted: auto-generated The branch stable/14 has been updated by andrew: URL: https://cgit.FreeBSD.org/src/commit/?id=bfd435bfe8e1b740d932355831b8a3cc644c14ed commit bfd435bfe8e1b740d932355831b8a3cc644c14ed Author: Andrew Turner AuthorDate: 2024-08-19 09:07:19 +0000 Commit: Andrew Turner CommitDate: 2024-09-02 08:49:38 +0000 buf_ring: Use atomic operations with br_prod_tail As with br_cons_tail use an atomic load acquire to read br_prod_tail in buf_ring_dequeue_mc and buf_ring_peek*. On dequeue we need to ensure we don't read the entry from the buf_ring until it is available and prod_tail has updated. There is already an appropriate store in the enqueue path and an appropriate load in the single consumer dequeue, we just need one in the other functions that read from the buf_ring. Reviewed by: imp, markj Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46154 (cherry picked from commit 947754af55edc217c10e3456d97558c4eb6d0f99) --- sys/sys/buf_ring.h | 45 ++++++++++++++++----------------------------- 1 file changed, 16 insertions(+), 29 deletions(-) diff --git a/sys/sys/buf_ring.h b/sys/sys/buf_ring.h index ad463fba27cb..66e1e55bc5e9 100644 --- a/sys/sys/buf_ring.h +++ b/sys/sys/buf_ring.h @@ -131,7 +131,7 @@ static __inline void * buf_ring_dequeue_mc(struct buf_ring *br) { uint32_t cons_head, cons_next, cons_idx; - uint32_t mask; + uint32_t prod_tail, mask; void *buf; critical_enter(); @@ -139,8 +139,9 @@ buf_ring_dequeue_mc(struct buf_ring *br) do { cons_head = br->br_cons_head; cons_next = cons_head + 1; + prod_tail = atomic_load_acq_32(&br->br_prod_tail); - if (cons_head == br->br_prod_tail) { + if (cons_head == prod_tail) { critical_exit(); return (NULL); } @@ -266,29 +267,26 @@ buf_ring_putback_sc(struct buf_ring *br, void *new) static __inline void * buf_ring_peek(struct buf_ring *br) { - uint32_t mask; + uint32_t cons_head, prod_tail, mask; #if defined(DEBUG_BUFRING) && defined(_KERNEL) if ((br->br_lock != NULL) && !mtx_owned(br->br_lock)) panic("lock not held on single consumer dequeue"); #endif mask = br->br_cons_mask; - /* - * I believe it is safe to not have a memory barrier - * here because we control cons and tail is worst case - * a lagging indicator so we worst case we might - * return NULL immediately after a buffer has been enqueued - */ - if (br->br_cons_head == br->br_prod_tail) + prod_tail = atomic_load_acq_32(&br->br_prod_tail); + cons_head = br->br_cons_head; + + if (cons_head == prod_tail) return (NULL); - return (br->br_ring[br->br_cons_head & mask]); + return (br->br_ring[cons_head & mask]); } static __inline void * buf_ring_peek_clear_sc(struct buf_ring *br) { - uint32_t mask; + uint32_t cons_head, prod_tail, mask; void *ret; #if defined(DEBUG_BUFRING) && defined(_KERNEL) @@ -297,30 +295,19 @@ buf_ring_peek_clear_sc(struct buf_ring *br) #endif mask = br->br_cons_mask; - if (br->br_cons_head == br->br_prod_tail) - return (NULL); + prod_tail = atomic_load_acq_32(&br->br_prod_tail); + cons_head = br->br_cons_head; -#if defined(__arm__) || defined(__aarch64__) - /* - * The barrier is required there on ARM and ARM64 to ensure, that - * br->br_ring[br->br_cons_head] will not be fetched before the above - * condition is checked. - * Without the barrier, it is possible, that buffer will be fetched - * before the enqueue will put mbuf into br, then, in the meantime, the - * enqueue will update the array and the br_prod_tail, and the - * conditional check will be true, so we will return previously fetched - * (and invalid) buffer. - */ - atomic_thread_fence_acq(); -#endif + if (cons_head == prod_tail) + return (NULL); - ret = br->br_ring[br->br_cons_head & mask]; + ret = br->br_ring[cons_head & mask]; #ifdef DEBUG_BUFRING /* * Single consumer, i.e. cons_head will not move while we are * running, so atomic_swap_ptr() is not necessary here. */ - br->br_ring[br->br_cons_head & mask] = NULL; + br->br_ring[cons_head & mask] = NULL; #endif return (ret); }