From nobody Tue Aug 12 12:56:13 2025 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4c1Wht3G6Bz64jVC; Tue, 12 Aug 2025 12:56:14 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4c1Wht08MBz3KHb; Tue, 12 Aug 2025 12:56:14 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1755003374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Y/YK1xhhzgpIsYS4DZ9KxSPchxP49nFXXATg37VPtV0=; b=F44XzK1eR/IaDXViS2GaFyvjXeMAPAsKR2ftqyXIcQbijIKQYVr0DYWsfjsmGuxqqLxrnf 1MuCYjPTSVREY6YuunMWdYC8R+FcidPdV+4j/bNVxzrOVJ1+DIhcbKIo8QrnN+smkX/EPG g5uW5fbOYAbNZDupaRmqQ2ZqaxQrOE7eDEJs0nWU2zobsbEfCgG+iO7Y3BfWKTzMayrwzd 7NwIQgNHxbJME3+3OePKc8vVh/Q99Sh7864fgfVlTChUhsCzf6EpvXEpvQFGeIPIMM9hwE 9iFUfGw9RwkH1UFF06/zvj4YCePPgL2CLOu3sU1AO4/sC/KWoQBjzOjJCEar+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1755003374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Y/YK1xhhzgpIsYS4DZ9KxSPchxP49nFXXATg37VPtV0=; b=ZCVZDwTZ3/KL/ZcuRJNFZCdCEyQDFDp3axD3J7X9NMYnEay0fFlgHACMVE0DyedaMBLTTy U/Sgsshx85jATe/87daat/tj9gZxGxt5Vin8bD12anScyfjZvhBAfnsT7GnYTz6LXBTcvz VYHrDhP4klfOdPLTmI2P67ROQ0ylrnrRtBTNcIICFIU8czxKSry4/cSf0QGqRsvqA7TvSt gIs5GL69YR9tA7jEMpS+1E2EFxpM48yTqOqny16ZscrZC2LVuzobv6qVjvKgpyVc356gIQ psPFAueuonLjL2GixtOJnN4NmpaKA3JCJL2AgHa2wA5jWCimi9tag39aNT64vQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1755003374; a=rsa-sha256; cv=none; b=uJjRg2fvYXlEZllASewO8MQpVseIcem39vQEsFWRItAYoOazHeAKZ7iWCuzGq3ksU4AEru 3FJJox7smftranpoVIzIaJSYb2+fXx7RMeQ77MO3mvFgcvfjD9hPzmTSCyZiXy1CCQwCpK vGKpp9Tn8K0DdZdRff/On2/AbwO2upr1VzjUuxaARi3pHmXT3cOXQkOGjkVP+doYFyU5rX tmP2lw0dMy8BnvEd5f5eDCfOxKToVjL9TRpGI1KscmSSyy2oRbLSOjz2ppR200IP2L1trY 6i6upaVu02f6cbkwIr4ILA/lUzrM7q1XGmA2RSeTOILCBaQf+DcyoHNLKC1B0A== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4c1Whs6mtgzfn4; Tue, 12 Aug 2025 12:56:13 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 57CCuDDp088444; Tue, 12 Aug 2025 12:56:13 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 57CCuDHO088441; Tue, 12 Aug 2025 12:56:13 GMT (envelope-from git) Date: Tue, 12 Aug 2025 12:56:13 GMT Message-Id: <202508121256.57CCuDHO088441@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Mark Johnston Subject: git: 9280aa57a5d2 - stable/14 - random: Change the entropy harvest event queuing scheme List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-branches@freebsd.org Sender: owner-dev-commits-src-branches@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 9280aa57a5d262f011030a042aabec97f2de3bf6 Auto-Submitted: auto-generated The branch stable/14 has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=9280aa57a5d262f011030a042aabec97f2de3bf6 commit 9280aa57a5d262f011030a042aabec97f2de3bf6 Author: Mark Johnston AuthorDate: 2025-07-07 15:02:36 +0000 Commit: Mark Johnston CommitDate: 2025-08-12 12:49:42 +0000 random: Change the entropy harvest event queuing scheme The entropy queue stores entropy gathered from environmental sources. Periodically (every 100ms currently), the random kthread will drain this queue and mix it into the CSPRNG's entropy pool(s). The old scheme uses a ring buffer with a mutex to serialize producers, while the sole consumer, the random kthread, avoids using a mutex on the basis that no serialization is needed since nothing else is updating the consumer index. On platforms without total store ordering, however, this isn't sufficient: when a producer inserts a queue entry and updates `ring.in`, there is no guarantee that the consumer will see the updated queue entry upon observing the updated producer index. That is, the update to `ring.in` may be visible before the updated queue entry is visible. As a result, we could end up mixing in zero'ed queue entries, though this race is fairly unlikely in practice given how infrequently the kthread runs. The easiest way to fix this is to make the kthread acquire the mutex as well, and hold it while processing queue entries. However, this might result in a long hold time if there are many queue entries, and we really want the hold times to be short, e.g., to avoid delaying interrupt processing. We could introduce a proper MPSC queue, but this is probably overcomplicated for a consumer which runs at 10Hz. Instead, define two buffers, always with one designated as the "active" buffer. Producers queue entries in the active buffer, and the kthread uses the mutex to atomically flip the two buffers, so it can process entries from the inactive buffer without holding the mutex. This requires more memory, but keeps mutex hold times short and lets us keep the queue implementation very simple. Reviewed by: cem MFC after: 1 month Sponsored by: Stormshield Sponsored by: Klara, Inc. Differential Revision: https://reviews.freebsd.org/D51112 (cherry picked from commit 9940c974029ba53fc00696b3fa1784725c48a9e9) --- sys/dev/random/random_harvestq.c | 103 +++++++++++++++++---------------------- 1 file changed, 46 insertions(+), 57 deletions(-) diff --git a/sys/dev/random/random_harvestq.c b/sys/dev/random/random_harvestq.c index 4cf40edb0ab4..f38fd8e92c36 100644 --- a/sys/dev/random/random_harvestq.c +++ b/sys/dev/random/random_harvestq.c @@ -132,30 +132,14 @@ static struct harvest_context { /* The context of the kernel thread processing harvested entropy */ struct proc *hc_kthread_proc; /* - * Lockless ring buffer holding entropy events - * If ring.in == ring.out, - * the buffer is empty. - * If ring.in != ring.out, - * the buffer contains harvested entropy. - * If (ring.in + 1) == ring.out (mod RANDOM_RING_MAX), - * the buffer is full. - * - * NOTE: ring.in points to the last added element, - * and ring.out points to the last consumed element. - * - * The ring.in variable needs locking as there are multiple - * sources to the ring. Only the sources may change ring.in, - * but the consumer may examine it. - * - * The ring.out variable does not need locking as there is - * only one consumer. Only the consumer may change ring.out, - * but the sources may examine it. + * A pair of buffers for queued events. New events are added to the + * active queue while the kthread processes the other one in parallel. */ - struct entropy_ring { + struct entropy_buffer { struct harvest_event ring[RANDOM_RING_MAX]; - volatile u_int in; - volatile u_int out; - } hc_entropy_ring; + u_int pos; + } hc_entropy_buf[2]; + u_int hc_active_buf; struct fast_entropy_accumulator { volatile u_int pos; uint32_t buf[RANDOM_ACCUM_MAX]; @@ -184,37 +168,41 @@ random_harvestq_fast_process_event(struct harvest_event *event) static void random_kthread(void) { - u_int maxloop, ring_out, i; + struct harvest_context *hc; - /* - * Locking is not needed as this is the only place we modify ring.out, and - * we only examine ring.in without changing it. Both of these are volatile, - * and this is a unique thread. - */ + hc = &harvest_context; for (random_kthread_control = 1; random_kthread_control;) { - /* Deal with events, if any. Restrict the number we do in one go. */ - maxloop = RANDOM_RING_MAX; - while (harvest_context.hc_entropy_ring.out != harvest_context.hc_entropy_ring.in) { - ring_out = (harvest_context.hc_entropy_ring.out + 1)%RANDOM_RING_MAX; - random_harvestq_fast_process_event(harvest_context.hc_entropy_ring.ring + ring_out); - harvest_context.hc_entropy_ring.out = ring_out; - if (!--maxloop) - break; - } + struct entropy_buffer *buf; + u_int entries; + + /* Deal with queued events. */ + RANDOM_HARVEST_LOCK(); + buf = &hc->hc_entropy_buf[hc->hc_active_buf]; + entries = buf->pos; + buf->pos = 0; + hc->hc_active_buf = (hc->hc_active_buf + 1) % + nitems(hc->hc_entropy_buf); + RANDOM_HARVEST_UNLOCK(); + for (u_int i = 0; i < entries; i++) + random_harvestq_fast_process_event(&buf->ring[i]); + + /* Poll sources of noise. */ random_sources_feed(); + /* XXX: FIX!! Increase the high-performance data rate? Need some measurements first. */ - for (i = 0; i < RANDOM_ACCUM_MAX; i++) { - if (harvest_context.hc_entropy_fast_accumulator.buf[i]) { - random_harvest_direct(harvest_context.hc_entropy_fast_accumulator.buf + i, sizeof(harvest_context.hc_entropy_fast_accumulator.buf[0]), RANDOM_UMA); - harvest_context.hc_entropy_fast_accumulator.buf[i] = 0; + for (u_int i = 0; i < RANDOM_ACCUM_MAX; i++) { + if (hc->hc_entropy_fast_accumulator.buf[i]) { + random_harvest_direct(&hc->hc_entropy_fast_accumulator.buf[i], + sizeof(hc->hc_entropy_fast_accumulator.buf[0]), RANDOM_UMA); + hc->hc_entropy_fast_accumulator.buf[i] = 0; } } /* XXX: FIX!! This is a *great* place to pass hardware/live entropy to random(9) */ - tsleep_sbt(&harvest_context.hc_kthread_proc, 0, "-", + tsleep_sbt(&hc->hc_kthread_proc, 0, "-", SBT_1S/RANDOM_KTHREAD_HZ, 0, C_PREL(1)); } random_kthread_control = -1; - wakeup(&harvest_context.hc_kthread_proc); + wakeup(&hc->hc_kthread_proc); kproc_exit(0); /* NOTREACHED */ } @@ -435,7 +423,7 @@ random_harvestq_init(void *unused __unused) hc_source_mask = almost_everything_mask; RANDOM_HARVEST_INIT_LOCK(); - harvest_context.hc_entropy_ring.in = harvest_context.hc_entropy_ring.out = 0; + harvest_context.hc_active_buf = 0; } SYSINIT(random_device_h_init, SI_SUB_RANDOM, SI_ORDER_THIRD, random_harvestq_init, NULL); @@ -540,9 +528,9 @@ SYSUNINIT(random_device_h_init, SI_SUB_RANDOM, SI_ORDER_THIRD, random_harvestq_d * This is supposed to be fast; do not do anything slow in here! * It is also illegal (and morally reprehensible) to insert any * high-rate data here. "High-rate" is defined as a data source - * that will usually cause lots of failures of the "Lockless read" - * check a few lines below. This includes the "always-on" sources - * like the Intel "rdrand" or the VIA Nehamiah "xstore" sources. + * that is likely to fill up the buffer in much less than 100ms. + * This includes the "always-on" sources like the Intel "rdrand" + * or the VIA Nehamiah "xstore" sources. */ /* XXXRW: get_cyclecount() is cheap on most modern hardware, where cycle * counters are built in, but on older hardware it will do a real time clock @@ -551,28 +539,29 @@ SYSUNINIT(random_device_h_init, SI_SUB_RANDOM, SI_ORDER_THIRD, random_harvestq_d void random_harvest_queue_(const void *entropy, u_int size, enum random_entropy_source origin) { + struct harvest_context *hc; + struct entropy_buffer *buf; struct harvest_event *event; - u_int ring_in; - KASSERT(origin >= RANDOM_START && origin < ENTROPYSOURCE, ("%s: origin %d invalid\n", __func__, origin)); + KASSERT(origin >= RANDOM_START && origin < ENTROPYSOURCE, + ("%s: origin %d invalid", __func__, origin)); + + hc = &harvest_context; RANDOM_HARVEST_LOCK(); - ring_in = (harvest_context.hc_entropy_ring.in + 1)%RANDOM_RING_MAX; - if (ring_in != harvest_context.hc_entropy_ring.out) { - /* The ring is not full */ - event = harvest_context.hc_entropy_ring.ring + ring_in; + buf = &hc->hc_entropy_buf[hc->hc_active_buf]; + if (buf->pos < RANDOM_RING_MAX) { + event = &buf->ring[buf->pos++]; event->he_somecounter = random_get_cyclecount(); event->he_source = origin; - event->he_destination = harvest_context.hc_destination[origin]++; + event->he_destination = hc->hc_destination[origin]++; if (size <= sizeof(event->he_entropy)) { event->he_size = size; memcpy(event->he_entropy, entropy, size); - } - else { + } else { /* Big event, so squash it */ event->he_size = sizeof(event->he_entropy[0]); event->he_entropy[0] = jenkins_hash(entropy, size, (uint32_t)(uintptr_t)event); } - harvest_context.hc_entropy_ring.in = ring_in; } RANDOM_HARVEST_UNLOCK(); }