From nobody Fri Jul 24 22:36:23 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4h6NBb4yH1z6mrJs for ; Fri, 24 Jul 2026 22:36:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4h6NBb4G49z3rBj for ; Fri, 24 Jul 2026 22:36:23 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1784932583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=5FBIp+R9v1hlV1wapj9auGje/bc8PQnScFRQoedyW2U=; b=pXbKq8EsYki3jMhC55UWQqSDmPT+UuN4KHPa/9G+TCdz7HwpJewrPG0hkkP0dNpZHGP2oC FWKdcnEjkwdbHayu8LOwch+EVAXORgZcm2IEfakRbCO2nAp1B48hsbjik2jYoDQtELUdSg ASgYepS6ARCPbKWi7+Q6igdKGwgnozWVwhkihAYyhiNAPYTxNNVJ/aw7rpMtE+ppchpNWg YEZID0GH/8Q/T/EaC98zsDSpOAFgxgdsD09TYEQWQVzgGgjFK+6NjUieaAE+ugHcOml3TZ MEAwiqh6GwNBK3QZOn+8c/umE+1U57b7DIvLuAnAVeSMqZKnJHSqOOXks3IE/A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1784932583; a=rsa-sha256; cv=none; b=QXVGMRteYAnn9uw+QiDX5xmKX/7HE15BTi63fnTSV+oVz/HELpc+TiaKlIw5dY7gKpkLrC Vbk9uRL9e8w1x4Al3MhJwRZohYzpKw0tbcTP3OBfT6bZ8xigJX517liZFmH01XlxTpiH2X lEWF3L1KARQGKNE+4fDI+h1/E+5xGWh/KmWyqu93R0RyYZZz2pQZ26xTUXXzIxi6cjhm57 6UFgCw6Sj/tRmJGwNZHR6FjorGkl+/F0ases7hPFLFlmqoyxTdtJeZXGEkLOgp+SFzy5Ri a8SmBGeV8z74hccPS46Rxj8QWAL3mbcv0q7N/Lgz4dOL0TCUxPAdVlXZteLrYA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1784932583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=5FBIp+R9v1hlV1wapj9auGje/bc8PQnScFRQoedyW2U=; b=eC3LHdLhBPN/IZS10xPv67xkP/ffluKKL6PIeR7RMpndfPmQcLQaoEx6H/wt2cYXjoSHbz j3MqSpAvqvCVkV4lHHxPGbDxAlGgchG/IAjO+s76NmiInofuWxpCoGea2GoVajhideiUIv fTIaP4RdoF1V3ixU7tJ8Bb3Y1F8CQ5jzns1QTOmLxKPmG+x4ncoHoH6pHOKbD2y/srPoer aPSIRjpiwdwUHWgl4G09InC4WngDMM2YUrDD1xKBMvcC2nt7e3KMCaaxhIuVtRgR3FPMnv wHzik5c6DjgfOFDksv54H80bqbrPi46vPbmZDMLoAVniBk58AsDtyHoxx1DGtg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4h6NBb2rWYz19TN for ; Fri, 24 Jul 2026 22:36:23 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 188ba by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Fri, 24 Jul 2026 22:36:23 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: 6337ca19a363 - main - uma: Factor out the implementations of uma_zfree_{arg,smr}() List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 6337ca19a3637aa72eddf4d62a2eaf7d8df51638 Auto-Submitted: auto-generated Date: Fri, 24 Jul 2026 22:36:23 +0000 Message-Id: <6a63e8e7.188ba.5fd30e5@gitrepo.freebsd.org> The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=6337ca19a3637aa72eddf4d62a2eaf7d8df51638 commit 6337ca19a3637aa72eddf4d62a2eaf7d8df51638 Author: Mark Johnston AuthorDate: 2026-07-24 21:11:58 +0000 Commit: Mark Johnston CommitDate: 2026-07-24 21:11:58 +0000 uma: Factor out the implementations of uma_zfree_{arg,smr}() The two function both free an item to a UMA zone, but uma_zfree_arg() does so in such as way as to ensure that the item will be the first one returned by a subsequent allocation, while uma_zfree_smr() must defer reuse of the item and therefore never frees to the per-CPU alloc bucket. When KASAN is enabled, we actually want uma_zfree_arg() to behave like uma_zfree_smr(): to improve the reliability of use-after-free detection, reuse of the newly freed item should be deferred for some time. Refactor a bit to make it easier to improve KASAN along these lines: introduce two helper functions, cache_free_item() and cache_free_smr(), which handle most of the work of interacting with the per-CPU caches. A subsequent commit will let uma_zfree_arg() use cache_free_smr() when KASAN is enabled. No functional change intended. Reviewed by: rlibby MFC after: 1 month Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D58268 --- sys/vm/uma_core.c | 159 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 91 insertions(+), 68 deletions(-) diff --git a/sys/vm/uma_core.c b/sys/vm/uma_core.c index 410b4f5a5142..02b2df89f636 100644 --- a/sys/vm/uma_core.c +++ b/sys/vm/uma_core.c @@ -4447,37 +4447,82 @@ fail: return (NULL); } -/* See uma.h */ -void -uma_zfree_smr(uma_zone_t zone, void *item) +static __always_inline bool +cache_free_item(uma_zone_t zone, int uz_flags, void *item, void *udata) { uma_cache_t cache; - uma_cache_bucket_t bucket; int itemdomain; + + /* + * If possible, free to the per-CPU cache. There are two + * requirements for safe access to the per-CPU cache: (1) the thread + * accessing the cache must not be preempted or yield during access, + * and (2) the thread must not migrate CPUs without switching which + * cache it accesses. We rely on a critical section to prevent + * preemption and migration. We release the critical section in + * order to acquire the zone mutex if we are unable to free to the + * current cache; when we re-acquire the critical section, we must + * detect and handle migration if it has occurred. + */ + itemdomain = 0; #ifdef NUMA - int uz_flags; + if ((uz_flags & UMA_ZONE_FIRSTTOUCH) != 0) + itemdomain = item_domain(item); #endif - CTR3(KTR_UMA, "uma_zfree_smr zone %s(%p) item %p", - zone->uz_name, zone, item); + critical_enter(); + do { + uma_cache_bucket_t bucket; -#ifdef UMA_ZALLOC_DEBUG - KASSERT((zone->uz_flags & UMA_ZONE_SMR) != 0, - ("uma_zfree_smr: called with non-SMR zone.")); - KASSERT(item != NULL, ("uma_zfree_smr: Called with NULL pointer.")); - SMR_ASSERT_NOT_ENTERED(zone->uz_smr); - if (uma_zfree_debug(zone, item, NULL) == EJUSTRETURN) - return; + cache = &zone->uz_cpu[curcpu]; + /* + * Try to free into the allocbucket first to give LIFO + * ordering for cache-hot datastructures. Spill over + * into the freebucket if necessary. Alloc will swap + * them if one runs dry. + */ + bucket = &cache->uc_allocbucket; +#ifdef NUMA + if ((uz_flags & UMA_ZONE_FIRSTTOUCH) != 0 && + PCPU_GET(domain) != itemdomain) { + bucket = &cache->uc_crossbucket; + } else #endif - cache = &zone->uz_cpu[curcpu]; + if (bucket->ucb_cnt == bucket->ucb_entries && + cache->uc_freebucket.ucb_cnt < + cache->uc_freebucket.ucb_entries) + cache_bucket_swap(&cache->uc_freebucket, + &cache->uc_allocbucket); + if (__predict_true(bucket->ucb_cnt < bucket->ucb_entries)) { + cache_bucket_push(cache, bucket, item); + critical_exit(); + return (true); + } + } while (cache_free(zone, cache, udata, itemdomain)); + critical_exit(); + + return (false); +} + +static __always_inline bool +cache_free_smr(uma_zone_t zone, void *item, void *udata) +{ + uma_cache_t cache; + int itemdomain; +#ifdef NUMA + int uz_flags; +#endif + itemdomain = 0; #ifdef NUMA - uz_flags = cache_uz_flags(cache); + uz_flags = cache_uz_flags(&zone->uz_cpu[curcpu]); if ((uz_flags & UMA_ZONE_FIRSTTOUCH) != 0) itemdomain = item_domain(item); #endif critical_enter(); do { + uma_cache_bucket_t bucket; + cache = &zone->uz_cpu[curcpu]; /* SMR Zones must free to the free bucket. */ bucket = &cache->uc_freebucket; @@ -4490,11 +4535,33 @@ uma_zfree_smr(uma_zone_t zone, void *item) if (__predict_true(bucket->ucb_cnt < bucket->ucb_entries)) { cache_bucket_push(cache, bucket, item); critical_exit(); - return; + return (true); } - } while (cache_free(zone, cache, NULL, itemdomain)); + } while (cache_free(zone, cache, udata, itemdomain)); critical_exit(); + return (false); +} + +/* See uma.h */ +void +uma_zfree_smr(uma_zone_t zone, void *item) +{ + CTR3(KTR_UMA, "uma_zfree_smr zone %s(%p) item %p", + zone->uz_name, zone, item); + +#ifdef UMA_ZALLOC_DEBUG + KASSERT((zone->uz_flags & UMA_ZONE_SMR) != 0, + ("uma_zfree_smr: called with non-SMR zone.")); + KASSERT(item != NULL, ("uma_zfree_smr: Called with NULL pointer.")); + SMR_ASSERT_NOT_ENTERED(zone->uz_smr); + if (uma_zfree_debug(zone, item, NULL) == EJUSTRETURN) + return; +#endif + + if (cache_free_smr(zone, item, NULL)) + return; + /* * If nothing else caught this, we'll just do an internal free. */ @@ -4506,8 +4573,7 @@ void uma_zfree_arg(uma_zone_t zone, void *item, void *udata) { uma_cache_t cache; - uma_cache_bucket_t bucket; - int itemdomain, uz_flags; + int uz_flags; /* Enable entropy collection for RANDOM_ENABLE_UMA kernel option */ random_harvest_fast_uma(&zone, sizeof(zone), RANDOM_UMA); @@ -4540,55 +4606,12 @@ uma_zfree_arg(uma_zone_t zone, void *item, void *udata) * The race here is acceptable. If we miss it we'll just have to wait * a little longer for the limits to be reset. */ - if (__predict_false(uz_flags & UMA_ZFLAG_LIMIT)) { - if (atomic_load_32(&zone->uz_sleepers) > 0) - goto zfree_item; - } + if (__predict_false(uz_flags & UMA_ZFLAG_LIMIT) && + atomic_load_32(&zone->uz_sleepers) > 0) + goto zfree_item; - /* - * If possible, free to the per-CPU cache. There are two - * requirements for safe access to the per-CPU cache: (1) the thread - * accessing the cache must not be preempted or yield during access, - * and (2) the thread must not migrate CPUs without switching which - * cache it accesses. We rely on a critical section to prevent - * preemption and migration. We release the critical section in - * order to acquire the zone mutex if we are unable to free to the - * current cache; when we re-acquire the critical section, we must - * detect and handle migration if it has occurred. - */ - itemdomain = 0; -#ifdef NUMA - if ((uz_flags & UMA_ZONE_FIRSTTOUCH) != 0) - itemdomain = item_domain(item); -#endif - critical_enter(); - do { - cache = &zone->uz_cpu[curcpu]; - /* - * Try to free into the allocbucket first to give LIFO - * ordering for cache-hot datastructures. Spill over - * into the freebucket if necessary. Alloc will swap - * them if one runs dry. - */ - bucket = &cache->uc_allocbucket; -#ifdef NUMA - if ((uz_flags & UMA_ZONE_FIRSTTOUCH) != 0 && - PCPU_GET(domain) != itemdomain) { - bucket = &cache->uc_crossbucket; - } else -#endif - if (bucket->ucb_cnt == bucket->ucb_entries && - cache->uc_freebucket.ucb_cnt < - cache->uc_freebucket.ucb_entries) - cache_bucket_swap(&cache->uc_freebucket, - &cache->uc_allocbucket); - if (__predict_true(bucket->ucb_cnt < bucket->ucb_entries)) { - cache_bucket_push(cache, bucket, item); - critical_exit(); - return; - } - } while (cache_free(zone, cache, udata, itemdomain)); - critical_exit(); + if (cache_free_item(zone, uz_flags, item, udata)) + return; /* * If nothing else caught this, we'll just do an internal free.