From nobody Sun Jun 8 18:36:48 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4bFkKs1lrlz5xwrr; Sun, 08 Jun 2025 18:36:49 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4bFkKs0FQvz42bg; Sun, 08 Jun 2025 18:36:49 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1749407809; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=U+NHATqZM0v3urQaevNgDmv+N7fNGKsqX7ELUT7C0Ak=; b=CK1hmvgLS9+eRkO+V4foIoM+kciCAo1Aa/YdjcNArduySHFTjRkHv5LawPeHVtDMqXnPxR 81B9VSNqbjrtECC1RCG4q37/ygGNAIAVquXEdf47dhlZYPmG5comWgvXxZVdDggt1WPQXw VWcIxOhqhJCxrb9kUtDMxS4B6xvEtyZbQHAY8kFgm1nhzaspNZjxidaPJscDofEA9cVYC0 HC3eSTm46/yhKmNvCZpLc+Fn3ij21pU8u6C+5SX+to1PXUTUcZurW+a4V86+bOlaFIYf6M pZ7Jreau3y0GWeAtZbi5WR93ynrhk7CFEoIgA7yZzmmdFYDxHQAVMI71Zs+MaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1749407809; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=U+NHATqZM0v3urQaevNgDmv+N7fNGKsqX7ELUT7C0Ak=; b=ZX+UhXE28TpVWOL4KBeUwuPNYwt2/N8LuHY4/qwLjH8uKPaBwj2wZNKLUxx6ipe07HRNgy eiYFjSulGlHZONnvWLzGij4+I8fwQOFjGZiUKb1mNQbdXj8BWiOWD9oM3q3Pq66wCzmPGF FYGbfdaxmgmP1xAsJ31/1xfnp/W1kj/WGjBrncEAn07yBBLzMDMVgi4yAniJF0PoFk35pg uvoDloyRaDHWuywn+LEzef3B0wLVBJuOK+1SMPOXnNdpGlbyyviHcgrpcFk1ZXIyyTnlJQ fuaD/AIFEMniyLY2klAPLnLsazkBfgJc8Pzw6t2Qvusji5eMFemNExoRjXiElA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1749407809; a=rsa-sha256; cv=none; b=EE0o3EyxLmGP5my3ppIj97nE/w9RdzTctaJsy0vGm+7W+Jjl17386zwL+sytqTTK/J7peY JM1xR9RkeN5D6lFEK/OjAFdXvy0GJJ67x1jnrHPBPl7lWKWoe39TsYjo6+s08V/X39PvUe KKTEfQcdIY7ya/7nZ3GMVViHn1Fa8mcv6+X3UTCXhBHVhG7ecrspqSEev+vgMlXsGSjGnc mqhhplGaXcv90Iohvrg1Nru3xQl1WDf+NwvNoaIBhlJPa6M9nt9UdXPr03Zl7rgKRFtr13 WH9vXFltYkK2o+mXjiQS0GKOwg6u6q8U7XQWEuNU30OYxtSGnitBH0o2/E7s1A== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4bFkKr6Whlz68f; Sun, 08 Jun 2025 18:36:48 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 558IamHr008412; Sun, 8 Jun 2025 18:36:48 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 558IamHL008409; Sun, 8 Jun 2025 18:36:48 GMT (envelope-from git) Date: Sun, 8 Jun 2025 18:36:48 GMT Message-Id: <202506081836.558IamHL008409@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alan Cox Subject: git: 1fee99800a79 - main - vm_page: Retire its listq field List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: alc X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 1fee99800a79887b9037749a34d09f2acab082c0 Auto-Submitted: auto-generated The branch main has been updated by alc: URL: https://cgit.FreeBSD.org/src/commit/?id=1fee99800a79887b9037749a34d09f2acab082c0 commit 1fee99800a79887b9037749a34d09f2acab082c0 Author: Alan Cox AuthorDate: 2025-05-27 08:27:16 +0000 Commit: Alan Cox CommitDate: 2025-06-08 18:35:56 +0000 vm_page: Retire its listq field Over the life cycle of a vm_page, its listq field has been used for two distinct purposes. First, linking together all of the pages allocated to a vm_object. Recently, c8d56817b80f ("vm_object: drop memq field") completed the elimination of this use case, using pctrie iterators in place of iteration over the listq. Second, linking together power-of- two-sized chunks of free pages within vm_phys. This change eliminates that use case. In essence, this change reverts vm_phys back to using the plinks.q field, like it did before 5cd29d0f3cda ("Improve VM page queue scalability."), but with a twist to maintain scalability. Just before vm_phys uses the plinks.q field, it ensures that any lazy dequeue from a paging queue, e.g., PQ_ACTIVE, has completed. Typically, the dequeue has completed, so vm_page_dequeue() is infrequently called by vm_freelist_add(). The reason being that vm_phys only needs to use the plinks.q field within the first page of any power-of-two-sized chunk, so the rest of the pages can still have pending dequeues until the chunk is split. This change saves a non-trivial amount of memory, since we have an instance of struct vm_page for every dynamically allocatable physical page. Bump __FreeBSD_version, since third-party modules that use the inline accessors in vm_page.h may need to be recompiled. Reviewed by: dougm, kib, markj Differential Revision: https://reviews.freebsd.org/D50515 --- sys/sys/param.h | 2 +- sys/vm/vm_page.c | 22 +++++++++++++++------- sys/vm/vm_page.h | 1 - sys/vm/vm_phys.c | 20 +++++++++++++++----- 4 files changed, 31 insertions(+), 14 deletions(-) diff --git a/sys/sys/param.h b/sys/sys/param.h index da2089918323..e167c96cf9f8 100644 --- a/sys/sys/param.h +++ b/sys/sys/param.h @@ -73,7 +73,7 @@ * cannot include sys/param.h and should only be updated here. */ #undef __FreeBSD_version -#define __FreeBSD_version 1500045 +#define __FreeBSD_version 1500046 /* * __FreeBSD_kernel__ indicates that this system uses the kernel of FreeBSD, diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index 79eaf8563208..128a1ef7ca54 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -341,7 +341,7 @@ vm_page_blacklist_add(vm_paddr_t pa, bool verbose) vm_domain_free_unlock(vmd); if (found) { vm_domain_freecnt_inc(vmd, -1); - TAILQ_INSERT_TAIL(&blacklist_head, m, listq); + TAILQ_INSERT_TAIL(&blacklist_head, m, plinks.q); if (verbose) printf("Skipping page with pa 0x%jx\n", (uintmax_t)pa); } @@ -411,7 +411,7 @@ sysctl_vm_page_blacklist(SYSCTL_HANDLER_ARGS) if (error != 0) return (error); sbuf_new_for_sysctl(&sbuf, NULL, 128, req); - TAILQ_FOREACH(m, &blacklist_head, listq) { + TAILQ_FOREACH(m, &blacklist_head, plinks.q) { sbuf_printf(&sbuf, "%s%#jx", first ? "" : ",", (uintmax_t)m->phys_addr); first = 0; @@ -2470,6 +2470,13 @@ again: } found: + /* + * If the page comes from the free page cache, then it might still + * have a pending deferred dequeue. Specifically, when the page is + * imported from a different pool by vm_phys_alloc_npages(), the + * second, third, etc. pages in a non-zero order set could have + * pending deferred dequeues. + */ vm_page_dequeue(m); vm_page_alloc_check(m); @@ -2536,17 +2543,18 @@ vm_page_alloc_nofree_domain(int domain, int req) return (NULL); } m->ref_count = count - 1; - TAILQ_INSERT_HEAD(&vmd->vmd_nofreeq, m, listq); + TAILQ_INSERT_HEAD(&vmd->vmd_nofreeq, m, plinks.q); VM_CNT_ADD(v_nofree_count, count); } m = TAILQ_FIRST(&vmd->vmd_nofreeq); - TAILQ_REMOVE(&vmd->vmd_nofreeq, m, listq); + TAILQ_REMOVE(&vmd->vmd_nofreeq, m, plinks.q); if (m->ref_count > 0) { vm_page_t m_next; m_next = &m[1]; + vm_page_dequeue(m_next); m_next->ref_count = m->ref_count - 1; - TAILQ_INSERT_HEAD(&vmd->vmd_nofreeq, m_next, listq); + TAILQ_INSERT_HEAD(&vmd->vmd_nofreeq, m_next, plinks.q); m->ref_count = 0; } vm_domain_free_unlock(vmd); @@ -2566,7 +2574,7 @@ vm_page_free_nofree(struct vm_domain *vmd, vm_page_t m) { vm_domain_free_lock(vmd); MPASS(m->ref_count == 0); - TAILQ_INSERT_HEAD(&vmd->vmd_nofreeq, m, listq); + TAILQ_INSERT_HEAD(&vmd->vmd_nofreeq, m, plinks.q); vm_domain_free_unlock(vmd); VM_CNT_ADD(v_nofree_count, 1); } @@ -3971,7 +3979,7 @@ vm_page_dequeue(vm_page_t m) old = vm_page_astate_load(m); do { - if (old.queue == PQ_NONE) { + if (__predict_true(old.queue == PQ_NONE)) { KASSERT((old.flags & PGA_QUEUE_STATE_MASK) == 0, ("%s: page %p has unexpected queue state", __func__, m)); diff --git a/sys/vm/vm_page.h b/sys/vm/vm_page.h index 4bcd8d9f8236..4f82a69ebe25 100644 --- a/sys/vm/vm_page.h +++ b/sys/vm/vm_page.h @@ -229,7 +229,6 @@ struct vm_page { void *zone; } uma; } plinks; - TAILQ_ENTRY(vm_page) listq; /* pages in same object (O) */ vm_object_t object; /* which object am I in (O) */ vm_pindex_t pindex; /* offset into object (O,P) */ vm_paddr_t phys_addr; /* physical address of page (C) */ diff --git a/sys/vm/vm_phys.c b/sys/vm/vm_phys.c index 95bf6b61fe19..ba16ae551093 100644 --- a/sys/vm/vm_phys.c +++ b/sys/vm/vm_phys.c @@ -393,13 +393,23 @@ static void vm_freelist_add(struct vm_freelist *fl, vm_page_t m, int order, int pool, int tail) { + /* + * The paging queues and the free page lists utilize the same field, + * plinks.q, within the vm_page structure. When a physical page is + * freed, it is lazily removed from the paging queues to reduce the + * cost of removal through batching. Here, we must ensure that any + * deferred dequeue on the physical page has completed before using + * its plinks.q field. + */ + if (__predict_false(vm_page_astate_load(m).queue != PQ_NONE)) + vm_page_dequeue(m); m->order = order; m->pool = pool; if (tail) - TAILQ_INSERT_TAIL(&fl[order].pl, m, listq); + TAILQ_INSERT_TAIL(&fl[order].pl, m, plinks.q); else - TAILQ_INSERT_HEAD(&fl[order].pl, m, listq); + TAILQ_INSERT_HEAD(&fl[order].pl, m, plinks.q); fl[order].lcnt++; } @@ -407,7 +417,7 @@ static void vm_freelist_rem(struct vm_freelist *fl, vm_page_t m, int order) { - TAILQ_REMOVE(&fl[order].pl, m, listq); + TAILQ_REMOVE(&fl[order].pl, m, plinks.q); fl[order].lcnt--; m->order = VM_NFREEORDER; } @@ -1582,7 +1592,7 @@ vm_phys_find_freelist_contig(struct vm_freelist *fl, u_long npages, * check if there are enough free blocks starting at a properly aligned * block. Thus, no block is checked for free-ness more than twice. */ - TAILQ_FOREACH(m, &fl[max_order].pl, listq) { + TAILQ_FOREACH(m, &fl[max_order].pl, plinks.q) { /* * Skip m unless it is first in a sequence of free max page * blocks >= low in its segment. @@ -1655,7 +1665,7 @@ vm_phys_find_queues_contig( for (oind = order; oind < VM_NFREEORDER; oind++) { for (pind = vm_default_freepool; pind < VM_NFREEPOOL; pind++) { fl = (*queues)[pind]; - TAILQ_FOREACH(m_ret, &fl[oind].pl, listq) { + TAILQ_FOREACH(m_ret, &fl[oind].pl, plinks.q) { /* * Determine if the address range starting at pa * is within the given range, satisfies the