From nobody Fri Aug 7 02:52:39 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4hGTGJ1QdTz6nk2T for ; Fri, 07 Aug 2026 02:52:40 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4hGTGJ0l7Wz42Vs for ; Fri, 07 Aug 2026 02:52:40 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1786071160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=g99tRUyA5/UxtJY3KVhMtXtX2DdgQKbrxD/7pHqekmA=; b=tt4BdZYCq70Yw7cwiMWXFDPWS3++Iq5NKw3ouc5AAgyFlOPZKUE25WQdAErFsEI+Fb/liu D7za7PDaWN4rH971MBc+88oEZ2wzDqWVSNYYyzzFFpxXbw1YCaCeLjYxduE40tikAkOkkB 0YjxbPkf0gGqEtt7Tq/YLcCl+Ka/quiRnJAFT3escg9W/lr+l+IF+y+Mbq13ezQb72k+it Zi7chtn+nO18IXWRe7NFrq9Bijlj3YCIKOZjMNX7YD5XNvirhiqbnrQ/taCqAJOLUhHaqY RB7iCEWhUWGwbYSwKjXIhycnPI48AURvtuP3O3EQRD/Oj1j8b0mU4fPSK/OBrg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1786071160; a=rsa-sha256; cv=none; b=RMga4hkNMc/pdXXpsCrkw7JsQBQhNth7T9Loz9dj5xGq2t01KnfkMJ2zXOSz5fseAAda2r d30dJk4PLlX4jejc+Ah2iavCzqIyI1VBYXAUTyOpZcQ6i8/F7cdhW3GFAoQMKnfu3VoKx3 eQMstfCdD9l9xpLhGe9egEa/L5gU9SUCmDp2iEBjlMcaNopJlhnCZUq/ZIEmBINM+aHrGW /eBY2LbY70l7F+tB0l0OanoFA7rAcaoW6Fyxb9OBtnfl5i1geu9Hg6AQN1qXrgGKdW8Emd vTk+0qXJyb8MR9UlwKHa0FnPRBOyEF5/hsHmDgo6+hIISpGlLhV1LHl5j7WhqA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1786071160; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=g99tRUyA5/UxtJY3KVhMtXtX2DdgQKbrxD/7pHqekmA=; b=XdA53cvQJA/+gI/qlUrpdHnIFW8qFxORKIZ5miU+BVD06rIT6331wOREGgxis4uz3O5Oxn 73qoOpQauXBksryj17hxz6WhZM9iyrVOK0TaMPG8mbSuAJjaHHiDTsLOVxY58WN6nRIeIQ Jvxtf5sE+xIdJQOElXL8fMdFeX3GCcayPTJWLLqTX+3bCXmAn3NC1/qPDNjl1bqbcRyqOF S4sa4CDmZLmeQe2hJV6HDEzw/IaOvQPau9Qm648Q21gwEAbmBUyrAAK67HTp+0T50gp488 z4vNW39UCLoc3qNY7FOyP+Cp7gRMcCM00ixDoZo6oRppOi07pFBLUNTSNDznYA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4hGTGH6rGnzprG for ; Fri, 07 Aug 2026 02:52:39 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 1a3f7 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Fri, 07 Aug 2026 02:52:39 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Justin Hibbits Subject: git: b83204edf101 - main - powerpc/pmap: Support booke64 kernel pmap growing List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: jhibbits X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: b83204edf101d2c8ad40005ca2fe9a76796cd4f1 Auto-Submitted: auto-generated Date: Fri, 07 Aug 2026 02:52:39 +0000 Message-Id: <6a754877.1a3f7.3865dab@gitrepo.freebsd.org> The branch main has been updated by jhibbits: URL: https://cgit.FreeBSD.org/src/commit/?id=b83204edf101d2c8ad40005ca2fe9a76796cd4f1 commit b83204edf101d2c8ad40005ca2fe9a76796cd4f1 Author: Justin Hibbits AuthorDate: 2026-08-05 03:47:27 +0000 Commit: Justin Hibbits CommitDate: 2026-08-07 02:52:28 +0000 powerpc/pmap: Support booke64 kernel pmap growing In preparation of increasing the KVA on powerpc64 to 2TB to mirror amd64's, rework the 64-bit Book-E pmap to not allocate all page table pages at boot time, since that would be a waste of a lot of memory. Instead, allocate all page table pages for the higher levels, leaving the leaves (page directories) for dynamic allocation. This cuts the boot-time page table size down from ~64MB to ~8MB with the current 32GB KVA size, and bumping to 2TB KVA the boot-time page table is still ~8MB instead of ballooning to ~4GB of mostly wasted space. --- sys/powerpc/booke/pmap.c | 18 +++++++-- sys/powerpc/booke/pmap_64.c | 95 ++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 101 insertions(+), 12 deletions(-) diff --git a/sys/powerpc/booke/pmap.c b/sys/powerpc/booke/pmap.c index 545416921d09..ce16566a6f13 100644 --- a/sys/powerpc/booke/pmap.c +++ b/sys/powerpc/booke/pmap.c @@ -359,6 +359,9 @@ static int mmu_booke_decode_kernel_ptr(vm_offset_t addr, static void mmu_booke_page_array_startup(long); static bool mmu_booke_page_is_mapped(vm_page_t m); static bool mmu_booke_ps_enabled(pmap_t pmap); +#ifdef __powerpc64__ +static int mmu_booke_growkernel(vm_offset_t); +#endif static struct pmap_funcs mmu_booke_methods = { /* pmap dispatcher interface */ @@ -402,6 +405,9 @@ static struct pmap_funcs mmu_booke_methods = { .page_array_startup = mmu_booke_page_array_startup, .page_is_mapped = mmu_booke_page_is_mapped, .ps_enabled = mmu_booke_ps_enabled, +#ifdef __powerpc64__ + .growkernel_nopanic = mmu_booke_growkernel, +#endif /* Internal interfaces */ .bootstrap = mmu_booke_bootstrap, @@ -700,7 +706,6 @@ mmu_booke_bootstrap(vm_offset_t start, vm_offset_t kernelend) (uintptr_t)msgbufp, data_end); data_end = round_page(data_end); - data_end = round_page(mmu_booke_alloc_kernel_pgtables(data_end)); /* Retrieve phys/avail mem regions */ mem_regions(&physmem_regions, &physmem_regions_sz, @@ -709,7 +714,6 @@ mmu_booke_bootstrap(vm_offset_t start, vm_offset_t kernelend) if (PHYS_AVAIL_ENTRIES < availmem_regions_sz) panic("mmu_booke_bootstrap: phys_avail too small"); - data_end = round_page(data_end); vm_page_array = (vm_page_t)data_end; /* * Get a rough idea (upper bound) on the size of the page array. The @@ -723,6 +727,14 @@ mmu_booke_bootstrap(vm_offset_t start, vm_offset_t kernelend) sz = (round_page(sz) / (PAGE_SIZE + sizeof(struct vm_page))); data_end += round_page(sz * sizeof(struct vm_page)); + /* + * Reserve kernel page-table pages last, so their reservation size can + * be computed from the final bootstrap data_end (on 64-bit, only leaf + * ptbls covering [VM_MIN_KERNEL_ADDRESS, data_end + slack] are + * pre-allocated; the rest are added on demand by pmap_growkernel()). + */ + data_end = round_page(mmu_booke_alloc_kernel_pgtables(data_end)); + /* Pre-round up to 1MB. This wastes some space, but saves TLB entries */ data_end = roundup2(data_end, 1 << 20); @@ -1216,7 +1228,7 @@ mmu_booke_kremove(vm_offset_t va) pte = pte_find(kernel_pmap, va); - if (!PTE_ISVALID(pte)) { + if (pte == NULL || !PTE_ISVALID(pte)) { CTR1(KTR_PMAP, "%s: invalid pte", __func__); return; diff --git a/sys/powerpc/booke/pmap_64.c b/sys/powerpc/booke/pmap_64.c index 97f9a5967167..3f2db07ce931 100644 --- a/sys/powerpc/booke/pmap_64.c +++ b/sys/powerpc/booke/pmap_64.c @@ -113,6 +113,13 @@ unsigned int kernel_pdirs; static uma_zone_t ptbl_root_zone; static pte_t ****kernel_ptbl_root; +/* + * Slack space beyond the bootstrap data_end for which leaf ptbls are + * pre-allocated. Must cover post-alloc growth (1MB roundup + TLB1 grow + + * kstack + guard pages). Rounded up to PDIR_SIZE by the reservation code. + */ +#define BOOTSTRAP_LEAF_SLACK (2 * PDIR_SIZE) + /* * Base of the pmap_mapdev() region. On 32-bit it immediately follows the * userspace address range. On On 64-bit it's far above, at (1 << 63), and @@ -262,14 +269,16 @@ get_pgtbl_page(pmap_t pmap, void **ptr_tbl, uint32_t index, vm_page_t m; page = ptr_tbl[index]; - KASSERT(page != 0 || pmap != kernel_pmap, - ("NULL page table page found in kernel pmap!")); if (page == NULL) { page = mmu_booke_alloc_page(pmap, index, nosleep); if (ptr_tbl[index] == NULL) { *isnew = true; ptr_tbl[index] = page; - if (hold_parent) { + /* + * Kernel page-table pages are never freed, so we do + * not maintain refcounts on their parents. + */ + if (hold_parent && pmap != kernel_pmap) { m = PHYS_TO_VM_PAGE(pmap_kextract((vm_offset_t)ptr_tbl)); m->ref_count++; } @@ -518,7 +527,8 @@ kernel_pte_alloc(vm_offset_t data_end, vm_offset_t addr) { pte_t *pte; vm_size_t kva_size; - int kernel_pdirs, kernel_pgtbls, pdir_l1s; + int kernel_pdirs, pdir_l1s; + unsigned int kernel_pgtbls; vm_offset_t va, l1_va, pdir_va, ptbl_va; int i, j, k; @@ -526,7 +536,8 @@ kernel_pte_alloc(vm_offset_t data_end, vm_offset_t addr) kernel_pmap->pm_root = kernel_ptbl_root; pdir_l1s = howmany(kva_size, PG_ROOT_SIZE); kernel_pdirs = howmany(kva_size, PDIR_L1_SIZE); - kernel_pgtbls = howmany(kva_size, PDIR_SIZE); + kernel_pgtbls = howmany(kernel_vm_end - VM_MIN_KERNEL_ADDRESS, + PDIR_SIZE); /* Initialize kernel pdir */ l1_va = (vm_offset_t)kernel_ptbl_root + @@ -537,7 +548,8 @@ kernel_pte_alloc(vm_offset_t data_end, vm_offset_t addr) printf("ptbl_root_va: %#lx\n", (vm_offset_t)kernel_ptbl_root); printf("l1_va: %#lx (%d entries)\n", l1_va, pdir_l1s); printf("pdir_va: %#lx(%d entries)\n", pdir_va, kernel_pdirs); - printf("ptbl_va: %#lx(%d entries)\n", ptbl_va, kernel_pgtbls); + printf("ptbl_va: %#lx(%u entries)\n", ptbl_va, kernel_pgtbls); + printf("kernel_vm_end: %#lx\n", kernel_vm_end); } va = VM_MIN_KERNEL_ADDRESS; @@ -550,8 +562,18 @@ kernel_pte_alloc(vm_offset_t data_end, vm_offset_t addr) kernel_pmap->pm_root[i][j] = (pte_t **)pdir_va; for (k = 0; k < PDIR_NENTRIES && va < VM_MAX_KERNEL_ADDRESS; - k++, va += PDIR_SIZE, ptbl_va += PAGE_SIZE) - kernel_pmap->pm_root[i][j][k] = (pte_t *)ptbl_va; + k++, va += PDIR_SIZE) { + /* + * Only wire up leaf ptbl pages for the + * pre-allocated bootstrap KVA range; the rest + * are populated lazily by mmu_booke_growkernel(). + */ + if (va < kernel_vm_end) { + kernel_pmap->pm_root[i][j][k] = + (pte_t *)ptbl_va; + ptbl_va += PAGE_SIZE; + } + } } } /* @@ -572,16 +594,71 @@ static vm_offset_t mmu_booke_alloc_kernel_pgtables(vm_offset_t data_end) { vm_size_t kva_size = VM_MAX_KERNEL_ADDRESS - VM_MIN_KERNEL_ADDRESS; + unsigned int leaves; + kernel_ptbl_root = (pte_t ****)data_end; data_end += round_page(PG_ROOT_NENTRIES * sizeof(pte_t ***)); data_end += howmany(kva_size, PG_ROOT_SIZE) * PAGE_SIZE; data_end += howmany(kva_size, PDIR_L1_SIZE) * PAGE_SIZE; - data_end += howmany(kva_size, PDIR_SIZE) * PAGE_SIZE; + /* + * Reserve leaf page-table pages only for the current bootstrap KVA + * range plus a small slack for post-alloc growth (1MB TLB roundup, + * kstack + guard). Leaves for the rest of KVA are allocated on + * demand by mmu_booke_growkernel(). The two-step howmany() converges + * because leaves themselves add at most one extra leaf per 512 leaves. + */ + leaves = howmany(data_end - VM_MIN_KERNEL_ADDRESS + + BOOTSTRAP_LEAF_SLACK, PDIR_SIZE); + leaves = howmany(data_end - VM_MIN_KERNEL_ADDRESS + + BOOTSTRAP_LEAF_SLACK + leaves * PAGE_SIZE, PDIR_SIZE); + data_end += leaves * PAGE_SIZE; + + kernel_vm_end = VM_MIN_KERNEL_ADDRESS + leaves * PDIR_SIZE; return (data_end); } +/* + * Grow the kernel page-table by allocating additional leaf ptbl pages so + * that pte_find(kernel_pmap, va) can succeed for [kernel_vm_end, addr]. + * + * All upper levels (root, pdir_l1, pdir) are pre-populated for the whole + * KVA at bootstrap, so only the leaf level needs to be allocated here. + */ +static int +mmu_booke_growkernel(vm_offset_t addr) +{ + pte_t **pdir; + vm_page_t m; + vm_offset_t va; + + if (addr <= kernel_vm_end) + return (KERN_SUCCESS); + + addr = roundup2(addr, PDIR_SIZE); + if (addr - 1 >= vm_map_max(kernel_map)) + addr = vm_map_max(kernel_map); + + for (va = kernel_vm_end; va < addr; va += PDIR_SIZE) { + pdir = kernel_pmap->pm_root[PG_ROOT_IDX(va)][PDIR_L1_IDX(va)]; + KASSERT(pdir != NULL, + ("mmu_booke_growkernel: NULL pdir at va %#lx", va)); + if (pdir[PDIR_IDX(va)] != NULL) + continue; + m = vm_page_alloc_noobj(VM_ALLOC_INTERRUPT | VM_ALLOC_WIRED | + VM_ALLOC_ZERO); + if (m == NULL) { + kernel_vm_end = va; + return (KERN_RESOURCE_SHORTAGE); + } + m->pindex = va >> PDIR_SHIFT; + pdir[PDIR_IDX(va)] = (pte_t *)VM_PAGE_TO_DMAP(m); + } + kernel_vm_end = addr; + return (KERN_SUCCESS); +} + /* * Initialize a preallocated and zeroed pmap structure, * such as one in a vmspace structure.