From owner-svn-src-all@freebsd.org Tue Sep 3 13:18:52 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8FAB6DABE8; Tue, 3 Sep 2019 13:18:52 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46N6wX3D72z4LXJ; Tue, 3 Sep 2019 13:18:52 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 51CEE332E; Tue, 3 Sep 2019 13:18:52 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x83DIqJl024298; Tue, 3 Sep 2019 13:18:52 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x83DIpWb024294; Tue, 3 Sep 2019 13:18:51 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201909031318.x83DIpWb024294@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Tue, 3 Sep 2019 13:18:51 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r351742 - in head/sys: amd64/amd64 amd64/include vm X-SVN-Group: head X-SVN-Commit-Author: markj X-SVN-Commit-Paths: in head/sys: amd64/amd64 amd64/include vm X-SVN-Commit-Revision: 351742 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Sep 2019 13:18:52 -0000 Author: markj Date: Tue Sep 3 13:18:51 2019 New Revision: 351742 URL: https://svnweb.freebsd.org/changeset/base/351742 Log: Map the vm_page array into KVA on amd64. r351198 allows the kernel to use domain-local memory to back the vm_page array (up to 2MB boundaries) and reserves a separate PML4 entry for that purpose. One consequence of that change is that the vm_page array is no longer present in minidumps, which only adds pages mapped above VM_MIN_KERNEL_ADDRESS. To avoid the friction caused by having kernel data structures mapped below VM_MIN_KERNEL_ADDRESS, map the vm_page array starting at VM_MIN_KERNEL_ADDRESS instead of using a dedicated PML4 entry. Reviewed by: kib Discussed with: jeff Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21491 Modified: head/sys/amd64/amd64/pmap.c head/sys/amd64/include/vmparam.h head/sys/vm/vm_kern.c head/sys/vm/vm_page.c Modified: head/sys/amd64/amd64/pmap.c ============================================================================== --- head/sys/amd64/amd64/pmap.c Tue Sep 3 12:54:51 2019 (r351741) +++ head/sys/amd64/amd64/pmap.c Tue Sep 3 13:18:51 2019 (r351742) @@ -384,9 +384,6 @@ static u_int64_t DMPDphys; /* phys addr of direct mapp static u_int64_t DMPDPphys; /* phys addr of direct mapped level 3 */ static int ndmpdpphys; /* number of DMPDPphys pages */ -static uint64_t PAPDPphys; /* phys addr of page array level 3 */ -static int npapdpphys; /* number of PAPDPphys pages */ - static vm_paddr_t KERNend; /* phys addr of end of bootstrap data */ /* @@ -1431,16 +1428,6 @@ create_pagetables(vm_paddr_t *firstaddr) pml4_entry_t *p4_p; uint64_t DMPDkernphys; - npapdpphys = howmany(ptoa(Maxmem) / sizeof(struct vm_page), NBPML4); - if (npapdpphys > NPAPML4E) { - printf("NDMPML4E limits system to %lu GB\n", - (NDMPML4E * 512) * (PAGE_SIZE / sizeof(struct vm_page))); - npapdpphys = NPAPML4E; - Maxmem = atop(NPAPML4E * NBPML4 * - (PAGE_SIZE / sizeof(struct vm_page))); - } - PAPDPphys = allocpages(firstaddr, npapdpphys); - /* Allocate page table pages for the direct map */ ndmpdp = howmany(ptoa(Maxmem), NBPDP); if (ndmpdp < 4) /* Minimum 4GB of dirmap */ @@ -1587,12 +1574,6 @@ create_pagetables(vm_paddr_t *firstaddr) p4_p[KPML4BASE + i] = KPDPphys + ptoa(i); p4_p[KPML4BASE + i] |= X86_PG_RW | X86_PG_V; } - - /* Connect the page array slots up to the pml4. */ - for (i = 0; i < npapdpphys; i++) { - p4_p[PAPML4I + i] = PAPDPphys + ptoa(i); - p4_p[PAPML4I + i] |= X86_PG_RW | X86_PG_V | pg_nx; - } } /* @@ -3456,11 +3437,6 @@ pmap_pinit_pml4(vm_page_t pml4pg) X86_PG_V; } - for (i = 0; i < npapdpphys; i++) { - pm_pml4[PAPML4I + i] = (PAPDPphys + ptoa(i)) | X86_PG_RW | - X86_PG_V; - } - /* install self-referential address mapping entry(s) */ pm_pml4[PML4PML4I] = VM_PAGE_TO_PHYS(pml4pg) | X86_PG_V | X86_PG_RW | X86_PG_A | X86_PG_M; @@ -3817,8 +3793,6 @@ pmap_release(pmap_t pmap) pmap->pm_pml4[KPML4BASE + i] = 0; for (i = 0; i < ndmpdpphys; i++)/* Direct Map */ pmap->pm_pml4[DMPML4I + i] = 0; - for (i = 0; i < npapdpphys; i++) - pmap->pm_pml4[PAPML4I + i] = 0; pmap->pm_pml4[PML4PML4I] = 0; /* Recursive Mapping */ for (i = 0; i < lm_ents; i++) /* Large Map */ pmap->pm_pml4[LMSPML4I + i] = 0; @@ -3856,6 +3830,10 @@ kvm_free(SYSCTL_HANDLER_ARGS) SYSCTL_PROC(_vm, OID_AUTO, kvm_free, CTLTYPE_LONG|CTLFLAG_RD, 0, 0, kvm_free, "LU", "Amount of KVM free"); +/* + * Allocate physical memory for the vm_page array and map it into KVA, + * attempting to back the vm_pages with domain-local memory. + */ void pmap_page_array_startup(long pages) { @@ -3868,14 +3846,15 @@ pmap_page_array_startup(long pages) vm_page_array_size = pages; - start = va = PA_MIN_ADDRESS; - end = va + (pages * sizeof(struct vm_page)); + start = va = VM_MIN_KERNEL_ADDRESS; + end = va + pages * sizeof(struct vm_page); while (va < end) { - pfn = first_page + ((va - start) / sizeof(struct vm_page)); + pfn = first_page + (va - start) / sizeof(struct vm_page); domain = _vm_phys_domain(ctob(pfn)); pdpe = pmap_pdpe(kernel_pmap, va); if ((*pdpe & X86_PG_V) == 0) { pa = vm_phys_early_alloc(domain, PAGE_SIZE); + dump_add_page(pa); bzero((void *)PHYS_TO_DMAP(pa), PAGE_SIZE); *pdpe = (pdp_entry_t)(pa | X86_PG_V | X86_PG_RW | X86_PG_A | X86_PG_M); @@ -3892,6 +3871,7 @@ pmap_page_array_startup(long pages) pde_store(pde, newpdir); va += NBPDR; } + vm_page_array = (vm_page_t)start; } /* Modified: head/sys/amd64/include/vmparam.h ============================================================================== --- head/sys/amd64/include/vmparam.h Tue Sep 3 12:54:51 2019 (r351741) +++ head/sys/amd64/include/vmparam.h Tue Sep 3 13:18:51 2019 (r351742) @@ -160,13 +160,12 @@ * 0xffff808000000000 - 0xffff847fffffffff large map (can be tuned up) * 0xffff848000000000 - 0xfffff7ffffffffff unused (large map extends there) * 0xfffff80000000000 - 0xfffffbffffffffff 4TB direct map - * 0xfffffc0000000000 - 0xfffffcffffffffff unused - * 0xfffffd0000000000 - 0xfffffd7fffffffff page array 512GB - * 0xfffffd8000000000 - 0xfffffdffffffffff unused + * 0xfffffc0000000000 - 0xfffffdffffffffff unused * 0xfffffe0000000000 - 0xffffffffffffffff 2TB kernel map * * Within the kernel map: * + * 0xfffffe0000000000 vm_page_array * 0xffffffff80000000 KERNBASE */ @@ -216,10 +215,10 @@ (x) & ~DMAP_MIN_ADDRESS; }) /* - * amd64 statically allocates the page array address so that it can - * be more easily allocated on the correct memory domains. + * amd64 maps the page array into KVA so that it can be more easily + * allocated on the correct memory domains. */ -#define PMAP_HAS_PAGE_ARRAY 1 +#define PMAP_HAS_PAGE_ARRAY 1 /* * How many physical pages per kmem arena virtual page. Modified: head/sys/vm/vm_kern.c ============================================================================== --- head/sys/vm/vm_kern.c Tue Sep 3 12:54:51 2019 (r351741) +++ head/sys/vm/vm_kern.c Tue Sep 3 13:18:51 2019 (r351742) @@ -759,7 +759,7 @@ kmem_init(vm_offset_t start, vm_offset_t end) vm_map_lock(m); /* N.B.: cannot use kgdb to debug, starting with this assignment ... */ kernel_map = m; - (void) vm_map_insert(m, NULL, (vm_ooffset_t) 0, + (void)vm_map_insert(m, NULL, 0, #ifdef __amd64__ KERNBASE, #else @@ -767,6 +767,18 @@ kmem_init(vm_offset_t start, vm_offset_t end) #endif start, VM_PROT_ALL, VM_PROT_ALL, MAP_NOFAULT); /* ... and ending with the completion of the above `insert' */ + +#ifdef __amd64__ + /* + * Mark KVA used for the page array as allocated. Other platforms + * that handle vm_page_array allocation can simply adjust virtual_avail + * instead. + */ + (void)vm_map_insert(m, NULL, 0, (vm_offset_t)vm_page_array, + (vm_offset_t)vm_page_array + round_2mpage(vm_page_array_size * + sizeof(struct vm_page)), + VM_PROT_RW, VM_PROT_RW, MAP_NOFAULT); +#endif vm_map_unlock(m); /* Modified: head/sys/vm/vm_page.c ============================================================================== --- head/sys/vm/vm_page.c Tue Sep 3 12:54:51 2019 (r351741) +++ head/sys/vm/vm_page.c Tue Sep 3 13:18:51 2019 (r351742) @@ -135,11 +135,7 @@ static int vm_pageproc_waiters; */ vm_page_t bogus_page; -#ifdef PMAP_HAS_PAGE_ARRAY -vm_page_t vm_page_array = (vm_page_t)PA_MIN_ADDRESS; -#else vm_page_t vm_page_array; -#endif long vm_page_array_size; long first_page;