From owner-freebsd-hackers@FreeBSD.ORG Mon Jul 1 21:47:37 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 78FAA28B; Mon, 1 Jul 2013 21:47:37 +0000 (UTC) (envelope-from torek@torek.net) Received: from elf.torek.net (50-73-42-1-utah.hfc.comcastbusiness.net [50.73.42.1]) by mx1.freebsd.org (Postfix) with ESMTP id 3C00C13CC; Mon, 1 Jul 2013 21:47:36 +0000 (UTC) Received: from elf.torek.net (localhost [127.0.0.1]) by elf.torek.net (8.14.5/8.14.5) with ESMTP id r61LlUr7002898; Mon, 1 Jul 2013 15:47:30 -0600 (MDT) (envelope-from torek@torek.net) Message-Id: <201307012147.r61LlUr7002898@elf.torek.net> From: Chris Torek To: freebsd-hackers@freebsd.org Subject: Re: expanding amd64 past the 1TB limit In-reply-to: Your message of "Fri, 28 Jun 2013 14:33:55 -0600." <201306282033.r5SKXtYK053022@elf.torek.net> Date: Mon, 01 Jul 2013 15:47:30 -0600 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (elf.torek.net [127.0.0.1]); Mon, 01 Jul 2013 15:47:30 -0600 (MDT) Cc: Konstantin Belousov , alc@freebsd.org, kib@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jul 2013 21:47:37 -0000 I went ahead and implemented the "ndmpdpphys" change I had thought about. Here's that patch by itself. At this point it might be reasonable to use the patch without the AMD64_HUGE option, just increasing the KVM area, as the direct map no longer consumes extra pages. (There's a glitch in the comment in the original patch, I'll fix that if/when there's any agreement on whether it gets applied in some form :-) ) Chris diff --git a/amd64/amd64/pmap.c b/amd64/amd64/pmap.c index acf5af2..f1ed8b6 100644 --- a/amd64/amd64/pmap.c +++ b/amd64/amd64/pmap.c @@ -232,6 +232,7 @@ u_int64_t KPML4phys; /* phys addr of kernel level 4 */ static u_int64_t DMPDphys; /* phys addr of direct mapped level 2 */ static u_int64_t DMPDPphys; /* phys addr of direct mapped level 3 */ +static int ndmpdpphys; /* number of DMPDPphys pages */ static struct rwlock_padalign pvh_global_lock; @@ -543,7 +544,18 @@ create_pagetables(vm_paddr_t *firstaddr) ndmpdp = (ptoa(Maxmem) + NBPDP - 1) >> PDPSHIFT; if (ndmpdp < 4) /* Minimum 4GB of dirmap */ ndmpdp = 4; - DMPDPphys = allocpages(firstaddr, NDMPML4E); + ndmpdpphys = howmany(ndmpdp, NPML4EPG); + if (ndmpdpphys > NDMPML4E) { + /* + * Each NDMPML4E allows 512 GB, so limit to that, + * and then readjust ndmpdp and ndmpdpphys. + */ + printf("NDMPML4E limits system to %d GB\n", NDMPML4E * 512); + Maxmem = atop(NDMPML4E * NBPML4); + ndmpdpphys = NDMPML4E; + ndmpdp = NDMPML4E * NPDEPG; + } + DMPDPphys = allocpages(firstaddr, ndmpdpphys); ndm1g = 0; if ((amd_feature & AMDID_PAGE1GB) != 0) ndm1g = ptoa(Maxmem) >> PDPSHIFT; @@ -626,7 +638,7 @@ create_pagetables(vm_paddr_t *firstaddr) p4_p[PML4PML4I] |= PG_RW | PG_V | PG_U; /* Connect the Direct Map slot(s) up to the PML4. */ - for (i = 0; i < NDMPML4E; i++) { + for (i = 0; i < ndmpdpphys; i++) { p4_p[DMPML4I + i] = DMPDPphys + ptoa(i); p4_p[DMPML4I + i] |= PG_RW | PG_V | PG_U; } @@ -1698,7 +1710,7 @@ pmap_pinit(pmap_t pmap) pmap->pm_pml4[KPML4BASE + i] = (KPDPphys + (i << PAGE_SHIFT)) | PG_RW | PG_V | PG_U; } - for (i = 0; i < NDMPML4E; i++) { + for (i = 0; i < ndmpdpphys; i++) { pmap->pm_pml4[DMPML4I + i] = (DMPDPphys + (i << PAGE_SHIFT)) | PG_RW | PG_V | PG_U; } @@ -1955,7 +1967,7 @@ pmap_release(pmap_t pmap) for (i = 0; i < NKPML4E; i++) /* KVA */ pmap->pm_pml4[KPML4BASE + i] = 0; - for (i = 0; i < NDMPML4E; i++) /* Direct Map */ + for (i = 0; i < ndmpdpphys; i++)/* Direct Map */ pmap->pm_pml4[DMPML4I + i] = 0; pmap->pm_pml4[PML4PML4I] = 0; /* Recursive Mapping */