From owner-freebsd-current@FreeBSD.ORG Wed Jun 19 08:32:36 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 85A87478 for ; Wed, 19 Jun 2013 08:32:36 +0000 (UTC) (envelope-from chris.torek@gmail.com) Received: from elf.torek.net (50-73-42-1-utah.hfc.comcastbusiness.net [50.73.42.1]) by mx1.freebsd.org (Postfix) with ESMTP id 64B8D10F5 for ; Wed, 19 Jun 2013 08:32:36 +0000 (UTC) Received: from elf.torek.net (localhost [127.0.0.1]) by elf.torek.net (8.14.5/8.14.5) with ESMTP id r5J8WZFE082135 for ; Wed, 19 Jun 2013 02:32:35 -0600 (MDT) (envelope-from chris.torek@gmail.com) Message-Id: <201306190832.r5J8WZFE082135@elf.torek.net> From: Chris Torek To: freebsd-current@freebsd.org Subject: expanding past 1 TB on amd64 Date: Wed, 19 Jun 2013 02:32:35 -0600 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (elf.torek.net [127.0.0.1]); Wed, 19 Jun 2013 02:32:35 -0600 (MDT) X-Mailman-Approved-At: Wed, 19 Jun 2013 11:33:27 +0000 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 19 Jun 2013 08:32:36 -0000 In src/sys/amd64/include/vmparam.h is this handy map: * 0x0000000000000000 - 0x00007fffffffffff user map * 0x0000800000000000 - 0xffff7fffffffffff does not exist (hole) * 0xffff800000000000 - 0xffff804020100fff recursive page table (512GB slot) * 0xffff804020101000 - 0xfffffdffffffffff unused * 0xfffffe0000000000 - 0xfffffeffffffffff 1TB direct map * 0xffffff0000000000 - 0xffffff7fffffffff unused * 0xffffff8000000000 - 0xffffffffffffffff 512GB kernel map showing that the system can deal with at most 1 TB of address space (because of the direct map), using at most half of that for kernel memory (less, really, due to the inevitable VM fragmentation). New boards are coming soonish that will have the ability to go past that (24 DIMMs of 64 GB each = 1.5 TB). Or, if some crazy people :-) might want to use a most of a 768 GB board (24 DIMMs of 32 GB each, possible today although the price is kind of staggering) as wired-down kernel memory, the 512 GB VM area is already a problem. I have not wrapped my head around the amd64 pmap code but figured I'd ask: what might need to change to support larger spaces? Obviously NKPML4E in amd64/include/pmap.h, for the kernel start address; and NDMPML4E for the direct map. It looks like this would adjust KERNBASE and the direct map appropriately. But would that suffice, or have I missed something? For that matter, if these are changed to make space for future expansion, what would be a good expansion size? Perhaps multiply the sizes by 16? (If memory doubles roughly every 18 months, that should give room for at least 5 years.) Chris