From owner-freebsd-current@FreeBSD.ORG Tue Jul 16 23:12:46 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7A0614FB; Tue, 16 Jul 2013 23:12:46 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id 25AB6823; Tue, 16 Jul 2013 23:12:45 +0000 (UTC) Received: from mail.cicely.de ([10.1.1.37]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id r6GMxDHd040350 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Wed, 17 Jul 2013 00:59:13 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (cicely7.cicely.de [10.1.1.9]) by mail.cicely.de (8.14.5/8.14.4) with ESMTP id r6GMx6K6090096 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 17 Jul 2013 00:59:06 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.14.2/8.14.2) with ESMTP id r6GMx6cD093364; Wed, 17 Jul 2013 00:59:06 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.14.2/8.14.2/Submit) id r6GMx6SS093363; Wed, 17 Jul 2013 00:59:06 +0200 (CEST) (envelope-from ticso) Date: Wed, 17 Jul 2013 00:59:06 +0200 From: Bernd Walter To: alc@freebsd.org Subject: Max mmap space (was: expanding past 1 TB on amd64) Message-ID: <20130716225906.GT76364@cicely7.cicely.de> References: <51E553C4.9000207@pix.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD cicely7.cicely.de 7.0-STABLE i386 User-Agent: Mutt/1.5.11 X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED=-1, BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01 autolearn=ham version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on spamd.cicely.de Cc: freebsd-current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: ticso@cicely.de List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jul 2013 23:12:46 -0000 On Tue, Jul 16, 2013 at 02:12:42PM -0700, Alan Cox wrote: > On Tue, Jul 16, 2013 at 7:08 AM, Kurt Lidl wrote: > > > On Wed, Jun 19, 2013 at 1:32 AM, Chris Torek > >> wrote: > >> > >> In src/sys/amd64/include/vmparam.**h is this handy map: > >>> > >>> * 0x0000000000000000 - 0x00007fffffffffff user map > >>> * 0x0000800000000000 - 0xffff7fffffffffff does not exist (hole) > >>> * 0xffff800000000000 - 0xffff804020100fff recursive page table (512GB > >>> slot) > >>> * 0xffff804020101000 - 0xfffffdffffffffff unused > >>> * 0xfffffe0000000000 - 0xfffffeffffffffff 1TB direct map > >>> * 0xffffff0000000000 - 0xffffff7fffffffff unused > >>> * 0xffffff8000000000 - 0xffffffffffffffff 512GB kernel map > >> > >> The actual data that I've seen shows that DIMMs are doubling in size at > >> about half that pace, about every three years. For example, see > >> http://users.ece.cmu.edu/~**omutlu/pub/mutlu_memory-** > >> scaling_imw13_invited-talk.**pdfslide > >> #8. So, I think that a factor of 16 is a lot more than we'll need in > >> the next five years. I would suggest configuring the kernel virtual > >> address space for 4 TB. Once you go beyond 512 GB, 4 TB is the net > >> "plateau" in terms of address translation cost. At 4 TB all of the PML4 > >> entries for the kernel virtual address space will reside in the same L2 > >> cache line, so a page table walk on a TLB miss for an instruction fetch > >> will effectively prefetch the PML4 entry for the kernel heap and vice > >> versa. > >> > > > > The largest commodity motherboards that are shipping today support > > 24 DIMMs, at a max size of 32GB per DIMM. That's 768GB, right now. > > (So FreeBSD is already "out of bits" in terms of supporting current > > shipping hardware.) > > > > Actually, this scenario with 768 GB of RAM on amd64 as it is today is > analogous to the typical 32-bit i386 machine, where the amount of RAM has > long exceeded the default 1 GB size of the kernel virtual address space. > In theory, we could currently handle up to 1 TB of RAM, but the kernel > virtual address space would only be 512 GB. Talking about virtual address space. I plan to permanently mmap serveral nGB sized files (all together 6-8TB) into a single process address space. Actually I see the user map is 128TB, so I shouldn't get into trouble by doing this and also have lot of additional space left to avoid problems because of fragmentation. The system has 192G physical memory, so mapping tables have enough space. Is there anything else to worry about? -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.