From owner-freebsd-arm@FreeBSD.ORG Sat Feb 16 19:35:21 2013 Return-Path: Delivered-To: arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6A6624F1 for ; Sat, 16 Feb 2013 19:35:21 +0000 (UTC) (envelope-from ian@FreeBSD.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 251609E6 for ; Sat, 16 Feb 2013 19:35:20 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.6/8.14.6) with ESMTP id r1GJZK58057024 for ; Sat, 16 Feb 2013 12:35:20 -0700 (MST) (envelope-from ian@FreeBSD.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id r1GJYvBX046635; Sat, 16 Feb 2013 12:34:57 -0700 (MST) (envelope-from ian@FreeBSD.org) Subject: Re: kernel address space & auto-tuning From: Ian Lepore To: Alan Cox In-Reply-To: <511FD88E.2020403@rice.edu> References: <51192C44.1060204@rice.edu> <1361039490.1164.36.camel@revolution.hippie.lan> <511FD88E.2020403@rice.edu> Content-Type: text/plain; charset="us-ascii" Date: Sat, 16 Feb 2013 12:34:57 -0700 Message-ID: <1361043297.1164.43.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: "arm@freebsd.org" , Kostik Belousov X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Feb 2013 19:35:21 -0000 On Sat, 2013-02-16 at 13:05 -0600, Alan Cox wrote: > On 02/16/2013 12:31, Ian Lepore wrote: > > On Mon, 2013-02-11 at 11:37 -0600, Alan Cox wrote: > >> Ian and others here, > >> > >> Could you please test the attached patch? However, before you apply > >> this patch, can you run > >> > >> sysctl vm.max_kernel_address > >> > >> and record the output. I'd like to know how this output changes after > >> the patch is applied. > >> > >> Here is an explanation of what the patch does: > >> > >> 1. Recently, a #define for VM_KMEM_SIZE_SCALE was added to > >> arm/include/vmparam.h. This is good. However, it will create a problem > >> as soon as someone gets arm hardware with more than ~1.5GB of RAM. This > >> patch introduces a cap on the size of the kmem submap so that booting on > >> future larger-memory systems won't fail. > >> > >> 2. It appears that arm is more like sparc64 than x86 in the respect that > >> the ending address of the kernel address space can vary from machine to > >> machine. To be more precise, I'm talking about the kernel address space > >> into which we can dynamically map and unmap code/data and I'm not > >> including regions for device access or direct maps. Thus, the current > >> #define for VM_MAX_KERNEL_ADDRESS on arm is really incorrect or at least > >> inconsistent with how this is defined on other architectures. This > >> patch borrows from sparc64 in defining a global variable, > >> vm_max_kernel_address, rather than a constant #define to represent the > >> end of the kernel address space. > >> > >> Thanks, > >> Alan > >> > > Took me a while to get to this, here's the results... > > > > DreamPlug (armv5) > > > > real memory = 536870912 (512 MB) > > avail memory = 516612096 (492 MB) > > > > before: vm.max_kernel_address: 4294967295 0xFFFFFFFF > > after: vm.max_kernel_address: 4026531840 0xF0000000 > > > > rpi (armv6) > > > > real memory = 536870912 (512 MB) > > avail memory = 386789376 (368 MB) > > > > before: vm.max_kernel_address: 4294967295 0xFFFFFFFF > > after: vm.max_kernel_address: 3741319168 0xDF000000 > > > > Beaglebone (armv7) > > > > real memory = 268435456 (256 MB) > > avail memory = 254808064 (243 MB) > > > > before: vm.max_kernel_address: 4294967295 0xFFFFFFFF > > after: vm.max_kernel_address: 3741319168 0xDF000000 > > > > It appears that the values that go into making the max kernel address on > > various arm platforms may be choosen somewhat arbitrarily. Most SoCs > > map the on-chip register space "somewhere" way up near the end of the > > 32-bit virtual address space, and set the max kernel address to that > > point. 0xD0000000, 0xE0000000, and 0xF0000000 all seem to be popular > > choices. I just changed the value for DreamPlug, which only needs 1MB > > of register mappings, from 0xF0000000 to 0xFE000000 and it seems to be > > working fine. > > > > So, given that the variable ending address may be arbitrary and not much > > different than the current 0xFFFFFFFF constant, I wonder if your changes > > are going to have the intended effect? > > There are two parts to my change. One is making VM_MAX_KERNEL_ADDRESS > accurately reflect the end of the kernel's address space by making it a > variable. The other is placing a cap on the size of the kernel's heap, > i.e., the kmem submap. Without that cap, machines with larger physical > memories won't boot. In fact, I just saw an e-mail a few hours ago from > Oleksandr Tymoshenko saying that his 1GB Beaglebone won't boot anymore. > > > ... It may be that the important > > part of the change would come next: pick better arbitrary ending > > addresses. > > Is the location of the register mapping something that the kernel > chooses? Or is it something that is passed to the kernel at boot-time? > As near as I can tell, it's a pretty much arbitrary choice on a per-SoC basis, always hard-coded with a named constant of some sort in kernel source code. 0xE0000000 seems to be a popular choice, with comments about it being the address used to bootstrap the initial devmap. That appears to be true for only one SoC. I think maybe there's been some cut and paste propagation here. -- Ian