From owner-freebsd-arm@FreeBSD.ORG Sat Feb 16 19:51:23 2013 Return-Path: Delivered-To: arm@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 5CB1385E; Sat, 16 Feb 2013 19:51:23 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh2.mail.rice.edu (mh2.mail.rice.edu [128.42.201.21]) by mx1.freebsd.org (Postfix) with ESMTP id 0BFF3A5C; Sat, 16 Feb 2013 19:51:22 +0000 (UTC) Received: from mh2.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh2.mail.rice.edu (Postfix) with ESMTP id BD07450012F; Sat, 16 Feb 2013 13:51:22 -0600 (CST) Received: from mh2.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh2.mail.rice.edu (Postfix) with ESMTP id BBA57500129; Sat, 16 Feb 2013 13:51:22 -0600 (CST) X-Virus-Scanned: by amavis-2.7.0 at mh2.mail.rice.edu, auth channel Received: from mh2.mail.rice.edu ([127.0.0.1]) by mh2.mail.rice.edu (mh2.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id HUTS3RkPuuW4; Sat, 16 Feb 2013 13:51:22 -0600 (CST) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh2.mail.rice.edu (Postfix) with ESMTPSA id 4E39750011D; Sat, 16 Feb 2013 13:51:22 -0600 (CST) Message-ID: <511FE339.1010703@rice.edu> Date: Sat, 16 Feb 2013 13:51:21 -0600 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:17.0) Gecko/20130127 Thunderbird/17.0.2 MIME-Version: 1.0 To: Ian Lepore Subject: Re: kernel address space & auto-tuning References: <51192C44.1060204@rice.edu> <1361039490.1164.36.camel@revolution.hippie.lan> <511FD88E.2020403@rice.edu> <1361043297.1164.43.camel@revolution.hippie.lan> In-Reply-To: <1361043297.1164.43.camel@revolution.hippie.lan> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "arm@freebsd.org" , Kostik Belousov X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Feb 2013 19:51:23 -0000 On 02/16/2013 13:34, Ian Lepore wrote: > On Sat, 2013-02-16 at 13:05 -0600, Alan Cox wrote: >> On 02/16/2013 12:31, Ian Lepore wrote: >>> On Mon, 2013-02-11 at 11:37 -0600, Alan Cox wrote: >>>> Ian and others here, >>>> >>>> Could you please test the attached patch? However, before you apply >>>> this patch, can you run >>>> >>>> sysctl vm.max_kernel_address >>>> >>>> and record the output. I'd like to know how this output changes after >>>> the patch is applied. >>>> >>>> Here is an explanation of what the patch does: >>>> >>>> 1. Recently, a #define for VM_KMEM_SIZE_SCALE was added to >>>> arm/include/vmparam.h. This is good. However, it will create a problem >>>> as soon as someone gets arm hardware with more than ~1.5GB of RAM. This >>>> patch introduces a cap on the size of the kmem submap so that booting on >>>> future larger-memory systems won't fail. >>>> >>>> 2. It appears that arm is more like sparc64 than x86 in the respect that >>>> the ending address of the kernel address space can vary from machine to >>>> machine. To be more precise, I'm talking about the kernel address space >>>> into which we can dynamically map and unmap code/data and I'm not >>>> including regions for device access or direct maps. Thus, the current >>>> #define for VM_MAX_KERNEL_ADDRESS on arm is really incorrect or at least >>>> inconsistent with how this is defined on other architectures. This >>>> patch borrows from sparc64 in defining a global variable, >>>> vm_max_kernel_address, rather than a constant #define to represent the >>>> end of the kernel address space. >>>> >>>> Thanks, >>>> Alan >>>> >>> Took me a while to get to this, here's the results... >>> >>> DreamPlug (armv5) >>> >>> real memory = 536870912 (512 MB) >>> avail memory = 516612096 (492 MB) >>> >>> before: vm.max_kernel_address: 4294967295 0xFFFFFFFF >>> after: vm.max_kernel_address: 4026531840 0xF0000000 >>> >>> rpi (armv6) >>> >>> real memory = 536870912 (512 MB) >>> avail memory = 386789376 (368 MB) >>> >>> before: vm.max_kernel_address: 4294967295 0xFFFFFFFF >>> after: vm.max_kernel_address: 3741319168 0xDF000000 >>> >>> Beaglebone (armv7) >>> >>> real memory = 268435456 (256 MB) >>> avail memory = 254808064 (243 MB) >>> >>> before: vm.max_kernel_address: 4294967295 0xFFFFFFFF >>> after: vm.max_kernel_address: 3741319168 0xDF000000 >>> >>> It appears that the values that go into making the max kernel address on >>> various arm platforms may be choosen somewhat arbitrarily. Most SoCs >>> map the on-chip register space "somewhere" way up near the end of the >>> 32-bit virtual address space, and set the max kernel address to that >>> point. 0xD0000000, 0xE0000000, and 0xF0000000 all seem to be popular >>> choices. I just changed the value for DreamPlug, which only needs 1MB >>> of register mappings, from 0xF0000000 to 0xFE000000 and it seems to be >>> working fine. >>> >>> So, given that the variable ending address may be arbitrary and not much >>> different than the current 0xFFFFFFFF constant, I wonder if your changes >>> are going to have the intended effect? >> There are two parts to my change. One is making VM_MAX_KERNEL_ADDRESS >> accurately reflect the end of the kernel's address space by making it a >> variable. The other is placing a cap on the size of the kernel's heap, >> i.e., the kmem submap. Without that cap, machines with larger physical >> memories won't boot. In fact, I just saw an e-mail a few hours ago from >> Oleksandr Tymoshenko saying that his 1GB Beaglebone won't boot anymore. >> >>> ... It may be that the important >>> part of the change would come next: pick better arbitrary ending >>> addresses. >> Is the location of the register mapping something that the kernel >> chooses? Or is it something that is passed to the kernel at boot-time? >> > As near as I can tell, it's a pretty much arbitrary choice on a per-SoC > basis, always hard-coded with a named constant of some sort in kernel > source code. 0xE0000000 seems to be a popular choice, with comments > about it being the address used to bootstrap the initial devmap. That > appears to be true for only one SoC. I think maybe there's been some > cut and paste propagation here. > Then, it sounds like VM_MAX_KERNEL_ADDRESS doesn't need to be defined as a variable (as it is on sparc64). It just needs to be correctly defined by each SoC configuration.