From owner-freebsd-ppc@FreeBSD.ORG Mon Feb 25 03:08:32 2013 Return-Path: Delivered-To: ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4A26A380; Mon, 25 Feb 2013 03:08:32 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh10.mail.rice.edu (mh10.mail.rice.edu [128.42.201.30]) by mx1.freebsd.org (Postfix) with ESMTP id F00F796F; Mon, 25 Feb 2013 03:08:31 +0000 (UTC) Received: from mh10.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh10.mail.rice.edu (Postfix) with ESMTP id D1BF2603D8; Sun, 24 Feb 2013 21:08:30 -0600 (CST) Received: from mh10.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh10.mail.rice.edu (Postfix) with ESMTP id B29AE603E0; Sun, 24 Feb 2013 21:08:30 -0600 (CST) X-Virus-Scanned: by amavis-2.7.0 at mh10.mail.rice.edu, auth channel Received: from mh10.mail.rice.edu ([127.0.0.1]) by mh10.mail.rice.edu (mh10.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id Um-0YLlEr2IH; Sun, 24 Feb 2013 21:08:30 -0600 (CST) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh10.mail.rice.edu (Postfix) with ESMTPSA id 1B220603D8; Sun, 24 Feb 2013 21:08:30 -0600 (CST) Message-ID: <512AD5AD.3090303@rice.edu> Date: Sun, 24 Feb 2013 21:08:29 -0600 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:17.0) Gecko/20130127 Thunderbird/17.0.2 MIME-Version: 1.0 To: Nathan Whitehorn Subject: Re: kernel address space and auto-sizing References: <512AAB65.1000402@rice.edu> <512AB347.4030603@freebsd.org> In-Reply-To: <512AB347.4030603@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Kostik Belousov , ppc@freebsd.org, grehan@freebsd.org X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Feb 2013 03:08:32 -0000 On 02/24/2013 18:41, Nathan Whitehorn wrote: > This is bringing back bad memories. Let me explain about > VM_MAX_SAFE_KERNEL_ADDRESS and mmu_oea64.c briefly. > > mmu_oea64 is the PMAP code for both PPC32 and PPC64 running on 64-bit > CPUs. Both AIM32 and AIM64 in general have a direct map. *However*, > 32-bit kernels running on 64-bit CPUs do *not* have a direct map. > > The memory layout when the kernel is booted is has the low 4 GB > organized into 16 256 MB segments. We keep all firmware mappings and try > to fit in a direct map. Segments 12 and 13 (0xc, 0xd) are guaranteed to > be free. OF allocates its own mappings at the end of segment 14 (0xe). > Everything else can't be touched. Without a direct map, having only 512 > MB of KVA causes the kernel to regularly run out of VA space -- this was > crashing package building. To ameliorate that, I added the > MAX_SAFE_ADDRESS stuff and it then expands the KVA space from there into > segment 14 until it finds memory the firmware is actually using and ends > KVA there. I suspect this patch will cause a return to the crashing. Did > I miss something? Let's ignore the patch for the moment. Have you tried booting a 32-bit kernel on a 64-bit CPU since Andre's commit changing the mbuf auto-sizing? As I mention below, 32-bit AIM defines VM_KMEM_SIZE to 12 MB. Before Andre's commit, we sized the kmem submap as VM_KMEM_SIZE plus sufficient space for the maximum number of mbuf clusters, nmbclusters. Now, after Andre's change, nmbclusters hasn't been computed at the point the kmem submap is created. Since 32-bit AIM doesn't define VM_KMEM_SIZE_SCALE, the kmem submap will be stuck at 12 MB. Alan > > On 02/24/13 18:08, Alan Cox wrote: >> After the fallout from the recent auto-sizing changes, I've been >> reviewing the kernel address space configuration on all of our supported >> architectures. This review turned up a couple of odd things on the AIM >> platform: >> >> 1. Everywhere else, except for 32- and 64-bit AIM, VM_MAX_KERNEL_ADDRESS >> denotes the end of the kernel map. Instead, 32- and 64-bit AIM define >> VM_MAX_SAFE_KERNEL_ADDRESS and uses this to initialize the end of the >> kernel map. On occasion, we use VM_MAX_KERNEL_ADDRESS to implement >> auto-sizing based on the size of the kernel map, so this difference >> interferes with that. >> >> 2. The size of the kmem submap on 32-bit AIM is hardwired to only 12 >> MB! ARM had the same issue. There, the consequences were much worse >> because ARM, unlike 32-bit AIM, doesn't have a direct map, so >> uma_small_alloc() can't be used on arm. However, even with >> uma_small_alloc(), this 12 MB cap on the kmem submap still unnecessarily >> limits multipage allocations by the kernel. Even with a direct map, >> those come from the kmem submap. >> >> The attached patch does the following things to AIM: >> >> 1. On 32-bit AIM, it redefines VM_MAX_KERNEL_ADDRESS to be the address >> that should be the end of the kernel map and eliminates >> VM_MAX_SAFE_KERNEL_ADDRESS. On 64-bit AIM, VM_MAX_KERNEL_ADDRESS and >> VM_MAX_SAFE_KERNEL_ADDRESS were the same, so it simply eliminates >> VM_MAX_SAFE_KERNEL_ADDRESS. >> >> 2. It enables auto-sizing of the kmem submap on 32-bit AIM, but places a >> cap on its maximum size so that it will not consume the entire kernel >> map. (64-bit AIM already has auto-sizing and a cap for the kmem submap.) >> >> I would appreciate review and/or testing of this patch. In particular, >> I'd like to see the output from two sysctl's before and after applying >> the patch: vm.max_kernel_address, vm.kmem_size, and vm.kmem_size_max >> >> Thanks, >> Alan >> >> P.S. On 64-bit AIM, the following bit of code in the pmap bootstrap >> function was already a NOP because VM_MAX_KERNEL_ADDRESS and >> VM_MAX_SAFE_KERNEL_ADDRESS were the same. I didn't touch it, but I >> thought that I would mention it. >> >> #ifndef __powerpc64__ /* KVA is in high memory on PPC64 */ >> PMAP_LOCK(kernel_pmap); >> while (virtual_end < VM_MAX_KERNEL_ADDRESS && >> moea64_pvo_find_va(kernel_pmap, virtual_end+1) == NULL) >> virtual_end += PAGE_SIZE; >> PMAP_UNLOCK(kernel_pmap); >> #endif >> >> >