Date: Sun, 24 Feb 2013 21:08:29 -0600 From: Alan Cox <alc@rice.edu> To: Nathan Whitehorn <nwhitehorn@freebsd.org> Cc: Kostik Belousov <kostikbel@gmail.com>, ppc@freebsd.org, grehan@freebsd.org Subject: Re: kernel address space and auto-sizing Message-ID: <512AD5AD.3090303@rice.edu> In-Reply-To: <512AB347.4030603@freebsd.org> References: <512AAB65.1000402@rice.edu> <512AB347.4030603@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 02/24/2013 18:41, Nathan Whitehorn wrote: > This is bringing back bad memories. Let me explain about > VM_MAX_SAFE_KERNEL_ADDRESS and mmu_oea64.c briefly. > > mmu_oea64 is the PMAP code for both PPC32 and PPC64 running on 64-bit > CPUs. Both AIM32 and AIM64 in general have a direct map. *However*, > 32-bit kernels running on 64-bit CPUs do *not* have a direct map. > > The memory layout when the kernel is booted is has the low 4 GB > organized into 16 256 MB segments. We keep all firmware mappings and try > to fit in a direct map. Segments 12 and 13 (0xc, 0xd) are guaranteed to > be free. OF allocates its own mappings at the end of segment 14 (0xe). > Everything else can't be touched. Without a direct map, having only 512 > MB of KVA causes the kernel to regularly run out of VA space -- this was > crashing package building. To ameliorate that, I added the > MAX_SAFE_ADDRESS stuff and it then expands the KVA space from there into > segment 14 until it finds memory the firmware is actually using and ends > KVA there. I suspect this patch will cause a return to the crashing. Did > I miss something? Let's ignore the patch for the moment. Have you tried booting a 32-bit kernel on a 64-bit CPU since Andre's commit changing the mbuf auto-sizing? As I mention below, 32-bit AIM defines VM_KMEM_SIZE to 12 MB. Before Andre's commit, we sized the kmem submap as VM_KMEM_SIZE plus sufficient space for the maximum number of mbuf clusters, nmbclusters. Now, after Andre's change, nmbclusters hasn't been computed at the point the kmem submap is created. Since 32-bit AIM doesn't define VM_KMEM_SIZE_SCALE, the kmem submap will be stuck at 12 MB. Alan > > On 02/24/13 18:08, Alan Cox wrote: >> After the fallout from the recent auto-sizing changes, I've been >> reviewing the kernel address space configuration on all of our supported >> architectures. This review turned up a couple of odd things on the AIM >> platform: >> >> 1. Everywhere else, except for 32- and 64-bit AIM, VM_MAX_KERNEL_ADDRESS >> denotes the end of the kernel map. Instead, 32- and 64-bit AIM define >> VM_MAX_SAFE_KERNEL_ADDRESS and uses this to initialize the end of the >> kernel map. On occasion, we use VM_MAX_KERNEL_ADDRESS to implement >> auto-sizing based on the size of the kernel map, so this difference >> interferes with that. >> >> 2. The size of the kmem submap on 32-bit AIM is hardwired to only 12 >> MB! ARM had the same issue. There, the consequences were much worse >> because ARM, unlike 32-bit AIM, doesn't have a direct map, so >> uma_small_alloc() can't be used on arm. However, even with >> uma_small_alloc(), this 12 MB cap on the kmem submap still unnecessarily >> limits multipage allocations by the kernel. Even with a direct map, >> those come from the kmem submap. >> >> The attached patch does the following things to AIM: >> >> 1. On 32-bit AIM, it redefines VM_MAX_KERNEL_ADDRESS to be the address >> that should be the end of the kernel map and eliminates >> VM_MAX_SAFE_KERNEL_ADDRESS. On 64-bit AIM, VM_MAX_KERNEL_ADDRESS and >> VM_MAX_SAFE_KERNEL_ADDRESS were the same, so it simply eliminates >> VM_MAX_SAFE_KERNEL_ADDRESS. >> >> 2. It enables auto-sizing of the kmem submap on 32-bit AIM, but places a >> cap on its maximum size so that it will not consume the entire kernel >> map. (64-bit AIM already has auto-sizing and a cap for the kmem submap.) >> >> I would appreciate review and/or testing of this patch. In particular, >> I'd like to see the output from two sysctl's before and after applying >> the patch: vm.max_kernel_address, vm.kmem_size, and vm.kmem_size_max >> >> Thanks, >> Alan >> >> P.S. On 64-bit AIM, the following bit of code in the pmap bootstrap >> function was already a NOP because VM_MAX_KERNEL_ADDRESS and >> VM_MAX_SAFE_KERNEL_ADDRESS were the same. I didn't touch it, but I >> thought that I would mention it. >> >> #ifndef __powerpc64__ /* KVA is in high memory on PPC64 */ >> PMAP_LOCK(kernel_pmap); >> while (virtual_end < VM_MAX_KERNEL_ADDRESS && >> moea64_pvo_find_va(kernel_pmap, virtual_end+1) == NULL) >> virtual_end += PAGE_SIZE; >> PMAP_UNLOCK(kernel_pmap); >> #endif >> >> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?512AD5AD.3090303>