From owner-svn-src-user@FreeBSD.ORG Thu Feb 14 15:23:02 2013 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B999EFBE; Thu, 14 Feb 2013 15:23:02 +0000 (UTC) (envelope-from attilio@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) by mx1.freebsd.org (Postfix) with ESMTP id 936A9A9C; Thu, 14 Feb 2013 15:23:02 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.5/8.14.5) with ESMTP id r1EFN2dK057865; Thu, 14 Feb 2013 15:23:02 GMT (envelope-from attilio@svn.freebsd.org) Received: (from attilio@localhost) by svn.freebsd.org (8.14.5/8.14.5/Submit) id r1EFN1ER057854; Thu, 14 Feb 2013 15:23:01 GMT (envelope-from attilio@svn.freebsd.org) Message-Id: <201302141523.r1EFN1ER057854@svn.freebsd.org> From: Attilio Rao Date: Thu, 14 Feb 2013 15:23:01 +0000 (UTC) To: src-committers@freebsd.org, svn-src-user@freebsd.org Subject: svn commit: r246794 - in user/attilio/vmc-playground: . sys/vm X-SVN-Group: user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Feb 2013 15:23:02 -0000 Author: attilio Date: Thu Feb 14 15:23:00 2013 New Revision: 246794 URL: http://svnweb.freebsd.org/changeset/base/246794 Log: The radix preallocation pages can overfow the biggestone segment, so use a different scheme for preallocation: reserve few KB of nodes to be used to cater page allocations before the memory can be efficiently pre-allocated by UMA. This at all effects remove boot_pages further carving and along with this modifies to the boot_pages allocation system and necessity to initialize the UMA zone before pmap_init(). Reported by: pho, jhb Modified: user/attilio/vmc-playground/UPDATING user/attilio/vmc-playground/sys/vm/uma_core.c user/attilio/vmc-playground/sys/vm/uma_int.h user/attilio/vmc-playground/sys/vm/vm_init.c user/attilio/vmc-playground/sys/vm/vm_page.c user/attilio/vmc-playground/sys/vm/vm_radix.c user/attilio/vmc-playground/sys/vm/vm_radix.h Modified: user/attilio/vmc-playground/UPDATING ============================================================================== --- user/attilio/vmc-playground/UPDATING Thu Feb 14 15:17:53 2013 (r246793) +++ user/attilio/vmc-playground/UPDATING Thu Feb 14 15:23:00 2013 (r246794) @@ -26,13 +26,6 @@ NOTE TO PEOPLE WHO THINK THAT FreeBSD 10 disable the most expensive debugging functionality run "ln -s 'abort:false,junk:false' /etc/malloc.conf".) -20130211: - The vm.boot_pages tunable and sysctl are removed and new - vm.initial_boot_pages tunable and sysctl are introduced. The new - adds represent the number of pages that are initially reserved to the - system for startup. The number of pages can however be extended if - necessary by the system during the boot stage. - 20130129: A BSD-licensed patch(1) variant has been added and is installed as bsdpatch, being the GNU version the default patch. Modified: user/attilio/vmc-playground/sys/vm/uma_core.c ============================================================================== --- user/attilio/vmc-playground/sys/vm/uma_core.c Thu Feb 14 15:17:53 2013 (r246793) +++ user/attilio/vmc-playground/sys/vm/uma_core.c Thu Feb 14 15:23:00 2013 (r246794) @@ -329,7 +329,7 @@ bucket_alloc(int entries, int bflags) /* * This is to stop us from allocating per cpu buckets while we're - * running out of boot_pages. Otherwise, we would exhaust the + * running out of vm.boot_pages. Otherwise, we would exhaust the * boot pages. This also prevents us from allocating buckets in * low memory situations. */ @@ -984,7 +984,7 @@ startup_alloc(uma_zone_t zone, int bytes } mtx_unlock(&uma_boot_pages_mtx); if (booted < UMA_STARTUP2) - panic("UMA: Increase vm.initial_boot_pages"); + panic("UMA: Increase vm.boot_pages"); /* * Now that we've booted reset these users to their real allocator. */ Modified: user/attilio/vmc-playground/sys/vm/uma_int.h ============================================================================== --- user/attilio/vmc-playground/sys/vm/uma_int.h Thu Feb 14 15:17:53 2013 (r246793) +++ user/attilio/vmc-playground/sys/vm/uma_int.h Thu Feb 14 15:23:00 2013 (r246794) @@ -118,8 +118,7 @@ #define UMA_SLAB_MASK (PAGE_SIZE - 1) /* Mask to get back to the page */ #define UMA_SLAB_SHIFT PAGE_SHIFT /* Number of bits PAGE_MASK */ -/* Initial pages allocated for startup */ -#define UMA_INIT_BOOT_PAGES 64 +#define UMA_BOOT_PAGES 64 /* Pages allocated for startup */ /* Max waste before going to off page slab management */ #define UMA_MAX_WASTE (UMA_SLAB_SIZE / 10) Modified: user/attilio/vmc-playground/sys/vm/vm_init.c ============================================================================== --- user/attilio/vmc-playground/sys/vm/vm_init.c Thu Feb 14 15:17:53 2013 (r246793) +++ user/attilio/vmc-playground/sys/vm/vm_init.c Thu Feb 14 15:23:00 2013 (r246794) @@ -82,7 +82,6 @@ __FBSDID("$FreeBSD$"); #include #include #include -#include #include #include #include @@ -124,7 +123,6 @@ vm_mem_init(dummy) vm_object_init(); vm_map_startup(); kmem_init(virtual_avail, virtual_end); - vm_radix_init(); pmap_init(); vm_pager_init(); } Modified: user/attilio/vmc-playground/sys/vm/vm_page.c ============================================================================== --- user/attilio/vmc-playground/sys/vm/vm_page.c Thu Feb 14 15:17:53 2013 (r246793) +++ user/attilio/vmc-playground/sys/vm/vm_page.c Thu Feb 14 15:23:00 2013 (r246794) @@ -145,10 +145,10 @@ long vm_page_array_size; long first_page; int vm_page_zero_count; -static int boot_pages = UMA_INIT_BOOT_PAGES; -TUNABLE_INT("vm.initial_boot_pages", &boot_pages); -SYSCTL_INT(_vm, OID_AUTO, initial_boot_pages, CTLFLAG_RD, &boot_pages, 0, - "Initial number of pages allocated for bootstrapping the VM system"); +static int boot_pages = UMA_BOOT_PAGES; +TUNABLE_INT("vm.boot_pages", &boot_pages); +SYSCTL_INT(_vm, OID_AUTO, boot_pages, CTLFLAG_RD, &boot_pages, 0, + "number of pages allocated for bootstrapping the VM system"); static int pa_tryrelock_restart; SYSCTL_INT(_vm, OID_AUTO, tryrelock_restart, CTLFLAG_RD, @@ -307,7 +307,7 @@ vm_page_startup(vm_offset_t vaddr) low_water = 0; #endif - new_end = phys_avail[biggestone+1]; + end = phys_avail[biggestone+1]; /* * Initialize the page and queue locks. @@ -318,6 +318,17 @@ vm_page_startup(vm_offset_t vaddr) for (i = 0; i < PQ_COUNT; i++) vm_pagequeue_init_lock(&vm_pagequeues[i]); + /* + * Allocate memory for use when boot strapping the kernel memory + * allocator. + */ + new_end = end - (boot_pages * UMA_SLAB_SIZE); + new_end = trunc_page(new_end); + mapped = pmap_map(&vaddr, new_end, end, + VM_PROT_READ | VM_PROT_WRITE); + bzero((void *)mapped, end - new_end); + uma_startup((void *)mapped, boot_pages); + #if defined(__amd64__) || defined(__i386__) || defined(__arm__) || \ defined(__mips__) /* @@ -373,20 +384,6 @@ vm_page_startup(vm_offset_t vaddr) end = new_end; /* - * Allocate memory for use when boot strapping the kernel memory - * allocator. - */ - boot_pages += howmany(vm_radix_allocphys_size(page_range), - UMA_SLAB_SIZE - UMA_MAX_WASTE); - new_end = end - (boot_pages * UMA_SLAB_SIZE); - new_end = trunc_page(new_end); - mapped = pmap_map(&vaddr, new_end, end, - VM_PROT_READ | VM_PROT_WRITE); - bzero((void *)mapped, end - new_end); - uma_startup((void *)mapped, boot_pages); - end = new_end; - - /* * Reserve an unmapped guard page to trap access to vm_page_array[-1]. */ vaddr += PAGE_SIZE; Modified: user/attilio/vmc-playground/sys/vm/vm_radix.c ============================================================================== --- user/attilio/vmc-playground/sys/vm/vm_radix.c Thu Feb 14 15:17:53 2013 (r246793) +++ user/attilio/vmc-playground/sys/vm/vm_radix.c Thu Feb 14 15:23:00 2013 (r246794) @@ -67,6 +67,8 @@ #include #endif +#define VM_RADIX_BOOT_CACHE 1500 + /* * Such sizes should permit to keep node children contained into a single * cache-line, or to at least not span many of those. @@ -102,6 +104,14 @@ struct vm_radix_node { static uma_zone_t vm_radix_node_zone; +/* + * Boot-time cache of struct vm_radix_node objects. + * This cache is used to cater page allocations before the UMA zone is + * actually setup and pre-allocated (ie. pmap_init()). + */ +static u_int boot_cache_cnt; +static struct vm_radix_node boot_cache[VM_RADIX_BOOT_CACHE]; + #ifdef INVARIANTS /* * Radix node zone destructor. @@ -127,22 +137,29 @@ vm_radix_node_get(vm_pindex_t owner, uin { struct vm_radix_node *rnode; - rnode = uma_zalloc(vm_radix_node_zone, M_NOWAIT | M_ZERO); + if (boot_cache_cnt <= VM_RADIX_BOOT_CACHE) { + if (boot_cache_cnt == VM_RADIX_BOOT_CACHE) + panic("%s: Increase VM_RADIX_BOOT_CACHE", __func__); + rnode = &boot_cache[boot_cache_cnt]; + boot_cache_cnt++; + } else { + rnode = uma_zalloc(vm_radix_node_zone, M_NOWAIT | M_ZERO); - /* - * The required number of nodes might be already correctly - * pre-allocated in vm_radix_init(). However, UMA can reserve few - * nodes on per-cpu specific buckets, which will not be accessible - * from the curcpu. The allocation could then return NULL when the - * pre-allocation pool is close to be exhausted. - * Anyway, in practice this should never be a problem because a new - * node is not always required for insert, thus the pre-allocation - * pool should already have some extra-pages that indirectly deal with - * this situation. - */ - if (rnode == NULL) - panic("%s: uma_zalloc() returned NULL for a new node", - __func__); + /* + * The required number of nodes might be already correctly + * pre-allocated in vm_radix_init(). However, UMA can reserve + * few nodes on per-cpu specific buckets, which will not be + * accessible from the curcpu. The allocation could then + * return NULL when the pre-allocation pool is close to be + * exhausted. Anyway, in practice this should never be a + * problem because a new node is not always required for + * insert, thus the pre-allocation pool should already have + * some extra-pages that indirectly deal with this situation. + */ + if (rnode == NULL) + panic("%s: uma_zalloc() returned NULL for a new node", + __func__); + } rnode->rn_owner = owner; rnode->rn_count = count; rnode->rn_clev = clevel; @@ -156,6 +173,8 @@ static __inline void vm_radix_node_put(struct vm_radix_node *rnode) { + if (rnode > boot_cache && rnode <= &boot_cache[VM_RADIX_BOOT_CACHE]) + return; uma_zfree(vm_radix_node_zone, rnode); } @@ -341,21 +360,12 @@ vm_radix_reclaim_allnodes_int(struct vm_ } /* - * Returns the amount of requested memory to satisfy nodes pre-allocation. - */ -size_t -vm_radix_allocphys_size(size_t nitems) -{ - - return (nitems * sizeof(struct vm_radix_node)); -} - -/* * Pre-allocate intermediate nodes from the UMA slab zone. */ -void -vm_radix_init(void) +static void +vm_radix_init(void *arg __unused) { + int nitems; vm_radix_node_zone = uma_zcreate("RADIX NODE", sizeof(struct vm_radix_node), NULL, @@ -365,8 +375,11 @@ vm_radix_init(void) NULL, #endif NULL, NULL, VM_RADIX_PAD, UMA_ZONE_VM | UMA_ZONE_NOFREE); - uma_prealloc(vm_radix_node_zone, vm_page_array_size); + nitems = uma_zone_set_max(vm_radix_node_zone, vm_page_array_size); + uma_prealloc(vm_radix_node_zone, nitems); + boot_cache_cnt = VM_RADIX_BOOT_CACHE + 1; } +SYSINIT(vm_radix_init, SI_SUB_KMEM, SI_ORDER_SECOND, vm_radix_init, NULL); /* * Inserts the key-value pair in to the trie. Modified: user/attilio/vmc-playground/sys/vm/vm_radix.h ============================================================================== --- user/attilio/vmc-playground/sys/vm/vm_radix.h Thu Feb 14 15:17:53 2013 (r246793) +++ user/attilio/vmc-playground/sys/vm/vm_radix.h Thu Feb 14 15:23:00 2013 (r246794) @@ -34,8 +34,6 @@ #ifdef _KERNEL -size_t vm_radix_allocphys_size(size_t nitems); -void vm_radix_init(void); void vm_radix_insert(struct vm_radix *rtree, vm_pindex_t index, vm_page_t page); vm_page_t vm_radix_lookup(struct vm_radix *rtree, vm_pindex_t index);