From owner-svn-src-all@FreeBSD.ORG Wed Dec 11 21:48:05 2013 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 39FA8FAB; Wed, 11 Dec 2013 21:48:05 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2694B1B2E; Wed, 11 Dec 2013 21:48:05 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.7/8.14.7) with ESMTP id rBBLm5dA066055; Wed, 11 Dec 2013 21:48:05 GMT (envelope-from mav@svn.freebsd.org) Received: (from mav@localhost) by svn.freebsd.org (8.14.7/8.14.7/Submit) id rBBLm5vP066054; Wed, 11 Dec 2013 21:48:05 GMT (envelope-from mav@svn.freebsd.org) Message-Id: <201312112148.rBBLm5vP066054@svn.freebsd.org> From: Alexander Motin Date: Wed, 11 Dec 2013 21:48:04 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r259232 - head/sys/kern X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Dec 2013 21:48:05 -0000 Author: mav Date: Wed Dec 11 21:48:04 2013 New Revision: 259232 URL: http://svnweb.freebsd.org/changeset/base/259232 Log: Create own free list for each of the first 32 possible allocation sizes. In case of 4K allocation quantum that means for allocations up to 128K. With growth of memory fragmentation these lists may grow to quite a large sizes (tenths and hundreds of thousands items). Having in one list items of different sizes in worst case may require full linear list traversal, that may be very expensive. Having lists for items of single size means that unless user specify some alignment or border requirements (that are very rare cases) first item found on the list should satisfy the request. While running SPEC NFS benchmark on top of ZFS on 24-core machine with 84GB RAM this change reduces CPU time spent in vmem_xalloc() from 8% and lock congestion spinning around it from 20% to invisible levels. And that all is by the cost of just 26 more pointers per vmem instance. If at some point our kernel will start to actively use KVA allocations with odd sizes above 128K, something may need to be done to bigger lists also. Modified: head/sys/kern/subr_vmem.c Modified: head/sys/kern/subr_vmem.c ============================================================================== --- head/sys/kern/subr_vmem.c Wed Dec 11 21:41:21 2013 (r259231) +++ head/sys/kern/subr_vmem.c Wed Dec 11 21:48:04 2013 (r259232) @@ -70,7 +70,10 @@ __FBSDID("$FreeBSD$"); #include #include -#define VMEM_MAXORDER (sizeof(vmem_size_t) * NBBY) +#define VMEM_OPTORDER 5 +#define VMEM_OPTVALUE (1 << VMEM_OPTORDER) +#define VMEM_MAXORDER \ + (VMEM_OPTVALUE - 1 + sizeof(vmem_size_t) * NBBY - VMEM_OPTORDER) #define VMEM_HASHSIZE_MIN 16 #define VMEM_HASHSIZE_MAX 131072 @@ -200,8 +203,10 @@ static LIST_HEAD(, vmem) vmem_list = LIS #define VMEM_CROSS_P(addr1, addr2, boundary) \ ((((addr1) ^ (addr2)) & -(boundary)) != 0) -#define ORDER2SIZE(order) ((vmem_size_t)1 << (order)) -#define SIZE2ORDER(size) ((int)flsl(size) - 1) +#define ORDER2SIZE(order) ((order) < VMEM_OPTVALUE ? ((order) + 1) : \ + (vmem_size_t)1 << ((order) - (VMEM_OPTVALUE - VMEM_OPTORDER - 1))) +#define SIZE2ORDER(size) ((size) <= VMEM_OPTVALUE ? ((size) - 1) : \ + (flsl(size) + (VMEM_OPTVALUE - VMEM_OPTORDER - 2))) /* * Maximum number of boundary tags that may be required to satisfy an @@ -334,11 +339,14 @@ bt_free(vmem_t *vm, bt_t *bt) /* * freelist[0] ... [1, 1] - * freelist[1] ... [2, 3] - * freelist[2] ... [4, 7] - * freelist[3] ... [8, 15] + * freelist[1] ... [2, 2] * : - * freelist[n] ... [(1 << n), (1 << (n + 1)) - 1] + * freelist[29] ... [30, 30] + * freelist[30] ... [31, 31] + * freelist[31] ... [32, 63] + * freelist[33] ... [64, 127] + * : + * freelist[n] ... [(1 << (n - 26)), (1 << (n - 25)) - 1] * : */ @@ -979,6 +987,7 @@ vmem_init(vmem_t *vm, const char *name, int i; MPASS(quantum > 0); + MPASS((quantum & (quantum - 1)) == 0); bzero(vm, sizeof(*vm)); @@ -988,8 +997,7 @@ vmem_init(vmem_t *vm, const char *name, LIST_INIT(&vm->vm_freetags); strlcpy(vm->vm_name, name, sizeof(vm->vm_name)); vm->vm_quantum_mask = quantum - 1; - vm->vm_quantum_shift = SIZE2ORDER(quantum); - MPASS(ORDER2SIZE(vm->vm_quantum_shift) == quantum); + vm->vm_quantum_shift = flsl(quantum) - 1; vm->vm_nbusytag = 0; vm->vm_size = 0; vm->vm_inuse = 0;