From owner-svn-src-head@freebsd.org Wed Feb 15 01:51:00 2017 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE187CDFCEF; Wed, 15 Feb 2017 01:51:00 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8875BBA5; Wed, 15 Feb 2017 01:51:00 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id v1F1oxFm063795; Wed, 15 Feb 2017 01:50:59 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id v1F1owRa063794; Wed, 15 Feb 2017 01:50:58 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201702150150.v1F1owRa063794@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Wed, 15 Feb 2017 01:50:58 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r313756 - head/sys/kern X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Feb 2017 01:51:00 -0000 Author: markj Date: Wed Feb 15 01:50:58 2017 New Revision: 313756 URL: https://svnweb.freebsd.org/changeset/base/313756 Log: Apply MADV_FREE to exec_map entries only after a lowmem event. This effectively provides the same benefit as applying MADV_FREE inline upon every execve, since the page daemon invokes lowmem handlers prior to scanning the inactive queue. It also has less overhead; the cost of applying MADV_FREE is very noticeable on many-CPU systems since it includes that of a TLB shootdown of global PTEs. For instance, this change nearly halves the system CPU usage during a buildkernel on a 128-vCPU EC2 instance (with some other patches applied). Benchmarked by: cperciva (earlier version) Reviewed by: kib MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D9586 Modified: head/sys/kern/kern_exec.c Modified: head/sys/kern/kern_exec.c ============================================================================== --- head/sys/kern/kern_exec.c Wed Feb 15 01:35:26 2017 (r313755) +++ head/sys/kern/kern_exec.c Wed Feb 15 01:50:58 2017 (r313756) @@ -1320,6 +1320,7 @@ err_exit: struct exec_args_kva { vm_offset_t addr; + u_int gen; SLIST_ENTRY(exec_args_kva) next; }; @@ -1327,6 +1328,7 @@ static DPCPU_DEFINE(struct exec_args_kva static SLIST_HEAD(, exec_args_kva) exec_args_kva_freelist; static struct mtx exec_args_kva_mtx; +static u_int exec_args_gen; static void exec_prealloc_args_kva(void *arg __unused) @@ -1339,6 +1341,7 @@ exec_prealloc_args_kva(void *arg __unuse for (i = 0; i < exec_map_entries; i++) { argkva = malloc(sizeof(*argkva), M_PARGS, M_WAITOK); argkva->addr = kmap_alloc_wait(exec_map, exec_map_entry_size); + argkva->gen = exec_args_gen; SLIST_INSERT_HEAD(&exec_args_kva_freelist, argkva, next); } } @@ -1364,15 +1367,16 @@ exec_alloc_args_kva(void **cookie) } static void -exec_free_args_kva(void *cookie) +exec_release_args_kva(struct exec_args_kva *argkva, u_int gen) { - struct exec_args_kva *argkva; vm_offset_t base; - argkva = cookie; base = argkva->addr; - - vm_map_madvise(exec_map, base, base + exec_map_entry_size, MADV_FREE); + if (argkva->gen != gen) { + vm_map_madvise(exec_map, base, base + exec_map_entry_size, + MADV_FREE); + argkva->gen = gen; + } if (!atomic_cmpset_ptr((uintptr_t *)DPCPU_PTR(exec_args_kva), (uintptr_t)NULL, (uintptr_t)argkva)) { mtx_lock(&exec_args_kva_mtx); @@ -1382,6 +1386,46 @@ exec_free_args_kva(void *cookie) } } +static void +exec_free_args_kva(void *cookie) +{ + + exec_release_args_kva(cookie, exec_args_gen); +} + +static void +exec_args_kva_lowmem(void *arg __unused) +{ + SLIST_HEAD(, exec_args_kva) head; + struct exec_args_kva *argkva; + u_int gen; + int i; + + gen = atomic_fetchadd_int(&exec_args_gen, 1) + 1; + + /* + * Force an madvise of each KVA range. Any currently allocated ranges + * will have MADV_FREE applied once they are freed. + */ + SLIST_INIT(&head); + mtx_lock(&exec_args_kva_mtx); + SLIST_SWAP(&head, &exec_args_kva_freelist, exec_args_kva); + mtx_unlock(&exec_args_kva_mtx); + while ((argkva = SLIST_FIRST(&head)) != NULL) { + SLIST_REMOVE_HEAD(&head, next); + exec_release_args_kva(argkva, gen); + } + + CPU_FOREACH(i) { + argkva = (void *)atomic_readandclear_ptr( + (uintptr_t *)DPCPU_ID_PTR(i, exec_args_kva)); + if (argkva != NULL) + exec_release_args_kva(argkva, gen); + } +} +EVENTHANDLER_DEFINE(vm_lowmem, exec_args_kva_lowmem, NULL, + EVENTHANDLER_PRI_ANY); + /* * Allocate temporary demand-paged, zero-filled memory for the file name, * argument, and environment strings.