From owner-svn-src-head@freebsd.org Sat Jul 7 01:54:46 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 70780102367A; Sat, 7 Jul 2018 01:54:46 +0000 (UTC) (envelope-from jeff@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 237587B887; Sat, 7 Jul 2018 01:54:46 +0000 (UTC) (envelope-from jeff@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DF36110F07; Sat, 7 Jul 2018 01:54:45 +0000 (UTC) (envelope-from jeff@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w671sjbt070522; Sat, 7 Jul 2018 01:54:45 GMT (envelope-from jeff@FreeBSD.org) Received: (from jeff@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w671sjZK070521; Sat, 7 Jul 2018 01:54:45 GMT (envelope-from jeff@FreeBSD.org) Message-Id: <201807070154.w671sjZK070521@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jeff set sender to jeff@FreeBSD.org using -f From: Jeff Roberson Date: Sat, 7 Jul 2018 01:54:45 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r336055 - head/sys/vm X-SVN-Group: head X-SVN-Commit-Author: jeff X-SVN-Commit-Paths: head/sys/vm X-SVN-Commit-Revision: 336055 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Jul 2018 01:54:46 -0000 Author: jeff Date: Sat Jul 7 01:54:45 2018 New Revision: 336055 URL: https://svnweb.freebsd.org/changeset/base/336055 Log: Use the ticks since the last update to reduce hysteresis in the partpopq and contention on the vm_reserv_domain lock. This gives a roughly 8x speedup on will-it-scale fault1 on a 16 core machine. Reviewed by: alc, kib, markj Modified: head/sys/vm/vm_reserv.c Modified: head/sys/vm/vm_reserv.c ============================================================================== --- head/sys/vm/vm_reserv.c Sat Jul 7 00:41:04 2018 (r336054) +++ head/sys/vm/vm_reserv.c Sat Jul 7 01:54:45 2018 (r336055) @@ -116,6 +116,12 @@ typedef u_long popmap_t; #define NPOPMAP howmany(VM_LEVEL_0_NPAGES, NBPOPMAP) /* + * Number of elapsed ticks before we update the LRU queue position. Used + * to reduce contention and churn on the list. + */ +#define PARTPOPSLOP 1 + +/* * Clear a bit in the population map. */ static __inline void @@ -183,6 +189,7 @@ struct vm_reserv { vm_page_t pages; /* (c) first page */ uint16_t domain; /* (c) NUMA domain. */ uint16_t popcnt; /* (r) # of pages in use */ + int lasttick; /* (r) last pop update tick. */ char inpartpopq; /* (d) */ popmap_t popmap[NPOPMAP]; /* (r) bit vector, used pages */ }; @@ -394,6 +401,7 @@ vm_reserv_insert(vm_reserv_t rv, vm_object_t object, v vm_reserv_object_lock(object); rv->pindex = pindex; rv->object = object; + rv->lasttick = ticks; LIST_INSERT_HEAD(&object->rvq, rv, objq); vm_reserv_object_unlock(object); } @@ -430,16 +438,20 @@ vm_reserv_depopulate(vm_reserv_t rv, int index) } popmap_clear(rv->popmap, index); rv->popcnt--; - vm_reserv_domain_lock(rv->domain); - if (rv->inpartpopq) { - TAILQ_REMOVE(&vm_rvq_partpop[rv->domain], rv, partpopq); - rv->inpartpopq = FALSE; + if ((unsigned)(ticks - rv->lasttick) >= PARTPOPSLOP || + rv->popcnt == 0) { + vm_reserv_domain_lock(rv->domain); + if (rv->inpartpopq) { + TAILQ_REMOVE(&vm_rvq_partpop[rv->domain], rv, partpopq); + rv->inpartpopq = FALSE; + } + if (rv->popcnt != 0) { + rv->inpartpopq = TRUE; + TAILQ_INSERT_TAIL(&vm_rvq_partpop[rv->domain], rv, partpopq); + } + vm_reserv_domain_unlock(rv->domain); + rv->lasttick = ticks; } - if (rv->popcnt != 0) { - rv->inpartpopq = TRUE; - TAILQ_INSERT_TAIL(&vm_rvq_partpop[rv->domain], rv, partpopq); - } - vm_reserv_domain_unlock(rv->domain); vmd = VM_DOMAIN(rv->domain); if (rv->popcnt == 0) { vm_reserv_remove(rv); @@ -536,6 +548,10 @@ vm_reserv_populate(vm_reserv_t rv, int index) rv, rv->domain)); popmap_set(rv->popmap, index); rv->popcnt++; + if ((unsigned)(ticks - rv->lasttick) < PARTPOPSLOP && + rv->inpartpopq && rv->popcnt != VM_LEVEL_0_NPAGES) + return; + rv->lasttick = ticks; vm_reserv_domain_lock(rv->domain); if (rv->inpartpopq) { TAILQ_REMOVE(&vm_rvq_partpop[rv->domain], rv, partpopq);