From owner-svn-src-head@freebsd.org Tue Jul 2 19:51:42 2019 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB13D15DBF90; Tue, 2 Jul 2019 19:51:41 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8AC6D6D706; Tue, 2 Jul 2019 19:51:41 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 55A3E1003; Tue, 2 Jul 2019 19:51:41 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x62JpfDW018485; Tue, 2 Jul 2019 19:51:41 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x62JpfAl018483; Tue, 2 Jul 2019 19:51:41 GMT (envelope-from markj@FreeBSD.org) Message-Id: <201907021951.x62JpfAl018483@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Tue, 2 Jul 2019 19:51:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r349612 - head/sys/vm X-SVN-Group: head X-SVN-Commit-Author: markj X-SVN-Commit-Paths: head/sys/vm X-SVN-Commit-Revision: 349612 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 8AC6D6D706 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.95 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; NEURAL_HAM_SHORT(-0.96)[-0.956,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jul 2019 19:51:42 -0000 Author: markj Date: Tue Jul 2 19:51:40 2019 New Revision: 349612 URL: https://svnweb.freebsd.org/changeset/base/349612 Log: Mark pages allocated from the per-CPU cache. Only free pages to the cache when they were allocated from that cache. This mitigates rapid fragmentation of physical memory seen during poudriere's dependency calculation phase. In particular, pages belonging to broken reservations are no longer freed to the per-CPU cache, so they get a chance to coalesce with freed pages during the break. Otherwise, the optimized CoW handler may create object chains in which multiple objects contain pages from the same reservation, and the order in which we do object termination means that the reservation is broken before all of those pages are freed, so some of them end up in the per-CPU cache and thus permanently fragment physical memory. The flag may also be useful for eliding calls to vm_reserv_free_page(), thus avoiding memory accesses for data that is likely not present in the CPU caches. Reviewed by: alc Discussed with: jeff MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D20763 Modified: head/sys/vm/vm_page.c head/sys/vm/vm_page.h Modified: head/sys/vm/vm_page.c ============================================================================== --- head/sys/vm/vm_page.c Tue Jul 2 19:32:48 2019 (r349611) +++ head/sys/vm/vm_page.c Tue Jul 2 19:51:40 2019 (r349612) @@ -1812,8 +1812,9 @@ vm_page_alloc_domain_after(vm_object_t object, vm_pind if (object != NULL) VM_OBJECT_ASSERT_WLOCKED(object); -again: + flags = 0; m = NULL; +again: #if VM_NRESERVLEVEL > 0 /* * Can we allocate the page from a reservation? @@ -1829,8 +1830,10 @@ again: vmd = VM_DOMAIN(domain); if (object != NULL && vmd->vmd_pgcache != NULL) { m = uma_zalloc(vmd->vmd_pgcache, M_NOWAIT); - if (m != NULL) + if (m != NULL) { + flags |= PG_PCPU_CACHE; goto found; + } } if (vm_domain_allocate(vmd, req, 1)) { /* @@ -1858,10 +1861,8 @@ again: } /* - * At this point we had better have found a good page. + * At this point we had better have found a good page. */ - KASSERT(m != NULL, ("missing page")); - found: vm_page_dequeue(m); vm_page_alloc_check(m); @@ -1869,10 +1870,8 @@ found: /* * Initialize the page. Only the PG_ZERO flag is inherited. */ - flags = 0; if ((req & VM_ALLOC_ZERO) != 0) - flags = PG_ZERO; - flags &= m->flags; + flags |= (m->flags & PG_ZERO); if ((req & VM_ALLOC_NODUMP) != 0) flags |= PG_NODUMP; m->flags = flags; @@ -2018,6 +2017,7 @@ vm_page_alloc_contig_domain(vm_object_t object, vm_pin * Can we allocate the pages without the number of free pages falling * below the lower bound for the allocation class? */ + m_ret = NULL; again: #if VM_NRESERVLEVEL > 0 /* @@ -2031,7 +2031,6 @@ again: goto found; } #endif - m_ret = NULL; vmd = VM_DOMAIN(domain); if (vm_domain_allocate(vmd, req, npages)) { /* @@ -3506,7 +3505,7 @@ vm_page_free_toq(vm_page_t m) return; vmd = vm_pagequeue_domain(m); - if (m->pool == VM_FREEPOOL_DEFAULT && vmd->vmd_pgcache != NULL) { + if ((m->flags & PG_PCPU_CACHE) != 0 && vmd->vmd_pgcache != NULL) { uma_zfree(vmd->vmd_pgcache, m); return; } Modified: head/sys/vm/vm_page.h ============================================================================== --- head/sys/vm/vm_page.h Tue Jul 2 19:32:48 2019 (r349611) +++ head/sys/vm/vm_page.h Tue Jul 2 19:51:40 2019 (r349612) @@ -379,6 +379,7 @@ extern struct mtx_padalign pa_lock[]; * Page flags. If changed at any other time than page allocation or * freeing, the modification must be protected by the vm_page lock. */ +#define PG_PCPU_CACHE 0x0001 /* was allocated from per-CPU caches */ #define PG_FICTITIOUS 0x0004 /* physical page doesn't exist */ #define PG_ZERO 0x0008 /* page is zeroed */ #define PG_MARKER 0x0010 /* special queue marker page */