From owner-svn-src-all@freebsd.org Sun Sep 29 20:44:14 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 09497129EC2; Sun, 29 Sep 2019 20:44:14 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46hHZP6Qc4z3xPZ; Sun, 29 Sep 2019 20:44:13 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C086C1FBA2; Sun, 29 Sep 2019 20:44:13 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x8TKiDRB024951; Sun, 29 Sep 2019 20:44:13 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x8TKiDf8024950; Sun, 29 Sep 2019 20:44:13 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <201909292044.x8TKiDf8024950@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Sun, 29 Sep 2019 20:44:13 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r352874 - head/sys/amd64/amd64 X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: head/sys/amd64/amd64 X-SVN-Commit-Revision: 352874 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Sep 2019 20:44:14 -0000 Author: mjg Date: Sun Sep 29 20:44:13 2019 New Revision: 352874 URL: https://svnweb.freebsd.org/changeset/base/352874 Log: amd64 pmap: batch chunk removal in pmap_remove_pages pv list lock is the main bottleneck during poudriere -j 104 and pmap_remove_pages is the most impactful consumer. It frees chunks with the lock held even though it plays no role in correctness. Moreover chunks are often freed in groups, sample counts during buildkernel (0-sized frees removed): value ------------- Distribution ------------- count 0 | 0 1 | 8 2 |@@@@@@@ 19329 4 |@@@@@@@@@@@@@@@@@@@@@@ 58517 8 | 1085 16 | 71 32 |@@@@@@@@@@ 24919 64 | 899 128 | 7 256 | 2 512 | 0 Thus: 1. batch freeing 2. move it past unlocking pv list Reviewed by: alc (previous version), markj (previous version), kib Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D21832 Modified: head/sys/amd64/amd64/pmap.c Modified: head/sys/amd64/amd64/pmap.c ============================================================================== --- head/sys/amd64/amd64/pmap.c Sun Sep 29 20:08:14 2019 (r352873) +++ head/sys/amd64/amd64/pmap.c Sun Sep 29 20:44:13 2019 (r352874) @@ -1105,7 +1105,10 @@ static caddr_t crashdumpmap; #define MAPDEV_FLUSHCACHE 0x0000001 /* Flush cache after mapping. */ #define MAPDEV_SETATTR 0x0000002 /* Modify existing attrs. */ +TAILQ_HEAD(pv_chunklist, pv_chunk); + static void free_pv_chunk(struct pv_chunk *pc); +static void free_pv_chunk_batch(struct pv_chunklist *batch); static void free_pv_entry(pmap_t pmap, pv_entry_t pv); static pv_entry_t get_pv_entry(pmap_t pmap, struct rwlock **lockp); static int popcnt_pc_map_pq(uint64_t *map); @@ -4248,13 +4251,10 @@ free_pv_entry(pmap_t pmap, pv_entry_t pv) } static void -free_pv_chunk(struct pv_chunk *pc) +free_pv_chunk_dequeued(struct pv_chunk *pc) { vm_page_t m; - mtx_lock(&pv_chunks_mutex); - TAILQ_REMOVE(&pv_chunks, pc, pc_lru); - mtx_unlock(&pv_chunks_mutex); PV_STAT(atomic_subtract_int(&pv_entry_spare, _NPCPV)); PV_STAT(atomic_subtract_int(&pc_chunk_count, 1)); PV_STAT(atomic_add_int(&pc_chunk_frees, 1)); @@ -4265,6 +4265,35 @@ free_pv_chunk(struct pv_chunk *pc) vm_page_free(m); } +static void +free_pv_chunk(struct pv_chunk *pc) +{ + + mtx_lock(&pv_chunks_mutex); + TAILQ_REMOVE(&pv_chunks, pc, pc_lru); + mtx_unlock(&pv_chunks_mutex); + free_pv_chunk_dequeued(pc); +} + +static void +free_pv_chunk_batch(struct pv_chunklist *batch) +{ + struct pv_chunk *pc, *npc; + + if (TAILQ_EMPTY(batch)) + return; + + mtx_lock(&pv_chunks_mutex); + TAILQ_FOREACH(pc, batch, pc_list) { + TAILQ_REMOVE(&pv_chunks, pc, pc_lru); + } + mtx_unlock(&pv_chunks_mutex); + + TAILQ_FOREACH_SAFE(pc, batch, pc_list, npc) { + free_pv_chunk_dequeued(pc); + } +} + /* * Returns a new PV entry, allocating a new PV chunk from the system when * needed. If this PV chunk allocation fails and a PV list lock pointer was @@ -6865,6 +6894,7 @@ pmap_remove_pages(pmap_t pmap) pt_entry_t *pte, tpte; pt_entry_t PG_M, PG_RW, PG_V; struct spglist free; + struct pv_chunklist free_chunks; vm_page_t m, mpte, mt; pv_entry_t pv; struct md_page *pvh; @@ -6900,6 +6930,7 @@ pmap_remove_pages(pmap_t pmap) PG_V = pmap_valid_bit(pmap); PG_RW = pmap_rw_bit(pmap); + TAILQ_INIT(&free_chunks); SLIST_INIT(&free); PMAP_LOCK(pmap); TAILQ_FOREACH_SAFE(pc, &pmap->pm_pvchunk, pc_list, npc) { @@ -7027,13 +7058,14 @@ pmap_remove_pages(pmap_t pmap) PV_STAT(atomic_subtract_long(&pv_entry_count, freed)); if (allfree) { TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); - free_pv_chunk(pc); + TAILQ_INSERT_TAIL(&free_chunks, pc, pc_list); } } if (lock != NULL) rw_wunlock(lock); pmap_invalidate_all(pmap); pmap_pkru_deassign_all(pmap); + free_pv_chunk_batch(&free_chunks); PMAP_UNLOCK(pmap); vm_page_free_pages_toq(&free, true); }