From owner-svn-src-all@freebsd.org Wed May 8 16:05:20 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF162158BAA6; Wed, 8 May 2019 16:05:19 +0000 (UTC) (envelope-from jhibbits@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 86D9C82941; Wed, 8 May 2019 16:05:19 +0000 (UTC) (envelope-from jhibbits@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 58C1F818C; Wed, 8 May 2019 16:05:19 +0000 (UTC) (envelope-from jhibbits@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x48G5JRV024633; Wed, 8 May 2019 16:05:19 GMT (envelope-from jhibbits@FreeBSD.org) Received: (from jhibbits@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x48G5JdC024632; Wed, 8 May 2019 16:05:19 GMT (envelope-from jhibbits@FreeBSD.org) Message-Id: <201905081605.x48G5JdC024632@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: jhibbits set sender to jhibbits@FreeBSD.org using -f From: Justin Hibbits Date: Wed, 8 May 2019 16:05:19 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r347350 - head/sys/powerpc/booke X-SVN-Group: head X-SVN-Commit-Author: jhibbits X-SVN-Commit-Paths: head/sys/powerpc/booke X-SVN-Commit-Revision: 347350 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 86D9C82941 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.977,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 May 2019 16:05:20 -0000 Author: jhibbits Date: Wed May 8 16:05:18 2019 New Revision: 347350 URL: https://svnweb.freebsd.org/changeset/base/347350 Log: powerpc/booke: Do as much work outside of TLB locks as possible Reduce the surface area of the TLB locks. Unfortunately the same trick for serializing the tlbie instruction on OEA64 cannot be used here to reduce the scope of the tlbivax mutex to the tlbsync only, as the mutex also serializes the TLB miss lock as a side effect, so contention on this lock may not be reducible any further. Modified: head/sys/powerpc/booke/pmap.c Modified: head/sys/powerpc/booke/pmap.c ============================================================================== --- head/sys/powerpc/booke/pmap.c Wed May 8 16:04:33 2019 (r347349) +++ head/sys/powerpc/booke/pmap.c Wed May 8 16:05:18 2019 (r347350) @@ -1382,7 +1382,7 @@ pte_enter(mmu_t mmu, pmap_t pmap, vm_page_t m, vm_offs unsigned int pp2d_idx = PP2D_IDX(va); unsigned int pdir_idx = PDIR_IDX(va); unsigned int ptbl_idx = PTBL_IDX(va); - pte_t *ptbl, *pte; + pte_t *ptbl, *pte, pte_tmp; pte_t **pdir; /* Get the page directory pointer. */ @@ -1400,12 +1400,13 @@ pte_enter(mmu_t mmu, pmap_t pmap, vm_page_t m, vm_offs KASSERT(nosleep, ("nosleep and NULL ptbl")); return (ENOMEM); } + pte = &ptbl[ptbl_idx]; } else { /* * Check if there is valid mapping for requested va, if there * is, remove it. */ - pte = &pdir[pdir_idx][ptbl_idx]; + pte = &ptbl[ptbl_idx]; if (PTE_ISVALID(pte)) { pte_remove(mmu, pmap, va, PTBL_HOLD); } else { @@ -1437,14 +1438,16 @@ pte_enter(mmu_t mmu, pmap_t pmap, vm_page_t m, vm_offs pv_insert(pmap, va, m); } + pmap->pm_stats.resident_count++; + + pte_tmp = PTE_RPN_FROM_PA(VM_PAGE_TO_PHYS(m)); + pte_tmp |= (PTE_VALID | flags); + mtx_lock_spin(&tlbivax_mutex); tlb_miss_lock(); tlb0_flush_entry(va); - pmap->pm_stats.resident_count++; - pte = &pdir[pdir_idx][ptbl_idx]; - *pte = PTE_RPN_FROM_PA(VM_PAGE_TO_PHYS(m)); - *pte |= (PTE_VALID | flags); + *pte = pte_tmp; tlb_miss_unlock(); mtx_unlock_spin(&tlbivax_mutex); @@ -1583,7 +1586,7 @@ pte_enter(mmu_t mmu, pmap_t pmap, vm_page_t m, vm_offs { unsigned int pdir_idx = PDIR_IDX(va); unsigned int ptbl_idx = PTBL_IDX(va); - pte_t *ptbl, *pte; + pte_t *ptbl, *pte, pte_tmp; CTR4(KTR_PMAP, "%s: su = %d pmap = %p va = %p", __func__, pmap == kernel_pmap, pmap, va); @@ -1598,6 +1601,8 @@ pte_enter(mmu_t mmu, pmap_t pmap, vm_page_t m, vm_offs KASSERT(nosleep, ("nosleep and NULL ptbl")); return (ENOMEM); } + pmap->pm_pdir[pdir_idx] = ptbl; + pte = &ptbl[ptbl_idx]; } else { /* * Check if there is valid mapping for requested @@ -1629,20 +1634,14 @@ pte_enter(mmu_t mmu, pmap_t pmap, vm_page_t m, vm_offs pmap->pm_stats.resident_count++; + pte_tmp = PTE_RPN_FROM_PA(VM_PAGE_TO_PHYS(m)); + pte_tmp |= (PTE_VALID | flags | PTE_PS_4KB); /* 4KB pages only */ + mtx_lock_spin(&tlbivax_mutex); tlb_miss_lock(); tlb0_flush_entry(va); - if (pmap->pm_pdir[pdir_idx] == NULL) { - /* - * If we just allocated a new page table, hook it in - * the pdir. - */ - pmap->pm_pdir[pdir_idx] = ptbl; - } - pte = &(pmap->pm_pdir[pdir_idx][ptbl_idx]); - *pte = PTE_RPN_FROM_PA(VM_PAGE_TO_PHYS(m)); - *pte |= (PTE_VALID | flags | PTE_PS_4KB); /* 4KB pages only */ + *pte = pte_tmp; tlb_miss_unlock(); mtx_unlock_spin(&tlbivax_mutex);