From owner-svn-src-all@freebsd.org Fri May 25 16:29:23 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFDC2EED244; Fri, 25 May 2018 16:29:23 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 57B3068394; Fri, 25 May 2018 16:29:23 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1DD3C233B3; Fri, 25 May 2018 16:29:23 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w4PGTMHi093977; Fri, 25 May 2018 16:29:22 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w4PGTMVs093975; Fri, 25 May 2018 16:29:22 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201805251629.w4PGTMVs093975@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Fri, 25 May 2018 16:29:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r334211 - in head/sys/i386: i386 include X-SVN-Group: head X-SVN-Commit-Author: kib X-SVN-Commit-Paths: in head/sys/i386: i386 include X-SVN-Commit-Revision: 334211 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 May 2018 16:29:23 -0000 Author: kib Date: Fri May 25 16:29:22 2018 New Revision: 334211 URL: https://svnweb.freebsd.org/changeset/base/334211 Log: Optimize i386 pmap_extract_and_hold(). In particular, stop using pmap_pte() to read non-promoted pte while walking the page table. pmap_pte() needs to shoot down the kernel mapping globally which causes IPI broadcast. Since pmap_extract_and_hold() is used for slow copyin(9), it is very significant hit for the 4/4 kernels. Instead, create single purpose per-processor page frame and use it to locally map page table page inside the critical section, to avoid reuse of the frame by other thread if context switched. Measurement demostrated very significant improvements in any load that utilizes copyin/copyout. Found and benchmarked by: bde Sponsored by: The FreeBSD Foundation Modified: head/sys/i386/i386/pmap.c head/sys/i386/include/pcpu.h Modified: head/sys/i386/i386/pmap.c ============================================================================== --- head/sys/i386/i386/pmap.c Fri May 25 16:24:20 2018 (r334210) +++ head/sys/i386/i386/pmap.c Fri May 25 16:29:22 2018 (r334211) @@ -692,6 +692,10 @@ pmap_init_reserved_pages(void) pc->pc_copyout_saddr = kva_alloc(ptoa(2)); if (pc->pc_copyout_saddr == 0) panic("unable to allocate sleepable copyout KVA"); + pc->pc_pmap_eh_va = kva_alloc(ptoa(1)); + if (pc->pc_pmap_eh_va == 0) + panic("unable to allocate pmap_extract_and_hold KVA"); + pc->pc_pmap_eh_ptep = (char *)vtopte(pc->pc_pmap_eh_va); /* * Skip if the mappings have already been initialized, @@ -1598,8 +1602,8 @@ pmap_extract(pmap_t pmap, vm_offset_t va) vm_page_t pmap_extract_and_hold(pmap_t pmap, vm_offset_t va, vm_prot_t prot) { - pd_entry_t pde; - pt_entry_t pte, *ptep; + pd_entry_t pde, newpf; + pt_entry_t *eh_ptep, pte, *ptep; vm_page_t m; vm_paddr_t pa; @@ -1619,9 +1623,17 @@ retry: vm_page_hold(m); } } else { - ptep = pmap_pte(pmap, va); + newpf = pde & PG_FRAME; + critical_enter(); + eh_ptep = (pt_entry_t *)PCPU_GET(pmap_eh_ptep); + if ((*eh_ptep & PG_FRAME) != newpf) { + *eh_ptep = newpf | PG_RW | PG_V | PG_A | PG_M; + invlcaddr((void *)PCPU_GET(pmap_eh_va)); + } + ptep = (pt_entry_t *)PCPU_GET(pmap_eh_va) + + (i386_btop(va) & (NPTEPG - 1)); pte = *ptep; - pmap_pte_release(ptep); + critical_exit(); if (pte != 0 && ((pte & PG_RW) || (prot & VM_PROT_WRITE) == 0)) { if (vm_page_pa_tryrelock(pmap, pte & PG_FRAME, Modified: head/sys/i386/include/pcpu.h ============================================================================== --- head/sys/i386/include/pcpu.h Fri May 25 16:24:20 2018 (r334210) +++ head/sys/i386/include/pcpu.h Fri May 25 16:29:22 2018 (r334211) @@ -76,9 +76,11 @@ struct mtx pc_copyout_mlock; \ struct sx pc_copyout_slock; \ char *pc_copyout_buf; \ + vm_offset_t pc_pmap_eh_va; \ + caddr_t pc_pmap_eh_ptep; \ uint32_t pc_smp_tlb_done; /* TLB op acknowledgement */ \ uint32_t pc_ibpb_set; \ - char __pad[546] + char __pad[538] #ifdef _KERNEL