From owner-svn-src-stable@freebsd.org Sat May 12 11:18:15 2018 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A97F8FD4C0E; Sat, 12 May 2018 11:18:15 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 56F7F86A65; Sat, 12 May 2018 11:18:15 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1E44B1D99C; Sat, 12 May 2018 11:18:15 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w4CBIFhV098812; Sat, 12 May 2018 11:18:15 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w4CBIElb098811; Sat, 12 May 2018 11:18:14 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201805121118.w4CBIElb098811@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Sat, 12 May 2018 11:18:14 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r333527 - stable/11/sys/amd64/amd64 X-SVN-Group: stable-11 X-SVN-Commit-Author: kib X-SVN-Commit-Paths: stable/11/sys/amd64/amd64 X-SVN-Commit-Revision: 333527 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 May 2018 11:18:15 -0000 Author: kib Date: Sat May 12 11:18:14 2018 New Revision: 333527 URL: https://svnweb.freebsd.org/changeset/base/333527 Log: MFC r332450: Optimize context switch for PTI on PCID pmap. Approved by: re (marius) Modified: stable/11/sys/amd64/amd64/pmap.c Directory Properties: stable/11/ (props changed) Modified: stable/11/sys/amd64/amd64/pmap.c ============================================================================== --- stable/11/sys/amd64/amd64/pmap.c Sat May 12 11:06:59 2018 (r333526) +++ stable/11/sys/amd64/amd64/pmap.c Sat May 12 11:18:14 2018 (r333527) @@ -7293,8 +7293,9 @@ pmap_pcid_alloc(pmap_t pmap, u_int cpuid) CRITICAL_ASSERT(curthread); gen = PCPU_GET(pcid_gen); - if (!pti && (pmap->pm_pcids[cpuid].pm_pcid == PMAP_PCID_KERN || - pmap->pm_pcids[cpuid].pm_gen == gen)) + if (pmap->pm_pcids[cpuid].pm_pcid == PMAP_PCID_KERN) + return (pti ? 0 : CR3_PCID_SAVE); + if (pmap->pm_pcids[cpuid].pm_gen == gen) return (CR3_PCID_SAVE); pcid_next = PCPU_GET(pcid_next); KASSERT((!pti && pcid_next <= PMAP_PCID_OVERMAX) || @@ -7321,7 +7322,7 @@ pmap_activate_sw(struct thread *td) { pmap_t oldpmap, pmap; struct invpcid_descr d; - uint64_t cached, cr3, kcr3, ucr3; + uint64_t cached, cr3, kcr3, kern_pti_cached, ucr3; register_t rflags; u_int cpuid; @@ -7370,11 +7371,10 @@ pmap_activate_sw(struct thread *td) if (!invpcid_works) rflags = intr_disable(); - if (!cached || (cr3 & ~CR3_PCID_MASK) != pmap->pm_cr3) { + kern_pti_cached = pti ? 0 : cached; + if (!kern_pti_cached || (cr3 & ~CR3_PCID_MASK) != pmap->pm_cr3) { load_cr3(pmap->pm_cr3 | pmap->pm_pcids[cpuid].pm_pcid | - cached); - if (cached) - PCPU_INC(pm_save_cnt); + kern_pti_cached); } PCPU_SET(curpmap, pmap); if (pti) { @@ -7382,13 +7382,13 @@ pmap_activate_sw(struct thread *td) ucr3 = pmap->pm_ucr3 | pmap->pm_pcids[cpuid].pm_pcid | PMAP_PCID_USER_PT; - /* - * Manually invalidate translations cached - * from the user page table, which are not - * flushed by reload of cr3 with the kernel - * page table pointer above. - */ - if (pmap->pm_ucr3 != PMAP_NO_CR3) { + if (!cached && pmap->pm_ucr3 != PMAP_NO_CR3) { + /* + * Manually invalidate translations cached + * from the user page table. They are not + * flushed by reload of cr3 with the kernel + * page table pointer above. + */ if (invpcid_works) { d.pcid = PMAP_PCID_USER_PT | pmap->pm_pcids[cpuid].pm_pcid; @@ -7405,6 +7405,8 @@ pmap_activate_sw(struct thread *td) } if (!invpcid_works) intr_restore(rflags); + if (cached) + PCPU_INC(pm_save_cnt); } else if (cr3 != pmap->pm_cr3) { load_cr3(pmap->pm_cr3); PCPU_SET(curpmap, pmap);