From owner-dev-commits-src-all@freebsd.org Tue Feb 9 08:36:59 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF00F535EFD; Tue, 9 Feb 2021 08:36:59 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DZbpx6sltz4gLH; Tue, 9 Feb 2021 08:36:57 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4C035274F7; Tue, 9 Feb 2021 08:36:56 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1198auXa013965; Tue, 9 Feb 2021 08:36:56 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1198auf9013964; Tue, 9 Feb 2021 08:36:56 GMT (envelope-from git) Date: Tue, 9 Feb 2021 08:36:56 GMT Message-Id: <202102090836.1198auf9013964@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Konstantin Belousov Subject: git: 4ab4f06952ea - stable/12 - tsc: add RDTSCP or faster variants of get_timecount() MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/stable/12 X-Git-Reftype: branch X-Git-Commit: 4ab4f06952eaee865326e0da7512f49c580d8a32 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Feb 2021 08:37:00 -0000 The branch stable/12 has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=4ab4f06952eaee865326e0da7512f49c580d8a32 commit 4ab4f06952eaee865326e0da7512f49c580d8a32 Author: Konstantin Belousov AuthorDate: 2021-01-05 21:00:14 +0000 Commit: Konstantin Belousov CommitDate: 2021-02-09 08:36:08 +0000 tsc: add RDTSCP or faster variants of get_timecount() Tested by: pho (cherry picked from commit 9e680e4005b77e3028d28377ee3722a5260f4422) (cherry picked from commit a013e285dfd6b89b1908ca13febb0fdb0a7f3b1f) (cherry picked from commit 9f47eeffa3cfdcb512e2011fb00fc23c7c1a7d75) --- sys/x86/x86/tsc.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/sys/x86/x86/tsc.c b/sys/x86/x86/tsc.c index cb1ba0758db9..f14e0707651c 100644 --- a/sys/x86/x86/tsc.c +++ b/sys/x86/x86/tsc.c @@ -96,6 +96,8 @@ static u_int tsc_get_timecount_lfence(struct timecounter *tc); static u_int tsc_get_timecount_low_lfence(struct timecounter *tc); static u_int tsc_get_timecount_mfence(struct timecounter *tc); static u_int tsc_get_timecount_low_mfence(struct timecounter *tc); +static u_int tscp_get_timecount(struct timecounter *tc); +static u_int tscp_get_timecount_low(struct timecounter *tc); static void tsc_levels_changed(void *arg, int unit); static uint32_t x86_tsc_vdso_timehands(struct vdso_timehands *vdso_th, struct timecounter *tc); @@ -632,7 +634,18 @@ init_TSC_tc(void) init: for (shift = 0; shift <= 31 && (tsc_freq >> shift) > max_freq; shift++) ; - if ((cpu_feature & CPUID_SSE2) != 0 && mp_ncpus > 1) { + + /* + * Timecounter implementation selection, top to bottom: + * - If RDTSCP is available, use RDTSCP. + * - If fence instructions are provided (SSE2), use LFENCE;RDTSC + * on Intel, and MFENCE;RDTSC on AMD. + * - For really old CPUs, just use RDTSC. + */ + if ((amd_feature & AMDID_RDTSCP) != 0) { + tsc_timecounter.tc_get_timecount = shift > 0 ? + tscp_get_timecount_low : tscp_get_timecount; + } else if ((cpu_feature & CPUID_SSE2) != 0 && mp_ncpus > 1) { if (cpu_vendor_id == CPU_VENDOR_AMD || cpu_vendor_id == CPU_VENDOR_HYGON) { tsc_timecounter.tc_get_timecount = shift > 0 ? @@ -785,6 +798,13 @@ tsc_get_timecount(struct timecounter *tc __unused) return (rdtsc32()); } +static u_int +tscp_get_timecount(struct timecounter *tc __unused) +{ + + return (rdtscp32()); +} + static inline u_int tsc_get_timecount_low(struct timecounter *tc) { @@ -795,6 +815,16 @@ tsc_get_timecount_low(struct timecounter *tc) return (rv); } +static u_int +tscp_get_timecount_low(struct timecounter *tc) +{ + uint32_t rv; + + __asm __volatile("rdtscp; movl %1, %%ecx; shrd %%cl, %%edx, %0" + : "=&a" (rv) : "m" (tc->tc_priv) : "ecx", "edx"); + return (rv); +} + static u_int tsc_get_timecount_lfence(struct timecounter *tc __unused) {