From owner-dev-commits-src-main@freebsd.org Sun Jan 10 02:43:21 2021 Return-Path: Delivered-To: dev-commits-src-main@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CCF794EADD5; Sun, 10 Jan 2021 02:43:19 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DD1Nk6tbMz4qYv; Sun, 10 Jan 2021 02:43:18 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0ECBA20E13; Sun, 10 Jan 2021 02:43:17 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 10A2hGBm058149; Sun, 10 Jan 2021 02:43:16 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 10A2hGk9058148; Sun, 10 Jan 2021 02:43:16 GMT (envelope-from git) Date: Sun, 10 Jan 2021 02:43:16 GMT Message-Id: <202101100243.10A2hGk9058148@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Konstantin Belousov Subject: git: cd240c9cf100 - main - x86 vdso gettc: Add RDTSCP support MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: cd240c9cf100bec3def38ceb4a320611b1d02693 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-main@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for the main branch of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2021 02:43:22 -0000 The branch main has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=cd240c9cf100bec3def38ceb4a320611b1d02693 commit cd240c9cf100bec3def38ceb4a320611b1d02693 Author: Konstantin Belousov AuthorDate: 2021-01-06 15:05:09 +0000 Commit: Konstantin Belousov CommitDate: 2021-01-10 02:42:34 +0000 x86 vdso gettc: Add RDTSCP support Detect and use RDTSCP if available, instead of fence+RDTSC. For AMD Zens+, use LFENCE+RDTSC instead of RDTSCP (or MFENCE;RDTSC previously). Reviewed by: gallatin, markj Tested by: pho MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D27986 --- lib/libc/x86/sys/__vdso_gettc.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/lib/libc/x86/sys/__vdso_gettc.c b/lib/libc/x86/sys/__vdso_gettc.c index 4e0fc9f7e841..7f224f8758cb 100644 --- a/lib/libc/x86/sys/__vdso_gettc.c +++ b/lib/libc/x86/sys/__vdso_gettc.c @@ -63,6 +63,16 @@ rdtsc_low(const struct vdso_timehands *th) return (rv); } +static inline u_int +rdtscp_low(const struct vdso_timehands *th) +{ + u_int rv; + + __asm __volatile("rdtscp; movl %%edi,%%ecx; shrd %%cl, %%edx, %0" + : "=a" (rv) : "D" (th->th_x86_shift) : "ecx", "edx"); + return (rv); +} + static u_int rdtsc_low_mb_lfence(const struct vdso_timehands *th) { @@ -103,6 +113,12 @@ rdtsc32_mb_none(void) return (rdtsc32()); } +static u_int +rdtscp32_(void) +{ + return (rdtscp32()); +} + struct tsc_selector_tag { u_int (*ts_rdtsc32)(void); u_int (*ts_rdtsc_low)(const struct vdso_timehands *); @@ -121,6 +137,10 @@ static const struct tsc_selector_tag tsc_selector[] = { .ts_rdtsc32 = rdtsc32_mb_none, .ts_rdtsc_low = rdtsc_low_mb_none, }, + [3] = { /* RDTSCP */ + .ts_rdtsc32 = rdtscp32_, + .ts_rdtsc_low = rdtscp_low, + }, }; static int @@ -144,6 +164,9 @@ tsc_selector_idx(u_int cpu_feature) do_cpuid(1, p); cpu_id = p[0]; + if (amd_cpu && CPUID_TO_FAMILY(cpu_id) >= 0x17) + return (0); + if (cpu_feature != 0) { do_cpuid(0x80000000, p); cpu_exthigh = p[0]; @@ -157,6 +180,8 @@ tsc_selector_idx(u_int cpu_feature) amd_feature = 0; } + if ((amd_feature & AMDID_RDTSCP) != 0) + return (3); if ((cpu_feature & CPUID_SSE2) == 0) return (2); return (amd_cpu ? 1 : 0);