From owner-dev-commits-src-main@freebsd.org Sat Aug 14 12:58:12 2021 Return-Path: Delivered-To: dev-commits-src-main@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F180666CA9E; Sat, 14 Aug 2021 12:58:11 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gn0pW3tzhz3lk8; Sat, 14 Aug 2021 12:58:11 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6D48B22983; Sat, 14 Aug 2021 12:58:11 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 17ECwBFM031111; Sat, 14 Aug 2021 12:58:11 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 17ECwBK7031110; Sat, 14 Aug 2021 12:58:11 GMT (envelope-from git) Date: Sat, 14 Aug 2021 12:58:11 GMT Message-Id: <202108141258.17ECwBK7031110@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Konstantin Belousov Subject: git: 652ae7b11485 - main - x86: cpufunc: Add rdtsc_ordered() MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: kib X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 652ae7b11485d9f991ea66a56aa79d4dd9cd1103 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-main@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for the main branch of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Aug 2021 12:58:12 -0000 The branch main has been updated by kib: URL: https://cgit.FreeBSD.org/src/commit/?id=652ae7b11485d9f991ea66a56aa79d4dd9cd1103 commit 652ae7b11485d9f991ea66a56aa79d4dd9cd1103 Author: Adam Fenn AuthorDate: 2021-07-28 17:12:00 +0000 Commit: Konstantin Belousov CommitDate: 2021-08-14 12:57:53 +0000 x86: cpufunc: Add rdtsc_ordered() Add a variant of 'rdtsc()' that performs the ordered version of 'rdtsc' appropriate for the invoking x86 variant. Also, expose the 'lfence'-ed and 'mfence'-ed 'rdtsc()' variants needed by 'rdtsc_ordered()' for general use. Sponsored by: Juniper Networks, Inc. Sponsored by: Klara, Inc. Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D31416 --- sys/amd64/include/cpufunc.h | 14 ++++++++++++++ sys/i386/include/cpufunc.h | 14 ++++++++++++++ sys/x86/include/x86_var.h | 1 + sys/x86/x86/cpu_machdep.c | 16 ++++++++++++++++ 4 files changed, 45 insertions(+) diff --git a/sys/amd64/include/cpufunc.h b/sys/amd64/include/cpufunc.h index 49975e826fb9..bca74d8ead67 100644 --- a/sys/amd64/include/cpufunc.h +++ b/sys/amd64/include/cpufunc.h @@ -355,6 +355,20 @@ rdtsc(void) return (low | ((uint64_t)high << 32)); } +static __inline uint64_t +rdtsc_ordered_lfence(void) +{ + lfence(); + return (rdtsc()); +} + +static __inline uint64_t +rdtsc_ordered_mfence(void) +{ + mfence(); + return (rdtsc()); +} + static __inline uint64_t rdtscp(void) { diff --git a/sys/i386/include/cpufunc.h b/sys/i386/include/cpufunc.h index 7b99978facc3..79cdd3004b77 100644 --- a/sys/i386/include/cpufunc.h +++ b/sys/i386/include/cpufunc.h @@ -394,6 +394,20 @@ rdtsc(void) return (rv); } +static __inline uint64_t +rdtsc_ordered_lfence(void) +{ + lfence(); + return (rdtsc()); +} + +static __inline uint64_t +rdtsc_ordered_mfence(void) +{ + mfence(); + return (rdtsc()); +} + static __inline uint64_t rdtscp(void) { diff --git a/sys/x86/include/x86_var.h b/sys/x86/include/x86_var.h index ec268503634b..d695822ddb7f 100644 --- a/sys/x86/include/x86_var.h +++ b/sys/x86/include/x86_var.h @@ -154,6 +154,7 @@ int user_dbreg_trap(register_t dr6); int minidumpsys(struct dumperinfo *); struct pcb *get_pcb_td(struct thread *td); void x86_set_fork_retval(struct thread *td); +uint64_t rdtsc_ordered(void); /* * MSR ops for x86_msr_op() diff --git a/sys/x86/x86/cpu_machdep.c b/sys/x86/x86/cpu_machdep.c index bfc1a697cde0..1924342c8e83 100644 --- a/sys/x86/x86/cpu_machdep.c +++ b/sys/x86/x86/cpu_machdep.c @@ -76,6 +76,7 @@ __FBSDID("$FreeBSD$"); #include #include +#include #include #include #include @@ -88,6 +89,7 @@ __FBSDID("$FreeBSD$"); #include #endif #include +#include #include #include @@ -1508,3 +1510,17 @@ acpi_get_fadt_bootflags(uint16_t *flagsp) return (false); #endif } + +DEFINE_IFUNC(, uint64_t, rdtsc_ordered, (void)) +{ + bool cpu_is_amd = cpu_vendor_id == CPU_VENDOR_AMD || + cpu_vendor_id == CPU_VENDOR_HYGON; + + if ((amd_feature & AMDID_RDTSCP) != 0) + return (rdtscp); + else if ((cpu_feature & CPUID_SSE2) != 0) + return (cpu_is_amd ? rdtsc_ordered_mfence : + rdtsc_ordered_lfence); + else + return (rdtsc); +}