From owner-svn-src-head@freebsd.org Thu Feb 7 02:56:11 2019 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 488B214BDAE9; Thu, 7 Feb 2019 02:56:11 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DF6DA6A84C; Thu, 7 Feb 2019 02:56:10 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BA3E7EBFC; Thu, 7 Feb 2019 02:56:10 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x172uAhF093940; Thu, 7 Feb 2019 02:56:10 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x172uAQH093939; Thu, 7 Feb 2019 02:56:10 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201902070256.x172uAQH093939@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Thu, 7 Feb 2019 02:56:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r343855 - head/lib/libc/x86/sys X-SVN-Group: head X-SVN-Commit-Author: kib X-SVN-Commit-Paths: head/lib/libc/x86/sys X-SVN-Commit-Revision: 343855 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: DF6DA6A84C X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.978,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-0.999,0] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Feb 2019 02:56:11 -0000 Author: kib Date: Thu Feb 7 02:56:10 2019 New Revision: 343855 URL: https://svnweb.freebsd.org/changeset/base/343855 Log: Use ifunc to select the barrier instruction for RDTSC. This optimizes out runtime switch and removes yet another cpuid from libc. Note that this is the first use of ifunc in i386 libc, so ifunc-capable toolchain is required for building runnable userspace on i386, same as on amd64. Discussed with: emaste Sponsored by: The FreeBSD Foundation MFC after: 1 week Modified: head/lib/libc/x86/sys/__vdso_gettc.c Modified: head/lib/libc/x86/sys/__vdso_gettc.c ============================================================================== --- head/lib/libc/x86/sys/__vdso_gettc.c Thu Feb 7 02:36:59 2019 (r343854) +++ head/lib/libc/x86/sys/__vdso_gettc.c Thu Feb 7 02:56:10 2019 (r343855) @@ -1,6 +1,6 @@ /*- * Copyright (c) 2012 Konstantin Belousov - * Copyright (c) 2016, 2017 The FreeBSD Foundation + * Copyright (c) 2016, 2017, 2019 The FreeBSD Foundation * All rights reserved. * * Portions of this software were developed by Konstantin Belousov @@ -50,15 +50,9 @@ __FBSDID("$FreeBSD$"); #ifdef WANT_HYPERV #include #endif +#include #include "libc_private.h" -static enum LMB { - LMB_UNKNOWN, - LMB_NONE, - LMB_MFENCE, - LMB_LFENCE -} lfence_works = LMB_UNKNOWN; - static void cpuidp(u_int leaf, u_int p[4]) { @@ -84,68 +78,35 @@ cpuidp(u_int leaf, u_int p[4]) : "0" (leaf)); } -static enum LMB -select_lmb(void) +static void +rdtsc_mb_lfence(void) { - u_int p[4]; - static const char intel_id[] = "GenuntelineI"; - cpuidp(0, p); - return (memcmp(p + 1, intel_id, sizeof(intel_id) - 1) == 0 ? - LMB_LFENCE : LMB_MFENCE); + lfence(); } static void -init_fence(void) +rdtsc_mb_mfence(void) { -#if defined(__i386__) - u_int cpuid_supported, p[4]; - lfence_works = LMB_NONE; - __asm __volatile( - " pushfl\n" - " popl %%eax\n" - " movl %%eax,%%ecx\n" - " xorl $0x200000,%%eax\n" - " pushl %%eax\n" - " popfl\n" - " pushfl\n" - " popl %%eax\n" - " xorl %%eax,%%ecx\n" - " je 1f\n" - " movl $1,%0\n" - " jmp 2f\n" - "1: movl $0,%0\n" - "2:\n" - : "=r" (cpuid_supported) : : "eax", "ecx", "cc"); - if (cpuid_supported) { - cpuidp(0x1, p); - if ((p[3] & CPUID_SSE2) != 0) - lfence_works = select_lmb(); - } -#elif defined(__amd64__) - lfence_works = select_lmb(); -#else -#error "Arch" -#endif + mfence(); } static void -rdtsc_mb(void) +rdtsc_mb_none(void) { +} -again: - if (__predict_true(lfence_works == LMB_LFENCE)) { - lfence(); - return; - } else if (lfence_works == LMB_MFENCE) { - mfence(); - return; - } else if (lfence_works == LMB_NONE) { - return; - } - init_fence(); - goto again; +DEFINE_UIFUNC(static, void, rdtsc_mb, (void), static) +{ + u_int p[4]; + static const char intel_id[] = "GenuntelineI"; + + if ((cpu_feature & CPUID_SSE2) == 0) + return (rdtsc_mb_none); + cpuidp(0, p); + return (memcmp(p + 1, intel_id, sizeof(intel_id) - 1) == 0 ? + rdtsc_mb_lfence : rdtsc_mb_mfence); } static u_int