From owner-svn-src-head@freebsd.org Thu Jul 30 00:13:21 2015 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66BCD9AEFB0; Thu, 30 Jul 2015 00:13:21 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 57079A95; Thu, 30 Jul 2015 00:13:21 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.14.9/8.14.9) with ESMTP id t6U0DL8n017923; Thu, 30 Jul 2015 00:13:21 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.14.9/8.14.9/Submit) id t6U0DKml017921; Thu, 30 Jul 2015 00:13:20 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201507300013.t6U0DKml017921@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Thu, 30 Jul 2015 00:13:20 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r286051 - in head/sys/i386: i386 include X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2015 00:13:21 -0000 Author: kib Date: Thu Jul 30 00:13:20 2015 New Revision: 286051 URL: https://svnweb.freebsd.org/changeset/base/286051 Log: Use private cache line for the locked nop in *mb() on i386. Suggested by: alc Reviewed by: alc, bde Tested by: pho Sponsored by: The FreeBSD Foundation MFC after: 2 weeks Modified: head/sys/i386/i386/vm_machdep.c head/sys/i386/include/atomic.h Modified: head/sys/i386/i386/vm_machdep.c ============================================================================== --- head/sys/i386/i386/vm_machdep.c Wed Jul 29 23:59:17 2015 (r286050) +++ head/sys/i386/i386/vm_machdep.c Thu Jul 30 00:13:20 2015 (r286051) @@ -111,8 +111,8 @@ _Static_assert(OFFSETOF_CURTHREAD == off "OFFSETOF_CURTHREAD does not correspond with offset of pc_curthread."); _Static_assert(OFFSETOF_CURPCB == offsetof(struct pcpu, pc_curpcb), "OFFSETOF_CURPCB does not correspond with offset of pc_curpcb."); -_Static_assert(OFFSETOF_MONITORBUF == offsetof(struct pcpu, pc_monitorbuf), - "OFFSETOF_MONINORBUF does not correspond with offset of pc_monitorbuf."); +_Static_assert(__OFFSETOF_MONITORBUF == offsetof(struct pcpu, pc_monitorbuf), + "__OFFSETOF_MONINORBUF does not correspond with offset of pc_monitorbuf."); static void cpu_reset_real(void); #ifdef SMP Modified: head/sys/i386/include/atomic.h ============================================================================== --- head/sys/i386/include/atomic.h Wed Jul 29 23:59:17 2015 (r286050) +++ head/sys/i386/include/atomic.h Thu Jul 30 00:13:20 2015 (r286051) @@ -37,9 +37,31 @@ #include #endif -#define mb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory", "cc") -#define wmb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory", "cc") -#define rmb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory", "cc") +#ifndef __OFFSETOF_MONITORBUF +/* + * __OFFSETOF_MONITORBUF == __pcpu_offset(pc_monitorbuf). + * + * The open-coded number is used instead of the symbolic expression to + * avoid a dependency on sys/pcpu.h in machine/atomic.h consumers. + * An assertion in i386/vm_machdep.c ensures that the value is correct. + */ +#define __OFFSETOF_MONITORBUF 0x180 + +static __inline void +__mbk(void) +{ + + __asm __volatile("lock; addl $0,%%fs:%0" + : "+m" (*(u_int *)__OFFSETOF_MONITORBUF) : : "memory", "cc"); +} + +static __inline void +__mbu(void) +{ + + __asm __volatile("lock; addl $0,(%%esp)" : : : "memory", "cc"); +} +#endif /* * Various simple operations on memory, each of which is atomic in the @@ -246,40 +268,15 @@ atomic_testandset_int(volatile u_int *p, * reordering accesses in a way that violates the semantics of acquire * and release. */ -#if defined(_KERNEL) - -/* - * OFFSETOF_MONITORBUF == __pcpu_offset(pc_monitorbuf). - * - * The open-coded number is used instead of the symbolic expression to - * avoid a dependency on sys/pcpu.h in machine/atomic.h consumers. - * An assertion in i386/vm_machdep.c ensures that the value is correct. - */ -#define OFFSETOF_MONITORBUF 0x180 +#if defined(_KERNEL) #if defined(SMP) -static __inline void -__storeload_barrier(void) -{ - - __asm __volatile("lock; addl $0,%%fs:%0" - : "+m" (*(u_int *)OFFSETOF_MONITORBUF) : : "memory", "cc"); -} +#define __storeload_barrier() __mbk() #else /* _KERNEL && UP */ -static __inline void -__storeload_barrier(void) -{ - - __compiler_membar(); -} +#define __storeload_barrier() __compiler_membar() #endif /* SMP */ #else /* !_KERNEL */ -static __inline void -__storeload_barrier(void) -{ - - __asm __volatile("lock; addl $0,(%%esp)" : : : "memory", "cc"); -} +#define __storeload_barrier() __mbu() #endif /* _KERNEL*/ #define ATOMIC_LOAD(TYPE) \ @@ -776,4 +773,14 @@ u_long atomic_swap_long(volatile u_long #endif /* !WANT_FUNCTIONS */ +#if defined(_KERNEL) +#define mb() __mbk() +#define wmb() __mbk() +#define rmb() __mbk() +#else +#define mb() __mbu() +#define wmb() __mbu() +#define rmb() __mbu() +#endif + #endif /* !_MACHINE_ATOMIC_H_ */