From owner-svn-src-all@freebsd.org Wed Jul 8 18:12:28 2015 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C9C1995484; Wed, 8 Jul 2015 18:12:28 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E3C581EAF; Wed, 8 Jul 2015 18:12:27 +0000 (UTC) (envelope-from kib@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.70]) by repo.freebsd.org (8.14.9/8.14.9) with ESMTP id t68ICRYh058956; Wed, 8 Jul 2015 18:12:27 GMT (envelope-from kib@FreeBSD.org) Received: (from kib@localhost) by repo.freebsd.org (8.14.9/8.14.9/Submit) id t68ICPkC058944; Wed, 8 Jul 2015 18:12:25 GMT (envelope-from kib@FreeBSD.org) Message-Id: <201507081812.t68ICPkC058944@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: kib set sender to kib@FreeBSD.org using -f From: Konstantin Belousov Date: Wed, 8 Jul 2015 18:12:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r285283 - in head/sys: amd64/include arm/include arm64/include i386/include mips/include powerpc/include sparc64/include X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jul 2015 18:12:28 -0000 Author: kib Date: Wed Jul 8 18:12:24 2015 New Revision: 285283 URL: https://svnweb.freebsd.org/changeset/base/285283 Log: Add the atomic_thread_fence() family of functions with intent to provide a semantic defined by the C11 fences with corresponding memory_order. atomic_thread_fence_acq() gives r | r, w, where r and w are read and write accesses, and | denotes the fence itself. atomic_thread_fence_rel() is r, w | w. atomic_thread_fence_acq_rel() is the combination of the acquire and release in single operation. Note that reads after the acq+rel fence could be made visible before writes preceeding the fence. atomic_thread_fence_seq_cst() orders all accesses before/after the fence, and the fence itself is globally ordered against other sequentially consistent atomic operations. Reviewed by: alc Discussed with: bde Sponsored by: The FreeBSD Foundation MFC after: 3 weeks Modified: head/sys/amd64/include/atomic.h head/sys/arm/include/atomic.h head/sys/arm64/include/atomic.h head/sys/i386/include/atomic.h head/sys/mips/include/atomic.h head/sys/powerpc/include/atomic.h head/sys/sparc64/include/atomic.h Modified: head/sys/amd64/include/atomic.h ============================================================================== --- head/sys/amd64/include/atomic.h Wed Jul 8 17:45:59 2015 (r285282) +++ head/sys/amd64/include/atomic.h Wed Jul 8 18:12:24 2015 (r285283) @@ -84,6 +84,10 @@ u_int atomic_fetchadd_int(volatile u_int u_long atomic_fetchadd_long(volatile u_long *p, u_long v); int atomic_testandset_int(volatile u_int *p, u_int v); int atomic_testandset_long(volatile u_long *p, u_int v); +void atomic_thread_fence_acq(void); +void atomic_thread_fence_acq_rel(void); +void atomic_thread_fence_rel(void); +void atomic_thread_fence_seq_cst(void); #define ATOMIC_LOAD(TYPE) \ u_##TYPE atomic_load_acq_##TYPE(volatile u_##TYPE *p) @@ -328,6 +332,34 @@ atomic_store_rel_##TYPE(volatile u_##TYP } \ struct __hack +static __inline void +atomic_thread_fence_acq(void) +{ + + __compiler_membar(); +} + +static __inline void +atomic_thread_fence_rel(void) +{ + + __compiler_membar(); +} + +static __inline void +atomic_thread_fence_acq_rel(void) +{ + + __compiler_membar(); +} + +static __inline void +atomic_thread_fence_seq_cst(void) +{ + + __storeload_barrier(); +} + #endif /* KLD_MODULE || !__GNUCLIKE_ASM */ ATOMIC_ASM(set, char, "orb %b1,%0", "iq", v); Modified: head/sys/arm/include/atomic.h ============================================================================== --- head/sys/arm/include/atomic.h Wed Jul 8 17:45:59 2015 (r285282) +++ head/sys/arm/include/atomic.h Wed Jul 8 18:12:24 2015 (r285283) @@ -1103,6 +1103,34 @@ atomic_store_long(volatile u_long *dst, *dst = src; } +static __inline void +atomic_thread_fence_acq(void) +{ + + dmb(); +} + +static __inline void +atomic_thread_fence_rel(void) +{ + + dmb(); +} + +static __inline void +atomic_thread_fence_acq_rel(void) +{ + + dmb(); +} + +static __inline void +atomic_thread_fence_seq_cst(void) +{ + + dmb(); +} + #define atomic_clear_ptr atomic_clear_32 #define atomic_set_ptr atomic_set_32 #define atomic_cmpset_ptr atomic_cmpset_32 Modified: head/sys/arm64/include/atomic.h ============================================================================== --- head/sys/arm64/include/atomic.h Wed Jul 8 17:45:59 2015 (r285282) +++ head/sys/arm64/include/atomic.h Wed Jul 8 18:12:24 2015 (r285283) @@ -728,6 +728,34 @@ atomic_subtract_rel_64(volatile uint64_t ); } +static __inline void +atomic_thread_fence_acq(void) +{ + + dmb(ld); +} + +static __inline void +atomic_thread_fence_rel(void) +{ + + dmb(sy); +} + +static __inline void +atomic_thread_fence_acq_rel(void) +{ + + dmb(sy); +} + +static __inline void +atomic_thread_fence_seq_cst(void) +{ + + dmb(sy); +} + #define atomic_add_rel_long atomic_add_rel_64 #define atomic_clear_rel_long atomic_clear_rel_64 #define atomic_cmpset_rel_long atomic_cmpset_rel_64 Modified: head/sys/i386/include/atomic.h ============================================================================== --- head/sys/i386/include/atomic.h Wed Jul 8 17:45:59 2015 (r285282) +++ head/sys/i386/include/atomic.h Wed Jul 8 18:12:24 2015 (r285283) @@ -86,6 +86,10 @@ void atomic_##NAME##_barr_##TYPE(volatil int atomic_cmpset_int(volatile u_int *dst, u_int expect, u_int src); u_int atomic_fetchadd_int(volatile u_int *p, u_int v); int atomic_testandset_int(volatile u_int *p, u_int v); +void atomic_thread_fence_acq(void); +void atomic_thread_fence_acq_rel(void); +void atomic_thread_fence_rel(void); +void atomic_thread_fence_seq_cst(void); #define ATOMIC_LOAD(TYPE) \ u_##TYPE atomic_load_acq_##TYPE(volatile u_##TYPE *p) @@ -310,6 +314,34 @@ atomic_store_rel_##TYPE(volatile u_##TYP } \ struct __hack +static __inline void +atomic_thread_fence_acq(void) +{ + + __compiler_membar(); +} + +static __inline void +atomic_thread_fence_rel(void) +{ + + __compiler_membar(); +} + +static __inline void +atomic_thread_fence_acq_rel(void) +{ + + __compiler_membar(); +} + +static __inline void +atomic_thread_fence_seq_cst(void) +{ + + __storeload_barrier(); +} + #ifdef _KERNEL #ifdef WANT_FUNCTIONS Modified: head/sys/mips/include/atomic.h ============================================================================== --- head/sys/mips/include/atomic.h Wed Jul 8 17:45:59 2015 (r285282) +++ head/sys/mips/include/atomic.h Wed Jul 8 18:12:24 2015 (r285283) @@ -496,6 +496,34 @@ atomic_fetchadd_64(__volatile uint64_t * } #endif +static __inline void +atomic_thread_fence_acq(void) +{ + + mips_sync(); +} + +static __inline void +atomic_thread_fence_rel(void) +{ + + mips_sync(); +} + +static __inline void +atomic_thread_fence_acq_rel(void) +{ + + mips_sync(); +} + +static __inline void +atomic_thread_fence_seq_cst(void) +{ + + mips_sync(); +} + /* Operations on chars. */ #define atomic_set_char atomic_set_8 #define atomic_set_acq_char atomic_set_acq_8 Modified: head/sys/powerpc/include/atomic.h ============================================================================== --- head/sys/powerpc/include/atomic.h Wed Jul 8 17:45:59 2015 (r285282) +++ head/sys/powerpc/include/atomic.h Wed Jul 8 18:12:24 2015 (r285283) @@ -730,4 +730,45 @@ atomic_swap_64(volatile u_long *p, u_lon #undef __ATOMIC_REL #undef __ATOMIC_ACQ +static __inline void +atomic_thread_fence_acq(void) +{ + + /* See above comment about lwsync being broken on Book-E. */ +#ifdef __powerpc64__ + __asm __volatile("lwsync" : : : "memory"); +#else + __asm __volatile("sync" : : : "memory"); +#endif +} + +static __inline void +atomic_thread_fence_rel(void) +{ + +#ifdef __powerpc64__ + __asm __volatile("lwsync" : : : "memory"); +#else + __asm __volatile("sync" : : : "memory"); +#endif +} + +static __inline void +atomic_thread_fence_acq_rel(void) +{ + +#ifdef __powerpc64__ + __asm __volatile("lwsync" : : : "memory"); +#else + __asm __volatile("sync" : : : "memory"); +#endif +} + +static __inline void +atomic_thread_fence_seq_cst(void) +{ + + __asm __volatile("sync" : : : "memory"); +} + #endif /* ! _MACHINE_ATOMIC_H_ */ Modified: head/sys/sparc64/include/atomic.h ============================================================================== --- head/sys/sparc64/include/atomic.h Wed Jul 8 17:45:59 2015 (r285282) +++ head/sys/sparc64/include/atomic.h Wed Jul 8 18:12:24 2015 (r285283) @@ -279,6 +279,35 @@ atomic_store_rel_ ## name(volatile ptype atomic_st_rel((p), (v), sz); \ } +static __inline void +atomic_thread_fence_acq(void) +{ + + __compiler_membar(); +} + +static __inline void +atomic_thread_fence_rel(void) +{ + + __compiler_membar(); +} + +static __inline void +atomic_thread_fence_acq_rel(void) +{ + + __compiler_membar(); +} + +static __inline void +atomic_thread_fence_seq_cst(void) +{ + + membar(LoadLoad | LoadStore | StoreStore | StoreLoad); +} + + ATOMIC_GEN(int, u_int *, u_int, u_int, 32); ATOMIC_GEN(32, uint32_t *, uint32_t, uint32_t, 32);