From owner-svn-src-head@freebsd.org Sat Jul 25 10:29:49 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3FE733A6123; Sat, 25 Jul 2020 10:29:49 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BDMl10wN9z42sQ; Sat, 25 Jul 2020 10:29:49 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DE2B81A1CF; Sat, 25 Jul 2020 10:29:48 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 06PATmfG037394; Sat, 25 Jul 2020 10:29:48 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 06PATmVJ037392; Sat, 25 Jul 2020 10:29:48 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <202007251029.06PATmVJ037392@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Sat, 25 Jul 2020 10:29:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r363516 - head/sys/sys X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: head/sys/sys X-SVN-Commit-Revision: 363516 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jul 2020 10:29:49 -0000 Author: mjg Date: Sat Jul 25 10:29:48 2020 New Revision: 363516 URL: https://svnweb.freebsd.org/changeset/base/363516 Log: seqc: add a sleepable variant and convert some routines to macros This temporarily duplicates some code. Macro conversion convinces clang to carry predicts into consumers. Added: head/sys/sys/_seqc.h (contents, props changed) Modified: head/sys/sys/seqc.h Added: head/sys/sys/_seqc.h ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/sys/_seqc.h Sat Jul 25 10:29:48 2020 (r363516) @@ -0,0 +1,11 @@ +/*- + * This file is in the public domain. + */ +/* $FreeBSD$ */ + +#ifndef _SYS__SEQC_H_ +#define _SYS__SEQC_H_ + +typedef uint32_t seqc_t; + +#endif /* _SYS__SEQC_H */ Modified: head/sys/sys/seqc.h ============================================================================== --- head/sys/sys/seqc.h Sat Jul 25 09:28:38 2020 (r363515) +++ head/sys/sys/seqc.h Sat Jul 25 10:29:48 2020 (r363516) @@ -36,7 +36,7 @@ /* * seqc_t may be included in structs visible to userspace */ -typedef uint32_t seqc_t; +#include #ifdef _KERNEL @@ -45,13 +45,15 @@ typedef uint32_t seqc_t; #include -static __inline bool -seqc_in_modify(seqc_t seqcp) -{ +/* + * Predicts from inline functions are not honored by clang. + */ +#define seqc_in_modify(seqc) ({ \ + seqc_t __seqc = (seqc); \ + \ + __predict_false(__seqc & 1); \ +}) - return (seqcp & 1); -} - static __inline void seqc_write_begin(seqc_t *seqcp) { @@ -86,7 +88,7 @@ seqc_read(const seqc_t *seqcp) for (;;) { ret = seqc_read_any(seqcp); - if (__predict_false(seqc_in_modify(ret))) { + if (seqc_in_modify(ret)) { cpu_spinwait(); continue; } @@ -96,19 +98,38 @@ seqc_read(const seqc_t *seqcp) return (ret); } -static __inline bool -seqc_consistent_nomb(const seqc_t *seqcp, seqc_t oldseqc) +#define seqc_consistent_nomb(seqcp, oldseqc) ({ \ + const seqc_t *__seqcp = (seqcp); \ + seqc_t __oldseqc = (oldseqc); \ + \ + MPASS(!(seqc_in_modify(__oldseqc))); \ + __predict_true(*__seqcp == __oldseqc); \ +}) + +#define seqc_consistent(seqcp, oldseqc) ({ \ + atomic_thread_fence_acq(); \ + seqc_consistent_nomb(seqcp, oldseqc); \ +}) + +/* + * Variant which does not critical enter/exit. + */ +static __inline void +seqc_sleepable_write_begin(seqc_t *seqcp) { - return (*seqcp == oldseqc); + MPASS(!seqc_in_modify(*seqcp)); + *seqcp += 1; + atomic_thread_fence_rel(); } -static __inline bool -seqc_consistent(const seqc_t *seqcp, seqc_t oldseqc) +static __inline void +seqc_sleepable_write_end(seqc_t *seqcp) { - atomic_thread_fence_acq(); - return (seqc_consistent_nomb(seqcp, oldseqc)); + atomic_thread_fence_rel(); + *seqcp += 1; + MPASS(!seqc_in_modify(*seqcp)); } #endif /* _KERNEL */