From owner-svn-src-head@freebsd.org Thu May 17 01:13:42 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E1FFEE3668; Thu, 17 May 2018 01:13:42 +0000 (UTC) (envelope-from mmacy@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D5AF9880B4; Thu, 17 May 2018 01:13:41 +0000 (UTC) (envelope-from mmacy@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 9DD3021F83; Thu, 17 May 2018 01:13:41 +0000 (UTC) (envelope-from mmacy@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w4H1DfkL060140; Thu, 17 May 2018 01:13:41 GMT (envelope-from mmacy@FreeBSD.org) Received: (from mmacy@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w4H1Df6x060138; Thu, 17 May 2018 01:13:41 GMT (envelope-from mmacy@FreeBSD.org) Message-Id: <201805170113.w4H1Df6x060138@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mmacy set sender to mmacy@FreeBSD.org using -f From: Matt Macy Date: Thu, 17 May 2018 01:13:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r333697 - in head/sys: kern sys X-SVN-Group: head X-SVN-Commit-Author: mmacy X-SVN-Commit-Paths: in head/sys: kern sys X-SVN-Commit-Revision: 333697 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2018 01:13:42 -0000 Author: mmacy Date: Thu May 17 01:13:40 2018 New Revision: 333697 URL: https://svnweb.freebsd.org/changeset/base/333697 Log: epoch(9): make recursion lighter weight There isn't any real work to do except bump td_epochnest when recursing. Skip the additional work in this case. Approved by: sbruno Modified: head/sys/kern/subr_epoch.c head/sys/sys/epoch.h Modified: head/sys/kern/subr_epoch.c ============================================================================== --- head/sys/kern/subr_epoch.c Thu May 17 00:52:48 2018 (r333696) +++ head/sys/kern/subr_epoch.c Thu May 17 01:13:40 2018 (r333697) @@ -284,21 +284,15 @@ epoch_free(epoch_t epoch) } while (0) void -epoch_enter(epoch_t epoch) +epoch_enter_internal(epoch_t epoch, struct thread *td) { struct epoch_pcpu_state *eps; - struct thread *td; INIT_CHECK(epoch); - - td = curthread; critical_enter(); eps = epoch->e_pcpu[curcpu]; - td->td_epochnest++; - MPASS(td->td_epochnest < UCHAR_MAX - 2); - if (td->td_epochnest == 1) - TAILQ_INSERT_TAIL(&eps->eps_record.er_tdlist, td, td_epochq); #ifdef INVARIANTS + MPASS(td->td_epochnest < UCHAR_MAX - 2); if (td->td_epochnest > 1) { struct thread *curtd; int found = 0; @@ -307,38 +301,31 @@ epoch_enter(epoch_t epoch) if (curtd == td) found = 1; KASSERT(found, ("recursing on a second epoch")); - } -#endif - if (td->td_epochnest > 1) { critical_exit(); return; } +#endif + TAILQ_INSERT_TAIL(&eps->eps_record.er_tdlist, td, td_epochq); sched_pin(); ck_epoch_begin(&eps->eps_record.er_record, (ck_epoch_section_t*)&td->td_epoch_section); critical_exit(); } void -epoch_exit(epoch_t epoch) +epoch_exit_internal(epoch_t epoch, struct thread *td) { struct epoch_pcpu_state *eps; - struct thread *td; td = curthread; + MPASS(td->td_epochnest == 0); INIT_CHECK(epoch); - MPASS(td->td_epochnest); critical_enter(); eps = epoch->e_pcpu[curcpu]; - td->td_epochnest--; - if (td->td_epochnest == 0) - TAILQ_REMOVE(&eps->eps_record.er_tdlist, td, td_epochq); - else { - critical_exit(); - return; - } - sched_unpin(); + ck_epoch_end(&eps->eps_record.er_record, (ck_epoch_section_t*)&td->td_epoch_section); + TAILQ_REMOVE(&eps->eps_record.er_tdlist, td, td_epochq); eps->eps_record.er_gen++; + sched_unpin(); critical_exit(); } Modified: head/sys/sys/epoch.h ============================================================================== --- head/sys/sys/epoch.h Thu May 17 00:52:48 2018 (r333696) +++ head/sys/sys/epoch.h Thu May 17 01:13:40 2018 (r333697) @@ -29,6 +29,8 @@ #ifndef _SYS_EPOCH_H_ #define _SYS_EPOCH_H_ +#include +#include struct epoch; typedef struct epoch *epoch_t; @@ -43,10 +45,35 @@ typedef struct epoch_context *epoch_context_t; epoch_t epoch_alloc(void); void epoch_free(epoch_t epoch); -void epoch_enter(epoch_t epoch); -void epoch_exit(epoch_t epoch); +void epoch_enter_internal(epoch_t epoch, struct thread *td); +void epoch_exit_internal(epoch_t epoch, struct thread *td); void epoch_wait(epoch_t epoch); void epoch_call(epoch_t epoch, epoch_context_t ctx, void (*callback) (epoch_context_t)); int in_epoch(void); + +static __inline void +epoch_enter(epoch_t epoch) +{ + struct thread *td; + int nesting; + + td = curthread; + nesting = td->td_epochnest++; +#ifndef INVARIANTS + if (nesting == 0) +#endif + epoch_enter_internal(epoch, td); +} + +static __inline void +epoch_exit(epoch_t epoch) +{ + struct thread *td; + + td = curthread; + MPASS(td->td_epochnest); + if (td->td_epochnest-- == 1) + epoch_exit_internal(epoch, td); +} #endif