From owner-svn-src-head@freebsd.org Thu May 17 00:45:36 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7D398EE1ACF; Thu, 17 May 2018 00:45:36 +0000 (UTC) (envelope-from mmacy@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C95186CF5; Thu, 17 May 2018 00:45:36 +0000 (UTC) (envelope-from mmacy@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0733221A96; Thu, 17 May 2018 00:45:36 +0000 (UTC) (envelope-from mmacy@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w4H0jZJp044723; Thu, 17 May 2018 00:45:35 GMT (envelope-from mmacy@FreeBSD.org) Received: (from mmacy@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w4H0jZSW044721; Thu, 17 May 2018 00:45:35 GMT (envelope-from mmacy@FreeBSD.org) Message-Id: <201805170045.w4H0jZSW044721@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mmacy set sender to mmacy@FreeBSD.org using -f From: Matt Macy Date: Thu, 17 May 2018 00:45:35 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r333695 - in head/sys: kern sys X-SVN-Group: head X-SVN-Commit-Author: mmacy X-SVN-Commit-Paths: in head/sys: kern sys X-SVN-Commit-Revision: 333695 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2018 00:45:36 -0000 Author: mmacy Date: Thu May 17 00:45:35 2018 New Revision: 333695 URL: https://svnweb.freebsd.org/changeset/base/333695 Log: epoch(9): Guarantee forward progress on busy sections Add epoch section to struct thread. We can use this to ennable epoch counter to advance even if a section is perpetually occupied by a thread. Approved by: sbruno Modified: head/sys/kern/subr_epoch.c head/sys/sys/proc.h Modified: head/sys/kern/subr_epoch.c ============================================================================== --- head/sys/kern/subr_epoch.c Wed May 16 23:42:02 2018 (r333694) +++ head/sys/kern/subr_epoch.c Thu May 17 00:45:35 2018 (r333695) @@ -54,7 +54,7 @@ __FBSDID("$FreeBSD$"); static MALLOC_DEFINE(M_EPOCH, "epoch", "epoch based reclamation"); /* arbitrary --- needs benchmarking */ -#define MAX_ADAPTIVE_SPIN 5000 +#define MAX_ADAPTIVE_SPIN 1000 #define EPOCH_EXITING 0x1 #ifdef __amd64__ @@ -63,6 +63,7 @@ static MALLOC_DEFINE(M_EPOCH, "epoch", "epoch based re #define EPOCH_ALIGN CACHE_LINE_SIZE #endif +CTASSERT(sizeof(epoch_section_t) == sizeof(ck_epoch_section_t)); SYSCTL_NODE(_kern, OID_AUTO, epoch, CTLFLAG_RW, 0, "epoch information"); SYSCTL_NODE(_kern_epoch, OID_AUTO, stats, CTLFLAG_RW, 0, "epoch stats"); @@ -308,8 +309,12 @@ epoch_enter(epoch_t epoch) KASSERT(found, ("recursing on a second epoch")); } #endif + if (td->td_epochnest > 1) { + critical_exit(); + return; + } sched_pin(); - ck_epoch_begin(&eps->eps_record.er_record, NULL); + ck_epoch_begin(&eps->eps_record.er_record, (ck_epoch_section_t*)&td->td_epoch_section); critical_exit(); } @@ -324,11 +329,15 @@ epoch_exit(epoch_t epoch) MPASS(td->td_epochnest); critical_enter(); eps = epoch->e_pcpu[curcpu]; - sched_unpin(); - ck_epoch_end(&eps->eps_record.er_record, NULL); td->td_epochnest--; if (td->td_epochnest == 0) TAILQ_REMOVE(&eps->eps_record.er_tdlist, td, td_epochq); + else { + critical_exit(); + return; + } + sched_unpin(); + ck_epoch_end(&eps->eps_record.er_record, (ck_epoch_section_t*)&td->td_epoch_section); eps->eps_record.er_gen++; critical_exit(); } Modified: head/sys/sys/proc.h ============================================================================== --- head/sys/sys/proc.h Wed May 16 23:42:02 2018 (r333694) +++ head/sys/sys/proc.h Thu May 17 00:45:35 2018 (r333695) @@ -76,6 +76,18 @@ /* + * A section object may be passed to every begin-end pair to allow for + * forward progress guarantees with-in prolonged active sections. + * + * We can't include ck_epoch.h so we define our own variant here and + * then CTASSERT that it's the same size in subr_epoch.c + */ +struct epoch_section { + unsigned int bucket; +}; +typedef struct epoch_section epoch_section_t; + +/* * One structure allocated per session. * * List of locks @@ -352,6 +364,7 @@ struct thread { struct proc *td_rfppwait_p; /* (k) The vforked child */ struct vm_page **td_ma; /* (k) uio pages held */ int td_ma_cnt; /* (k) size of *td_ma */ + epoch_section_t td_epoch_section; /* (t) epoch section object */ void *td_emuldata; /* Emulator state data */ int td_lastcpu; /* (t) Last cpu we were on. */ int td_oncpu; /* (t) Which cpu we are on. */