From owner-svn-src-head@freebsd.org Mon May 7 23:36:17 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 609DAFC15C3; Mon, 7 May 2018 23:36:17 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0DE62721F6; Mon, 7 May 2018 23:36:17 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 028C122245; Mon, 7 May 2018 23:36:17 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w47NaGWv001866; Mon, 7 May 2018 23:36:16 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w47NaGXH001864; Mon, 7 May 2018 23:36:16 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <201805072336.w47NaGXH001864@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Mon, 7 May 2018 23:36:16 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r333344 - in head/sys: kern sys X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: in head/sys: kern sys X-SVN-Commit-Revision: 333344 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 May 2018 23:36:17 -0000 Author: mjg Date: Mon May 7 23:36:16 2018 New Revision: 333344 URL: https://svnweb.freebsd.org/changeset/base/333344 Log: Inlined sched_userret. The tested condition is rarely true and it induces a function call on each return to userspace. Bumps getuid rate by about 1% on Broadwell. Modified: head/sys/kern/sched_4bsd.c head/sys/kern/sched_ule.c head/sys/sys/sched.h Modified: head/sys/kern/sched_4bsd.c ============================================================================== --- head/sys/kern/sched_4bsd.c Mon May 7 23:23:11 2018 (r333343) +++ head/sys/kern/sched_4bsd.c Mon May 7 23:36:16 2018 (r333344) @@ -1481,25 +1481,13 @@ sched_preempt(struct thread *td) } void -sched_userret(struct thread *td) +sched_userret_slowpath(struct thread *td) { - /* - * XXX we cheat slightly on the locking here to avoid locking in - * the usual case. Setting td_priority here is essentially an - * incomplete workaround for not setting it properly elsewhere. - * Now that some interrupt handlers are threads, not setting it - * properly elsewhere can clobber it in the window between setting - * it here and returning to user mode, so don't waste time setting - * it perfectly here. - */ - KASSERT((td->td_flags & TDF_BORROWING) == 0, - ("thread with borrowed priority returning to userland")); - if (td->td_priority != td->td_user_pri) { - thread_lock(td); - td->td_priority = td->td_user_pri; - td->td_base_pri = td->td_user_pri; - thread_unlock(td); - } + + thread_lock(td); + td->td_priority = td->td_user_pri; + td->td_base_pri = td->td_user_pri; + thread_unlock(td); } void Modified: head/sys/kern/sched_ule.c ============================================================================== --- head/sys/kern/sched_ule.c Mon May 7 23:23:11 2018 (r333343) +++ head/sys/kern/sched_ule.c Mon May 7 23:36:16 2018 (r333344) @@ -2356,26 +2356,14 @@ sched_preempt(struct thread *td) * to static priorities in msleep() or similar. */ void -sched_userret(struct thread *td) +sched_userret_slowpath(struct thread *td) { - /* - * XXX we cheat slightly on the locking here to avoid locking in - * the usual case. Setting td_priority here is essentially an - * incomplete workaround for not setting it properly elsewhere. - * Now that some interrupt handlers are threads, not setting it - * properly elsewhere can clobber it in the window between setting - * it here and returning to user mode, so don't waste time setting - * it perfectly here. - */ - KASSERT((td->td_flags & TDF_BORROWING) == 0, - ("thread with borrowed priority returning to userland")); - if (td->td_priority != td->td_user_pri) { - thread_lock(td); - td->td_priority = td->td_user_pri; - td->td_base_pri = td->td_user_pri; - tdq_setlowpri(TDQ_SELF(), td); - thread_unlock(td); - } + + thread_lock(td); + td->td_priority = td->td_user_pri; + td->td_base_pri = td->td_user_pri; + tdq_setlowpri(TDQ_SELF(), td); + thread_unlock(td); } /* Modified: head/sys/sys/sched.h ============================================================================== --- head/sys/sys/sched.h Mon May 7 23:23:11 2018 (r333343) +++ head/sys/sys/sched.h Mon May 7 23:36:16 2018 (r333344) @@ -103,13 +103,32 @@ void sched_switch(struct thread *td, struct thread *ne void sched_throw(struct thread *td); void sched_unlend_prio(struct thread *td, u_char prio); void sched_user_prio(struct thread *td, u_char prio); -void sched_userret(struct thread *td); +void sched_userret_slowpath(struct thread *td); void sched_wakeup(struct thread *td); #ifdef RACCT #ifdef SCHED_4BSD fixpt_t sched_pctcpu_delta(struct thread *td); #endif #endif + +static inline void +sched_userret(struct thread *td) +{ + + /* + * XXX we cheat slightly on the locking here to avoid locking in + * the usual case. Setting td_priority here is essentially an + * incomplete workaround for not setting it properly elsewhere. + * Now that some interrupt handlers are threads, not setting it + * properly elsewhere can clobber it in the window between setting + * it here and returning to user mode, so don't waste time setting + * it perfectly here. + */ + KASSERT((td->td_flags & TDF_BORROWING) == 0, + ("thread with borrowed priority returning to userland")); + if (__predict_false(td->td_priority != td->td_user_pri)) + sched_userret_slowpath(td); +} /* * Threads are moved on and off of run queues