From owner-svn-src-all@FreeBSD.ORG Fri Feb 22 09:47:21 2013 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id AFDE0485; Fri, 22 Feb 2013 09:47:21 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) by mx1.freebsd.org (Postfix) with ESMTP id 97195239; Fri, 22 Feb 2013 09:47:21 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.5/8.14.5) with ESMTP id r1M9lLnf007895; Fri, 22 Feb 2013 09:47:21 GMT (envelope-from mav@svn.freebsd.org) Received: (from mav@localhost) by svn.freebsd.org (8.14.5/8.14.5/Submit) id r1M9lLmY007894; Fri, 22 Feb 2013 09:47:21 GMT (envelope-from mav@svn.freebsd.org) Message-Id: <201302220947.r1M9lLmY007894@svn.freebsd.org> From: Alexander Motin Date: Fri, 22 Feb 2013 09:47:21 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r247150 - stable/9/sys/kern X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Feb 2013 09:47:21 -0000 Author: mav Date: Fri Feb 22 09:47:21 2013 New Revision: 247150 URL: http://svnweb.freebsd.org/changeset/base/247150 Log: MFC r242852, r243069: Several optimizations to sched_idletd(): - Do not try to steal load from other CPUs if there was no context switches on this CPU (i.e. it was idle all the time and woke up just for bus mastering or TLB shutdown). If current CPU was idle, then it is quite unlikely that some other CPU has load to steal. Under high I/O rate, when TLB shutdowns cause numerous CPU wakeups, on 24-CPU system load stealing code may consume up to 25% of all CPU time without giving any benefits. - Change code that implements spinning for load to restart spin in case of context switch. Previous code periodically called cpu_idle() even under high interrupt/context switch rate. - Rise spinning threshold to 10KHz, where it gives at least some effect that may worth consumed power. Modified: stable/9/sys/kern/sched_ule.c Directory Properties: stable/9/sys/ (props changed) Modified: stable/9/sys/kern/sched_ule.c ============================================================================== --- stable/9/sys/kern/sched_ule.c Fri Feb 22 08:56:48 2013 (r247149) +++ stable/9/sys/kern/sched_ule.c Fri Feb 22 09:47:21 2013 (r247150) @@ -1407,7 +1407,7 @@ sched_initticks(void *dummy) affinity = SCHED_AFFINITY_DEFAULT; #endif if (sched_idlespinthresh < 0) - sched_idlespinthresh = imax(16, 2 * hz / realstathz); + sched_idlespinthresh = 2 * max(10000, 6 * hz) / realstathz; } @@ -1849,7 +1849,8 @@ sched_switch(struct thread *td, struct t preempted = !(td->td_flags & TDF_SLICEEND); td->td_flags &= ~(TDF_NEEDRESCHED | TDF_SLICEEND); td->td_owepreempt = 0; - tdq->tdq_switchcnt++; + if (!TD_IS_IDLETHREAD(td)) + tdq->tdq_switchcnt++; /* * The lock pointer in an idle thread should never change. Reset it * to CAN_RUN as well. @@ -2575,18 +2576,30 @@ sched_idletd(void *dummy) { struct thread *td; struct tdq *tdq; - int switchcnt; + int oldswitchcnt, switchcnt; int i; mtx_assert(&Giant, MA_NOTOWNED); td = curthread; tdq = TDQ_SELF(); + oldswitchcnt = -1; for (;;) { + if (tdq->tdq_load) { + thread_lock(td); + mi_switch(SW_VOL | SWT_IDLE, NULL); + thread_unlock(td); + } + switchcnt = tdq->tdq_switchcnt + tdq->tdq_oldswitchcnt; #ifdef SMP - if (tdq_idled(tdq) == 0) - continue; -#endif + if (switchcnt != oldswitchcnt) { + oldswitchcnt = switchcnt; + if (tdq_idled(tdq) == 0) + continue; + } switchcnt = tdq->tdq_switchcnt + tdq->tdq_oldswitchcnt; +#else + oldswitchcnt = switchcnt; +#endif /* * If we're switching very frequently, spin while checking * for load rather than entering a low power state that @@ -2601,20 +2614,26 @@ sched_idletd(void *dummy) cpu_spinwait(); } } + + /* If there was context switch during spin, restart it. */ switchcnt = tdq->tdq_switchcnt + tdq->tdq_oldswitchcnt; - if (tdq->tdq_load == 0) { - tdq->tdq_cpu_idle = 1; - if (tdq->tdq_load == 0) { - cpu_idle(switchcnt > sched_idlespinthresh * 4); - tdq->tdq_switchcnt++; - } - tdq->tdq_cpu_idle = 0; - } - if (tdq->tdq_load) { - thread_lock(td); - mi_switch(SW_VOL | SWT_IDLE, NULL); - thread_unlock(td); - } + if (tdq->tdq_load != 0 || switchcnt != oldswitchcnt) + continue; + + /* Run main MD idle handler. */ + tdq->tdq_cpu_idle = 1; + cpu_idle(switchcnt * 4 > sched_idlespinthresh); + tdq->tdq_cpu_idle = 0; + + /* + * Account thread-less hardware interrupts and + * other wakeup reasons equal to context switches. + */ + switchcnt = tdq->tdq_switchcnt + tdq->tdq_oldswitchcnt; + if (switchcnt != oldswitchcnt) + continue; + tdq->tdq_switchcnt++; + oldswitchcnt++; } }