From owner-svn-src-all@FreeBSD.ORG Sat Sep 7 15:16:30 2013 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id A541E2DA; Sat, 7 Sep 2013 15:16:30 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 928032628; Sat, 7 Sep 2013 15:16:30 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.7/8.14.7) with ESMTP id r87FGUmH064853; Sat, 7 Sep 2013 15:16:30 GMT (envelope-from mav@svn.freebsd.org) Received: (from mav@localhost) by svn.freebsd.org (8.14.7/8.14.5/Submit) id r87FGURB064852; Sat, 7 Sep 2013 15:16:30 GMT (envelope-from mav@svn.freebsd.org) Message-Id: <201309071516.r87FGURB064852@svn.freebsd.org> From: Alexander Motin Date: Sat, 7 Sep 2013 15:16:30 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r255363 - head/sys/kern X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Sep 2013 15:16:30 -0000 Author: mav Date: Sat Sep 7 15:16:30 2013 New Revision: 255363 URL: http://svnweb.freebsd.org/changeset/base/255363 Log: Micro-optimize cpu_search(), allowing compiler to use more efficient inline ffsl() implementation, when it is available, instead of homegrown iteration. On dual-E5645 amd64 system (2x6x2 cores) under heavy I/O load that reduces time spent inside cpu_search() from 19% to 13%, while IOPS increased by 5%. Modified: head/sys/kern/sched_ule.c Modified: head/sys/kern/sched_ule.c ============================================================================== --- head/sys/kern/sched_ule.c Sat Sep 7 14:15:13 2013 (r255362) +++ head/sys/kern/sched_ule.c Sat Sep 7 15:16:30 2013 (r255363) @@ -667,10 +667,14 @@ cpu_search(const struct cpu_group *cg, s } /* Iterate through the child CPU groups and then remaining CPUs. */ - for (i = cg->cg_children, cpu = mp_maxid; i >= 0; ) { + for (i = cg->cg_children, cpu = mp_maxid; ; ) { if (i == 0) { +#ifdef HAVE_INLINE_FFSL + cpu = CPU_FFS(&cpumask) - 1; +#else while (cpu >= 0 && !CPU_ISSET(cpu, &cpumask)) cpu--; +#endif if (cpu < 0) break; child = NULL; @@ -695,6 +699,7 @@ cpu_search(const struct cpu_group *cg, s break; } } else { /* Handle child CPU. */ + CPU_CLR(cpu, &cpumask); tdq = TDQ_CPU(cpu); load = tdq->tdq_load * 256; rndptr = DPCPU_PTR(randomval); @@ -742,8 +747,11 @@ cpu_search(const struct cpu_group *cg, s i--; if (i == 0 && CPU_EMPTY(&cpumask)) break; - } else + } +#ifndef HAVE_INLINE_FFSL + else cpu--; +#endif } return (total); }