From owner-svn-src-head@FreeBSD.ORG Thu Jul 3 11:06:28 2014 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 54264B2F; Thu, 3 Jul 2014 11:06:28 +0000 (UTC) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 28F652A0B; Thu, 3 Jul 2014 11:06:28 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.8/8.14.8) with ESMTP id s63B6Stm090424; Thu, 3 Jul 2014 11:06:28 GMT (envelope-from kib@svn.freebsd.org) Received: (from kib@localhost) by svn.freebsd.org (8.14.8/8.14.8/Submit) id s63B6SRS090423; Thu, 3 Jul 2014 11:06:28 GMT (envelope-from kib@svn.freebsd.org) Message-Id: <201407031106.s63B6SRS090423@svn.freebsd.org> From: Konstantin Belousov Date: Thu, 3 Jul 2014 11:06:28 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r268211 - head/sys/kern X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2014 11:06:28 -0000 Author: kib Date: Thu Jul 3 11:06:27 2014 New Revision: 268211 URL: http://svnweb.freebsd.org/changeset/base/268211 Log: Micro-manage clang to get the expected inlining for cpu_search(). Mark cpu_search_lowest/cpu_search_highest/cpu_search_both as noinline, while cpu_search() gets always_inline. With the attributes set, cpu_search() is inlined in wrappers, and if()s with constant conditionals are optimized. On some tests on many-core machine, the hwpmc reported samples for cpu_search*() are reduced from 25% total to 9%. Submitted by: "Rang, Anton" MFC after: 1 week Modified: head/sys/kern/sched_ule.c Modified: head/sys/kern/sched_ule.c ============================================================================== --- head/sys/kern/sched_ule.c Thu Jul 3 10:59:42 2014 (r268210) +++ head/sys/kern/sched_ule.c Thu Jul 3 11:06:27 2014 (r268211) @@ -622,12 +622,14 @@ struct cpu_search { for ((cpu) = 0; (cpu) <= mp_maxid; (cpu)++) \ if (CPU_ISSET(cpu, &mask)) -static __inline int cpu_search(const struct cpu_group *cg, struct cpu_search *low, - struct cpu_search *high, const int match); -int cpu_search_lowest(const struct cpu_group *cg, struct cpu_search *low); -int cpu_search_highest(const struct cpu_group *cg, struct cpu_search *high); -int cpu_search_both(const struct cpu_group *cg, struct cpu_search *low, +static __always_inline int cpu_search(const struct cpu_group *cg, + struct cpu_search *low, struct cpu_search *high, const int match); +int __noinline cpu_search_lowest(const struct cpu_group *cg, + struct cpu_search *low); +int __noinline cpu_search_highest(const struct cpu_group *cg, struct cpu_search *high); +int __noinline cpu_search_both(const struct cpu_group *cg, + struct cpu_search *low, struct cpu_search *high); /* * Search the tree of cpu_groups for the lowest or highest loaded cpu @@ -640,7 +642,7 @@ int cpu_search_both(const struct cpu_gro * match argument. It is reduced to the minimum set for each case. It is * also recursive to the depth of the tree. */ -static __inline int +static __always_inline int cpu_search(const struct cpu_group *cg, struct cpu_search *low, struct cpu_search *high, const int match) {