From owner-freebsd-hackers@freebsd.org Wed Feb 20 18:20:24 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3319614FB55E for ; Wed, 20 Feb 2019 18:20:24 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 49BE98C3F3 for ; Wed, 20 Feb 2019 18:20:13 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id x1KIK2M6053163 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 20 Feb 2019 19:20:02 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: farhan@farhan.codes Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x1KIK1rM054778 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Thu, 21 Feb 2019 01:20:01 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Optimize execution of processes by CPU core To: Farhan Khan , freebsd-hackers@freebsd.org References: <2fcd9aee092730e11880c3ae88de4898@farhan.codes> From: Eugene Grosbein Message-ID: Date: Thu, 21 Feb 2019 01:19:53 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <2fcd9aee092730e11880c3ae88de4898@farhan.codes> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 49BE98C3F3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.89 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.99)[-0.989,0]; IP_SCORE(-1.30)[ip: (-1.92), ipnet: 2a01:4f8::/29(-2.37), asn: 24940(-2.23), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Feb 2019 18:20:24 -0000 21.02.2019 0:35, Farhan Khan via freebsd-hackers wrote: > Hi all, > > I am trying to optimize the execution of a CPU-intensive workload where I am running multiple instances of a program. > The moment the program ends (expected behavior), the calling shell script verifies the results and if its good it reruns the program. > The machines I am running this on have 8 cores, but ps reports that some of the processes frequently run on the same CPU, > so I suspect I am not getting optimized performance. System scheduler switches useland processed from one CPU core to another often enough so each CPU has even load and you cannot see that with naked. But you can easily verify if load is even or not checking sysctl kern.cp_times that shows five monotonically increasing counters per each CPU core. For example, in case of dual-core system: $ sysctl kern.cp_times kern.cp_times: 14789486 132229 14016113 327160 949428773 14374865 139326 12056998 2941012 949179638 $ sysctl kern.clockrate kern.clockrate: { hz = 1000, tick = 1000, profhz = 8126, stathz = 127 } There are "stathz" ticks per second and for each core exactly one of five counters is incremented by one: user, nice, system, interrupt, idle. That is, each 5th counter is incremented if corresponding CPU core was idle during "the tick". You can save output of sysctl kern.cp_times, run your test, stop it and save output again. Then compare difference of each 5th counter and they should be approximately equal. You can even draw graphs if you periodically get samples of the sysctl, compute diffs with previous samples, divide diffs by period length in seconds and then divide again by "stathz" value. Multiply by 100 to get idle time of single CPU code in percents for the period. Repeat for each core. I use net-mgmt/mrtg to draw such per-CPU graphs for my servers, it works just fine.