From owner-freebsd-stable@freebsd.org Mon Sep 14 12:18:34 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D507B9CA34C for ; Mon, 14 Sep 2015 12:18:34 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B356314D3 for ; Mon, 14 Sep 2015 12:18:34 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id AFF819CA34A; Mon, 14 Sep 2015 12:18:34 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95BE79CA349 for ; Mon, 14 Sep 2015 12:18:34 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: from mail-lb0-x233.google.com (mail-lb0-x233.google.com [IPv6:2a00:1450:4010:c04::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1AA5C14D2 for ; Mon, 14 Sep 2015 12:18:34 +0000 (UTC) (envelope-from trtrmitya@gmail.com) Received: by lbbmp1 with SMTP id mp1so66259329lbb.1 for ; Mon, 14 Sep 2015 05:18:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=nzgJmunnJVKg2tq1Zrg26VCz242VNnJn+JjqYBjWgV0=; b=1BIbHzQU7qkfURE5hYctNKFEKiBI3wN4VsfpaFoQboFf7rGG2FQ+zkacq7c34vZZzh orL1lMIt7f6yQmQqHF0CuM8f+ospoCDULN0TlFLtBQvJmeaGV9L0Rn7hnmuNJjFoCDbk bZwLBEA8SZtZ1jmmxY1NxcgBl8h8PfXoV6tpkw1mII40sWZOFbLFubt5XR0dizNgKmUQ 7uj8C1KhsD7nB01mvjutR4YMPierrmomnQsx193F5EikmXHLjKE9XbMXusaUeKahu9am H54rr0bfN00DuUw6b0gJG+QmT/GuIgXPxGSgpF5vcQQ0LEoSSCCkBobIyjivJd9jWmAw ZHZw== X-Received: by 10.152.43.137 with SMTP id w9mr13801462lal.56.1442233111889; Mon, 14 Sep 2015 05:18:31 -0700 (PDT) Received: from ?IPv6:2a02:6b8::408:c59:1b6c:c596:27fc? ([2a02:6b8:0:408:c59:1b6c:c596:27fc]) by smtp.gmail.com with ESMTPSA id j8sm2338310lah.46.2015.09.14.05.18.30 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 14 Sep 2015 05:18:30 -0700 (PDT) Content-Type: text/plain; charset=koi8-r Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\)) Subject: Re: process scheduling and cpuset From: Dmitry Sivachenko In-Reply-To: <20150913164057.GT21849@zxy.spb.ru> Date: Mon, 14 Sep 2015 15:18:29 +0300 Cc: FreeBSD Stable ML Content-Transfer-Encoding: quoted-printable Message-Id: <6FCB48B4-02BD-4048-A3A5-3F86A25F5386@gmail.com> References: <623FA99E-04E7-4D29-953A-61EE7B35CBF6@gmail.com> <20150913130920.GR3158@zxy.spb.ru> <20150913164057.GT21849@zxy.spb.ru> To: Slawa Olhovchenkov X-Mailer: Apple Mail (2.2104) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Sep 2015 12:18:34 -0000 > On 13 =D3=C5=CE=D4. 2015 =C7., at 19:40, Slawa Olhovchenkov = wrote: >=20 > On Sun, Sep 13, 2015 at 04:44:40PM +0300, Dmitry Sivachenko wrote: >=20 >>=20 >>> On 13 =D3=C5=CE=D4. 2015 =C7., at 16:09, Slawa Olhovchenkov = wrote: >>>=20 >>> On Sun, Sep 13, 2015 at 02:52:08PM +0300, Dmitry Sivachenko wrote: >>>=20 >>>> Hello, >>>>=20 >>>> I have 32 processor machine (2x CPU E5-2650) running several = CPU-bound processes (ULE scheduler). >>>> 3 processes are 32-threaded, and 8 are single threaded. >>>>=20 >>>> I bind all 3 32-threaded processes to CPUs 0-24 (cpuset -C -l 0-24 = -p XXX). >>>>=20 >>>> I expect that the remaining 8 single-threaded processes will = (mostly) run on the remaining 25-31 CPU cores and use (almost) 100% cpu = each. >>>>=20 >>>> But this is not the case (according to top(1)): they spend a lot = of time on 0-24 CPUs and CPU Idle time is about 10%. >>>>=20 >>>> These are all purely computational programs, in idle system = single-threaded programs steadily consume 100% of a core, and = 32-threaded programs consume all 32 cores and idle time is zero. >>>>=20 >>>> Is it an ULE scheduler feature or am I doing something wrong? >>>>=20 >>>> The goal is to give a single-threaded program a chance to run when = somebody started several 32-threaded processes. >>>=20 >>> You don't have 32 processor machine, you have only 16 processor >>> machine. >>> SMT/hyperthreading don't give real processor, SMT "CPU" have >>> unpredicable power and his load depend on load parent CPU. >>>=20 >>> For example, for my case I see such condition (simpliy) on CPU 0 and = 1 >>> (SMT of one real core) with rise load: >>>=20 >>> load 0.1 0.1 >>> load 0.2 0.2 >>> load 0.3 0.3 >>> load 0.4 0.4 >>> load 0.45 0.45 >>> load 0.48 0.48 >>> load 1.00 1.00\ >>=20 >>=20 >> Yes I know about HT. But how does this explain why I have 10% of CPU = idle? >>=20 >> If I explicitly bind my single-threaded processes to the remaining = CPU cores (25-32), they start to receive expected 100% of CPU and = overall Idle decreases. >>=20 >> I just expect scheduler to do the same for me. >>=20 >=20 > Idle is not goal, goal is lessing task executing time. Thanks for the explanation. In my example SMT pairs are numbered with sequential numbers, so 0+1 is = one SMT group, 2+3 is second SMT group, and so on. So in 25-32 range there are several real CPU cores which remain idle = while processes are fighting for overloaded 0-24. When I explicitly pin my single-threaded processes to 25-32 range, they = start to receive 100% of CPU (and finish faster to be clear).