From owner-freebsd-arch@freebsd.org Sat Aug 26 18:12:11 2017 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B491BDD761C for ; Sat, 26 Aug 2017 18:12:11 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 70C56739B5; Sat, 26 Aug 2017 18:12:11 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id v7QIC2ZB074444; Sat, 26 Aug 2017 11:12:02 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id v7QIC2eJ074443; Sat, 26 Aug 2017 11:12:02 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201708261812.v7QIC2eJ074443@pdx.rh.CN85.dnsmgr.net> Subject: Re: ULE steal_idle questions In-Reply-To: <20170826094725.G1648@besplex.bde.org> To: Bruce Evans Date: Sat, 26 Aug 2017 11:12:02 -0700 (PDT) CC: Don Lewis , avg@freebsd.org, freebsd-arch@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Aug 2017 18:12:11 -0000 > On Fri, 25 Aug 2017, Don Lewis wrote: > > > ... > > Something else that I did not expect is the how frequently threads are > > stolen from the other SMT thread on the same core, even though I > > increased steal_thresh from 2 to 3 to account for the off-by-one > > problem. This is true even right after the system has booted and no > > significant load has been applied. My best guess is that because of > > affinity, both the parent and child processes run on the same CPU after > > fork(), and if a number of processes are forked() in quick succession, > > the run queue of that CPU can get really long. Forcing a thread > > migration in exec() might be a good solution. > > Since you are trying a lot of combinations, maybe you can tell us which > ones work best. SCHED_4BSD works better for me on an old 2-core system. > SCHED_ULE works better on a not-so old 4x2 core (Haswell) system, but I > don't like it due to its complexity. It makes differences of at most > +-2% except when mistuned it can give -5% for real time (but better for > CPU and presumably power). > > For SCHED_4BSD, I wrote fancy tuning for fork/exec and sometimes get > everything to like up for a 3% improvement (803 seconds instead of 823 > on the old system, with -current much slower at 840+ and old versions > of ULE before steal_idle taking 890+). This is very resource (mainly > cache associativity?) dependent and my tuning makes little difference > on the newer system. SCHED_ULE still has bugfeatures which tend to > help large builds by reducing context switching, e.g., by bogusly > clamping all CPU-bound threads to nearly maximal priority. That last bugfeature is probably what makes current systems interactive performance tank rather badly when under heavy loads. Would it be hard to fix? -- Rod Grimes rgrimes@freebsd.org