From owner-freebsd-arch@freebsd.org Sat Aug 26 18:18:18 2017 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AEB0BDD786F for ; Sat, 26 Aug 2017 18:18:18 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1b.ore.mailhop.org (outbound1b.ore.mailhop.org [54.200.247.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8D21A73EA3 for ; Sat, 26 Aug 2017 18:18:18 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-User: f1f4ae47-8a8a-11e7-950d-03a3531dacf2 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 73.78.92.27 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [73.78.92.27]) by outbound1.ore.mailhop.org (Halon) with ESMTPSA id f1f4ae47-8a8a-11e7-950d-03a3531dacf2; Sat, 26 Aug 2017 18:18:23 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id v7QIIEmO006472; Sat, 26 Aug 2017 12:18:14 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <1503771494.56799.49.camel@freebsd.org> Subject: Re: ULE steal_idle questions From: Ian Lepore To: "Rodney W. Grimes" , Bruce Evans Cc: Don Lewis , avg@freebsd.org, freebsd-arch@freebsd.org Date: Sat, 26 Aug 2017 12:18:14 -0600 In-Reply-To: <201708261812.v7QIC2eJ074443@pdx.rh.CN85.dnsmgr.net> References: <201708261812.v7QIC2eJ074443@pdx.rh.CN85.dnsmgr.net> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Aug 2017 18:18:18 -0000 On Sat, 2017-08-26 at 11:12 -0700, Rodney W. Grimes wrote: > > > > On Fri, 25 Aug 2017, Don Lewis wrote: > > > > > > > > ... > > > Something else that I did not expect is the how frequently > > > threads are > > > stolen from the other SMT thread on the same core, even though I > > > increased steal_thresh from 2 to 3 to account for the off-by-one > > > problem.  This is true even right after the system has booted and > > > no > > > significant load has been applied.  My best guess is that because > > > of > > > affinity, both the parent and child processes run on the same CPU > > > after > > > fork(), and if a number of processes are forked() in quick > > > succession, > > > the run queue of that CPU can get really long.  Forcing a thread > > > migration in exec() might be a good solution. > > Since you are trying a lot of combinations, maybe you can tell us > > which > > ones work best.  SCHED_4BSD works better for me on an old 2-core > > system. > > SCHED_ULE works better on a not-so old 4x2 core (Haswell) system, > > but I  > > don't like it due to its complexity.  It makes differences of at > > most > > +-2% except when mistuned it can give -5% for real time (but better > > for > > CPU and presumably power). > > > > For SCHED_4BSD, I wrote fancy tuning for fork/exec and sometimes > > get > > everything to like up for a 3% improvement (803 seconds instead of > > 823 > > on the old system, with -current much slower at 840+ and old > > versions > > of ULE before steal_idle taking 890+).  This is very resource > > (mainly > > cache associativity?) dependent and my tuning makes little > > difference > > on the newer system.  SCHED_ULE still has bugfeatures which tend to > > help large builds by reducing context switching, e.g., by bogusly > > clamping all CPU-bound threads to nearly maximal priority. > That last bugfeature is probably what makes current systems > interactive performance tank rather badly when under heavy > loads.  Would it be hard to fix? > I would second that sentiment... as time goes on, heavily loaded systems seem to become less and less interactive-friendly.  Also, running the heavy-load jobs such as builds with nice, even -n 20, doesn't seem to make any noticible difference in terms of making un- nice'd processes more responsive (not sure there's any relationship in the underlying causes of that, though). -- Ian