From owner-freebsd-current@freebsd.org Sun Jun 10 01:35:19 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8319B1008218 for ; Sun, 10 Jun 2018 01:35:19 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 15B5C82B77; Sun, 10 Jun 2018 01:35:19 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.15.2/8.15.2) with ESMTPS id w5A1ZBQf010701 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 9 Jun 2018 18:35:12 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.15.2/8.15.2/Submit) id w5A1ZBHL010700; Sat, 9 Jun 2018 18:35:11 -0700 (PDT) (envelope-from sgk) Date: Sat, 9 Jun 2018 18:35:11 -0700 From: Steve Kargl To: Don Lewis Cc: Stefan Esser , Andriy Gapon , FreeBSD Current , "M. Warner Losh" Subject: Re: Is kern.sched.preempt_thresh=0 a sensible default? Message-ID: <20180610013511.GA10634@troutmask.apl.washington.edu> Reply-To: sgk@troutmask.apl.washington.edu References: <1d188cb0-ebc8-075f-ed51-57641ede1fd6@freebsd.org> <49fa8de4-e164-0642-4e01-a6188992c32e@freebsd.org> <32d6305b-3d57-4d37-ba1b-51631e994520@FreeBSD.org> <93efc3e1-7ac3-fedc-a71e-66c99f8e8c1e@freebsd.org> <9aaec961-e604-303a-52f3-ee24e3a435d0@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.2 (2017-12-15) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jun 2018 01:35:19 -0000 On Sat, Jun 09, 2018 at 06:07:15PM -0700, Don Lewis wrote: > On 9 Jun, Stefan Esser wrote: > > > 3) Programs that evenly split the load on all available cores have been > > suffering from sub-optimal assignment of threads to cores. E.g. on a > > CPU with 8 (virtual) cores, this resulted in 6 cores running the load > > in nominal time, 1 core taking twice as long because 2 threads were > > scheduled to run on it, while 1 core was mostly idle. Even if the > > load was initially evenly distributed, a woken up process that ran on > > one core destroyed the symmetry and it was not recovered. (This was a > > problem e.g. for parallel programs using MPI or the like.) > > When a core is about to go idle or first enters the idle state it will > search for the most heavily loaded core and steal a thread from it. The > core will only go to sleep if it can't find a non-running thread to > steal. > > If there are N cores and N+1 runnable threads, there is a long term load > balancer than runs periodically. It searches for the most and least > loaded cores and moves a thread from the former to the latter. That > prevents the same pair of threads from having to share the same core > indefinitely. > > There is an observed bug where a low priority thread can get pinned to a > particular core that is already occupied by a high-priority CPU-bound > thread that never releases the CPU. The low priority thread can't > migrate to another core that subsequently becomes available because it > it is pinned. It is not known how the thread originally got into this > state. I don't see any reason for 4BSD to be immune to this problem. > It is a well-known problem that an over-subscribed ULE kernel has much worse performance than a 4BSD kernel. I've posted more than once with benchmark numbers that demonstrate the problem. -- Steve