From owner-freebsd-current@FreeBSD.ORG Mon Dec 19 15:50:56 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4015E1065673 for ; Mon, 19 Dec 2011 15:50:56 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from agogare.doit.wisc.edu (agogare.doit.wisc.edu [144.92.197.211]) by mx1.freebsd.org (Postfix) with ESMTP id 0E4198FC1D for ; Mon, 19 Dec 2011 15:50:54 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from avs-daemon.smtpauth2.wiscmail.wisc.edu by smtpauth2.wiscmail.wisc.edu (Sun Java(tm) System Messaging Server 7u2-7.05 32bit (built Jul 30 2009)) id <0LWG00M0KK0UBF00@smtpauth2.wiscmail.wisc.edu>; Mon, 19 Dec 2011 09:50:54 -0600 (CST) Received: from comporellon.tachypleus.net ([unknown] [76.210.77.223]) by smtpauth2.wiscmail.wisc.edu (Sun Java(tm) System Messaging Server 7u2-7.05 32bit (built Jul 30 2009)) with ESMTPSA id <0LWG00GLOK0RRM10@smtpauth2.wiscmail.wisc.edu>; Mon, 19 Dec 2011 09:50:51 -0600 (CST) Date: Mon, 19 Dec 2011 09:50:50 -0600 From: Nathan Whitehorn In-reply-to: To: freebsd-current@freebsd.org, freebsd-performance@freebsd.org Message-id: <4EEF5D5A.5050700@freebsd.org> X-Spam-Report: AuthenticatedSender=yes, SenderIP=76.210.77.223 X-Spam-PmxInfo: Server=avs-14, Version=5.6.1.2065439, Antispam-Engine: 2.7.2.376379, Antispam-Data: 2011.12.19.154214, SenderIP=76.210.77.223 References: <4EE1EAFE.3070408@m5p.com> <4EE22421.9060707@gmail.com> <4EE6060D.5060201@mail.zedat.fu-berlin.de> <4EE69C5A.3090005@FreeBSD.org> <20111213104048.40f3e3de@nonamehost> <20111213090051.GA3339@vniz.net> <4EED5200.20302@cran.org.uk> <20111218164924.L64681@sola.nimnet.asn.au> <20111218075241.GA45367@vniz.net> <20111218102401.GA42627@freebsd.org> <20111218102600.GA44118@freebsd.org> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:8.0) Gecko/20111113 Thunderbird/8.0 Cc: Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Dec 2011 15:50:56 -0000 On 12/18/11 04:34, Adrian Chadd wrote: > The trouble is that there's lots of anecdotal evidence, but noone's > really gone digging deep into _their_ example of why it's broken. The > developers who know this stuff don't see anything wrong. That hints to > me it may be something a little more creepy - as an example, the > interplay between netisr/swi/taskqueue/callbacks and such. It may be > that something is being starved that isn't obviously obvious. It's > just a stab in the dark, but it sounds somewhat plausible based on > what I've seen ULE do in my network throughput hacking. > > I applaud reppie for trying to make it as easy as possible for people > to use KTR to provide scheduler traces for him to go digging with, so > please, if you have these issues and you can absolutely reproduce > them, please follow his instructions and work with him to get him what > he needs. > The thing I've seen is that ULE is substantially more enthusiastic about migrating processes between cores than 4BSD. Often, this is a good thing, but can increase the rate of cache misses, hurting performance for cache-bound processes (I see this particularly in HPC-type scientific workloads). It might be interesting to add some kind of tunable here. Another more interesting and slightly longer-term possibility if someone wants a project would be to integrate scheduling decisions with hwpmc counters, to accumulate statistics on cache hits at each context switch and preferentially keep processes with a high hits/misses ratio on the same thread/cache domain relative to processes with a low one. -Nathan P.S. The other thing that could be very interesting from a research and scheduling standpoint would be to integrate heterogeneous SMP support into the operating system, with a FreeBSD-4 "Application Processor" syscall model. We seem to be going down the road where GPGPU computing has MMUs, timer interrupts, IPIs, etc. (the next AMD Fusions, IBM Cell), as well as potential systems with both x86 and ARM cores. This is something that no operating system currently supports well, and would be a place for BSD to shine. If anyone has a free graduate student...