From owner-freebsd-stable@FreeBSD.ORG Fri Dec 23 00:59:33 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D192106568A; Fri, 23 Dec 2011 00:59:33 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) by mx1.freebsd.org (Postfix) with ESMTP id CA8FB8FC08; Fri, 23 Dec 2011 00:59:32 +0000 (UTC) Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu [127.0.0.1]) by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id pBN0xWe6047892; Thu, 22 Dec 2011 16:59:32 -0800 (PST) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id pBN0xWmM047891; Thu, 22 Dec 2011 16:59:32 -0800 (PST) (envelope-from sgk) Date: Thu, 22 Dec 2011 16:59:32 -0800 From: Steve Kargl To: Adrian Chadd Message-ID: <20111223005932.GA47439@troutmask.apl.washington.edu> References: <4EE1EAFE.3070408@m5p.com> <20111215215554.GA87606@troutmask.apl.washington.edu> <20111222005250.GA23115@troutmask.apl.washington.edu> <20111222103145.GA42457@onelab2.iet.unipi.it> <20111222184531.GA36084@troutmask.apl.washington.edu> <4EF37E7B.4020505@FreeBSD.org> <20111222194740.GA36796@troutmask.apl.washington.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-stable@freebsd.org, Andriy Gapon Subject: Re: SCHED_ULE should not be the default X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Dec 2011 00:59:33 -0000 On Thu, Dec 22, 2011 at 04:23:29PM -0800, Adrian Chadd wrote: > On 22 December 2011 11:47, Steve Kargl wrote: > > > There is the additional observation in one of my 2008 > > emails (URLs have been posted) that if you have N+1 > > cpu-bound jobs with, say, job0 and job1 ping-ponging > > on cpu0 (due to ULE's cpu-affinity feature) and if I > > kill job2 running on cpu1, then neither job0 nor job1 > > will migrate to cpu1. ?So, one now has N cpu-bound > > jobs running on N-1 cpus. > > .. and this sounds like a pretty serious regression. Have you ever > filed a PR for it? No. I was interacting directly with jeffr in 2008. I got as far as setting up root access on a node for jeffr. Unfortunately, both jeffr and I got busy with real life, and 4BSD allowed me to get my work done. > > Finally, my initial post in this email thread was to > > tell O. Hartman to quit beating his head against > > a wall with ULE (in an HPC environment). ?Switch to > > 4BSD. ?This was based on my 2008 observations and > > I've now wasted 2 days gather additional information > > which only re-affirms my recommendation. > > I personally don't think this is time wasted. You've done something > that noone else has actually done - provided actual results from > real-life testing, rather than a hundred posts of "I remember seeing > X, so I don't use ULE." > > If you can definitely and consistently reproduce that N-1 cpu bound > job bug, you're now in a great position to easily test and re-report > KTR/schedtrace results to see what impact they have. Please don't > underestimate exactly how valuable this is. I'll try this tomorrow. I first need to modify the code I used in the 2008 test to disable IO, so that it is nearly completely cpu-bound. > How often are those two jobs migrating between CPUs? How am I supposed > to read "CPU load" ? Why isn't it just sitting at 100% the whole time? This is my 1st foray into ktr and schedgraph, so I may not have done something incorrectly. In particular, it seems that schedgraph takes the cpu clock as a command line argument, so there is probably some scaling that I'm missing. > Would you mind repeating this with 4BSD (the N+1 jobs) so we can see > how the jobs are scheduled/interleaved? Something tells me we'll see > it the jobs being scheduled evenly Sure, I'll do this tomorrow as well. -- Steve