From owner-freebsd-stable@FreeBSD.ORG  Thu Dec 22 18:45:34 2011
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id D48E8106564A;
	Thu, 22 Dec 2011 18:45:34 +0000 (UTC)
	(envelope-from sgk@troutmask.apl.washington.edu)
Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu
	[128.95.76.21])
	by mx1.freebsd.org (Postfix) with ESMTP id 9035D8FC0A;
	Thu, 22 Dec 2011 18:45:34 +0000 (UTC)
Received: from troutmask.apl.washington.edu (localhost.apl.washington.edu
	[127.0.0.1])
	by troutmask.apl.washington.edu (8.14.5/8.14.5) with ESMTP id
	pBMIjYkY036258; Thu, 22 Dec 2011 10:45:34 -0800 (PST)
	(envelope-from sgk@troutmask.apl.washington.edu)
Received: (from sgk@localhost)
	by troutmask.apl.washington.edu (8.14.5/8.14.5/Submit) id
	pBMIjVjp036255; Thu, 22 Dec 2011 10:45:31 -0800 (PST)
	(envelope-from sgk)
Date: Thu, 22 Dec 2011 10:45:31 -0800
From: Steve Kargl <sgk@troutmask.apl.washington.edu>
To: Luigi Rizzo <rizzo@iet.unipi.it>
Message-ID: <20111222184531.GA36084@troutmask.apl.washington.edu>
References: <4EE1EAFE.3070408@m5p.com>
	<CAJ-FndBSOS3hKYqmPnVkoMhPmowBBqy9-+eJJEMTdoVjdMTEdw@mail.gmail.com>
	<20111215215554.GA87606@troutmask.apl.washington.edu>
	<CAJ-FndD0vFWUnRPxz6CTR5JBaEaY3gh9y7-Dy6Gds69_aRgfpg@mail.gmail.com>
	<20111222005250.GA23115@troutmask.apl.washington.edu>
	<20111222103145.GA42457@onelab2.iet.unipi.it>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20111222103145.GA42457@onelab2.iet.unipi.it>
User-Agent: Mutt/1.4.2.3i
Cc: Attilio Rao <attilio@freebsd.org>, Andrey Chernov <ache@nagual.pp.ru>,
	George Mitchell <george+freebsd@m5p.com>,
	Doug Barton <dougb@freebsd.org>, freebsd-stable@freebsd.org
Subject: Re: SCHED_ULE should not be the default
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>, 
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 22 Dec 2011 18:45:34 -0000

On Thu, Dec 22, 2011 at 11:31:45AM +0100, Luigi Rizzo wrote:
> On Wed, Dec 21, 2011 at 04:52:50PM -0800, Steve Kargl wrote:
> > 
> > I have placed several files at
> > 
> > http://troutmask.apl.washington.edu/~kargl/freebsd
> > 
> > dmesg.txt      --> dmesg for ULE kernel
> > summary        --> A summary that includes top(1) output of all runs.
> > sysctl.ule.txt --> sysctl -a for the ULE kernel
> > ktr-ule-problem-kargl.out.gz 

I've replaced the original version of the ktr file with
a new version.  The old version was corrupt due to my
failure to set 'sysctl debug.ktr.mask=0' prior to the
dump.

> One explanation for taking 1.5-2x times is that with ULE the
> threads are not migrated properly, so you end up with idle cores
> and ready threads not running (the other possible explanation
> would be that there are migrations, but they are so frequent and
> expensive that they completely trash the caches. But this seems
> unlikely for this type of task).

I've used schedgraph to look at the ktrdump output.  A jpg is
available at http://troutmask.apl.washington.edu/~kargl/freebsd/ktr.jpg
This shows the ping-pong effect where here 3 processes appear to be
using 2 cpus while the remaining 2 processes are pinned to their
cpus.

-- 
Steve