From owner-freebsd-performance@FreeBSD.ORG Thu Jul 7 08:20:30 2011 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A55DB106564A for ; Thu, 7 Jul 2011 08:20:30 +0000 (UTC) (envelope-from case.vanrij@gmail.com) Received: from mail-yi0-f54.google.com (mail-yi0-f54.google.com [209.85.218.54]) by mx1.freebsd.org (Postfix) with ESMTP id 604538FC15 for ; Thu, 7 Jul 2011 08:20:30 +0000 (UTC) Received: by yic13 with SMTP id 13so364234yic.13 for ; Thu, 07 Jul 2011 01:20:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=p0D56lVfkElHKkrTW47sALAUTuW2yWCLJHDv8aPOgkw=; b=A6YzL7Yw+OuwKg+vBZhhjD5kkOd8IhwqAuM+Q1zBXBiHSUlFp3Wq60W1bDY2hu0F/N v1B0JAoAPcY7SWFV3Ug8/Sp4hzWGKmVd6HMTmY0tutXCQeTIynRTYO7ES8vh5/N7aOqR GrcwIilF1c9JlpmNNzzKcBkJvADOF4l7LuOtQ= MIME-Version: 1.0 Received: by 10.236.144.231 with SMTP id n67mr519270yhj.354.1310025337524; Thu, 07 Jul 2011 00:55:37 -0700 (PDT) Received: by 10.236.106.65 with HTTP; Thu, 7 Jul 2011 00:55:37 -0700 (PDT) In-Reply-To: <4E154A5D.8080009@zedat.fu-berlin.de> References: <4E1421D9.7080808@zedat.fu-berlin.de> <4E147F54.40908@zedat.fu-berlin.de> <20110706162811.GA68436@troutmask.apl.washington.edu> <20110706193636.GA69550@troutmask.apl.washington.edu> <4E14CCE5.4050906@zedat.fu-berlin.de> <4E154A5D.8080009@zedat.fu-berlin.de> Date: Thu, 7 Jul 2011 00:55:37 -0700 Message-ID: From: Case van Rij To: sgk@troutmask.apl.washington.edu Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-performance@freebsd.org" , Arnaud Lacombe Subject: Re: Heavy I/O blocks FreeBSD box for several seconds X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2011 08:20:30 -0000 On Wed, Jul 6, 2011 at 10:55 PM, Hartmann, O. wrote: > On 07/07/11 06:29, Arnaud Lacombe wrote: >> >> Hi, >> >> On Wed, Jul 6, 2011 at 5:00 PM, Hartmann, O. >> =A0wrote: >>> >>> On 07/06/11 21:36, Steve Kargl wrote: >>>> >>>> On Wed, Jul 06, 2011 at 03:18:35PM -0400, Arnaud Lacombe wrote: >>>>> >>>>> Hi, >>>>> >>>>> On Wed, Jul 6, 2011 at 12:28 PM, Steve Kargl >>>>> =A0 =A0wrote: >>>>>> >>>>>> On Wed, Jul 06, 2011 at 05:29:24PM +0200, O. Hartmann wrote: >>>>>>> >>>>>>> I use SCHED_ULE on all machines, since it is supposed to be >>>>>>> performing >>>>>>> better on multicore boxes, but there are lots of suggestions >>>>>>> switching >>>>>>> back to the old SCHED_4BSD scheduler. >>>>>>> >>>>>> If you are using MPI in numerical codes, then you want >>>>>> to use SCHED_4BSD. ?I've posted numerous times about ULE >>>>>> and its very poor performance when using MPI. >>>>>> >>>>>> http://lists.freebsd.org/pipermail/freebsd-hackers/2008-October/0263= 75.html >>>>>>>With ULE, 2 Test_mpi jobs are always scheduled on the same core whil= e one >>>>>>>core remains idle. Also, note the difference in the reported load a= verages. While possibly not the same issue you're seeing, I noticed a similar problem on 8 and 12 core machines with ULE, specifically with a relatively small number of threads runnable but waiting to run on a busy core while other cores were sitting idle. tdq_idled won't steal threads from a queue unless there are kern.sched.steal_thresh threads in that queue, where steal_thresh =3D min(fls(mp_ncpus) - 1, 3); ie. on an 8 core system you need 3 threads in the queue before idled steals one. Fortunately you can simply override steal_thresh at run time. 1 works great for me, ymmv.