From owner-freebsd-current@FreeBSD.ORG Wed Jul 6 18:11:24 2011 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0AC58106564A for ; Wed, 6 Jul 2011 18:11:24 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from mail.icecube.wisc.edu (trout.icecube.wisc.edu [128.104.255.119]) by mx1.freebsd.org (Postfix) with ESMTP id 9AE848FC14 for ; Wed, 6 Jul 2011 18:11:23 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.icecube.wisc.edu (Postfix) with ESMTP id B0F9358139; Wed, 6 Jul 2011 13:11:22 -0500 (CDT) X-Virus-Scanned: amavisd-new at icecube.wisc.edu Received: from mail.icecube.wisc.edu ([127.0.0.1]) by localhost (trout.icecube.wisc.edu [127.0.0.1]) (amavisd-new, port 10030) with ESMTP id X5dPG3QJKonj; Wed, 6 Jul 2011 13:11:22 -0500 (CDT) Received: from wanderer.tachypleus.net (i3-dhcp-172-16-223-128.icecube.wisc.edu [172.16.223.128]) by mail.icecube.wisc.edu (Postfix) with ESMTP id 8FA315811D; Wed, 6 Jul 2011 13:11:22 -0500 (CDT) Message-ID: <4E14A54A.4050106@freebsd.org> Date: Wed, 06 Jul 2011 13:11:22 -0500 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.18) Gecko/20110624 Thunderbird/3.1.11 MIME-Version: 1.0 To: Steve Kargl References: <20110706170132.GA68775@troutmask.apl.washington.edu> <5080.1309971941@critter.freebsd.dk> <20110706180001.GA69157@troutmask.apl.washington.edu> In-Reply-To: <20110706180001.GA69157@troutmask.apl.washington.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Poul-Henning Kamp , FreeBSD Current , "Hartmann, O." , arrowdodger <6yearold@gmail.com>, freebsd-questions@freebsd.org Subject: Re: Heavy I/O blocks FreeBSD box for several seconds X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2011 18:11:24 -0000 On 07/06/11 13:00, Steve Kargl wrote: > On Wed, Jul 06, 2011 at 05:05:41PM +0000, Poul-Henning Kamp wrote: >> In message<20110706170132.GA68775@troutmask.apl.washington.edu>, Steve Kargl w >> rites: >> >>> I periodically ran the same type test in the 2008 post over the >>> last three years. Nothing has changed. I even set up an account >>> on one node in my cluster for jeffr to use. He was too busy to >>> investigate at that time. >> >> Isn't this just the lemming-syncer hurling every dirty block over >> the cliff at the same time ? > > I don't know the answer. Of course, having no experience in > processing scheduling, I don't understand the question either ;-) > > AFAICT, it is a cpu affinity issue. If I launch n+1 MPI images > on a system with n cpus/cores, then 2 (and sometimes 3) images > are stuck on a cpu and those 2 (or 3) images ping-pong on that > cpu. I recall trying to use renice(8) to force some load > balancing, but vaguely remember that it did not help. I've seen exactly this problem with multi-threaded math libraries, as well. Using parallel GotoBLAS on FreeBSD gives terrible performance because the threads keep migrating between CPUs, causing frequent cache misses. -Nathan