From owner-freebsd-current@FreeBSD.ORG Tue Sep 16 04:32:30 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21A0A1065677 for ; Tue, 16 Sep 2008 04:32:30 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (dan.emsphone.com [199.67.51.101]) by mx1.freebsd.org (Postfix) with ESMTP id D08A08FC26 for ; Tue, 16 Sep 2008 04:32:29 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.3/8.14.3) with ESMTP id m8G4BiIL037745 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 15 Sep 2008 23:11:45 -0500 (CDT) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.3/8.14.3/Submit) id m8G4BhVo037744; Mon, 15 Sep 2008 23:11:43 -0500 (CDT) (envelope-from dan) Date: Mon, 15 Sep 2008 23:11:43 -0500 From: Dan Nelson To: Stephen Montgomery-Smith Message-ID: <20080916041142.GH3188@dan.emsphone.com> References: <48CDBC78.4010409@math.missouri.edu> <20080915195021.GA69528@cons.org> <48CEFF74.8020602@math.missouri.edu> <20080916033459.GA31220@troutmask.apl.washington.edu> <48CF2AEF.9070208@math.missouri.edu> <48CF2CA4.1000802@math.missouri.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <48CF2CA4.1000802@math.missouri.edu> X-OS: FreeBSD 7.1-PRERELEASE User-Agent: Mutt/1.5.18 (2008-05-17) Cc: Martin Cracauer , freebsd-current@freebsd.org, Steve Kargl Subject: Re: Improved multiprocessor usage on amd64 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Sep 2008 04:32:30 -0000 In the last episode (Sep 15), Stephen Montgomery-Smith said: > Stephen Montgomery-Smith wrote: > > Steve Kargl wrote: > >> On Mon, Sep 15, 2008 at 07:36:04PM -0500, Stephen Montgomery-Smith wrote: > >>> ... and each thread is a loop of the form > >>> > >>> while (1) { > >>> wait until told to start; > >>> do massive amounts of floating point arithmetic (only additions and > >>> multiplications) on large arrays; > >>> tell the master process that you are done; > >>> } > >>> > >>>> Do you have about as many threads as processor or more? > >>> Both ways. The time difference between the two approaches is > >>> negligible. > >>> > >> > >> Are you using ULE? With my MPI applications, if the number of > >> launched processes exceeds the number of cpus by 1, ULE falls > >> through the floor. I have a nagging feeling that there is a problem > >> with cpu affinity. > >> > >> http://lists.freebsd.org/pipermail/freebsd-current/2008-July/086917.html > > Let me say a little bit more. > > I have this gut feeling that the problem has a lot to do with cache > management. My program has each thread doing, in effect, huge matrix > multiplications, each one working on their own little bit. If a CPU > core changes from one thread to another, it then has to flush out the > cache to RAM, and read in a whole bunch of other RAM into cache. You can try playing with the new cpuset functions in HEAD and 7-STABLE to lock particular threads on certain CPUs. -- Dan Nelson dnelson@allantgroup.com