From owner-freebsd-smp@FreeBSD.ORG Wed Oct 17 16:59:31 2007 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D5B8816A417 for ; Wed, 17 Oct 2007 16:59:31 +0000 (UTC) (envelope-from estrabd@gmail.com) Received: from bc1.hpc.lsu.edu (bc1.hpc.lsu.edu [130.39.198.6]) by mx1.freebsd.org (Postfix) with ESMTP id 877A113C44B for ; Wed, 17 Oct 2007 16:59:31 +0000 (UTC) (envelope-from estrabd@gmail.com) Received: from bc1.hpc.lsu.edu (localhost.hpc.lsu.edu [127.0.0.1]) by bc1.hpc.lsu.edu (8.14.1/8.13.8) with ESMTP id l9HGg4Vh083167; Wed, 17 Oct 2007 16:42:04 GMT (envelope-from estrabd@gmail.com) Received: (from estrabd@localhost) by bc1.hpc.lsu.edu (8.14.1/8.13.8/Submit) id l9HGg4hu083166; Wed, 17 Oct 2007 16:42:04 GMT (envelope-from estrabd@gmail.com) X-Authentication-Warning: bc1.hpc.lsu.edu: estrabd set sender to estrabd@gmail.com using -f Date: Wed, 17 Oct 2007 16:42:04 +0000 From: "B. Estrade" To: freebsd-smp Message-ID: <20071017164204.GG76582@bc1.hpc.lsu.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i mailed-by: estrabd@lsu.edu Subject: multi-threading with gcc43 openmp on dual core X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 16:59:31 -0000 I am messing around with gcc43's openmp capabilities on an IBM Thinkcentre, which has a dual core Intel vPro. I am running FreeBSD 6.2 with SMP. At this stage in the game I am trying to push around the limits of the systems capabilities, and I was wondering if FreeBSD attempts to offload some of the threads to the 2nd core. I admittedly don't know the chip architecure or how FreeBSD handles man threads in SMP, but I'd like too, which is why I am playing with this. As the number of threads increases, one core seems to be oversubscribed according to top (i.e., wcpu > 100%) yet the system is said to be 50% idle (+/- a few points). The load average also seems to not get much higher than 1.0. While I am happy with the load handling, I am wondering: 1. why are threads not evenly distributed b/w CPU0 and CPU1 2. is there a way to explicitly facilitate this thread balancing (don't know if this is related to CPU affinity) 3. what is a good way to push the system? The program I am running spawns an increasing number of threads from 1 to OMP_NUM_THREADS, where each thread has its own loop that simply assigns the sum of j and i to k. My OpenMP-foo is not advanced, but it's steadily getting there. Here is the code I am using right now: #include int main (int argc, char *argv[]) { int i,j,k,m,n; int MAX=500; n=omp_get_max_threads(); for (m=1;m<=n;m++) { k=0; omp_set_num_threads(m); #pragma omp parallel private(i,j) shared(MAX) reduction(+:k) { for (i=0;i Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 98D6316A419 for ; Wed, 17 Oct 2007 17:31:07 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx1.freebsd.org (Postfix) with ESMTP id 1B06813C457; Wed, 17 Oct 2007 17:31:05 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <471646D9.5090508@FreeBSD.org> Date: Wed, 17 Oct 2007 19:31:05 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: "B. Estrade" References: <20071017164204.GG76582@bc1.hpc.lsu.edu> In-Reply-To: <20071017164204.GG76582@bc1.hpc.lsu.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-smp Subject: Re: multi-threading with gcc43 openmp on dual core X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 17:31:07 -0000 B. Estrade wrote: > I am messing around with gcc43's openmp capabilities on an IBM Thinkcentre, which has a dual core Intel vPro. I am running FreeBSD 6.2 with SMP. > > At this stage in the game I am trying to push around the limits of the systems capabilities, and I was wondering if FreeBSD attempts to offload some of the threads to the 2nd core. I admittedly don't know the chip architecure or how FreeBSD handles man threads in SMP, but I'd like too, which is why I am playing with this. > > As the number of threads increases, one core seems to be oversubscribed according to top (i.e., wcpu > 100%) yet the system is said to be 50% idle (+/- a few points). The load average also seems to not get much higher than 1.0. While I am happy with the load handling, I am wondering: > > 1. why are threads not evenly distributed b/w CPU0 and CPU1 > 2. is there a way to explicitly facilitate this thread balancing (don't know if this is related to CPU affinity) > 3. what is a good way to push the system? > > The program I am running spawns an increasing number of threads from 1 to OMP_NUM_THREADS, where each thread has its own loop that simply assigns the sum of j and i to k. My OpenMP-foo is not advanced, but it's steadily getting there. > > Here is the code I am using right now: > > #include > int main (int argc, char *argv[]) { > int i,j,k,m,n; > int MAX=500; > n=omp_get_max_threads(); > for (m=1;m<=n;m++) { > k=0; > omp_set_num_threads(m); > #pragma omp parallel private(i,j) shared(MAX) reduction(+:k) > { for (i=0;i for (j=0;j k = j+i; > } > #pragma omp barrier > } > } > printf("%8d threads: %d\n",m,k); > } > return 0; > } > > It is compiled and executed with the following command: > > % gcc43 -fopenmp test.c && export OMP_NUM_THREADS=1500 && ./a.out > > Any thoughts? > > Once I get familiar with what 6.x is doing, I plan on putting 7.0 on this system to see how it compares. Ultimately, I'd like to a straightforward code I can use to evaluate how a particular SMP system running FreeBSD scales. I know it is not a "real" workload, but it would give /me some information that I'd find useful. > > TIA && Cheers, > Brett > FreeBSD uses all CPUs for running processes and threads. This relies on your program being sufficiently parallel though. I don't know enough about openmp but maybe it is not able to parallelize your workload very well. Kris From owner-freebsd-smp@FreeBSD.ORG Wed Oct 17 18:05:08 2007 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E51F016A420 for ; Wed, 17 Oct 2007 18:05:08 +0000 (UTC) (envelope-from estrabd@gmail.com) Received: from bc1.hpc.lsu.edu (bc1.hpc.lsu.edu [130.39.198.6]) by mx1.freebsd.org (Postfix) with ESMTP id 97A3713C469 for ; Wed, 17 Oct 2007 18:05:08 +0000 (UTC) (envelope-from estrabd@gmail.com) Received: from bc1.hpc.lsu.edu (localhost.hpc.lsu.edu [127.0.0.1]) by bc1.hpc.lsu.edu (8.14.1/8.13.8) with ESMTP id l9HI1eom083525; Wed, 17 Oct 2007 18:01:40 GMT (envelope-from estrabd@gmail.com) Received: (from estrabd@localhost) by bc1.hpc.lsu.edu (8.14.1/8.13.8/Submit) id l9HI1ejR083524; Wed, 17 Oct 2007 18:01:40 GMT (envelope-from estrabd@gmail.com) X-Authentication-Warning: bc1.hpc.lsu.edu: estrabd set sender to estrabd@gmail.com using -f Date: Wed, 17 Oct 2007 18:01:40 +0000 From: "B. Estrade" To: Kris Kennaway Message-ID: <20071017180140.GI76582@bc1.hpc.lsu.edu> References: <20071017164204.GG76582@bc1.hpc.lsu.edu> <471646D9.5090508@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <471646D9.5090508@FreeBSD.org> User-Agent: Mutt/1.4.2.3i mailed-by: estrabd@lsu.edu Cc: freebsd-smp Subject: Re: multi-threading with gcc43 openmp on dual core X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 18:05:09 -0000 On Wed, Oct 17, 2007 at 07:31:05PM +0200, Kris Kennaway wrote: > B. Estrade wrote: snip > >1. why are threads not evenly distributed b/w CPU0 and CPU1 > >2. is there a way to explicitly facilitate this thread balancing (don't > >know if this is related to CPU affinity) > >3. what is a good way to push the system? snip > >The program I am running spawns an increasing number of threads from 1 to > >OMP_NUM_THREADS, where each thread has its own loop that simply assigns > >the sum of j and i to k. My OpenMP-foo is not advanced, but it's steadily > >getting there. snip > >Brett > > > > FreeBSD uses all CPUs for running processes and threads. This relies on > your program being sufficiently parallel though. I don't know enough > about openmp but maybe it is not able to parallelize your workload very > well. Thanks, Kris. This is what I am trying to determine. After some more observations, it seems as if there is some swapping of duties between CPU0 and CPU1 - but only one of them has the entire set of threads at any one point in time, while the other is idle. I am execeuting a code that uses only explicitly declared private variables, and this includes taking out the reduction shown in the code above from my original email. It should be obvious to the compiler and OS that these threads are not dependent on each other in any way. I am going to look deeper into this - but I am still very interested in anything that anyone wants to say. I think my question can be best summed up as: how does one make it obvious to FreeBSD that all threads should be shared among all CPUs equally? I don't think this is an OpenMP specific question - I am just using it as a way to generate the threading...is it unreasonable of me to expect that all threads should be shared by all CPUs? Thanks, Brett > > Kris -- HPC Enablement Group Louisiana Optical Network Initiative http://www.loni.org 225-578-1920 From owner-freebsd-smp@FreeBSD.ORG Wed Oct 17 18:34:09 2007 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09B6E16A473 for ; Wed, 17 Oct 2007 18:34:09 +0000 (UTC) (envelope-from kris@FreeBSD.org) Received: from weak.local (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx1.freebsd.org (Postfix) with ESMTP id B68A113C458; Wed, 17 Oct 2007 18:34:07 +0000 (UTC) (envelope-from kris@FreeBSD.org) Message-ID: <4716559F.9030908@FreeBSD.org> Date: Wed, 17 Oct 2007 20:34:07 +0200 From: Kris Kennaway User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: "B. Estrade" References: <20071017164204.GG76582@bc1.hpc.lsu.edu> <471646D9.5090508@FreeBSD.org> <20071017180140.GI76582@bc1.hpc.lsu.edu> In-Reply-To: <20071017180140.GI76582@bc1.hpc.lsu.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-smp Subject: Re: multi-threading with gcc43 openmp on dual core X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 18:34:09 -0000 B. Estrade wrote: > On Wed, Oct 17, 2007 at 07:31:05PM +0200, Kris Kennaway wrote: >> B. Estrade wrote: > snip >>> 1. why are threads not evenly distributed b/w CPU0 and CPU1 >>> 2. is there a way to explicitly facilitate this thread balancing (don't >>> know if this is related to CPU affinity) >>> 3. what is a good way to push the system? > snip >>> The program I am running spawns an increasing number of threads from 1 to >>> OMP_NUM_THREADS, where each thread has its own loop that simply assigns >>> the sum of j and i to k. My OpenMP-foo is not advanced, but it's steadily >>> getting there. > snip >>> Brett >>> >> FreeBSD uses all CPUs for running processes and threads. This relies on >> your program being sufficiently parallel though. I don't know enough >> about openmp but maybe it is not able to parallelize your workload very >> well. > > Thanks, Kris. This is what I am trying to determine. After some more observations, it seems as if there is some swapping of duties between CPU0 and CPU1 - but only one of them has the entire set of threads at any one point in time, while the other is idle. I am execeuting a code that uses only explicitly declared private variables, and this includes taking out the reduction shown in the code above from my original email. It should be obvious to the compiler and OS that these threads are not dependent on each other in any way. > > I am going to look deeper into this - but I am still very interested in anything that anyone wants to say. I think my question can be best summed up as: how does one make it obvious to FreeBSD that all threads should be shared among all CPUs equally? I don't think this is an OpenMP specific question - I am just using it as a way to generate the threading...is it unreasonable of me to expect that all threads should be shared by all CPUs? I dunno if there is something special you have to do with OpenMP, with pthreads there is nothing special one needs to do, apart from making sure your code is such that multiple threads can actually run at once (i.e. they are not blocking each other for shared resources). Kris P.S. Please wrap your lines, it makes your emails hard to read. From owner-freebsd-smp@FreeBSD.ORG Wed Oct 17 19:09:00 2007 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5DADD16A41B; Wed, 17 Oct 2007 19:09:00 +0000 (UTC) (envelope-from estrabd@gmail.com) Received: from bc1.hpc.lsu.edu (bc1.hpc.lsu.edu [130.39.198.6]) by mx1.freebsd.org (Postfix) with ESMTP id 1104F13C461; Wed, 17 Oct 2007 19:08:59 +0000 (UTC) (envelope-from estrabd@gmail.com) Received: from bc1.hpc.lsu.edu (localhost.hpc.lsu.edu [127.0.0.1]) by bc1.hpc.lsu.edu (8.14.1/8.13.8) with ESMTP id l9HJ5V6b083739; Wed, 17 Oct 2007 19:05:31 GMT (envelope-from estrabd@gmail.com) Received: (from estrabd@localhost) by bc1.hpc.lsu.edu (8.14.1/8.13.8/Submit) id l9HJ5VdJ083738; Wed, 17 Oct 2007 19:05:31 GMT (envelope-from estrabd@gmail.com) X-Authentication-Warning: bc1.hpc.lsu.edu: estrabd set sender to estrabd@gmail.com using -f Date: Wed, 17 Oct 2007 19:05:31 +0000 From: "B. Estrade" To: Kris Kennaway Message-ID: <20071017190531.GJ76582@bc1.hpc.lsu.edu> References: <20071017164204.GG76582@bc1.hpc.lsu.edu> <471646D9.5090508@FreeBSD.org> <20071017180140.GI76582@bc1.hpc.lsu.edu> <4716559F.9030908@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4716559F.9030908@FreeBSD.org> User-Agent: Mutt/1.4.2.3i mailed-by: estrabd@lsu.edu Cc: freebsd-smp Subject: Re: multi-threading with gcc43 openmp on dual core X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Oct 2007 19:09:00 -0000 On Wed, Oct 17, 2007 at 08:34:07PM +0200, Kris Kennaway wrote: > I dunno if there is something special you have to do with OpenMP, with > pthreads there is nothing special one needs to do, apart from making > sure your code is such that multiple threads can actually run at once > (i.e. they are not blocking each other for shared resources). I am using OpenMP because it is easy and coincides with some stuff I am already doing. I will take a look at Pthreads since my purpose is also to see how things work. Thanks for the suggestion. I'm going back my cave for a while to take a look at a few things. If I have any FreeBSD specific questions or find something that I think is interestig, I'll stick my head out again. Hope the lines are better now :) Cheers, Brett > > > Kris > > P.S. Please wrap your lines, it makes your emails hard to read. > -- HPC Enablement Group Louisiana Optical Network Initiative http://www.loni.org 225-578-1920