Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 28 Oct 2006 21:00:04 -0700
From:      Julian Elischer <julian@elischer.org>
To:        "Alexandre \"Sunny\" Kovalenko" <Alex.Kovalenko@verizon.net>
Cc:        Daniel Eischen <deischen@freebsd.org>, Paul Allen <nospam@ugcs.caltech.edu>, current@freebsd.org
Subject:   Re: Comments on the  KSE option
Message-ID:  <45442744.6080103@elischer.org>
In-Reply-To: <1162059531.872.45.camel@RabbitsDen.RabbitsLawn.verizon.net>
References:  <45425D92.8060205@elischer.org>	<20061027201838.GH30707@riyal.ugcs.caltech.edu>	<Pine.GSO.4.64.0610271634160.7105@sea.ntplx.net>	<1161998104.872.18.camel@RabbitsDen.RabbitsLawn.verizon.net>	<4542B171.8050601@elischer.org>	<1161999387.872.29.camel@RabbitsDen.RabbitsLawn.verizon.net>	<4542D3A8.1040500@elischer.org> <1162059531.872.45.camel@RabbitsDen.RabbitsLawn.verizon.net>

next in thread | previous in thread | raw e-mail | index | archive | help
Alexandre "Sunny" Kovalenko wrote:
> On Fri, 2006-10-27 at 20:51 -0700, Julian Elischer wrote:
>> Alexandre "Sunny" Kovalenko wrote:
>>> On Fri, 2006-10-27 at 18:25 -0700, Julian Elischer wrote:
>>>> Alexandre "Sunny" Kovalenko wrote:
>>>>> On Fri, 2006-10-27 at 16:41 -0400, Daniel Eischen wrote:
>>>>>> On Fri, 27 Oct 2006, Paul Allen wrote:
>>>>>>
>>>>>>>> From Julian Elischer <julian@elischer.org>, Fri, Oct 27, 2006 at 12:27:14PM -0700:
>>>>>>>> The aim of the fair scheduling code is to ensure that if you, as a user,
>>>>>>>> make a process that starts 1000 threads, and I as a user, make an
>>>>>>>> unthreaded process, then I can still get to the CPU at somewhat similar
>>>>>>>> rates to you.  A naive scheduler would give you 1000 cpu slots and me 1.
>>>>>>> Ah.  Let me be one of the first to take a crack at attacking this idea as
>>>>>>> a mistake.
>>>>>> No, it is POSIX.  You, the application, can write a program with
>>>>>> system scope or process scope threads and get whatever you behavior
>>>>>> you want, within rlimits of course.
>>>>>>
>>>>>> If you want unfair scheduling, then create your threads with
>>>>>> system scope contention, otherwise use process scope.  The
>>>>>> kernel should be designed to allow both, and have adjustable
>>>>>> limits in place for (at least) system scope threads.
>>>>>>
>>>>>> Noone is saying that you can't have as many system scope threads
>>>>>> as you want (and as allowed by limits), just that you must also
>>>>>> be able to have process scope threads (with probably higher limits
>>>>>> or possibly no limits).
>>>>>>
>>>>> I might be missing something here, but OP was separating M:N (which is
>>>>> what you are referring to above), and "fairness" (not giving process
>>>>> with 1000 *system scope* threads 1000 CPU scheduling slots). As far as I
>>>>> know the first one is POSIX and the second one is not. 
>>>>>
>>>>> FWIW: as an application programmer who spent considerable amount of time
>>>>> lately trying to make heavily multithreaded application run most
>>>>> efficiently on 32-way machine, I would rather not have to deal with
>>>>> "fairness" -- M:N is bad enough.
>>>>>
>>>> no,  fairness is making sure that 1000 process scope threads
>>>> do not negatively impact other processes.
>>>> 1000 system  scope threads are controlled by your ulimit settings
>>>> (Each one counts as a process.)
>>>>
>>>>
>>> I apologize for misinterpreting your words. But then, if I have M:N set
>>> to 10:1, I would expect application with 1000 process scope threads to
>>> have as many CPU slots as 100 processes, or, if I have 10 system scope
>>> threads and 990 process scope threads, I would expect application to
>>> have as many CPU slots as 109 processes. Is this what you refer to as
>>> "fairness"? 
>>>
>> M:N is not a ratio, but rather the notation to say that M user threads 
>> are enacted using N kernel schedulable entities (kernel threads).
>> usually N is limited to something like NCPU kernel schedulable entities 
>>   running at a time. (not including sleeping threads waiting for IO)
>> (NCPU  is the number of CPUs).
>>
>> so in fact M:N is usually M user threads over over some number like 4 or 
>> 8 kernel threads (depending on #cpus) plus the number of threads waiting 
>> for IO.
>>
>> Julian
> In the environment I am most familiar with -- IBM AIX -- M:N is the
> ratio and it is settable either systemwide or  for the specific user
> using environment variable, e.g.:
> 
> export AIXTHREAD_MNRATIO=8:1
> 
> with the minimum kernel threads allocated according to another setting:
> 
> export AIXTHREAD_MINKTHREADS=4
> 
> Neither one depends on the physical CPU count in the box (which could
> change in the middle of application execution anyway).
> 
> Both settings have known default values (8:1 and 8 respectively).
> 
> Between the two I can always tell how many kernel threads given amount
> of the process scope threads will use. This gives me both flexibility
> and predictability.
> 
> Am I understanding correctly that what you have implemented fixes number
> of kernel threads at boot time and changes the M:N ratio throughout the
> run time of the application? 

no. the "number of available slots" for concurency is a number that is 
tunable on the fly. I think we may even be able to tune it in a program 
but I would have to look to see if that ever got committed.

> 



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?45442744.6080103>