Date: Tue, 8 Aug 2006 11:33:53 -0400 From: John Baldwin <jhb@freebsd.org> To: Jeremie Le Hen <jeremie@le-hen.org> Cc: freebsd-threads@freebsd.org Subject: Re: [fbsd] Re: system scope vs. process scope Message-ID: <200608081133.54490.jhb@freebsd.org> In-Reply-To: <20060807213945.GQ4498@obiwan.tataz.chchile.org> References: <20060804140657.GK4498@obiwan.tataz.chchile.org> <200608041134.43979.john@baldwin.cx> <20060807213945.GQ4498@obiwan.tataz.chchile.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday 07 August 2006 17:39, Jeremie Le Hen wrote: > Hi John, >=20 > On Fri, Aug 04, 2006 at 11:34:43AM -0400, John Baldwin wrote: > > On Friday 04 August 2006 10:06, Jeremie Le Hen wrote: > > > Hi, > > >=20 > > > occasionally, I saw environnement variables LIBPTHREAD_SYSTEM_SCOPE a= nd > > > LIBPTHREAD_PROCESS_SCOPE mentionned hither and thither. I grep(1)'ed > > > through the source tree for some documentation, but I wasn't able to > > > find any. Read the source Luke... > > >=20 > > > It seems that the PTHREAD_SCOPE_SYSTEM thread attribute is the defaul= t. > > > Intuitivelely, I would say that setting the PTHREAD_SCOPE_PROCESS > > > attribute creates a new process with its address space being shared w= ith > > > the other userland threads (a la Linux) whereas the default behaviour= is > > > to create a new kernel thread. > > >=20 > > > Am I right ? What are the pros and cons of either methods ? > >=20 > > That's not what it means. :) It has to do with scheduling. A=20 > > PTHREAD_SCOPE_SYSTEM will compete for the CPU with other "system" threa= ds=20 > > (aka kernel threads). That is, each userland thread has a direct kerne= l=20 > > thread that is visible to the system (kernel). A PTHREAD_SCOPE_PROCESS= =20 > > thread competes for the CPU just within the current process. That is, = each=20 > > group of PTHREAD_SCOPE_PROCESS thread's all share (in theory) a single = kernel=20 > > thread that competes for CPU with other system threads. An example mig= ht=20 > > explain the theory more completely. > >=20 > > Suppose you have a system with 2 processes. One process is single-thre= aded=20 > > and is CPU bound. The other process has 2 threads both of which are al= so CPU=20 > > bound. If the threads in the second process are PTHREAD_SCOPE_SYSTEM, = then=20 > > each thread will get 33% of the CPU. If the threads in the second proc= ess=20 > > are PTHREAD_SCOPE_PROCESS, then the each process will get 50% of the CP= U. =20 > > The second process will then use its own algorithm to split it's 50% of= the=20 > > CPU up between it's two threads. If it divides it evenly, then each of= its'=20 > > threads will end up with 25% of the CPU whereas the thread for the firs= t=20 > > process has 50% of the CPU. The idea for this is that if you have a sy= stem=20 > > with several single-threaded processes and one process with 1000 thread= s, you=20 > > don't want that process to starve out all the other processes. >=20 > Thank you very much for your explanation. >=20 > I have some difficulties though to understand why this is a thread > attribute, not process-wide. This implies that threads inside one > process may be either PTHREAD_SCOPE_SYSTEM or PTHREAD_SCOPE_PROCESS. > I don't really get the point in doing this, except if this how > libpthread manages to achieve M:N threading. You can mix the two. You might have a dedicated SYSTEM_SCOPE thread for doing some task like running a CORBA ORB server over in a 3rd party library and then create a pool of PROCESS_SCOPE worker threads for your actual application. > > In practice things get much hairier, but suffice it to say that libpthr= ead=20 > > manages the scheduling of PTHREAD_SCOPE_PROCESS threads in userland, wh= ereas=20 > > libthr would have to depend on the kernel managing that. >=20 > How d those two attributes modify libpthread behaviour in regards > to M:N scheduling ? I am quite puzzled about this. >=20 > > (To some extent=20 > > libpthread needs some help from the kernel to provide this as well.) >=20 > Scheduler Activations ? It's not really related to SAs directly. It's hard to explain w/o pictures. However, try to imagine how you would actually implement this stuff. =46irst, let's look at the M:M case where each userland thread has its own kernel thread. Suppose you have a multithreaded process where all the threads are PROCESS_SCOPE and you have a multithreaded process where all the threads are SYSTEM_SCOPE. You have a fixed quantum of, say, 10ms. What that tells you is that each SYSTEM_SCOPE thread should run for 10ms before being preempted (assume no I/O or blocking for now), but that the PROCESS_SCOPE threads all have to share a single 10ms quantum. How do you guarantee that? Well, you can create a thread group type structure that the actual quantum gets assigned to. A SYSTEM_SCOPE thread has its own private group (so it gets the full quantum), whereas the PROCESS_SCOPE threads in a process share a single group. The theory then is that you schedule thread groups to run for a quantum, and the group splits that up among its threads. In the case of a SYSTEM_SCOPE thread, there's only 1 thread in the group, so you just run the 1 thread. In the case of PROCESS_SCOPE, you now have to come up with an algorithm for determining how to schedule the various threads during that quantum. All of this stuff so far has been done in the kernel, and in theory the 'ksegroup' structure in the kernel serves as the thread group type entity mentioned above. In practice I believe it is a lot more complex than the above simple version. Now, as far as Scheduler Activations, they are primarily a way to optimize the PROCESS_SCOPE case a bit and allow for userland to participate in scheduling the threads. With SA, when you create a new PROCESS_SCOPE thread, it doesn't create a kernel thread (assume for simplicity the UP case for now). Instead, the kernel thread for your process is "tied" to a specific PROCESS_SCOPE thread. You can sort of think of the kernel thread as being a CPU for your process at at this point, and libpthread has its own scheduler that chooses which userland thread to run. When a userland thread makes a system call, the kernel thread it is bound to goes into the kernel to do the work. If the kernel thread has to block for I/O, then the kernel creates a new kernel thread that goes back into userland to let another userland thread continue to use the quantum. If all of the userland threads are blocked, then the new kernel thread has nothing to do, so it can commit suicide. Thus, you only have kernel threads for any threads blocked in the kernel, plus 1 kernel thread to run any non-blocked threads. This can result in the kernel only having 1 runnable thread in its thread group at a given time (like it would for SYSTEM_SCOPE threads) thus making the kernel scheduler not have to do as much work. With SMP there is some extra complication in that you will actually give a process up to N kernel threads (where N =3D=3D number of CPUs) to run non-blocked threads to allow for concurrent work, but many of the basics are the same. =46rom this, you can see that KSE isn't just about scheduler activations (which many people think it is), but it also provides the framework for managing PROCESS_SCOPE vs SYSTEM_SCOPE threads, even for the M:M case. These two parts of KSE are fairly orthogonal. You could have SA (and thus M:N) without supporting PROCESS_SCOPE vs SYSTEM_SCOPE, and you could have scheduler fairness (PROCESS_SCOPE vs SYSTEM_SCOPE) without SA. =2D-=20 John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200608081133.54490.jhb>