Date: Thu, 26 Apr 2001 11:24:02 -0700 From: Julian Elischer <julian@elischer.org> To: Daniel Eischen <eischen@vigrid.com> Cc: Arch@freebsd.org, alfred@freebsd.org, Robert Watson <rwatson@freebsd.org> Subject: Re: KSE threading support (first parts) Message-ID: <3AE867C2.3B657214@elischer.org> References: <Pine.SUN.3.91.1010425141253.20558A-100000@pcnet1.pcnet.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Daniel Eischen wrote:
>
> On Wed, 25 Apr 2001, Julian Elischer wrote:
> > In this view, KSEGs are on the run queue and when they get some
> > quanta the KSEs hanging off them are run.
> > If 2 KSEs are running, the KSEG's quanta are exhausted a twice
> > the rate.
>
> Don't we eventually want per-CPU run queues? Then how do
> multiple KSEs hanging off a KSEG get scheduled then if the quanta
> are in the KSEG? Round robin?
Nominally yes we do, but I must admit that I see a single queue
with multiple processors reading off it as actually being easier
to use and implement and I think it may even have some better
use characteristics. I have nighmares thinking about doing it with
multiple (per processor) run queues. Allocating quanta and priority
between KSEs on different queues is much more tricky.
>
> > Each KSE has a very strong affinity for one processor
> > and KSECs have a weak affinity for a KSE. If a KSE runs out
> > of work but has time, it will 'poach' a KSEC from another KSE in the
> > same KSEG list.
>
> Again, if KSEs can have a strong affinity for 1 processor and there
> can be multiple KSEs hanging off a KSEG, then how do you schedule
> these KSEs when we have per-CPU run queues? It makes scheduling
> these KSEs more difficult than it needs to be.
In that case the KSEs are put on the run queues when there is at
least one KSEC ready to run for each of them in the KSEG. If you
have 3 KSECs ready to run and 4 processors, then you put three KSEs
on run queues. If you have 6 KSECs ready to run then you put 4 on
the run queues. The first 2 to complete or block will do a second KSEC
if there is still some of the quantum left. The priority they are
scheduled with is taken from the KSEG (probably).
>
> I still don't see the need to have multiple KSEs within a KSEG ;-)
KSEs in the same KSEG are using the same pool of quanta to
complete KSECs.
SOMETHING has to hold that information. A KSE is a virtual
processor where a KSEG is a virtual multiprocessor.
You allocate quanta to the KSEG. KSEs use these quanta.
The KSEG is competing (almost) fairly with other processes
in the system. If you want "system" thread scheduling, you
can create more KSEGs. They compete against other processes and
against each other for slices of the 'real' machine.
>
> > In this patch the linkages are not set up at all.
> > All that is done is that the structures are
> > defined and used instead of a monolithic 'proc' struct.
> > The new structures are 'included' in the proc structure
> > to maintain compatibility and to allow code to be changed slowely.
> >
> > What really needs to be done is for everyone who is interested to go over
> > rather arbitrary allocation of fields to structures that
> > I did and make suggested changes.
> >
> > Also I've punted on most things to do with signals as we haven't
> > really discussed how we want signals to be handled in a KSE world..
> > (ca each KSEG or KSE get individual signals? do we need to
> > define a special 'signal' KSE? If so is that all it does?
>
> Signals should be sent (via an upcall) to the first available
> KSE to return to userland (return from syscall, after preemption,
> etc.). The userland thread scheduler will pick a thread to
> receive the signal. If the thread is running or in one
> of the scheduling queues for the current KSEG, it will
> be able to handle it without any other assist from the kernel.
> If the thread is running or in one of the scheduling queues for
> another KSEG, it will mark the signal pending in the target
> thread and "signal" the appropriate KSEG with help from the
> kernel (one of the new user<->kernel interfaces or syscalls).
OK so 'signals' and everything to do with them are "Per process".
I may edit the patch to indicate this. This does indicate a mutex
with SMP so that if two processors return their KSEs to userland
at the same time, they don't deliver the same signal twice.
Can two KSEs (KSEs are on different processors) deliver
DIFFERENT signals to userland at the same time?
>
> (We may have to replace "KSEG" in the above with "KSE")
yes, you are correct.. it should read: (I think)
> Signals should be sent (via an upcall) to the first available
> KSE to return to userland (return from syscall, after preemption,
> etc.). The userland thread scheduler will pick a thread to
> receive the signal. If the thread is running or in one
> of the scheduling queues for the current KSEG, it will
> be able to handle it without any other assist from the kernel.
Is this what you mean?
This is tricky... when a KSE returns to userland it is running
NO threads. All threaded syscalls return to userland in the 'suspended'
state, so that the UTS can decide what to run. All syscalls return via
an upcall to the UTS (actually the original newkse() call returns
infinitly many times.. that is how the upcall is achieved). The return
values, error returns and data movements have been made to the appropriate
memory locations.. It's as if the thread did a 'yield()' immediatly
after returning from a normal syscall..
So we can be sure that THIS KSE isn't running the interrupt thread.
If the thread is however being run on a different KSE (regardless of
whether in this KSEG or not) then the signal must be noted so that
the thread can see it at some future time. If it's not running but in
another KSEG then it's treated as if running, (the signal noted) and the
UTS will make it runnable at the next opportunity that that KSEG
is runnable. (If we ran the thread on this KSE regardless of the fact
that it's from another KSEG, then it will be running with a priority
other than what the programmer assigned it. (maybe he wants lower
priority signal handling)).
> If the thread is running or in one of the scheduling queues for
> another KSE, it will mark the signal pending in the target
> thread and "signal" the appropriate KSEG with help from the
> kernel (one of the new user<->kernel interfaces or syscalls).
If a KSEG is not running because it had no work, then
yes, you need to wake up one of its KSEs to handle the signal.
>
> It might be nice to have a general way of sending messages
> between KSEGs (KSEs?).
Userland-to-kernel? or userland-to-userland?
"kind of like a signal?" :-)
>
> > What happens to the 'u-area'?
> >
> > how do we define a "cur-kse" similar to curproc?
> > (do we need one?)
> > presently the processor state is stored all over the place
> > when a process is suspended..
> > This needs to be brought together so it can be put into the KSEC.
> > Who understands that stuff?
> >
> > Some of the next steps would be:
> > 1/ figure out what we want for signals etc..
>
> Ask me for help in this area. I know what the userland scheduler
> has to do when dispatching signals to threads.
>
> > 2/ get the contexts actually stored in the KSEC structure
> > when a proces is suspended. (instead of some strange pcb in funny memory
> > near the u area)
> > 3/ Set up the linkages between these structures, and
> > 4/ start using 'kse' instead of 'proc' in a bunch of places
> > and using the linkages to find the appropriate other
> > structures when needed.
> > 5/ Add code to make new KSEs so that the 1:1 Mapping is no longer
> > true.
> > 6/ Add syscalls to start making KSEs other than the one that
> > is built into the process.
> > 7/ start making upcalls
>
> Can't we start with 7 ;-)
well, really, there are 4 new syscalls and the upcall is the multiple
return of one of them.
kse_id ksecreate(struct retblock *rblk, boolean newkseg);
kseyield(timeout); /* never returns.. comes back as upcall when awakened */
ksewakeup(kse_id sleeper);
ksefinish(); /* just never returns (unless we are last kse in which case,
upcalls) */
upcalls return with certain information in the retblock..
1/ why the KSE upcalled. (a bitmap of reasons (there may have been more
than one reason, (e.g. 3 returned syscalls and 2 signals and a wakeup).
2/ head of linked list of completed syscall status blocks.
(these should be allocated in the thread control blocks that the UTS uses
and will include room for a pointer that the kernel ignores but which the
UTS can use to find the start of that thread control block. Also enough
information so that the kernel can store enough thread run state so that
the thread can be made to look as if it has just done a 'yield()'. (so it
can be restarted in the same way that other threads can be restarted.))
>
> --
> Dan Eischen
--
__--_|\ Julian Elischer
/ \ julian@elischer.org
( OZ ) World tour 2000-2001
---> X_.---._/
v
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-arch" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3AE867C2.3B657214>